Remote control method and device based on head-mounted device, equipment and medium

Abstract

The invention discloses a remote control method and device based on head-mounted equipment, equipment and a medium, and relates to the technical field of head-mounted equipment. The method is applied to a head-mounted device, and comprises the following steps: establishing wireless communication connection between the head-mounted device and a controlled device; control authority verification information is sent to the controlled equipment through the wireless communication connection, and the equipment information of the controlled equipment is reported under the condition that the controlled equipment verifies that the control authority verification information is correct; and displaying the equipment information under the condition that the equipment information reported by the controlled equipment is received. According to the method, the problem of complex operation when a user checks the information of the controlled equipment can be solved.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and more specifically, to a remote control method, apparatus, device, and medium based on a head-mounted display device. Background Technology

[0002] In real-world applications, users frequently need to view information about certain devices. However, to do so, users typically need to download a relevant app on a computer or mobile phone beforehand and then open a specific interface within the app. This results in a cumbersome process for users when viewing device information. Summary of the Invention

[0003] One objective of this application is to provide a new technology solution for remote control based on a head-mounted device.

[0004] According to a first aspect of this application, a remote control method based on a head-mounted device is provided, the method being applied to the head-mounted device, comprising:

[0005] Establish a wireless communication connection between the head-mounted device and the controlled device;

[0006] The controlled device sends control permission verification information to the controlled device through the wireless communication connection, wherein the controlled device reports its device information if it verifies that the control permission verification information is correct.

[0007] Upon receiving the device information reported by the controlled device, the device information is displayed.

[0008] Optionally, establishing a wireless communication connection between the head-mounted device and the controlled device includes:

[0009] Acquire the first image of the external environment;

[0010] The device of a specified type contained in the first external environment image is identified as the controlled device;

[0011] Wireless connection information is extracted from the first external environment image, and the surface of the controlled device is provided with wireless connection information;

[0012] Based on the wireless connection information, a wireless communication connection is established between the head-mounted device and the controlled device.

[0013] Optionally, establishing a wireless communication connection between the head-mounted device and the controlled device includes:

[0014] Upon receiving a viewing mode selection instruction input by the wearer of the head-mounted device, a first image of the external environment is acquired.

[0015] Optionally, the method further includes:

[0016] Receive the first control command input by the wearer;

[0017] The device information displayed on the head-mounted device is controlled according to the first control command.

[0018] Optionally, establishing a wireless communication connection between the head-mounted device and the first controlled device includes:

[0019] Upon receiving a control mode selection command input by the wearer of the head-mounted device, a wireless communication connection is established between the head-mounted device and the controlled device.

[0020] The method further includes:

[0021] Receive a second control command input by the wearer of the head-mounted device;

[0022] The second control command is sent to the controlled device based on the wireless communication connection.

[0023] Optionally, receiving a second control command input by the wearer of the head-mounted device includes:

[0024] Acquire a second image of the external environment and an image of the wearer's eyes;

[0025] Based on the second external environment image and the wearer's eye image, determine the wearer's operating gestures and the operating position in the device information;

[0026] The second control command is determined based on the operation gesture and operation position.

[0027] Optionally, the head-mounted device includes at least two display areas, and establishing a wireless communication connection between the head-mounted device and the controlled device includes:

[0028] For any of the controlled devices, the wireless communication connection between the head-mounted device and the controlled device is established;

[0029] The display of the device information includes:

[0030] The device information of different controlled devices is displayed in different display areas.

[0031] Optionally, determining the device of a predetermined type contained in the first external environment image as the controlled device includes:

[0032] If a device of a specified type is included in all of the first external environment images within a preset time period, the device of the specified type is identified as the controlled device.

[0033] Optionally, the device information includes the current display content of the controlled device, and the method further includes:

[0034] If no device information is received from the controlled device, a third external environment image containing the current display content of the controlled device is acquired;

[0035] Extract the current display content of the controlled device from the third external environment image;

[0036] Display the current display content of the controlled device;

[0037] If the controlled device fails to verify the control permission verification information, it will not report the device information of the controlled device.

[0038] According to a second aspect of this application, a remote control device based on a head-mounted device is provided, the device being applied to the head-mounted device, comprising:

[0039] A module is established to establish a wireless communication connection between the head-mounted device and the controlled device.

[0040] The sending module is used to send control permission verification information to the controlled device through the wireless communication connection, wherein the controlled device reports its device information when it verifies that the control permission verification information is correct.

[0041] The display module is used to display the device information when the device information reported by the controlled device is received.

[0042] According to a third aspect of this application, a head-mounted device is provided, the head-mounted device comprising the means as described in the second aspect; or,

[0043] The head-mounted device includes a memory and a processor, the memory for storing computer instructions and the processor for retrieving the computer instructions from the memory to perform the method as described in any one of the first aspects.

[0044] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the first aspects.

[0045] This embodiment provides a remote control method based on a head-mounted device. The method, applied to the head-mounted device, includes: establishing a wireless communication connection between the head-mounted device and the controlled device; sending control permission verification information to the controlled device via the wireless communication connection, wherein the controlled device reports its device information if the control permission verification information is correct; and displaying the device information upon receiving the reported device information from the controlled device. Through this method, the wearer can view the current monitoring content of a camera via the head-mounted device without needing to download a related application on a computer or mobile phone and open a specific interface to view the camera's current monitoring content. This solves the problem of complex operations for users when viewing information from a controlled device.

[0046] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0048] Figure 1 This is a framework for the hardware configuration of a head-mounted device for implementing a remote control method based on a head-mounted device, according to an embodiment of this application. Figure 1 ;

[0049] Figure 2 This is a flowchart illustrating a remote control method based on a head-mounted device according to an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of a structure for implementing a remote control device based on a head-mounted device, according to an embodiment of this application.

[0051] Figure 4 This is a framework for the hardware configuration of a head-mounted device for implementing a remote control method based on a head-mounted device, according to an embodiment of this application. Figure 2 . Detailed Implementation

[0052] Various exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this application.

[0053] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0054] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0055] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0056] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0057] Figure 1 This is a framework for the hardware configuration of a head-mounted device for implementing a remote control method based on a head-mounted device, according to an embodiment of this application. Figure 1 .

[0058] The head-mounted device 1000 can be either a terminal or a server. Furthermore, the terminal can be a head-mounted device (such as an AR device, MR device, or XR device), a laptop, a tablet, a PDA, etc. The server can be a cloud server, etc.

[0059] The head-mounted device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and an image acquisition device 1900, etc. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a headphone jack, etc. The communication device 1400 may be capable of wired or wireless communication. The display device 1500 may be, for example, an LCD screen, a touch screen, etc. The input device 1600 may include, for example, a touch screen, a keyboard, etc. The user can input / output voice information through the speaker 1700 and the microphone 1800. The image acquisition device 1900 includes an external camera for capturing environmental images and an internal camera for capturing images of the wearer's eyes.

[0060] Despite Figure 1 The head-mounted device 1000 shows multiple devices, but this application may only involve some of them. For example, the head-mounted device 1000 may only involve the memory 1200, the processor 1100, and the image acquisition device 1900.

[0061] In the embodiments applied in this application, the memory 1200 of the head-mounted device 1000 is used to store instructions for controlling the processor 1100 to execute the remote control method based on the head-mounted device provided in the embodiments of this application.

[0062] In the above description, those skilled in the art can design instructions based on the scheme disclosed in this application. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0063] This application provides a remote control method based on a head-mounted device, which is applied to, for example... Figure 1 The head-mounted device shown, such as Figure 2 As shown, the method includes the following steps S2100 to S2300.

[0064] Step S2100: Establish a wireless communication connection between the head-mounted device and the controlled device.

[0065] In this embodiment, both the head-mounted device and the controlled device have Bluetooth / WIFI functionality, and a wireless communication connection is established between the head-mounted device and the controlled device based on the Bluetooth / WIFI functionality.

[0066] In one example, the controlled device could be an all-in-one monitor, camera, or other device located at a height.

[0067] Step S2200: Send control permission verification information to the controlled device via wireless communication connection.

[0068] Among them, the controlled device shall report the device information of the controlled device if the verification of the control authority information is correct.

[0069] Additionally, the control access verification information is used by the controlled device to verify whether the headset is an authenticated control device. In one example, the control access verification information is a verification code.

[0070] In one embodiment of this application, the specific implementation of step S2200 above can be as follows: control permission verification information is pre-stored in the head-mounted device; after the head-mounted device establishes a wireless communication connection with the controlled device, the head-mounted device actively sends the control permission verification information to the controlled device, or prompts the wearer to instruct the head-mounted device to send the control permission verification information to the controlled device.

[0071] In another embodiment of this application, the specific implementation of step S2200 above can also be: the wearer manually or verbally inputs control permission verification information to the head-mounted device, and after receiving the control permission verification information, the head-mounted device sends the control permission verification information to the controlled device.

[0072] In this embodiment, the head-mounted device sends control permission verification information to the controlled device based on the wireless communication connection between the head-mounted device and the controlled device. After receiving the control permission verification information sent by the head-mounted device, the controlled device verifies whether the control permission verification information is correct. If it is correct, the controlled device determines that the head-mounted device can be used as its control device, and at this time, the controlled device reports its own device information to the head-mounted device.

[0073] In one example, if the controlled device is an all-in-one monitor, the device information can specifically be the current display content of the all-in-one monitor. If the controlled device is a camera, the device information can specifically be the current monitoring content of the camera.

[0074] It is understandable that device information may also include device configuration information, etc. It should be noted that this application does not limit the specific content of the device information.

[0075] Step S2300: Upon receiving device information reported by the controlled device, display the device information.

[0076] Taking a camera as the controlled device and the camera's current monitoring content as the device information as an example, based on the above step S2300, the head-mounted device can display the camera's current monitoring content upon receiving it. In this way, the wearer can view the camera's current monitoring content through the head-mounted device without needing to download a related application on a computer or mobile phone and then open a specific interface to view the camera's current monitoring content. This solves the problem of complex operations for users when viewing information from controlled devices.

[0077] This embodiment provides a remote control method based on a head-mounted device. The method, applied to the head-mounted device, includes: establishing a wireless communication connection between the head-mounted device and the controlled device; sending control permission verification information to the controlled device via the wireless communication connection, wherein the controlled device reports its device information if the control permission verification information is correct; and displaying the device information upon receiving the reported device information from the controlled device. Through this method, the wearer can view the current monitoring content of a camera via the head-mounted device without needing to download a related application on a computer or mobile phone and open a specific interface to view the camera's current monitoring content. This solves the problem of complex operations for users when viewing information from a controlled device.

[0078] In one embodiment of this application, the remote control method based on a head-mounted device provided in this application further includes a step of selecting the controlled device before the above step S2100. This step is specifically implemented through the following steps S2110 to S2113.

[0079] Step S2110: Obtain the first external environment image.

[0080] In this embodiment, the head-mounted device includes an image acquisition unit, which includes an external camera for capturing environmental images. Based on this, a first external environmental image is acquired by obtaining the image captured by the external camera.

[0081] In one embodiment of this application, the control method based on a head-mounted device provided in this application may further include, before step S2110, outputting prompt information to the wearer to select the controlled device. The wearer controls the head-mounted device to face the controlled device based on this prompt information. In this case, when step S2110 is executed, a first external environment image including the controlled device can be acquired. Furthermore, this embodiment avoids the unnecessary acquisition of the first external environment image.

[0082] In one embodiment of this application, the above step S2110 can be specifically implemented by the following step S2110-1.

[0083] Step S2110-1: Upon receiving a viewing mode selection instruction input by the wearer of the head-mounted device, acquire the first external environment image.

[0084] In this embodiment, the head-mounted device has a viewing mode. When the wearer selects a viewing mode, the head-mounted device receives a viewing mode selection command input by the wearer. At this time, a first external environment image is acquired. Further, steps S2111 to S2113 are executed, and then steps S2200 and S2300 are executed. This allows the wearer to view the device information of the controlled device through the head-mounted device when selecting a viewing mode.

[0085] For the embodiment shown in step S2110-1 above, the remote control method based on a head-mounted device provided in this application further includes a step of controlling the device information displayed on the head-mounted device. Specifically, the remote control method based on a head-mounted device provided in this application further includes the following steps S2400 and S2500.

[0086] Step S2400: Receive the first control command input by the wearer.

[0087] Step S2500: Control the device information displayed on the head-mounted device according to the first control command.

[0088] In this embodiment, when the head-mounted device is in viewing mode, the head-mounted device records the control commands input by the wearer as the first control command. The first control command can be a specific gesture performed by the wearer, a specific voice, or a specific physical or virtual button triggered by the wearer.

[0089] In one example, when the first control command is a circle gesture, and the controlled device is an all-in-one display with the device information currently displayed, the controlled device performs a zoom-in operation on the currently displayed content of the all-in-one display. In this example, when the controlled device is located at a distance, the current display content of the controlled device on the head-mounted device can be zoomed in for the wearer to view clearly.

[0090] Step S2111: The device of a specified type contained in the first external environment image is identified as the controlled device.

[0091] The head-mounted device pre-stores the types of controlled devices that the head-mounted device can control. In this embodiment, the types of controlled devices that the head-mounted device can control are denoted as the set type.

[0092] After obtaining the first external environment image based on the above step S2110, the first environment image is identified to determine the device of a set type contained in the first external environment image as the controlled device.

[0093] In one embodiment of this application, the above step S2111 can be specifically implemented by the following step S2111-1.

[0094] Step S2111-1: If the device of the specified type is included in all the first external environment images within the preset time period, the device of the specified type is identified as the controlled device.

[0095] In this embodiment, when the wearer wants to designate a device of a certain type as the controlled device, they can face the device for at least a preset time to instruct the head-mounted device to identify the device as the controlled device. Based on this, if the first external environment image within the preset time period contains a device of the specified type, it indicates that the wearer has instructed that a device of the specified type, which is present in all of the first external environment images, be designated as the controlled device. At this time, the device of the specified type, which is present in all of the first external environment images, is determined to be the controlled device.

[0096] In one example of this application, the preset duration is 5 seconds.

[0097] Based on the above steps S2111-1, the problem of the head-mounted device mistakenly identifying a device as the controlled device when the wearer faces a device for a short period of time can be avoided.

[0098] Step S2112: Extract wireless connection information from the first environmental image.

[0099] The surface of the controlled device is equipped with wireless connection information.

[0100] In this embodiment, the surface of the controlled device is provided with wireless connection information. This wireless connection information, used to establish a wireless communication connection between the head-mounted device and the controlled device, includes parameters such as the controlled device's device identifier, device address, and UUID. The wireless connection information can be exemplarily a QR code.

[0101] When the controlled device is included in the first environmental image, the first environmental image contains wireless connection information disposed on the surface of the controlled device. Based on this, the wireless connection information can be extracted from the first environmental image.

[0102] Step S2113: Establish a wireless communication connection between the head-mounted device and the controlled device based on the wireless connection information.

[0103] In one embodiment of this application, the remote control method based on a head-mounted device provided in this application can also control the controlled device. Specifically, step S2100 is implemented through step S2120.

[0104] Step S2120: Upon receiving a control mode selection command input by the wearer of the head-mounted device, a wireless communication connection is established between the head-mounted device and the controlled device.

[0105] In this embodiment, the head-mounted device has a control mode. When the wearer selects a control mode, the head-mounted device receives a control mode selection command input by the wearer. At this time, a wireless communication connection is established between the head-mounted device and the controlled device, and then steps S2200 and S2300 are executed. This allows the wearer to view the device information of the controlled device through the head-mounted device when selecting a control mode. Based on this, the remote control method based on the head-mounted device provided in this application further includes the following steps S2600 and S2700.

[0106] Step S2600: Receive a second control command input by the wearer of the head-mounted device.

[0107] When the wearer wishes to control the device, they input a second control command into the head-mounted device. This second control command can be similar to the first control command, and can be a specific gesture, a specific voice command, or a specific physical or virtual button triggered by the wearer.

[0108] Understandably, the head-mounted device distinguishes between the first and second control commands based on the current operating mode.

[0109] Step S2700: Send a second control command to the controlled device based on the wireless communication connection.

[0110] After receiving the second control command sent by the head-mounted device, the controlled device responds to the second control command and performs the corresponding operation.

[0111] In one example, if the second control command is a left swipe gesture, and the controlled device is a camera and the device information is the current monitoring content, the controlled device will perform the operation of reviewing the historical monitoring content.

[0112] In another embodiment, step S2600 is specifically implemented by steps S2610 to S2612.

[0113] Step S2610: Obtain a second external environment image and an image of the wearer's eyes.

[0114] In this embodiment, after performing step S2300, the external camera of the head-mounted device captures an image of the external environment, and this image is recorded as the second external environment image. Simultaneously, the internal camera of the head-mounted device captures an image of the wearer's eyes.

[0115] Step S2611: Based on the second external environment image and the wearer's eye image, determine the wearer's operation gestures and the operation position in the device information.

[0116] In this embodiment, the wearer's gaze point is determined based on the wearer's eye image and eye-tracking algorithm. Furthermore, the wearer's gestures are determined by combining a second external environment image, and the wearer's operational position on the device information is determined by combining the second external image and the wearer's gaze point.

[0117] Step S2612: Determine the second control command based on the operation gesture and operation position.

[0118] In this embodiment, the corresponding control method is determined based on the operation gesture, and the second control instruction is determined as an instruction to execute the aforementioned control method for the operation position based on the control method and the operation position.

[0119] It is understood that in this embodiment, the second control command is specifically a gesture control command, and the wearer's gestures act as a mouse.

[0120] In one embodiment of this application, the head-mounted device includes at least two display areas. In one example, the head-mounted device includes a left-eye display screen and a right-eye display screen, each serving as a display area. Based on this, step S2100 can be specifically implemented through step S2130 below.

[0121] Step S2130: For any controlled device, establish a wireless communication connection between the head-mounted device and the controlled device.

[0122] In this embodiment, the head-mounted device can establish wireless communication connections with at least two controlled devices. Therefore, step S2300 is specifically implemented through step S2310.

[0123] Step S2310: Display the device information of different controlled devices in different display areas.

[0124] In conjunction with the above embodiments, taking two controlled devices, controlled device A and controlled device B, as examples, the remote control method based on a head-mounted device provided in this application specifically includes: outputting a prompt message to the wearer to select controlled device 1; acquiring a first external environment image containing controlled device 1; identifying a device of a set type contained in the first external environment image containing controlled device 1 as controlled device 1; extracting wireless connection information 1 of controlled device 1 from the first external environment image containing controlled device 1; establishing a wireless communication connection A between the head-mounted device and controlled device 1 based on the wireless connection information 1; sending control permission verification information a to controlled device 1 through wireless communication connection A; displaying the device information of controlled device 1 on the left eye display screen upon receiving device information reported by controlled device 1; repeating the above steps for controlled device 2, displaying the device information of controlled device 2 on the right eye display screen.

[0125] Based on the above, the wearer can view the device information of different controlled devices through the head-mounted device, so as to facilitate the wearer's comparison of the device information of different controlled devices.

[0126] In one embodiment of this application, step S2130 can be specifically implemented by step S2310-1 as described below.

[0127] Step S2310-1: Upon receiving a viewing comparison mode selection instruction input by the wearer of the head-mounted device, establish a wireless communication connection between the head-mounted device and the controlled device for any one of the controlled devices.

[0128] In another embodiment of this application, step S2130 can be specifically implemented by step S2310-2 as described below.

[0129] Step S2310-2: Upon receiving a control comparison mode selection instruction input by the wearer of the head-mounted device, establish a wireless communication connection between the head-mounted device and the controlled device for any one of the controlled devices.

[0130] The remote control method based on a head-mounted device provided in this application further includes: receiving a third control command input by the wearer of the head-mounted device for a target controlled device; and sending the third control command to the target controlled device based on a wireless communication connection with the target controlled device. Based on this, it is possible to control one of the controlled devices while displaying device information for different controlled devices on the head-mounted device.

[0131] In one embodiment of this application, the device information includes the current display content of the controlled device. The remote control method based on head-mounted devices provided in this application further includes the following steps S2800 to S21000.

[0132] Step S2800: If no device information is received from the controlled device, a third external environment image containing the current display content of the controlled device is obtained.

[0133] In cases where the verification information for the controlled device's control authority is incorrect, the device information of the controlled device will not be reported.

[0134] Step S2900: Extract the display content of the controlled device from the third external environment image.

[0135] Step S21000: Display the current display content of the controlled device.

[0136] In this embodiment, after receiving control permission verification information from the head-mounted device, the controlled device verifies the correctness of the verification information. If an error occurs, the controlled device determines that the head-mounted device cannot be used as its control device, and therefore does not report its current display content to the head-mounted device. At this time, the head-mounted device acquires a third external environment image containing the controlled device. Further, the current display content of the controlled device is extracted from the third external environment image and displayed on the head-mounted device. Thus, even when the head-mounted device cannot be used as the control device for the controlled device, the wearer can still view the current display content of the controlled device through the head-mounted device.

[0137] It is understood that any of the above embodiments can be reasonably combined. Based on this, by combining the embodiments shown in steps S2800 to S21000 and the embodiments shown in steps S2130 and S2310, it is possible to enable the wearer to view the current display content of at least two controlled devices through the head-mounted device even when the head-mounted device cannot act as a control device for at least two controlled devices, thereby achieving a comparison of the current display content of at least two controlled devices.

[0138] This application also provides a remote control device 300 based on a head-mounted device, such as... Figure 3As shown, the device is used in a head-mounted device and includes:

[0139] Module 310 is used to establish a wireless communication connection between the head-mounted device and the controlled device.

[0140] The sending module 320 is used to send control permission verification information to the controlled device through the wireless communication connection, wherein the controlled device reports its device information when it verifies that the control permission verification information is correct.

[0141] Display module 330 is used to display the device information when the device information reported by the controlled device is received.

[0142] In one embodiment of this application, the establishment module 310 is specifically used for:

[0143] Acquire the first image of the external environment;

[0144] The device of a specified type contained in the first external environment image is identified as the controlled device;

[0145] Wireless connection information is extracted from the first external environment image, and the surface of the controlled device is provided with wireless connection information;

[0146] Based on the wireless connection information, a wireless communication connection is established between the head-mounted device and the controlled device.

[0147] In one embodiment of this application, the establishment module 310 is specifically used for:

[0148] Upon receiving a viewing mode selection instruction input by the wearer of the head-mounted device, a first image of the external environment is acquired.

[0149] In one embodiment of this application, the remote control device 300 based on a head-mounted device provided in this application further includes:

[0150] The control module, used in the method, further includes:

[0151] Receive the first control command input by the wearer;

[0152] The device information displayed on the head-mounted device is controlled according to the first control command.

[0153] In one embodiment of this application, the establishment module 310 is specifically used for:

[0154] Upon receiving a control mode selection command input by the wearer of the head-mounted device, a wireless communication connection is established between the head-mounted device and the controlled device.

[0155] In this embodiment, the control module is further configured to receive a second control command input by the wearer of the head-mounted device;

[0156] The second control command is sent to the controlled device based on the wireless communication connection.

[0157] In one embodiment of this application, the control module is specifically used for:

[0158] Acquire a second image of the external environment and an image of the wearer's eyes;

[0159] Based on the second external environment image and the wearer's eye image, determine the wearer's operating gestures and the operating position in the device information;

[0160] The second control command is determined based on the operation gesture and operation position.

[0161] In one embodiment of this application, the head-mounted device includes at least two display areas, and the establishment module 310 is specifically used for:

[0162] For any of the controlled devices, the wireless communication connection between the head-mounted device and the controlled device is established;

[0163] The display module 330 is specifically used to display the device information of different controlled devices in different display areas.

[0164] In one embodiment of this application, the establishment module 310 is specifically used to: determine the device of the set type as the controlled device when the first external environment image within a preset time period contains a device of a set type.

[0165] In one embodiment of this application, the device information includes the current display content of the controlled device, and the display module 330 is further configured to:

[0166] If no device information is received from the controlled device, a third external environment image containing the current display content of the controlled device is acquired;

[0167] Extract the current display content of the controlled device from the third external environment image;

[0168] Display the current display content of the controlled device;

[0169] If the controlled device fails to verify the control permission verification information, it will not report the device information of the controlled device.

[0170] This application also provides a head-mounted device 400, which includes any of the remote control devices 300 based on the head-mounted device provided in the above device embodiments.

[0171] Or, such as Figure 4 As shown, the head-mounted device 410 includes a memory 410 and a processor 420. The memory 410 is used to store computer instructions, and the processor 420 is used to retrieve the computer instructions from the memory 410 to execute the method as described in any of the above method embodiments.

[0172] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the above-described method embodiments.

[0173] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0174] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0175] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0176] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0177] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0178] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0179] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0180] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0181] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A remote control method based on a head-mounted device, characterized in that, The method is applied to a head-mounted device and includes: Establish a wireless communication connection between the head-mounted device and the controlled device; The controlled device sends control permission verification information to the controlled device through the wireless communication connection, wherein the controlled device reports its device information if it verifies that the control permission verification information is correct. Upon receiving the device information reported by the controlled device, the device information is displayed.

2. The method according to claim 1, characterized in that, Establishing a wireless communication connection between the head-mounted device and the controlled device includes: Acquire the first image of the external environment; The device of a specified type contained in the first external environment image is identified as the controlled device; Wireless connection information is extracted from the first external environment image, and the surface of the controlled device is provided with wireless connection information; Based on the wireless connection information, a wireless communication connection is established between the head-mounted device and the controlled device.

3. The method according to claim 2, characterized in that, Establishing a wireless communication connection between the head-mounted device and the controlled device includes: Upon receiving a viewing mode selection instruction input by the wearer of the head-mounted device, a first image of the external environment is acquired.

4. The method according to claim 3, characterized in that, The method further includes: Receive the first control command input by the wearer; The device information displayed on the head-mounted device is controlled according to the first control command.

5. The method according to claim 1, characterized in that, Establishing a wireless communication connection between the head-mounted device and the first controlled device includes: Upon receiving a control mode selection command input by the wearer of the head-mounted device, a wireless communication connection is established between the head-mounted device and the controlled device. The method further includes: Receive a second control command input by the wearer of the head-mounted device; The second control command is sent to the controlled device based on the wireless communication connection.

6. The method according to claim 5, characterized in that, The receiving of a second control command input by the wearer of the head-mounted device includes: Acquire a second image of the external environment and an image of the wearer's eyes; Based on the second external environment image and the wearer's eye image, determine the wearer's operating gestures and the operating position in the device information; The second control command is determined based on the operation gesture and operation position.

7. The method according to claim 1, characterized in that, The head-mounted device includes at least two display areas, and establishing a wireless communication connection between the head-mounted device and the controlled device includes: For any of the controlled devices, the wireless communication connection between the head-mounted device and the controlled device is established; The display of the device information includes: The device information of different controlled devices is displayed in different display areas.

8. The method according to claim 2, characterized in that, The step of identifying the device of a predetermined type contained in the first external environment image as the controlled device includes: If a device of a specified type is included in all of the first external environment images within a preset time period, the device of the specified type is identified as the controlled device.

9. The method according to claim 1, characterized in that, The device information includes the current display content of the controlled device, and the method further includes: If no device information is received from the controlled device, a third external environment image containing the current display content of the controlled device is acquired; Extract the current display content of the controlled device from the third external environment image; Display the current display content of the controlled device; If the controlled device fails to verify the control permission verification information, it will not report the device information of the controlled device.

10. A remote control device based on a head-mounted display, characterized in that, The device is used in a head-mounted device and includes: A module is established to establish a wireless communication connection between the head-mounted device and the controlled device. The sending module is used to send control permission verification information to the controlled device through the wireless communication connection, wherein the controlled device reports its device information when it verifies that the control permission verification information is correct. The display module is used to display the device information when the device information reported by the controlled device is received.

11. A head-mounted device, characterized in that, The head-mounted device includes the apparatus as described in claim 10; or... The head-mounted device includes a memory and a processor, the memory for storing computer instructions, and the processor for retrieving the computer instructions from the memory to perform the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images