Electronic device control method, apparatus, device, and medium

Abstract

The application relates to the technical field of electronic equipment, and provides an electronic equipment control method and device, equipment and a storage medium, the method projects a screen display picture of a controlled device onto a projection display plane through a near-eye display device, constructs a screen coordinate system, can facilitate a user to directly view the display content of the controlled device through the near-eye display device, and realizes a screen coordinate system and a picture mapping relationship of the controlled device, and improves the operation accuracy of an eye movement tracking operation electronic equipment mode. An operation target is determined by recognizing a current instruction coordinate of a current user's eye fixation point in the screen coordinate system. An operation action of the user is recognized through a motion sensor, and then an instruction type is determined, so that the user can directly generate and issue an operation instruction through the near-eye display device, remote control of the controlled device is realized, a multi-end operation mode of the controlled device is avoided, and the control convenience of the electronic equipment is improved.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, and in particular to an electronic equipment control method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Currently, the control methods for electronic devices on the market, including mobile phones, computers, tablets, and smart TVs, generally fall into categories such as touch control, remote control, mouse, keyboard, and game controller. Except for smartphones, the others require additional devices to connect to the controlled device via wired / wireless connections. The control device then sends commands to the controlled device, which responds to the commands and performs the corresponding functions.

[0003] However, existing electronic device control methods have problems such as some control devices (e.g., remote controls) being easily lost, or some devices (keyboards) being too large and inconvenient to carry. Furthermore, operating electronic devices cannot be done without human hands; some devices (smartphones, keyboards, etc.) require two hands.

[0004] Therefore, improving the ease of operation of electronic devices has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for controlling electronic devices, aiming to improve the ease of operation of electronic devices.

[0006] In a first aspect, this application provides an electronic device control method, comprising:

[0007] When the near-eye display device is in the wearing state and the near-eye display device is in communication connection with the controlled device, the screen display image of the controlled device is projected onto the projection display plane of the near-eye display device to obtain the screen coordinate system;

[0008] Identify the current command coordinates of the current user's eye gaze point in the screen coordinate system;

[0009] Collect the current user's actions and determine the command type;

[0010] Based on the current command coordinates and the command type, a control command is generated and sent to the controlled device to control the controlled device to perform the operation corresponding to the control command.

[0011] Secondly, this application also provides an electronic device control device, comprising:

[0012] The screen coordinate system acquisition module is used to project the screen display image of the controlled device onto the projection display plane of the near-eye display device when the near-eye display device is in the wearing state and the near-eye display device is in a communication connection with the controlled device, so as to obtain the screen coordinate system.

[0013] The current command coordinate recognition module is used to identify the current command coordinates of the current user's eye gaze point in the screen coordinate system;

[0014] The instruction type determination module is used to collect the current user's operation actions and determine the instruction type.

[0015] The device control module is used to generate control commands based on the current command coordinates and the command type, and send the control commands to the controlled device to control the controlled device to perform the operation action corresponding to the control commands.

[0016] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the electronic device control method as described above.

[0017] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the electronic device control method described above.

[0018] This application provides a method, apparatus, device, and storage medium for controlling an electronic device. The method includes, when a near-eye display device is worn and communicatively connected to a controlled device, projecting the screen display of the controlled device onto the projection display plane of the near-eye display device to obtain a screen coordinate system; identifying the current command coordinates of the current user's eye gaze point in the screen coordinate system; acquiring the current user's operation actions and determining the command type; generating a control command based on the current command coordinates and the command type, and sending the control command to the controlled device to control the controlled device to execute the operation action corresponding to the control command. Through this method, this application projects the screen display of the controlled device onto the projection display plane using a near-eye display device to construct a screen coordinate system. This allows users to directly view the display content of the controlled device through the near-eye display device and establishes a mapping relationship between the screen coordinate system and the screen of the controlled device, improving the accuracy of operation based on eye-tracking operation of electronic devices. By identifying the current command coordinates of the current user's eye gaze point in the screen coordinate system and determining the operation target based on the mapping relationship between the screen coordinate system and the screen of the controlled device, hand-free operation is achieved, improving the ease of operation of the controlled device. By using motion sensors to identify user actions and determine the type of instruction, users can generate and issue operation instructions directly through near-eye display devices, thereby enabling remote control of the controlled device. This avoids the need for multiple operation methods on the controlled device and improves the ease of operation of electronic devices. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a first embodiment of an electronic device control method provided in this application.

[0021] Figure 2 A schematic diagram of coordinate tracking of human eye annotation points provided in an embodiment of this application;

[0022] Figure 3 This is a flowchart illustrating a second embodiment of an electronic device control method provided in this application.

[0023] Figure 4 This is a flowchart illustrating a third embodiment of an electronic device control method provided in this application.

[0024] Figure 5 This is a schematic diagram of the structure of a first embodiment of an electronic device control device provided in this application;

[0025] Figure 6 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0026] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0029] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0030] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a first embodiment of an electronic device control method provided in this application.

[0031] like Figure 1 As shown, the electronic device control method includes steps S101 to S104.

[0032] S101. When the near-eye display device is in the wearing state and the near-eye display device is in communication connection with the controlled device, the screen display image of the controlled device is projected onto the projection display plane of the near-eye display device to obtain the screen coordinate system.

[0033] In one embodiment, near-eye display, also known as head-mounted display or wearable display, can create virtual images in a monocular or binocular field of view. Near-eye display is a technology that uses a display device placed within the non-visual distance of the human eye to render light field information to the human eye, thereby reconstructing a virtual scene in front of the eyes.

[0034] For example, near-eye display devices may include wearable display devices such as smart glasses (such as AR glasses, VR glasses, etc.) and smart helmets.

[0035] In one embodiment, the controlled device may be a terminal device such as a mobile phone, tablet computer, or smart TV.

[0036] In one embodiment, the near-eye display device and the controlled device can communicate with each other via communication technologies such as data cable, Bluetooth, or WIFI.

[0037] In one embodiment, the wearing status of the near-eye display device can be detected by sensor devices such as pressure sensors and temperature sensors. For example, pressure sensors and / or temperature sensors can be set at the nose pad or ear pad of the smart glasses. When the pressure sensor detects continuous pressure, or the temperature sensor detects human body temperature, it can be determined that the near-eye display device is in the wearing state.

[0038] In one embodiment, the near-eye display device can also determine whether it is being worn using iris recognition technology. By detecting the current user's iris information, if the current user's iris information can be recognized and the current user's identity authentication is successful, the near-eye display device can be considered to be being worn and activated.

[0039] In one embodiment, the near-eye display device can also be manually activated by the current user, and it can be determined that the device is being worn upon activation.

[0040] In one embodiment, the wearing status of the near-eye display device can also be detected and determined by other methods. When the near-eye display device is detected to be in a wearing state, the device can search for nearby controllable devices that can be paired and connected (the controllable devices must have connection authorization functions and permissions enabled, such as Bluetooth, WIFI, etc.). When a controllable device is found, the device can actively choose to pair and establish a connection; or it can initiate a connection request to the controllable device. When the request is approved, a communication connection between the near-eye display device and the controllable device can be established.

[0041] Further, the screen display image of the controlled device is acquired; based on the near-eye display device, the screen display image is projected onto the projection display plane to obtain a projected display image; based on the pixel resolution of the projected display image, the screen coordinate system is constructed.

[0042] In one embodiment, the screen display of the controlled device can be transmitted to the near-eye display device via wired projection methods such as HDMI (High Definition Multimedia Interface) or DP (DisplayPort).

[0043] HDMI is a fully digital video and audio transmission interface that can send uncompressed audio and video signals. HDMI can be used in devices such as set-top boxes, DVD players, personal computers, televisions, game consoles, integrated amplifiers, digital audio systems, and televisions. HDMI can transmit audio and video signals simultaneously.

[0044] DisplayPort (DP) is a digital video interface standard developed by a consortium of PC and chip manufacturers and standardized by the Video Electronics Standards Association (VESA). DP is a display communication port that relies on packet-based data transmission technology. The DisplayPort protocol is based on small data packets called micro-packets, which can embed timer signals within the data stream. Its advantage is that it can achieve higher resolution with fewer pins. It can be used for both internal and external display connections. DP can transmit audio and video simultaneously, each of which can be transmitted independently without the other.

[0045] In one embodiment, the screen display of the controlled device can also be transmitted to the near-eye display device via wireless projection technology such as DLMA (Digital Living Network Alliance).

[0046] In one embodiment, DLNA provides wireless networking capabilities for audio and video devices, including computers, game consoles, digital camcorders, and portable / mobile products. It is used to enable the sharing of digital media and content services between compatible devices via a wireless network. DLNA requires interconnected devices to be on the same network segment, and because it is a protocol specifically designed for local area network (LAN) media playback, it offers high network transmission efficiency.

[0047] In one embodiment, the near-eye display device can also transmit the screen display of the controlled device to the near-eye display device through remote desktop technology or other means, as long as the same function can be achieved.

[0048] In one embodiment, the near-eye display device projects the display content of the controlled device onto a fixed plane (i.e., the projection display plane) at a distance Zk from the human eye to obtain a projected display image, such as... Figure 2 As shown.

[0049] In one embodiment, the content displayed on the projected screen is the same as the content displayed on the screen of the controlled device. Therefore, the coordinates of the user's gaze point in the projected screen can be mapped to the coordinates of the user's gaze point in the screen of the controlled device by capturing the coordinates of the user's gaze point in the projected screen.

[0050] In one embodiment, a screen coordinate system can be constructed based on the resolution of the projected display screen. For example, the first pixel at the top left corner of the projected display screen can be taken as the origin of the screen coordinate system. The horizontal edge of the projected display screen starting from the origin can be taken as the X-axis of the screen coordinate system. The vertical edge of the projected display screen starting from the origin can be taken as the Y-axis of the screen coordinate system. The straight line parallel to the normal of the projected display screen or the vertical line between the human eye and the projected display screen and starting from the origin can be taken as the Z-axis. The resolution of the projected display screen can be used as the coordinate points of the screen coordinate system in the X-axis and Y-axis directions.

[0051] S102. Identify the current command coordinates of the current user's eye gaze point in the screen coordinate system;

[0052] In one embodiment, eye-tracking technology can be used to track the current user's gaze point. Eye tracking is the process of measuring eye movements. The most important aspect of eye-tracking research is determining where humans or animals are looking (e.g., the "gaze point" or "fixation point"). More precisely, it involves using image processing techniques with instruments to locate the pupil position, obtain coordinates, and calculate the coordinates of the point where the eye is gazing or fixating using a specific algorithm.

[0053] For example, eye-tracking technology can be a "non-invasive" technology based on video-on-eye (VOG). Its basic principle is to point a beam of light (near-infrared light) and a camera at the subject's eyes, infer the direction of the subject's gaze through light and backend analysis, and the camera records the interaction process.

[0054] In one embodiment, the core of video-based eye tracking is one or more cameras that capture a series of images of the eyes. The eye-tracking software uses image processing algorithms to identify two key locations on each image sent by the eye-tracking camera—the pupil center and the corneal reflection center. The corneal reflection center is the point on the cornea where light emitted from a fixed light source (infrared illuminator) is reflected back from the cornea.

[0055] In one embodiment, the position of the pupil center on the camera sensor changes as the eye rotates; however, when the head is stable, the position of the corneal reflection center remains relatively fixed on the camera sensor (because the reflection source does not move relative to the camera). If the eye is completely fixed in space and simply rotates around its own center, the location of the gaze / fixation can be determined simply by tracking the change in the pupil center on the camera sensor.

[0056] In one embodiment, since the relationship between the camera and the eyes remains relatively fixed regardless of head movement when wearing a near-eye display device, gaze point recognition can be performed using only pupil-tracking eye-tracking technology in the near-eye display device.

[0057] In one embodiment, eye-tracking technology is used to identify the current user's gaze point and calculate the coordinates of the gaze point in the screen coordinate system as the current command coordinates. The control corresponding to the current command coordinates is then used as the target control to be manipulated.

[0058] S103. Collect the current user's operation actions and determine the instruction type;

[0059] In one embodiment, the instruction type may include movement, click, long press, and focus.

[0060] In one embodiment, the current user's actions can be collected by motion sensors, such as pressure sensors or inertial measurement units (IMUs).

[0061] In one embodiment, the pressure sensor can be located below the touch area of ​​the near-eye display device, and the touch area can be a button or a touch screen, etc.

[0062] For example, the touch area of ​​a near-eye display device can be configured according to the structure of the near-eye display device. For instance, smart glasses can have a button touch area on the temple or frame; helmet-style near-eye display devices can have a button touch area or a touch screen-type touch area on the side of the head.

[0063] For example, the touch area can be in button mode, with the pressure sensor located below the button. The user can activate the pressure sensor by pressing the button corresponding to the operation action, and generate a corresponding command signal so that the near-eye display device can recognize the command type.

[0064] For example, the touch area can be a touchscreen, and the pressure sensor can be placed below the touchscreen. Users can trigger operation commands by tapping controls on the touchscreen or by swiping, clicking, etc. The pressure sensor identifies the corresponding operation command by recognizing the user's operation action, such as swiping to switch pages or clicking controls to enter the details page.

[0065] In one embodiment, an inertial measurement unit (IMU) is a device that measures the three-axis attitude angles and accelerations of an object. An IMU typically includes a gyroscope, accelerometers, and sometimes a magnetometer. The gyroscope measures the angles / angular velocities along the three axes, the accelerometer measures the accelerations along the three axes, and the magnetometer provides information on the orientation of the magnetic field.

[0066] For example, an IMU can be installed in a near-eye display device to recognize user actions. For instance, a leftward yaw could represent movement, a rightward yaw a click, an upward yaw a long press, and a downward yaw a focus. If the IMU detects that the user yaws to the right at a certain angle, such as 60°, it can determine that the user has triggered a "click" command; or if the IMU detects that the user yaws to the right twice consecutively, it can determine that the user has triggered a "click" command.

[0067] For example, the IMU sensor can also be installed in a peripheral operating device, such as a smart ring. The smart ring communicates with a near-eye display device, and an operation pointer can be displayed on the projection display interface, corresponding to the position of the smart ring. The IMU sensor recognizes the user's hand gestures; for example, when the hand slides, the operation pointer moves synchronously on the projection display interface; when the hand makes a double-click, the operation pointer performs a click operation, etc. By collecting the user's hand gestures, the command type is determined.

[0068] S104. Based on the current instruction coordinates and the instruction type, generate a control instruction and send the control instruction to the controlled device to control the controlled device to perform the operation corresponding to the control instruction.

[0069] In one embodiment, the near-eye display device generates control commands based on the current command coordinates and command type. The near-eye display device can send control commands to the controlled device via wireless communication methods such as Bluetooth, infrared communication technology, and WIFI.

[0070] In one embodiment, after receiving a control command, the controlled device parses the current command coordinates and command type information contained in the control command. Then, based on the current command coordinates, it determines the target control; based on the command type, it determines the operation action; and then executes the control command to complete actions such as mouse clicks or cursor movement.

[0071] In this embodiment, the near-eye display device has the advantages of being small, easy to carry and store compared to operating devices such as remote controls and keyboards. Moreover, the head-mounted operating device can free up the user's hands to operate compared to operating devices such as remote controls and keyboards, thus improving the user's ease of operation of electronic devices.

[0072] This embodiment provides a method for controlling an electronic device. This method projects the screen display of the controlled device onto a projection display plane using a near-eye display device, establishing a screen coordinate system. This allows users to directly view the displayed content of the controlled device through the near-eye display device and achieves a mapping relationship between the screen coordinate system and the screen of the controlled device, improving the accuracy of operation based on eye-tracking. By identifying the current command coordinates of the user's eye gaze point in the screen coordinate system and determining the operation target based on the mapping relationship between the screen coordinate system and the screen of the controlled device, hand-free operation is achieved, improving the ease of operation of the controlled device. Through a motion sensor, the user's operation actions are identified, and the command type is determined, allowing the user to directly generate and issue operation commands through the near-eye display device. This enables remote control of the controlled device, avoiding multi-terminal operation methods and further improving the ease of operation of the electronic device.

[0073] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of an electronic device control method provided in this application.

[0074] like Figure 3 As shown, based on the above Figure 1 In the illustrated embodiment, step S102 specifically includes:

[0075] Further, step S102 specifically includes:

[0076] S201. Based on eye-tracking technology, identify the current gaze coordinates of the human eye's gaze point in the gaze plane coordinate system;

[0077] The gaze plane coordinate system is constructed on the gaze plane, which coincides with the projection display plane, and the origin of the gaze plane coordinate system coincides with the origin of the screen coordinate system.

[0078] In one embodiment, eye-tracking technology is used to identify the current user's gaze point and calculate the current gaze coordinates of the gaze point in the gaze plane coordinate system. It can be assumed that the origin of the coordinate system for the gaze point is O(X). o ,Y o Z o ) and the origin of the display device screen coordinate system (X a ,Y a Za (Overlap). The screen coordinates of the point of human eye fixation are the offset of the point of fixation relative to the origin of the screen coordinate system (the first pixel at the top left corner of the projected display); because the projected display plane is located at a fixed distance Z from the center line of the user's field of vision. k The position of the gaze point is such that the projection display plane and the annotation plane of the human eye are on the same plane. Therefore, the depth information (Z-axis coordinate) of the gaze point can be ignored, and the two-dimensional coordinates of the human eye's gaze point in the screen coordinate system can be directly identified.

[0079] Further, the size parameters and screen resolution of the screen display are obtained; based on the size parameters and screen resolution, the pixel density of the screen display is calculated.

[0080] In one embodiment, parameters of the display device of the controlled device are obtained, such as: width W and height H, in millimeters (mm), and screen resolution width S. w Height S h The unit is pixels (px).

[0081] In one embodiment, when the controlled device and the near-eye display device establish a communication connection, if this is the first time the controlled device and the near-eye display device have paired up, the near-eye display device can read the device information (such as device number and device name) of the controlled device and the parameters of the display device from the controlled device, and can save the device information and parameters of the controlled device through archiving or other methods. If this is not the first time the controlled device and the near-eye display device have paired up, the near-eye display device can obtain the device information of the controlled device and, based on the device information, query the parameters of the display device of the controlled device from the archive.

[0082] In one embodiment, pixel density (PPM) represents the number of pixels per inch. Therefore, a higher PPM value indicates that the display can show images at a higher density. Pixel density includes both horizontal and vertical pixel density.

[0083] In one embodiment, the size parameters of the screen display include the length and width of the screen display, and the screen resolution includes the length and width pixel values ​​of the screen display.

[0084] Further, based on the length dimension and the length pixel value, the pixel density of the screen display in the horizontal direction is calculated; based on the width dimension and the width pixel value, the pixel density of the screen display in the vertical direction is calculated.

[0085] In one embodiment, based on the screen display size parameters and screen resolution: the size width W and height H, in millimeters (mm), and the screen resolution width S... w Height S h The unit is pixels (px), which can be used to calculate the horizontal pixel density W of the screen display. ppd =S w / W, and the pixel density H in the vertical direction of the screen display. ppm =S h / H.

[0086] S202. Based on the current gaze coordinates and the pixel density of the screen display, calculate the current command coordinates of the human eye gaze point in the screen coordinate system.

[0087] In one specific embodiment, the coordinates of the human eye's gaze point in the gaze plane coordinate system are obtained using eye-tracking technology as K(X). k ,Y k Z k The unit is millimeters. Then the point of fixation for the human eye is K(X). k ,Y k Z k The screen coordinates (X, Y) on the display screen of the controlled device are calculated by the following formula, in pixels (px).

[0088]

[0089] In one embodiment, since the projected display image is obtained by projecting the display image of the controlled device, the parameters (resolution and pixel density) of the screen display image of the controlled device can be used as the image parameters of the projected display image to obtain the target.

[0090] In this embodiment, eye-tracking technology is used to obtain the coordinates of the human eye's gaze point, while screen projection technology is used to output the screen display content of the control device to the head-mounted display device. The gaze point coordinates are converted into the pixel coordinates of the controlled device's display screen to obtain the coordinates of the human eye's gaze point relative to the display device's screen, achieving a hovering effect similar to a mouse pointer or finger touch. This eliminates the dependence on "hand" operation; control of the device can be achieved solely using the eye's gaze point, improving the ease of operation of electronic devices.

[0091] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating a third embodiment of an electronic device control method provided in this application.

[0092] like Figure 4 As shown, based on the above Figure 1 In the illustrated embodiment, step S104 further includes:

[0093] S301. Determine the target control based on the preset control information table and the current instruction coordinates;

[0094] In one embodiment, after identifying the current user's eye gaze point in the screen coordinate system using eye-tracking technology, the corresponding control coordinate area can be queried in the control information table. For example, if the current command coordinate is (20,30), it can be determined that the current command coordinate is located in the coordinate area [(10,10),(40,40)], so the target control pointed to by the current command coordinate is control A.

[0095] Furthermore, before step S301, the method further includes: obtaining control information of at least one control in the screen display; determining the control coordinate region corresponding to each control based on the screen coordinate system; associating the control information and the control coordinate region to construct a control information table for the controlled device.

[0096] In one embodiment, the controlled device can be an electronic device such as a mobile phone, computer, or tablet. The electronic device contains at least one application control, such as a chat application control or a game application control, which can send the application control information from the controlled device to the near-eye display device. The application control refers to the encapsulated result of the application's data and operating method, and can serve as an interaction button between the near-eye display device and the application in the controlled device, used for inputting, editing, or displaying data.

[0097] In one embodiment, the control information may include information such as control name, control type, and control function.

[0098] In one embodiment, the position and size of controls on the screen display can be determined, and the control coordinate area in the screen coordinate system can be calculated. Since the coordinate axes in the screen coordinate system use the pixel resolution of the screen display as coordinate points, and controls typically occupy a uniform area on the screen display, containing multiple pixels, the control coordinates should be a coordinate area containing multiple pixel coordinate points. For example, control A[(10,10),(40,40)] indicates that the control coordinate area of ​​control A is the coordinate area enclosed by coordinate points (10,10), (10,40), (40,10), and (40,40).

[0099] In one embodiment, the control information is associated with the control's corresponding coordinate region, that is, a unique correspondence between the control and the control's coordinate region is established, and a control information table is created so that the corresponding coordinate region can be matched according to the coordinates of the human eye's gaze point, and then the target control can be queried.

[0100] S302. Based on the instruction type, determine the operation action corresponding to the control instruction;

[0101] In one embodiment, the instruction type may include control operation methods such as move, click, long press, focus, etc. The operation action that the current user wants to perform on the target control can be determined according to the instruction type, and then the controlled device can be controlled to perform the operation action on the target control.

[0102] S303. Based on the control command, control the controlled device to perform the operation action on the target control.

[0103] In one embodiment, the near-eye display device can package the current command coordinates and command type to generate a control command, send the control command to the controlled device, and the controlled device can parse the control command to determine the target control corresponding to the current command coordinates and the operation action corresponding to the command type, and then perform the operation action on the target control.

[0104] In one embodiment, the near-eye display device may also pre-configure a control information table for the controlled device, match the target control in the control information table according to the current instruction coordinates, package the target control and instruction type to generate a control instruction, send the control instruction to the controlled device, and have the controlled device parse the control instruction and perform the operation action corresponding to the instruction type on the target control.

[0105] In this embodiment, by constructing a control information table for the controlled device, querying the target control corresponding to the current instruction coordinates through the control information table, and then combining the target control and instruction type, a control instruction is generated to control the controlled device to perform the corresponding operation action on the target control, thereby improving the control efficiency and accuracy of the controlled device.

[0106] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a first embodiment of an electronic device control device provided in this application. The electronic device control device is used to execute the aforementioned electronic device control method.

[0107] like Figure 5 As shown, the electronic device control device 400 includes: a screen coordinate system acquisition module 401, a current command coordinate recognition module 402, a command type determination module 403, and a device control module 404.

[0108] The screen coordinate system acquisition module 401 is used to project the screen display image of the controlled device onto the projection display plane of the near-eye display device when the near-eye display device is in the wearing state and the near-eye display device is in communication connection with the controlled device, so as to obtain the screen coordinate system.

[0109] The current command coordinate recognition module 402 is used to identify the current command coordinates of the current user's eye gaze point in the screen coordinate system;

[0110] The instruction type determination module 403 is used to collect the current user's operation actions and determine the instruction type;

[0111] The device control module 404 is used to generate a control command based on the current command coordinates and the command type, and send the control command to the controlled device to control the controlled device to perform the operation action corresponding to the control command.

[0112] In one embodiment, the screen coordinate system acquisition module 401 includes:

[0113] A screen display acquisition unit is used to acquire the screen display of the controlled device.

[0114] The projection display image acquisition unit is used to project the screen display image onto the projection display plane based on the near-eye display device to obtain the projection display image;

[0115] The screen coordinate system construction unit is used to construct the screen coordinate system based on the pixel resolution of the projected display image.

[0116] In one embodiment, the current command coordinate recognition module 402 includes:

[0117] The current annotation coordinate recognition unit is used to identify the current gaze coordinates of the human eye gaze point in the gaze plane coordinate system based on eye-tracking technology;

[0118] The current command coordinate calculation unit is used to calculate the current command coordinates of the human eye's gaze point in the screen coordinate system based on the current gaze coordinates and the pixel density of the screen display.

[0119] The gaze plane coordinate system is constructed on the gaze plane, which coincides with the projection display plane, and the origin of the gaze plane coordinate system coincides with the origin of the screen coordinate system.

[0120] In one embodiment, the current command coordinate recognition module 402 further includes:

[0121] A screen parameter acquisition unit is used to acquire the size parameters and screen resolution of the screen display image;

[0122] A pixel density calculation unit is used to calculate the pixel density of the screen display based on the size parameters and the screen resolution.

[0123] In one embodiment, the size parameters of the screen display include the length and width of the screen display, and the screen resolution includes the length and width pixel values ​​of the screen display;

[0124] The pixel density calculation unit includes:

[0125] A horizontal pixel density calculation subunit is used to calculate the pixel density of the screen display in the horizontal direction based on the length dimension and the length pixel value;

[0126] A vertical pixel density calculation subunit is used to calculate the pixel density of the screen display in the vertical direction based on the width dimension and the width pixel value.

[0127] In one embodiment, the device control module 404 includes:

[0128] The target control determination unit is used to determine the target control based on a preset control information table and the current instruction coordinates;

[0129] An operation action determination unit is used to determine the operation action corresponding to the control instruction based on the instruction type.

[0130] The device control unit is used to control the controlled device to perform the operation action on the target control based on the control command.

[0131] In one embodiment, the electronic device control device 400 further includes:

[0132] A control information acquisition unit is used to acquire control information of at least one control in the screen display.

[0133] The control coordinate region determination unit is used to determine the control coordinate region corresponding to each control based on the screen coordinate system.

[0134] The control information table construction unit is used to associate the control information with the control coordinate area to construct the control information table of the controlled device.

[0135] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described device and each module can be referred to the corresponding processes in the aforementioned embodiments of the electronic device control method, and will not be repeated here.

[0136] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.

[0137] Please see Figure 6 , Figure 6 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0138] See Figure 6 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0139] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any electronic device control method.

[0140] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0141] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, they enable the processor to perform any electronic device control method.

[0142] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0143] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0144] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0145] When the near-eye display device is in the wearing state and the near-eye display device is in communication connection with the controlled device, the screen display image of the controlled device is projected onto the projection display plane of the near-eye display device to obtain the screen coordinate system;

[0146] Identify the current command coordinates of the current user's eye gaze point in the screen coordinate system;

[0147] Collect the current user's actions and determine the command type;

[0148] Based on the current command coordinates and the command type, a control command is generated and sent to the controlled device to control the controlled device to perform the operation corresponding to the control command.

[0149] In one embodiment, when the processor implements the near-eye display device to project the screen display image of the controlled device onto a projection display plane and obtain the screen coordinate system, it is used to:

[0150] Acquire the screen display of the controlled device;

[0151] Based on the near-eye display device, the screen display image is projected onto the projection display plane to obtain the projected display image;

[0152] The screen coordinate system is constructed based on the pixel resolution of the projected display image.

[0153] In one embodiment, when the processor implements the instruction to identify the current user's eye gaze point in the screen coordinate system, it is configured to:

[0154] Based on eye-tracking technology, the current gaze coordinates of the human eye's gaze point in the gaze plane coordinate system are identified;

[0155] Based on the current gaze coordinates and the pixel density of the screen display, calculate the current command coordinates of the human eye gaze point in the screen coordinate system;

[0156] The gaze plane coordinate system is constructed on the gaze plane, which coincides with the projection display plane, and the origin of the gaze plane coordinate system coincides with the origin of the screen coordinate system.

[0157] In one embodiment, before the processor calculates the current command coordinates of the human eye's gaze point in the screen coordinate system based on the current gaze coordinates and the pixel density of the screen display, it is further configured to:

[0158] Obtain the size parameters and screen resolution of the screen display;

[0159] The pixel density of the screen display is calculated based on the size parameters and the screen resolution.

[0160] In one embodiment, the size parameters of the screen display include the length and width of the screen display, and the screen resolution includes the length and width pixel values ​​of the screen display;

[0161] When the processor calculates the pixel density of the screen display based on the size parameters and the screen resolution, it is used to:

[0162] Based on the length dimension and the length pixel value, calculate the pixel density of the screen display in the horizontal direction;

[0163] Based on the width dimension and the width pixel value, the pixel density of the screen display image in the vertical direction is calculated.

[0164] In one embodiment, when the processor generates control instructions based on the current instruction coordinates and the instruction type, and sends the control instructions to the controlled device to control the controlled device to execute the operation action corresponding to the control instructions, it is configured to:

[0165] The target control is determined based on the preset control information table and the current instruction coordinates;

[0166] Based on the instruction type, determine the operation action corresponding to the control instruction;

[0167] Based on the control command, the controlled device is controlled to perform the operation action on the target control.

[0168] In one embodiment, before determining the target control based on the preset control information table and the current instruction coordinates, the processor is further configured to implement:

[0169] Obtain control information for at least one control in the screen display;

[0170] Based on the screen coordinate system, determine the control coordinate area corresponding to each control;

[0171] The control information and the control coordinate area are associated to construct the control information table of the controlled device.

[0172] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the electronic device control methods provided in the embodiments of this application.

[0173] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the computer device.

[0174] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for controlling an electronic device, characterized in that, The method includes: When the near-eye display device is in the wearing state and the near-eye display device is in communication connection with the controlled device, the screen display image of the controlled device is projected onto the projection display plane of the near-eye display device to obtain the screen coordinate system; Identify the current command coordinates of the current user's eye gaze point in the screen coordinate system; Collect the current user's actions and determine the command type; Based on the current command coordinates and the command type, a control command is generated and sent to the controlled device to control the controlled device to perform the operation action corresponding to the control command; The step of identifying the current user's eye gaze point in the screen coordinate system includes the following: Based on eye-tracking technology, the current gaze coordinates of the human eye's gaze point in the gaze plane coordinate system are identified; Based on the current gaze coordinates and the pixel density of the screen display, calculate the current command coordinates of the human eye gaze point in the screen coordinate system; The gaze plane coordinate system is constructed on the gaze plane, which coincides with the projection display plane, and the origin of the gaze plane coordinate system coincides with the origin of the screen coordinate system.

2. The electronic device control method according to claim 1, characterized in that, The step of projecting the screen display image of the controlled device onto the projection display plane of the near-eye display device to obtain the screen coordinate system includes: Acquire the screen display of the controlled device; Based on the near-eye display device, the screen display image is projected onto the projection display plane to obtain the projected display image; The screen coordinate system is constructed based on the pixel resolution of the projected display image.

3. The electronic device control method according to claim 1, characterized in that, Before calculating the current command coordinates of the human eye's gaze point in the screen coordinate system based on the current gaze coordinates and the pixel density of the screen display, the method further includes: Obtain the size parameters and screen resolution of the screen display; The pixel density of the screen display is calculated based on the size parameters and the screen resolution.

4. The electronic device control method according to claim 3, characterized in that, The screen display size parameters include the length and width of the screen display, and the screen resolution includes the length and width pixel values ​​of the screen display; The step of calculating the pixel density of the screen display based on the size parameter and the screen resolution includes: Based on the length dimension and the length pixel value, calculate the pixel density of the screen display in the horizontal direction; Based on the width dimension and the width pixel value, the pixel density of the screen display image in the vertical direction is calculated.

5. The electronic device control method according to claim 1, characterized in that, The step of generating a control command based on the current command coordinates and the command type, and sending the control command to the controlled device to control the controlled device to execute the operation action corresponding to the control command, includes: The target control is determined based on the preset control information table and the current instruction coordinates; Based on the instruction type, determine the operation action corresponding to the control instruction; Based on the control command, the controlled device is controlled to perform the operation action on the target control.

6. The electronic device control method according to claim 5, characterized in that, Before determining the target control based on the preset control information table and the current instruction coordinates, the process also includes: Obtain control information for at least one control in the screen display; Based on the screen coordinate system, determine the control coordinate area corresponding to each control; The control information and the control coordinate area are associated to construct the control information table of the controlled device.

7. An electronic device control device, characterized in that, The electronic device control unit includes: The screen coordinate system acquisition module is used to project the screen display image of the controlled device onto the projection display plane of the near-eye display device when the near-eye display device is in the wearing state and the near-eye display device is in a communication connection with the controlled device, so as to obtain the screen coordinate system. The current command coordinate recognition module is used to identify the current command coordinates of the current user's eye gaze point in the screen coordinate system; The instruction type determination module is used to collect the current user's operation actions and determine the instruction type. The device control module is used to generate control commands based on the current command coordinates and the command type, and send the control commands to the controlled device to control the controlled device to perform the operation action corresponding to the control commands; Specifically, the current command coordinate recognition module is further used for: Based on eye-tracking technology, the current gaze coordinates of the human eye's gaze point in the gaze plane coordinate system are identified; Based on the current gaze coordinates and the pixel density of the screen display, calculate the current command coordinates of the human eye gaze point in the screen coordinate system; The gaze plane coordinate system is constructed on the gaze plane, which coincides with the projection display plane, and the origin of the gaze plane coordinate system coincides with the origin of the screen coordinate system.

8. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the electronic device control method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the electronic device control method as described in any one of claims 1 to 6.

Images