Method for mapping a mouse pointer with a hand, and electronic device and readable medium therefor

Abstract

The application relates to the field of computer vision, and discloses a hand mapping mouse pointer method, an electronic device and a readable medium. The hand mapping mouse pointer method comprises the following steps: acquiring an image of a user of an electronic device, the image at least containing an image of a hand of the user, and performing hand recognition on the image of the user to determine the position of a hand joint; determining the current operating hand of the user according to the change of the position of the hand joint; mapping a first position where the current operating hand of the user is located to a second position on a screen of the electronic device, displaying a mouse pointer at the second position on the screen, and moving the mouse pointer on the screen of the electronic device according to the moving track of the current operating hand in a control coordinate system, so as to realize the movement of the mouse pointer on the screen of the electronic device controlled by the hand of the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a method for mapping a hand to a mouse pointer, and an electronic device and a readable medium. BACKGROUND

[0002] With the development of computer vision (CV) technology, various application technologies based on computer vision technology for tracking and processing human limb movements have emerged, and the air mouse is one of them. The air mouse takes the user's hand as a mouse, accurately identifies the user's hand and precisely tracks the hand movement trajectory through computer vision technology, and maps the movement of the user's hand to the movement of the corresponding mouse pointer on the screen of the electronic device, so as to realize the control of the electronic device by the user directly through the hand movement without the need of an additional mouse device. SUMMARY

[0003] The embodiments of the present application provide a method for mapping a hand to a mouse pointer, and an electronic device and a readable medium, which can map the user's hand to the mouse pointer on the screen of the electronic device, and then realize the control of the movement of the mouse pointer on the screen by the hand movement.

[0004] In a first aspect, the embodiments of the present application provide a method for mapping a hand to a mouse pointer, and the method comprises:

[0005] obtaining an image of a user of an electronic device, the image containing at least an image of a hand of the user, and performing hand recognition on the image of the user to determine the position of a hand joint; determining a current operating hand of the user according to the change of the position of the hand joint; mapping a first position where the current operating hand of the user is located to a second position on a screen of the electronic device, displaying a mouse pointer at the second position on the screen, and moving the mouse pointer on the screen according to the movement trajectory of the current operating hand of the user in a control coordinate system.

[0006] In a possible implementation of the first aspect, the obtaining of the image of the user of the electronic device comprises:

[0007] obtaining the image of the user of the electronic device through a camera.

[0008] In a possible implementation of the first aspect, the determining of the current operating hand of the user according to the change of the position of the hand joint comprises:

[0009] The user's moving hand is determined based on the changes in the position of the hand joints. Specifically, the changes in the position of the hand joints are obtained by comparing the positions of corresponding joints in multiple sets of hand joints. The positions of the multiple sets of hand joints are obtained by performing hand recognition on images containing at least the user's hand in multiple consecutive frames. The moving hand is identified as the user's current operating hand, and left or right hand recognition is performed to determine whether the user's current operating hand is the left hand or the right hand.

[0010] In one possible implementation of the first aspect described above, after determining the user's current operating hand based on changes in the position of the hand joints, the method further includes:

[0011] The user's current operating hand is compared with the previous operating hand. If the user's current operating hand is different from the previous operating hand, a new control coordinate system is established based on the user's current operating hand.

[0012] In one possible implementation of the first aspect described above, mapping the first position of the user's current operating hand to a second position on the screen of the electronic device, displaying a mouse pointer at the second position on the screen, and moving the mouse pointer on the screen of the electronic device according to the movement trajectory of the user's current operating hand in the control coordinate system includes:

[0013] Establish a corresponding control coordinate system based on the user's current operating hand, map the first coordinate of the user's current operating hand in the control coordinate system to the second coordinate of the second position on the screen of the electronic device, and display the mouse pointer at the second coordinate on the screen; move the mouse pointer on the screen of the electronic device accordingly based on the movement trajectory of the user's current operating hand in the control coordinate system.

[0014] In one possible implementation of the first aspect above, a corresponding control coordinate system is established based on the user's current operating hand, including:

[0015] The position of the origin of the control coordinate system is determined based on the user's physical characteristics, and the corresponding horizontal and vertical coordinate axes are determined based on the position of the origin.

[0016] In one possible implementation of the first aspect above, a corresponding control coordinate system is established based on the user's current operating hand, including:

[0017] The position of the origin of the control coordinate system is determined based on the natural position of the user's current operating hand, and the corresponding horizontal and vertical coordinate axes are determined based on the position of the origin.

[0018] In one possible implementation of the first aspect above, a corresponding control coordinate system is established based on the user's current operating hand, including:

[0019] The position of the origin of the control coordinate system is determined based on the operating area of ​​the electronic device's screen, and the corresponding horizontal and vertical coordinate axes are determined based on the position of the origin.

[0020] In one possible implementation of the first aspect above, the method further includes:

[0021] The position of the origin of the control coordinate system is adjusted based on the user's operating habits.

[0022] In one possible implementation of the first aspect above, the method further includes:

[0023] The control coordinate system is translated or rotated based on the user's position and posture.

[0024] In a second aspect, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the system, and a processor, one of the processors of the system, for performing any of the possible methods described in the first aspect.

[0025] Thirdly, embodiments of this application provide a machine-readable medium, characterized in that the machine-readable medium stores instructions that, when executed on a machine, cause the machine to perform any of the possible methods described in the first aspect. Attached Figure Description

[0026] Figure 1 According to some embodiments of this application, a scenario 10 of mapping a hand to a mouse pointer is shown.

[0027] Figure 2 According to some embodiments of this application, a comparative schematic diagram of mouse pointer positions generated by prior art is shown.

[0028] Figure 3 According to some embodiments of this application, a structural schematic diagram of a large-screen display device 200 is shown.

[0029] Figure 4 According to some embodiments of this application, a flowchart of a method for mapping a hand to a mouse pointer is shown.

[0030] Figure 5 According to some embodiments of this application, a schematic diagram of a human joint is shown.

[0031] Figure 6 According to some embodiments of this application, a schematic diagram of constructing a hand recognition model is shown.

[0032] Figure 7 According to some embodiments of this application, a schematic diagram of recognizing a hand image using a hand recognition model is shown.

[0033] Figure 8 According to some embodiments of this application, a schematic diagram of a hand joint is shown.

[0034] Figure 9 According to some embodiments of this application, a schematic diagram of hand movement is shown.

[0035] Figure 10 According to some embodiments of this application, a flowchart of a method for mapping a hand to a mouse pointer is shown.

[0036] Figure 11 According to some embodiments of this application, a flowchart of a method for mapping a hand to a mouse pointer is shown.

[0037] Figure 12 According to some embodiments of this application, a software structure block diagram of an electronic device is shown. Detailed Implementation

[0038] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of the embodiments. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of this application, and not all structures.

[0039] Figure 1 According to some embodiments of this application, a scenario 10 of mapping a hand to a mouse pointer is shown. Figure 1 As shown, user 100 needs to control the content displayed on electronic device 200. User 100 moves the mouse pointer 300 displayed on the screen of electronic device 200 by moving their hand, thus achieving human-computer interaction. For example, if user 100 moves their hand to the right, the mouse pointer 300 moves to the right accordingly; if user 100 moves their hand upward, the mouse pointer 300 moves upward accordingly; if user 100's hand movement traces a circle, the mouse pointer 300's movement traces a circle on the screen accordingly, and so on. By mapping the user's hand movement to the corresponding mouse pointer movement, there is no need to provide electronic device 200 with a dedicated mouse control device, while facilitating user 100's control of electronic device 200 and enhancing the user's human-computer interaction experience.

[0040] In existing technologies, when mapping a user's hands, the user's left and right hands are mapped to mouse pointers in a two-dimensional coordinate system (including X and Y coordinates), with the origin of this coordinate system fixed at the upper left corner of the screen of the electronic device 200. Because the origin of the two-dimensional coordinate system is fixed, and the user's left and right hands are mapped to their respective mouse pointers, the mapping of the left and right hands does not consider the most comfortable operating range for each hand. This makes it relatively difficult to use the left hand to operate interface elements on the right side of the screen of the electronic device 200 within the same coordinate system; the user may need to move their own position to move the mapped mouse pointer to the desired interface element. Similarly, it is also relatively difficult to use the right hand to operate interface elements on the left side of the screen. Furthermore, even if the user does not move their own position but moves the mapped mouse pointer through more complex body movements, the camera may not be able to capture a complete frontal image of the hand. For example, if the user uses their right hand to operate the interface elements on the left side of the screen of the electronic device 200, the mapped mouse pointer of the right hand may have difficulty reaching the corresponding interface elements. The user may try to extend their right hand to the left, making it impossible for their palm to be completely parallel to the screen. As a result, the hand image captured by the camera will be obstructed, which will adversely affect the subsequent user gesture recognition and lead to a decrease in gesture recognition accuracy.

[0041] This application discloses a hand-mapping mouse pointer method. Specifically, it determines the user's current operating hand by recognizing an image containing the user's hand, establishes a corresponding control coordinate system based on the user's current operating hand, and then maps the movement of the user's operating hand in the control coordinate system to the movement of the mouse pointer on the electronic device screen. This allows the user to move the mouse pointer across the entire electronic device screen by moving their current operating hand within a relatively small range, without the user needing to change their own position or perform any difficult physical movements, thus facilitating the user's control of the electronic device.

[0042] The effects of the technical solutions according to the embodiments of this application can be referred to Figure 2 The illustration. (See diagram.) Figure 2As shown, in the prior art scheme where the origin of the control coordinate system is fixed, mapping the coordinates of the user's right hand relative to the upper left corner of the screen to the corresponding mouse pointer 1302 requires the user's right hand to move a relatively long distance to move the mouse pointer 1302 to the left side of the screen. On the other hand, in a comparison, in the embodiment corresponding to this application where a right-hand control coordinate system is established based on the user's currently used right hand, and the coordinates of the user's right hand in the right-hand control coordinate system are mapped to the screen, the mouse pointer corresponding to the user's right hand is 1301. It can be seen that mouse pointer 1301 is positioned further to the left than mouse pointer 1302. Therefore, in the embodiment where the origin of the control coordinate system is not fixed, the user's right hand only needs to move a shorter distance within a more comfortable area closer to the right side of the body to move the mouse pointer 1301 to the left side of the screen, making operation of the entire screen space more convenient for the user.

[0043] It should be noted that, in specific embodiments of this application, the electronic device 200 includes, but is not limited to, large-screen display devices, smart TVs, tablet computers, smartphones, laptops, desktop computers, and other electronic devices. In the following description, for the sake of simplicity, a large-screen display device 200 will be used as an example to illustrate the technical solution of this application.

[0044] Figure 3 According to some embodiments of this application, a structural schematic diagram of a large-screen display device 200 is shown. Specifically, as... Figure 3 As shown, the large-screen display device 200 includes a processor 201, a memory 202, a camera 203, a power switch 204, a communication processing module 205, a large display screen 206, and an audio module 207, etc.

[0045] Processor 201 can be used to read and execute computer-readable instructions. In a specific implementation, processor 201 mainly includes a controller, an arithmetic logic unit (ALU), and registers. The controller is primarily responsible for instruction decoding and issuing control signals for the operations corresponding to the instructions. The ALU is primarily responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations, and can also perform address operations and translations. Registers are primarily responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. In a specific implementation, the hardware architecture of processor 201 can be an application-specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, or an NP architecture, etc.

[0046] Processor 201 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0047] For example, an NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by drawing inspiration from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs can enable intelligent cognitive applications on large-screen display devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0048] In some embodiments of this application, the NPU can recognize the user's human body contour through a human pose recognition model and annotate the coordinates of the user's human body contour joints. Furthermore, the NPU can recognize the user's hand contour through a gesture recognition model and annotate the coordinates of the user's hand joints.

[0049] Among them, the human pose recognition model includes, but is not limited to, single shot multibox detector (SSD), region convolutional neural network (R-CNN), fast region convolutional neural network (Fast R-CNN), etc., and the gesture recognition model can be a lightweight network (Mobile-Net). This application embodiment does not limit this.

[0050] Memory 202 is coupled to processor 201 and is used to store various software programs and / or multiple sets of instructions. In specific implementations, memory 202 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 202 may store an operating system, such as uCOS, VxWorks, RTLinux, or other embedded operating systems. Memory 202 may also store communication programs that can be used to communicate with a mobile phone, one or more servers, or additional devices.

[0051] Camera 203 is used to capture still images or videos. An object is projected onto a photosensitive element through the lens, generating an optical image. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats.

[0052] In some embodiments of this application, the large-screen display device 200 may include at least one camera. The camera 203 is used to capture images containing the user's hand. Then, the processor 201 performs joint point annotation on the captured images containing the user's hand and identifies the user's operating hand based on the joint point coordinates.

[0053] The power switch 204 can be used to control the power supply to various components of the smart TV 200.

[0054] The communication processing module 205 may include a wireless communication processing module (not shown) and a wired local area network (LAN) communication processing module (not shown).

[0055] The wireless communication processing module may also include a cellular mobile communication processing module (not shown). The cellular mobile communication processing module can communicate with other devices (such as servers) via cellular mobile communication technology.

[0056] The wired LAN communication processing module can be used to communicate with other devices in the same LAN via a wired LAN, and can also be used to connect to a wide area network (WAN) via a wired LAN to communicate with devices in the WAN.

[0057] The large display screen 206 can be used to display user control interfaces, images, videos, etc. It can be a single display or a large screen composed of multiple displays. Large display screens are typically used to present content to multiple people and are often used in new product launches, centralized control centers for monitoring areas, etc. The large display screen 206 can use liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, active-matrix organic light-emitting diode (AMOLED) displays, flexible light-emitting diode (FLED) displays, quantum dot light-emitting diode (QLED) displays, etc.

[0058] The audio module 210 can be used to output audio signals through the audio output interface, thus enabling the large-screen display device 200 to support audio playback. The audio module can also be used to receive audio data through the audio input interface. The audio module 207 includes, but is not limited to, a microphone, speaker, receiver, etc.

[0059] Understandable, Figure 3 The illustrated structure does not constitute a specific limitation on the large-screen display device 200. In other embodiments of this application, the large-screen display device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented by hardware or software, or a combination of software and hardware.

[0060] Based on the above Figure 3 The structure shown is based on Figure 4 The technical solution of this application is described in detail, taking into account specific scenarios. For example... Figure 4 As shown, the hand-mapped mouse pointer scheme of this application includes:

[0061] In step S101, the large screen display device 200 captures an image of the user 100 through the camera 203. The image of the user 100 includes an image of the user 100's hand.

[0062] In some embodiments of this application, the large-screen display device can capture images of user 100 through camera 203, and the images of user 100 include at least an image of user 100's hands. Specifically, the images of user 100's hands include at least one hand. It is understood that the camera 203 has a certain shooting range, and the images of user 100 captured by camera 203 may include images of user 100's entire body, images of the background surrounding user 100, and images of other users' entire bodies or parts thereof.

[0063] In addition, the camera 203 periodically captures images of the user 100. The capture frequency can be preset at the factory, such as capturing 24 frames of images per second. Users can also adjust the frequency as needed.

[0064] In step S102, the large-screen display device 200 identifies the image of user 100 using a preset hand recognition model and determines the joint positions of user 100's hand. Here, the large-screen display device 200 typically maps the mouse pointer based on the hand of user 100. The hand of user 100 used to map the mouse pointer is the user 100's current operating hand, and movement of the user 100's current operating hand will cause movement of the corresponding mouse pointer.

[0065] In some embodiments of this application, recognizing the image of user 100 can begin by identifying the image of user 100's hand, i.e., detecting user 100's hand. Since the image of user 100 contains much image content unrelated to the mapped mouse pointer, this content can affect the recognition of user 100's hand. Therefore, extracting the image of user 100's hand from the image and processing it separately not only reduces the computational resource consumption required for subsequent processing but also reduces the interference of irrelevant image content on user 100's hand detection, thus improving the accuracy of user 100's hand detection.

[0066] User 100's hand detection can be performed using various hand detection models, ultimately resulting in a hand image of User 100.

[0067] In some embodiments of this application, the hand detection model can be an extension of the human pose recognition model. This involves first performing human pose recognition on an image of user 100 to obtain the position of the wrist joint, and then determining the position of user 100's hand based on the wrist joint. The human pose recognition model is built on a deep neural network using supervised learning and is used to identify the positions of user 100's human joints. The deep neural network used in the human pose recognition model can include, but is not limited to, SSD, R-CNN, Fast R-CNN, etc. The embodiments of this application do not limit the human pose recognition model.

[0068] For example, the SSD model identifies the human joints of user 100 in an image containing user 100's hand, and obtains the positions of user 100's human joints.

[0069] In some embodiments of this application, such as Figure 5 As shown, the human body contour can correspond to the following joints: spinal base joint 0, spinal midpoint joint 1, spinal shoulder joint 2, neck joint 3, head joint 4, left shoulder joint 5, right shoulder joint 8, left elbow joint 6, right elbow joint 9, left wrist joint 7, right wrist joint 10, left hip joint 11, right hip joint 15, left knee joint 12, right knee joint 15, left ankle joint 13, right ankle joint 17, left foot joint 14, and right foot joint 18. Furthermore, the SSD model outputs the coordinates corresponding to each joint position; for example, the coordinates of spinal base joint 0 are (x0, y0), the coordinates of spinal midpoint joint 1 are (x1, y1), ..., the coordinates of right foot joint 18 are (x0, y0). 18 ,y 18 ).

[0070] It should be noted that, in order to improve the accuracy of the coordinates corresponding to the human joint positions of user 100, the image containing user 100's hand can be preprocessed before being input into the SSD model. The preprocessing includes normalization, image enhancement, and image denoising. Then, the preprocessed image containing user 100's hand is input into the SSD model. The SSD model identifies the human joints in the preprocessed image containing user 100's hand to obtain the coordinates corresponding to the human joint positions of user 100.

[0071] It is understandable that the location of user 100's human body joints can be... Figure 5 The positions of the 19 joints can be some of the joints in the coordinates of the above 19 joints, or they can be other joints in the human body. This application does not limit the position and number of human body joints.

[0072] After determining the positions of user 100's body joints, the hand detection model can then determine the position of user 100's hand based on the wrist joint position within the body joints. For example, ... Figure 5 As shown, the position of the left or right hand in the image containing the user 100's hand can be determined by the left wrist joint point 7 or the right wrist joint point 10. The position of the user 100's hand is marked with a dashed box in the figure.

[0073] In other embodiments of this application, the hand detection model can be an extension of the palm detection model. A human hand includes the palm and fingers. The palm detection model is used to detect the human palm in an image, and then the human hand is obtained by extending the palm. For example, the palm detection model can be the BlazePalm model in the MediaPipe framework. The BlazePalm model is a deep neural network model based on the SSD architecture. Using the BlazePalm model to detect an image containing user 100's hand, the palm position of user 100 is described by a square bounding box. Then, based on the palm position, it is extended to obtain the hand position of user 100 defined by a rectangular bounding box.

[0074] The image containing the user 100's hand and corresponding to the position of the user 100's hand is the user 100's hand image. In some embodiments of this application, the user 100's hand image is input into a preset hand recognition model to obtain the joint positions of the user 100's hand.

[0075] In some embodiments of this application, the hand recognition model is a pre-built deep neural network model, and the construction process of the hand recognition model can be as follows: Figure 6 As shown, a large number of images containing hands are first acquired as training data through image acquisition. Then, the training images are preprocessed, such as image enhancement and image denoising. The training images are then input into a deep neural network for feature extraction and selection. The model parameters are trained through multiple iterations, and finally, a trained hand recognition model is generated.

[0076] The process of inputting user 100's hand image into the hand recognition model for recognition is as follows: Figure 7 As shown, before this, the hand image can be preprocessed. The hand recognition model extracts and selects features from the input user image, then recognizes the hand joints, and finally obtains the recognition results of the joints on the hand image.

[0077] In some embodiments of this application, the hand recognition model can be Mobile-Net. The Mobile-Net model detects the hand image of user 100 to obtain the position information of the hand joints in the hand image. For example, Figure 8The image shows the result of recognizing a user's right hand using the Mobile-Net model. The right hand image includes 21 joints, labeled with numbers 0-20. For example, the base of the hand is labeled 0, and the joints from the first joint of the thumb to the last joint of the little finger are labeled 1-20. In a Cartesian coordinate system with point O as the origin, the horizontal x-axis and the vertical y-axis, the coordinates of the hand joints are (a1, b1), (a2, b2), ..., (a...). 19 ,b 19 ), (a 20 ,b 20 ).

[0078] In some other embodiments of this application, the hand recognition model can be the hand tagging model in the MediaPipe framework. The hand tagging model determines the coordinates corresponding to the positions of 21 hand joints by performing hand joint regression (i.e., direct coordinate prediction) on the hand image of user 100.

[0079] In step S103, the large screen display device 200 determines the user's current operating hand based on the change in the joint position of the user's hand, and establishes a corresponding control coordinate system based on the current operating hand.

[0080] (1) Determine the user's current operating mode

[0081] In some embodiments of this application, the large-screen display device 200 uses the user 100's moving hand as the user 100's current operating hand.

[0082] Processor 201 can determine the moving hand of user 100 by detecting changes in the position of hand joints, and designate the moving hand as the user 100's current operating hand. The following explains how processor 201 determines the moving hand of user 100.

[0083] Five consecutive frames of user 100's hand images are selected, and the hand joints are identified. The movement of user 100's hand is then determined based on the coordinate changes corresponding to these joints. The following explanation uses the root joint 0 of the hand image as an example. Figure 9 As shown, the five consecutive frames of the user's hand image are image frame 601, image frame 602, image frame 603, image frame 604, and image frame 605. The coordinates of the root joint point 0 in image frame 601 are... The coordinates of the root joint point 0 in image frame 602 are: The coordinates of the root joint point 0 in image frame 603 are: The coordinates of the root joint point 0 in image frame 604 are: The coordinates of the root joint point 0 in image frame 605 are: From image frame 601 to image frame 605, the x-axis coordinate of the root joint point 0 changes as follows: The change in the x-coordinate exceeds the preset threshold for x-coordinate change; the change in the y-coordinate of the root joint point 0 is... The change in the vertical axis also exceeded the preset vertical axis change threshold, so it can be considered that user 100's hand has moved, and this hand is a moving hand.

[0084] It should be noted that the selection of 5 consecutive frames of hand images in the above embodiments is only exemplary, and other numbers of hand images may also be used. This application does not limit this.

[0085] The above embodiments determine whether the hand has moved by detecting the coordinate changes corresponding to the positions of the joints at the root of the hand. It can be understood that detecting the coordinate changes corresponding to the positions of any one or more of the 21 joints can also be used to determine whether the hand has moved. This application does not limit this.

[0086] After identifying the moving hand, the device detects whether it is the left or right hand. If user 100 has only one moving hand, that hand is designated as the current operating hand. If user 100 has two moving hands, the device identifies the corresponding hand as the current operating hand based on a preset default operating hand, such as left or right. For example, if one of user 100's hands is moving and the other is stationary, the large-screen display device 200 identifies the moving hand as the current operating hand and performs left / right hand recognition. If user 100 moves both hands simultaneously, and the preset default operating hand is the right hand, the large-screen display device 200 identifies the right hand and designates it as the current operating hand.

[0087] In some embodiments of this application, left and right hand recognition of a moving hand can be performed using a preset left and right hand detection model. The left and right hand detection model is obtained by supervising learning by inputting a large number of images containing left and right hands as training data into a deep neural network. The deep neural network can be, for example, a convolutional neural network model or an extended model based on a convolutional neural network, etc. The embodiments of this application do not impose specific limitations on this.

[0088] Additionally, in some embodiments, user 100 typically moves their hand with the palm facing outwards, and the camera captures the user 100's palm. Therefore, the left or right hand can be determined based on the relative positions of the hand's knuckles. For example, the left or right hand can be determined based on the relative positions of the knuckles at the base of the little finger and the base of the index finger. If the x-coordinate of the little finger's knuckle is less than that of the index finger's knuckle, it is determined to be the right hand; if the x-coordinate is greater than that of the index finger's knuckle, it is determined to be the left hand. (Reference) Figure 8 In the figure, the horizontal coordinate of the little finger base joint 17 is less than the horizontal coordinate of the index finger base joint 5, therefore the hand shown in the figure is the right hand.

[0089] It is understood that the left and right hands can be determined based on the relative positions of the hand joints. Any pair of joints with different relative positions in the left and right hands can be used. This application does not limit the specific pair of joints used for left and right hand determination.

[0090] (2) Establish the corresponding control coordinate system based on the current operating hand of user 100.

[0091] In some embodiments of this application, a corresponding control coordinate system is established based on the user's current operating hand. For example, if the user's current operating hand is left-handed, a corresponding left-hand control coordinate system is established in the air; if the user's current operating hand is right-handed, a corresponding right-hand control coordinate system is established in the air. The origins of the left-hand and right-hand control coordinate systems are different. The left-hand control coordinate system covers the range that the user's left hand can move, and the right-hand control coordinate system covers the range that the user's right hand can move. The origin of the control coordinate system is at the upper left corner, the X-axis (horizontal axis) is horizontal to the right, and the Y-axis (vertical axis) is vertically downward. Furthermore, the control coordinate plane formed by the X-axis and Y-axis is usually parallel to the display screen, but in some cases it may not be parallel.

[0092] In some embodiments of this application, the origin of the control coordinate system can be determined based on the user's body characteristics. These body characteristics may include, but are not limited to, the aforementioned human joint points, height, shoulder width, and other features. For example, referring to... Figure 5 The left shoulder joint 5, one of the aforementioned human joint points, is designated as the origin of the right-hand control coordinate system. The horizontal direction to the right of the left shoulder joint 5 is used as the X-axis, and the vertical direction downwards is used as the Y-axis to establish the right-hand control coordinate system. Alternatively, the origin of the control coordinate system can be determined based on the user's shoulder width, and further reference can be made. Figure 5The user's shoulder width can be determined based on the horizontal distance between the left shoulder joint 5 and the right shoulder joint 8. The x-coordinate of the origin of the left-hand control coordinate system can be set to the x-coordinate of the left shoulder joint 5 minus the user's shoulder width, and the y-coordinate of the origin can be set to the y-coordinate of the left shoulder joint 5. Then, the left-hand control coordinate system can be established with the horizontal direction to the right of the origin as the X-axis and the vertical direction downward as the Y-axis.

[0093] This is understandable. The origin of the control coordinate system can be determined based on multiple body characteristics of the user. For example, the position of the origin can be determined based on the user's height and shoulder width. (Continue to refer to...) Figure 5 The user's height can be determined based on the vertical distance between the head joint (4) and the left ankle joint (14). The x-coordinate of the origin of the left-hand control coordinate system can be set to the x-coordinate of the left shoulder joint (5) minus the user's shoulder width, and the y-coordinate of the origin can be set to the user's height. A left-hand control coordinate system can then be established with the horizontal direction to the right from this origin as the X-axis and the vertical direction downwards as the Y-axis. Here, there are no specific limitations on how one or more user body features are used to determine the origin of the control coordinate system.

[0094] In some other embodiments of this application, the origin of the control coordinate system can be determined based on the natural position of the user's hand. The natural position of the user's hand is the position where the hand is raised from its naturally hanging position to a position where the user feels comfortable moving it in the air. For example, when the user raises their right hand above their right elbow but below their right shoulder, with the right elbow about a fist's distance from their body, this is the natural position of the right hand. Similarly, when the left hand is raised above its left elbow but below its left shoulder, with the left elbow about a fist's distance from its body, this is the natural position of the left hand. Here, the natural position of the left or right hand can be determined by pre-sampling the natural position of the left or right hand, or it can be inferred by identifying the user's joints, for example, by inferring the natural position of the right hand based on the positions of the right shoulder joint, right elbow joint, right hand joint, and the midpoint joint of the spine.

[0095] The user's natural right hand position can be used as the center of the right-hand control coordinate system. The user's right hand moves within this coordinate system from its natural position, providing the most comfortable operation. Therefore, the origin of the right-hand control coordinate system can be set to the upper left of the user's natural right hand position. For example, the position reached by the user's right hand after extending it as far to the upper left as possible from its natural position can be used as the origin. Then, a horizontal X-axis pointing to the right and a vertical Y-axis pointing downwards can be established based on the origin. Furthermore, the lengths of the X and Y axes can be defined. For example, the length of the X-axis can be determined as twice the horizontal distance between the user's natural right hand position and the origin, and the length of the Y-axis can be determined as twice the vertical distance between the user's natural right hand position and the origin.

[0096] It is understandable that the natural position of the left hand can be used as the center of the left-hand control coordinate system. The origin of the left-hand control coordinate system can be set at a position to the upper left of the user's natural left hand position. Then, based on the origin of the left-hand control coordinate system, a horizontal X-axis to the right and a vertical Y-axis downward can be established.

[0097] In some embodiments of this application, the origin of the control coordinate system can be adjusted according to the user's operating habits. Here, when multiple control coordinate systems are established for a specific user, the origin is not fixed but can be optimized based on the user's operating habits to better suit those habits and provide support for personalized operation. User operating habits may include, but are not limited to: the maximum distance the user's hand can move left / right / up / down, the user's preferred operating hand (e.g., left or right hand), the user's body posture during operation (e.g., standing facing the screen or standing sideways), the degree of bending of the user's operating arm, and the frequency of the user's hand appearing in different positions within the control coordinate system.

[0098] For example, by statistically analyzing users' historical operating habits, it might be discovered that users habitually operate in the right-hand control coordinate system, slightly to the right of the historical coordinate origin. In this case, the origin of the right-hand control coordinate system can be set to the right of the historical coordinate origin, placing the user's preferred operating area at the center of the right-hand control coordinate system, thus facilitating the user's operating habits. As another example, if it's found that users habitually operate with their left hand while facing the screen sideways, the origin of the left-hand control coordinate system can be set to the left of the historical coordinate origin, increasing the user's left-hand operable area to accommodate their preferred posture.

[0099] It is understandable that optimizing the historical coordinate origin of the control coordinate system cannot be accomplished in one or a few times. As user operation habit data is collected, the setting position of the coordinate origin will be continuously optimized. In this embodiment, the number of times the coordinate origin is optimized is not specifically limited.

[0100] In other embodiments of this application, the origin of the control coordinate system can be determined based on the operation area on the display interface. Here, the distribution of user-operable objects such as buttons and input boxes on the user interface displayed on the screen is uneven. Some areas of the interface contain many user-operable objects, while others have almost none. In this case, the frequency of the user's hand movement varies greatly in different areas of the control coordinate system. For example, in control coordinate system areas with more user-operable objects on the display interface, the user's hand will frequently move to that area. If this area is a relatively remote area in the control coordinate system, it increases the distance the user needs to move, resulting in a poor user experience. For instance, if the user-operable objects on the display interface are concentrated on the right side, the user's right hand needs to frequently move to the right-hand control coordinate system. In this case, the origin of the user's right-hand control coordinate system can be set to a position slightly to the left of the usual origin, thereby reducing the distance the user needs to move and improving the user experience.

[0101] It is understandable that optimizing the position of the origin of the control coordinate system corresponding to the corresponding operator based on the operation area of ​​the display interface is closely related to the distribution of the operation area on the display interface. When the current display interface and operation area change, the optimized position of the origin of the coordinate system also needs to be adjusted accordingly.

[0102] In some embodiments of this application, the control coordinate system can be translated and rotated according to the user's position and posture. Here, a historical control coordinate system is established based on the user's body reference point. After the position of the user's body reference point changes, the current control coordinate system is obtained by translating the historical control coordinate system. The user's body reference point can be a region or limb node on the user's body. For example, the user's torso can be the user's body reference point. When the user is sitting on the left side of the sofa, a control coordinate system A is established based on the position M of the user's torso. The user moves their hand within control coordinate system A. After the user sits on the right side of the sofa, the position of the user's torso changes from M to N. The historical control coordinate system is A, and the reference point for establishing the control coordinate system changes from M to N. Therefore, the historical control coordinate system A can be translated to obtain the current control coordinate system B with N as the reference point. The user moves their hand within control coordinate system B. Through the translation of the control coordinate system, the user can still control the entire display screen in a non-moving manner after the position changes.

[0103] Furthermore, users typically perform hand movements while facing the display screen. However, in situations where it's inconvenient for users to face the screen, such as when lying down on a sofa, it becomes very difficult for them to operate their hands within the control coordinate system facing the screen. Therefore, the control coordinate system can be rotated based on the user's current posture, for example, rotating it to a position parallel to the user's front to facilitate hand operations. User posture detection can utilize a pre-established human posture detection model. For instance, the joints of the current human body can be obtained based on the aforementioned joint detection model, and the positional relationships between these joints can determine whether the human body is standing, sitting, lying down, or in another posture. When rotating the control coordinate system, the origin can be fixed, and a new control coordinate system can be obtained by rotating the X and Y axes. Alternatively, the origin can be moved to a suitable position, and the X and Y axes established based on the new origin can be rotated accordingly to obtain a new control coordinate system.

[0104] Step S104: Map the user's current hand position to a first position on the screen of the large-screen display device 200, and display the mouse pointer at the second position. Here, the content displayed on the screen of the large-screen display device 200 is all located in a display coordinate system. The origin of the display coordinate system is fixed at the upper left corner of the screen, the X-axis is parallel to the horizontal edge of the display screen, and the Y-axis is parallel to the vertical edge of the display screen. Mapping the user's current hand position in the control coordinate system to the second position in the display coordinate system—for example, mapping the coordinates of the first position to the coordinates of the second position in the display coordinate system, and displaying the mouse pointer at the corresponding coordinates of the second position—completes the mapping from the user's current hand position to the mouse pointer.

[0105] In some embodiments of this application, the coordinates of the user's current hand are mapped to the display coordinate system of the large-screen display device 200 in a point-to-point manner. For example, if the user's current hand coordinates in the control coordinate system are (600, 200), the mouse pointer can be directly displayed at coordinates (600, 200) in the display coordinate system of the large-screen display device 200, thus completing the mapping between the two coordinate systems. Furthermore, if the resolution of the image captured by the camera differs from the display resolution of the large-screen display device 200, corresponding scaling can be performed during coordinate mapping. For example, if the resolution of the image captured by the camera is three times the display resolution of the large-screen display device 200, the horizontal and vertical coordinates of the user's current hand in the control coordinate system can be multiplied by a factor of three, and the multiplied coordinates can be used as the coordinates in the display coordinate system.

[0106] In some embodiments of this application, the position of any one of the hand joints can be used as the position of the user's current operating hand. For example, the position of the root joint 0 among the 21 hand joints can be selected as the position of the user's current operating hand, or the position of the root joint 2 of the thumb can be selected as the position of the user's current operating hand, etc.

[0107] In other embodiments of this application, a center point can be generated based on multiple hand joints, and this center point can be used as the position of the user's current operating hand. For example, a geometric mean point can be calculated based on the coordinates of 21 hand joints, and the coordinates of this geometric mean point can be used as the coordinates of the user's current operating hand. As another example, a geometric mean point can be calculated based on the root joint 0, thumb root joint 2, index finger root joint 5, middle finger root joint 9, ring finger root joint 13, and little finger root joint 17, and the coordinates of this geometric mean point can be used as the coordinates corresponding to the position of the user's current operating hand. In the embodiments of this application, there are no restrictions on the number of joints used to determine the position of the user's current operating hand or the specific selection thereof.

[0108] Step S105: Based on the movement trajectory of the user's current hand in the control coordinate system, move the mouse pointer accordingly on the large screen. Here, the camera continuously captures images of the user's current hand during the movement process, and performs the aforementioned hand joint point recognition on each image to obtain the current position of the user's current hand. Then, the current position of the user's current hand is mapped to the current position in the display coordinate system of the large screen display device 200. The mouse pointer is de-displayed at its previous position in the display coordinate system, and the mouse pointer is displayed at its current position in the display coordinate system, thereby realizing the movement of the mouse pointer.

[0109] In some embodiments of this application, the user's current operating hand can change during operation; for example, the current operating hand can change from the right hand to the left hand, or vice versa. The following is in conjunction with... Figure 10 The technical solution of this application is described, such as... Figure 10 As shown, the technical solutions for changing the user's current operating hand include:

[0110] Step S201: Acquire the user's image. For the specific acquisition process, please refer to step S101, which will not be repeated here.

[0111] Step S202: The user's image is recognized by a preset hand recognition model to obtain the joint positions of the user's hand. For details, please refer to step S102, which will not be repeated here.

[0112] Step S203: Determine the user's current operating hand based on the changes in the joint position of the user's hand. For details, please refer to step S103, which will not be repeated here.

[0113] Step S204: Check if the user's current operating hand has changed. If it has changed, proceed to step S205; otherwise, proceed to step S208.

[0114] In some embodiments of this application, checking whether the user's current operating hand has changed can be done by comparing the user's current operating hand with the recorded previous operating hand. The previous operating hand refers to the operating hand used by the user during the last operating hand check. Recording the operating hand used by the user at that time reveals the user's previous operating hand. If the comparison result between the user's current operating hand and the previous operating hand is different, it is considered that the user's current operating hand has changed, and step S205 is executed next; if the comparison result between the user's current operating hand and the previous operating hand is the same, it is considered that the user's current operating hand has not changed, and step S208 is executed.

[0115] Step S205: Establish a new control coordinate system based on the user's changed operating hand. Here, the user's current operating hand has changed relative to the previous operating hand, and is therefore the user's changed operating hand. At this time, the user's changed operating hand can be recorded in the previous operating hand for comparison during the next operating hand check.

[0116] Since the user previously operated within the control coordinate system corresponding to their previous hand, it becomes inconvenient to continue operating within that system after the hand changes. Therefore, it's necessary to re-determine the control coordinate system based on the user's new hand. For example, if the user's previous hand was left-hand and the new hand is right-hand, the control coordinate system needs to be adjusted from a left-handed to a right-handed system; conversely, if the user's previous hand was right-handed and the new hand is left-handed, the control coordinate system needs to be adjusted from a right-handed to a left-handed system. The origin of the new control coordinate system can be determined using the various methods described above, which will not be elaborated upon here.

[0117] Step S206: Map the user's changed operating hand to the first position in the new control coordinate system to the second position on the screen, and display the mouse pointer at the second position on the screen. For details, please refer to step S104, which will not be repeated here.

[0118] Step S207: Based on the user's changed hand movement trajectory in the new control coordinate system, move the mouse pointer accordingly on the large screen. For details, please refer to step S105, which will not be repeated here.

[0119] Step S208: Map the user's current hand position in the control coordinate system to the second position on the screen, and display the mouse pointer at the second position on the screen. For details, please refer to step S104, which will not be repeated here.

[0120] Step S209: Based on the movement trajectory of the user's current hand in the control coordinate system, move the mouse pointer accordingly on the large screen. For details, please refer to step S105; it will not be repeated here. By checking whether the user's current hand has changed during operation and establishing a corresponding control coordinate system based on the changed hand, user operation can be facilitated, and the user experience improved.

[0121] In some embodiments of this application, the large-screen display device 200 can map the hands of multiple users to corresponding mouse pointers on the screen, allowing each user to control their own mouse pointer. The following describes... Figure 11 The technical solution of this application is described, such as... Figure 11 As shown, the technical solutions for multi-user control include:

[0122] Step S301: Acquire images of multiple users. These images include hand images of multiple users. For the specific acquisition process, please refer to step S101, which will not be repeated here.

[0123] Step S302 involves recognizing images of multiple users using a preset hand recognition model to obtain the joint positions of each user's hand. Here, the hand images of multiple users can be extracted from the entire image and recognized separately. For the specific process of recognizing the hand image of each user, please refer to step S102, which will not be repeated here.

[0124] Step S303: Determine the current operating hand of each user based on the changes in the joint position of each user's hand, and establish a corresponding control coordinate system for each user based on the current operating hand. For the specific process of determining the current operating hand of each user, please refer to step S103, which will not be repeated here.

[0125] Step S304: Map the first position of each user's current operating hand in its respective control coordinate system to the second position on the screen, and display the mouse pointer at the second position on the screen. For details, please refer to step S104, which will not be repeated here.

[0126] Step S305: Based on the movement trajectory of each user's current operating hand in their respective control coordinate system, move the mouse pointer accordingly on the large screen. For details, please refer to step S105, which will not be repeated here.

[0127] For reference Figure 12This is a software structure block diagram of an electronic device according to an embodiment of this application. The electronic device 900 can be used to acquire the user's image through a camera and identify the user's current operating hand, then establish a corresponding control coordinate system based on the current operating hand, and map the position of the current operating hand in the control coordinate system to the position of the mouse pointer on the electronic device screen. Figure 12 The software system of this electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of the electronic device.

[0128] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0129] The application layer can include a series of application packages.

[0130] like Figure 12 As shown, the application package may include applications such as phone, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0131] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0132] like Figure 12 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0133] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0134] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0135] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0136] A phone manager is used to provide communication functions for terminal devices. For example, it manages call status (including connection and disconnection).

[0137] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0138] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating the device, and flashing indicator lights.

[0139] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0140] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0141] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0142] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0143] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0144] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0145] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0146] A 2D graphics engine is a graphics engine for 2D drawing.

[0147] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, sensor drivers, mouse drivers, etc.

[0148] In this specification, references to "some embodiments" or "embodiments" mean that a specific feature, structure, or characteristic described in connection with an embodiment is included in at least one exemplary implementation or technology according to this application. The phrase "in some embodiments" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0149] This application also relates to means for performing operations in text. Such means may be specifically constructed for the claimed purpose or may include a general-purpose computer selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored on a computer-readable medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, application-specific integrated circuits (ASICs), or any type of medium suitable for storing electronic instructions, and each may be coupled to a computer system bus. Furthermore, the computer mentioned in the specification may include a single processor or may be an architecture employing multiple processors for increased computing power.

[0150] The processes and displays presented in this application do not inherently relate to any specific computer or other device. Various general-purpose systems may also be used with the programs taught in this application, or it may prove convenient to construct more specialized devices to perform one or more method steps. Structures for various such systems are discussed in the following description. Furthermore, any specific programming language sufficient to implement the techniques and embodiments of this application may be used. Various programming languages ​​may be used to implement the embodiments of this application.

[0151] Furthermore, the language used in this specification has been primarily chosen for readability and instructional purposes and may not have been intended to depict or limit the disclosed subject matter. Therefore, this application is intended to illustrate, rather than limit, the scope of the concepts discussed herein.

Claims

1. A method for mapping a mouse pointer to a hand in an electronic device, characterized in that, include: The electronic device acquires an image of the user, the image of the user including at least an image of the user's hand, and performs hand recognition on the image of the user to determine the position of the hand joints; The user's current operating hand is determined based on the changes in the position of the hand joints; A control coordinate system is established based on the user's current operating hand and the user's operating habit data. The origin of the control coordinate system is offset in a first direction relative to the origin of the historical control coordinate system. The first direction is the direction in which the user's habitual operating area is offset relative to the center of the historical control coordinate system, so that the user's habitual operating area is closer to the center of the control coordinate system. The user's operating habit data is determined by statistics from the user's historical operating data. The first position of the user's current operating hand is mapped to a second position on the screen of the electronic device, and a mouse pointer is displayed at the second position on the screen. The mouse pointer is moved on the screen of the electronic device according to the movement trajectory of the user's current operating hand in the control coordinate system.

2. The method according to claim 1, characterized in that, Acquiring an image of the user of the electronic device includes: The electronic device acquires images of the user through a camera.

3. The method according to claim 1, characterized in that, Determining the user's current operating hand based on changes in the position of the hand joints includes: The user's moving hand is determined based on the changes in the position of the hand joints. The changes in the position of the hand joints are obtained by comparing the positions of corresponding joints in multiple sets of hand joints. The positions of the multiple sets of hand joints are obtained by performing hand recognition on images containing at least the user's hand in multiple consecutive frames. The moving hand is identified as the user's current operating hand, and the user's current operating hand is identified as either the left or right hand.

4. The method according to any one of claims 1 to 3, characterized in that, After determining the user's current operating hand based on the changes in the position of the hand joints, the method further includes: The user's current operating hand is compared with the previous operating hand. If the user's current operating hand is different from the previous operating hand, a new control coordinate system is established based on the user's current operating hand.

5. The method according to claim 1, characterized in that, Mapping the first position of the user's current operating hand to a second position on the screen of the electronic device, displaying a mouse pointer at the second position on the screen, and moving the mouse pointer on the screen of the electronic device according to the movement trajectory of the user's current operating hand in the control coordinate system, including: The first coordinate describing the first position of the user's current operating hand in the control coordinate system is mapped to the second coordinate describing the second position on the screen of the electronic device, and the mouse pointer is displayed at the second coordinate on the screen. Based on the current movement trajectory of the user's hand in the control coordinate system, the mouse pointer is moved accordingly on the screen of the electronic device.

6. The method according to claim 5, characterized in that, Establish a corresponding control coordinate system based on the user's current operating hand and the user's operating habits data, including: The position of the origin of the control coordinate system is determined based on the user's physical characteristics, and the corresponding horizontal and vertical coordinate axes are determined based on the position of the origin.

7. The method according to claim 5, characterized in that, Establish a corresponding control coordinate system based on the user's current operating hand and the user's operating habits data, including: The position of the origin of the control coordinate system is determined based on the natural position of the user's current operating hand, and the corresponding horizontal and vertical coordinate axes are determined based on the position of the origin.

8. The method according to claim 5, characterized in that, Establish a corresponding control coordinate system based on the user's current operating hand and the user's operating habits data, including: The position of the origin of the control coordinate system is determined based on the operating area of ​​the screen of the electronic device, and the corresponding horizontal and vertical coordinate axes are determined based on the position of the origin.

9. The method according to claim 5, characterized in that, Also includes: The control coordinate system is translated or rotated according to the user's position and posture.

10. An electronic device, comprising: A memory for storing instructions to be executed by one or more processors of the system, and a processor, one of the processors of the system, for performing the method of any one of claims 1 to 9.

11. A machine-readable medium, characterized in that, The machine-readable medium stores instructions that, when executed on a machine, cause the machine to perform the method of any one of claims 1 to 9.

Images