Stylus position estimation method and stylus

Abstract

The invention provides a stylus position estimation method and a stylus. The method includes the following steps. An image is acquired from the display through an image acquisition device disposed on the stylus. The distance between the stylus and the display is detected through a distance measuring sensor arranged on the stylus. A display bezel of the display in the image is detected. The spatial position of the stylus relative to the display is determined according to the display frame and the distance in the image. Therefore, the spatial position of the stylus relative to the display can be accurately obtained.

Description

Technical Field

[0001] The present invention relates to a stylus system, and in particular to a stylus position estimation method and a stylus. Background Art

[0002] Generally speaking, fingers and styluses are the most commonly used input tools for touch operations. Since styluses can produce finer and more accurate strokes, they are more suitable for drawing scenarios or some specific scenarios. However, as electronic products have more and more functions, the touch operation method of directly using a stylus to touch the screen has gradually failed to meet the user's operational needs. For example, the stylus must be close enough to the touch screen to be effective, which will directly limit the actual distance between the user and the electronic product. Therefore, how to make the stylus provide more diverse control methods is indeed an issue of concern to technical personnel in this field. Summary of the invention

[0003] The present invention relates to a stylus position estimation method and a stylus, which can accurately obtain the spatial position of the stylus relative to a display, so as to provide more diversified control modes according to the spatial position of the stylus relative to the display.

[0004] The embodiment of the present invention provides a method for estimating the position of a stylus pen, which includes the following steps. An image is acquired toward a display by an image acquisition device disposed on the stylus pen. The distance between the stylus pen and the display is detected by a distance measuring sensor disposed on the stylus pen. The display frame of the display in the image is detected. The spatial position of the stylus pen relative to the display is determined based on the display frame and the distance in the image.

[0005] The embodiment of the present invention provides a stylus pen, which includes an image acquisition device, a distance sensor and a controller. The controller is coupled to the image acquisition device and the distance sensor, and is configured to perform the following operations. An image is acquired toward a display through the image acquisition device disposed on the stylus pen. The distance between the stylus pen and the display is detected through the distance sensor disposed on the stylus pen. A display frame of the display in the image is detected. The spatial position of the stylus pen relative to the display is determined based on the display frame and the distance in the image.

[0006] Based on the above, in an embodiment of the present invention, the image acquisition device disposed on the stylus can acquire an image toward the display, and the distance sensor can detect the distance between the stylus and the display. According to the display frame of the display in the image and the above distance, the spatial position of the stylus relative to the display can be estimated. Therefore, the spatial position of the stylus relative to the display can be used to perform a control function to increase the diversification of the use of the stylus. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram of a touch pen and a display according to an embodiment of the present invention;

[0008] Figure 2 is a block diagram of a touch pen according to an embodiment of the present invention;

[0009] Figure 3 is a flow chart of a method for estimating a stylus position according to an embodiment of the present invention;

[0010] Figure 4 is a flow chart of a method for estimating a stylus position according to an embodiment of the present invention;

[0011] Figure 5 is a schematic diagram of depth sensing data according to an embodiment of the present invention;

[0012] Fig. 6A and Figure 6B is a schematic diagram of a display frame in an image according to an embodiment of the present invention;

[0013] Figure 7 FIG. 4 is a schematic diagram of determining the spatial position of a stylus according to an embodiment of the present invention.

[0014] Description of Reference Numerals

[0015] 10: stylus position estimation system;

[0016] 100: stylus;

[0017] 200: display;

[0018] C1: screen center;

[0019] N1: normal direction;

[0020] θ: relative angle;

[0021] 110: image acquisition device;

[0022] 120: distance measuring sensor;

[0023] 130: inertial sensor;

[0024] 150: transceiver;

[0025] 160: controller;

[0026] 140: storage device;

[0027] d00~d77: depth sensing value;

[0028] 51, D_d1: depth sensing data;

[0029] Img11, Img12, Img1: image;

[0030] B11, B12: display frame;

[0031] C11: center point;

[0032] RP1: first reference position information;

[0033] RP2: second reference position information;

[0034] RP3: third reference position information;

[0035] M_d1: inertial sensing data;

[0036] FP: spatial position;

[0037] S310~S340, S410~S470, S441~S442, S451~S453: steps. DETAILED DESCRIPTION

[0038] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0039] Figure 1 is a schematic diagram of a stylus position estimation system according to an embodiment of the present invention. Figure 1 , the stylus position estimation system 10 includes a stylus 100 and a display 200. The display 200 is, for example, a touch screen of a smart phone, a tablet computer, a laptop computer, an e-book, a game console, or a desktop computer, but the present invention is not limited thereto. The display 200 is, for example, a liquid crystal display (LCD), a thin film transistor (TFT)-LCD, an organic light-emitting diode (OLED), a flexible display, a three-dimensional (3D) display, etc., but is not limited thereto. Generally speaking, a user can use the stylus 100 to perform a touch input operation on the touch display 200.

[0040] It should be noted that, in the embodiment of the present invention, the spatial position of the stylus 100 relative to the display 200 can be located according to the sensing data of various sensors on the stylus 100. In some embodiments, the spatial position of the stylus 100 relative to the screen center C1 of the display 200 can be located. According to the sensing data of various sensors on the stylus 100, the tilt state of the stylus 100 can also be obtained. Therefore, even if the touch panel of the display 200 does not sense the contact or approach of the stylus 100, the user can still issue user operations to the electronic device including the display 200 by changing the gesture of holding the stylus 100. For example, when the touch panel of the display 200 does not sense the contact or approach of the stylus 100, the user can change the spatial position and tilt state of the stylus 100 by changing the gesture of holding the stylus 100, so as to interact with the display object of the display 200 through the gesture of holding the stylus 100.

[0041] Figure 2 is a block diagram of a stylus pen according to an embodiment of the present invention. Figure 2 The stylus pen 100 includes an image acquisition device 110 , a distance sensor 120 , an inertial sensor 130 , a storage device 140 , a transceiver 150 and a controller 160 .

[0042] The image acquisition device 110 is used to capture images, and may include a lens, an image sensor, and other components. The image sensor is used to provide an image sensing function, and may include a photosensitive component, such as a charge coupled device (CCD), a complementary metal-oxide semiconductor (CMOS) component, or other components, and the present invention is not limited to this. The lens can focus imaging light on the image sensor to achieve the purpose of acquiring an image. In some embodiments, the image acquisition device 110 may include a fisheye lens or a wide-angle lens. In some embodiments, the image acquisition device 110 may be disposed at a position close to the tip of the stylus 100. Therefore, when the tip of the stylus 100 points to the display 200, the image acquisition device 110 can acquire an image toward the display 200.

[0043] The distance sensor 120 is used to detect the distance between the stylus 100 and the display plane of the display 200. The distance sensor 120 may be an ultrasonic distance sensor, an infrared distance sensor, a laser distance sensor, or other types of distance sensors. In some embodiments, the distance sensor 120 may include a multi-region Time of Flight (TOF) distance sensor, which can provide measurement distances of multiple measurement regions. In some embodiments, the distance sensor 120 may be disposed near the tip of the stylus 100. Therefore, when the tip of the stylus 100 points to the display 200, the distance sensor 120 may detect the distance between the tip of the stylus 100 and the display plane of the display 200.

[0044] The inertial measurement unit sensor (IMU sensor) 130 is used to sense the motion state of the stylus 100, such as angular velocity and acceleration, etc. The inertial sensor 130 may include a gyroscope, an accelerometer, a magnetometer, or a combination thereof.

[0045] The storage device 140 is used to store data or software modules or instructions for access by the controller 160 , and may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory or a combination thereof.

[0046] The transceiver 150 can transmit and receive signals wirelessly or by wire. The transceiver can also perform operations such as low noise amplification, impedance matching, mixing, up or down frequency conversion, filtering, amplification, and the like. The stylus 100 can communicate with an electronic device including the display 100 through the transceiver 150 to receive and send data.

[0047] The controller 160 is coupled to the image acquisition device 110, the distance sensor 120, the inertial sensor 130, the storage device 140, and the transceiver 150 to control the overall operation of the stylus 100. The relevant functions of the controller 160 can be implemented in various logic blocks, modules, and circuits in one or more controllers, microcontrollers, microprocessors, application-specific integrated circuits (ASICs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), and / or other processing units.

[0048] However, in addition to Figure 2 In addition to the components shown, the stylus pen 100 may also include other components, such as a pressure sensor, a power management circuit, a vibration component or a button, etc., and the present invention is not limited thereto.

[0049] Figure 3 is a flow chart of a method for estimating the position of a stylus pen according to an embodiment of the present invention. Figure 3 The method of this embodiment is applicable to Figure 1 and Figure 2 The stylus pen 100 of the present invention is used as an example. The following describes the detailed steps of the stylus pen position estimation method of the present invention in conjunction with the various components in the stylus pen 100.

[0050] In step S310, the controller 160 acquires an image toward the display 200 through the image acquisition device 110 disposed on the stylus 100. Specifically, when the stylus 100 is held by the user and pointed at the display 200, the image acquisition device 110 on the stylus 100 may acquire an image toward the display 200. In some embodiments, the image acquisition device 110 may include a fisheye lens, and the field of view (FOV) of the fisheye lens may exceed 180 degrees, thereby greatly reducing the probability that the acquired image does not include the display 200. In some embodiments, the image acquisition device 110 may continuously capture an image sequence according to an acquisition frame rate.

[0051] In step S320, the controller 160 detects the distance between the stylus pen 100 and the display 200 through the distance measuring sensor 120 disposed on the stylus pen 100. Specifically, when the stylus pen 100 is held by the user and points to the display 200, the distance measuring sensor 120 on the stylus pen 100 can detect the distance between the stylus pen 100 and the display plane of the display 200 in real time. The light, ultrasonic wave or other electromagnetic signal emitted by the distance measuring sensor 120 can hit the display plane of the display 200, and the distance measuring sensor 120 can sense the reflected light, reflected ultrasonic wave or other reflected electromagnetic signal reflected by the display 200 to obtain the distance between the stylus pen 100 and the display 200.

[0052] In step S330, the controller 160 detects the display frame of the display 200 in the image. In some embodiments, the controller 160 may detect the display 200 in the image according to a pre-trained object recognition model to further obtain the display frame of the display 200 in the image. The object recognition model is a machine learning model trained according to a machine learning algorithm, such as a convolutional neural network model, a Yolo model, or an image segmentation model. In some embodiments, the controller 160 may detect the edge in the image by an edge detection algorithm, and further detect the straight edge. Afterwards, the controller 160 may detect the display frame in the image according to the size information (such as the aspect ratio) of the display 200. In addition, in some embodiments, the screen edge or screen corner of the display 200 presents an identification mark (such as a barcode or a specific pattern), and the controller 160 may locate the display frame of the display 200 in the image by detecting the identification mark in the image. In some embodiments, the identification mark may be implemented as a sticker attached to the display 200, or the display 200 may display the identification mark by itself.

[0053] In step S340, the controller 160 determines the spatial position of the stylus 100 relative to the display 200 according to the display frame and the distance in the image. In some embodiments, the spatial position of the stylus 100 relative to the display 200 can be represented by the coordinate position of the world coordinate system. In some embodiments, the center C1 of the display 200 is taken as the origin of the world coordinate system as an example. The Z-axis direction (i.e., the third axis) of this world coordinate system is the normal direction N1 of the display plane of the display 200. The X-axis direction (i.e., the first axis) of the world coordinate system is the X-direction of the display plane of the display 200, and the Y-axis direction (i.e., the second axis) of the world coordinate system is the Y-direction of the display plane of the display 200. The controller 160 can determine the Z-axis coordinate component of the stylus 100 in the world coordinate system according to the distance sensed by the ranging sensor 120. Furthermore, the controller 160 may determine the X-axis coordinate component and the Y-axis coordinate component of the stylus pen 100 in the world coordinate system according to the imaging status of the display frame in the image (eg, imaging position, imaging size, and imaging completeness).

[0054] Therefore, after locating the spatial position of the stylus pen 100 relative to the display 200, the controller 160 may perform a function according to the spatial position. In some embodiments, the stylus pen 100 may perform an interactive operation with an electronic device including the display 200 according to the spatial position relative to the display 200. In detail, the controller 160 may transmit the above spatial position to the electronic device including the display 200 through the transceiver 150 to interact with the above electronic device. For example, when the display 200 displays a 3D object model, the user can drag or rotate the 3D object model by moving the stylus pen 100.

[0055] It should be noted that the above embodiments are described with the stylus pen 100 as an example, but the present invention is not limited thereto. In other embodiments, the stylus pen 100 can provide the distance detected by the distance measuring sensor 120 and the image acquired by the image acquisition device 110 to an electronic device including a display 200, so that the electronic device can determine the spatial position of the stylus pen 100 relative to the display 200.

[0056] Figure 4 is a flow chart of a method for estimating the position of a stylus pen according to an embodiment of the present invention. Figure 4 The method of this embodiment is applicable to Figure 1 and Figure 2 The stylus pen 100 of the present invention is used. The following describes the detailed steps of the stylus pen position estimation method of the present invention in conjunction with the various components in the touch control system 10.

[0057] In step S410, the controller 160 acquires an image toward the display 200 through the image acquisition device 110 disposed on the stylus 100. In step S420, the controller 160 detects the distance between the stylus 100 and the display 200 through the distance measurement sensor 120 disposed on the stylus 100. The detailed implementation of steps S410 to S420 has been described in Figure 3 The embodiments are clearly described and will not be repeated here.

[0058] In step S430, the controller 160 detects the tilt state of the stylus pen 100. In some embodiments, the tilt state of the stylus pen 100 includes a relative angle θ of the pointing direction of the stylus pen 100 relative to the normal direction N1 of the display plane of the display 200. That is, when the relative angle θ is 0 degrees, it means that the pointing direction of the stylus pen 100 is perpendicular to the display plane of the display 200. The tilt state of the stylus pen 100 depends on the gesture of the user holding the stylus pen 100.

[0059] In some embodiments, the inertial sensor 130 is disposed on the stylus pen 100 and is used to detect the tilt state of the stylus pen 100. Specifically, the controller 160 may calculate the angle of the stylus pen 100 relative to the ground plane based on the sensing data of the gyroscope to obtain the tilt state of the stylus pen 100. In some embodiments, when the angle between the display plane of the display 200 and the ground plane is known, the controller 160 may further obtain the relative angle θ of the pointing direction of the stylus pen 100 relative to the normal direction N1 of the display plane of the display 200.

[0060] In some embodiments, the distance sensor 120 includes a multi-region TOF distance sensor, and the multi-region TOF distance sensor is used to detect the tilt state of the stylus 100. Specifically, the multi-region TOF distance sensor can perform multiple independent distance measurements and obtain distance measurement results of multiple different measurement areas. In other words, the multi-region TOF distance sensor may include sensing arrays corresponding to different measurement areas.

[0061] For example, Figure 5 is a schematic diagram of depth sensing data according to an embodiment of the present invention. Figure 5 The multi-region TOF distance measurement sensor is, for example, an 8*8 multi-region TOF distance measurement sensor. The depth sensing data 51 output by the multi-region TOF distance measurement sensor may include depth sensing values ​​d00-d77 corresponding to 64 measurement regions. The distance measurement sensor 120 is disposed near the tip of the stylus pen 100 and can be used to measure the distance between the tip of the stylus pen and the display 200.

[0062] It can be seen that when the stylus pen 100 is pointed close to perpendicular to the display plane of the display 200, the multiple depth sensing values ​​(e.g., depth sensing values ​​d00-d77) associated with the display 200 in the depth sensing data 51 are similar to each other and do not have much difference. In contrast, if the relative angle θ of the pointing direction of the stylus pen 100 relative to the normal direction N1 of the display plane of the display 200 is obvious, the multiple depth sensing values ​​(e.g., depth sensing values ​​d00-d77) associated with the display 200 in the depth sensing data 51 will have more obvious differences, and the size of the multiple depth sensing values ​​associated with the display 200 is related to the pointing direction of the stylus pen 100. Therefore, by analyzing the multiple depth sensing values ​​in the depth sensing data 51, the controller 160 can obtain the tilt state of the stylus pen 100. In addition, according to the comparison result of the multiple depth sensing values ​​in the depth sensing data 51, the controller 160 can also determine whether each depth sensing data in the depth sensing data 51 is generated in response to the display 200. For example, assuming that the difference between the first row of depth sensing values ​​d00, d10, ..., d70 and other depth sensing values ​​is greater than the critical value, it means that the first row of depth sensing values ​​d00, d10, ..., d70 is depth information irrelevant to the display 200, that is, the first row of depth sensing values ​​d00, d10, ..., d70 is not generated based on the reflected light or reflected signal reflected by the display 200.

[0063] In step S440, the controller 160 detects the display frame of the display 200 in the image. In some embodiments, the image acquisition device 110 includes a fisheye lens, and the image acquired by the image acquisition device 110 includes a fisheye image. Step S440 can be implemented as steps S441 and S442.

[0064] In step S441, the controller 160 performs image correction processing on the fisheye image according to the distance to obtain a corrected image. In step S442, the controller 160 detects the display frame of the display 200 in the corrected image. Specifically, according to the distance (i.e., depth information) sensed by the ranging sensor 120 and the feature points in the fisheye image, the controller 160 can use camera calibration technology or geometric reasoning methods to accurately estimate the fisheye distortion parameters. These fisheye distortion parameters may include the center of distortion, the field of view, and the radial / tangential distortion coefficients, etc. Afterwards, according to these fisheye distortion parameters, the controller 160 can project the fisheye image onto a plane and inversely transform the distortion to obtain a corrected image. The image object in the corrected image can be restored to its true shape and size. Thus, the controller 160 can detect the display frame of the display 200 in the corrected image in the manner mentioned in the aforementioned embodiment.

[0065] In this embodiment, the spatial position of the stylus 100 relative to the display 200 may include a first coordinate component on a first axis, a second coordinate component on a second axis, and a third coordinate component on a third axis. The third axis is the normal direction of the display plane of the display 200, and the first axis, the second axis, and the third axis are perpendicular to each other. Figure 1 For example, the first axis is the X axis; the second axis is the Y axis; and the third axis is the Z axis.

[0066] In step S450, the controller 160 determines a first coordinate component on the first axis and a second coordinate component on the second axis according to the display frame in the image (ie, the corrected image). Step S450 can be implemented as steps S451 and S453.

[0067] In step S451, the controller 160 determines whether the display frame in the image is complete. In other words, the controller 160 determines whether the image includes a portion of the display frame or the entire display frame. In some embodiments, the controller 160 may determine whether the display frame in the image is a closed quadrilateral to determine whether the display frame is complete.

[0068] If the judgment in step S451 is yes, in step S453, in response to the display frame in the image being complete, the controller 160 determines the first coordinate component and the second coordinate component according to the position information of the display frame in the image. The position information of the display frame in the image may include the corner point position of any corner point of the display frame. Alternatively, the position information of the display frame in the image may include the center point position of the center point of the display frame. In some embodiments, the controller 160 may determine the first coordinate component and the second coordinate component of the stylus 100 in the world coordinate system according to the position information of the display frame by looking up a table. In some embodiments, the controller 160 may determine the first coordinate component and the second coordinate component of the stylus 100 in the world coordinate system according to the ratio between the two vertical frames of the display frame. Alternatively, in some embodiments, the controller 160 may determine the first coordinate component and the second coordinate component of the stylus 100 in the world coordinate system according to the ratio between the two horizontal frames of the display frame.

[0069] Fig. 6A is a schematic diagram of a display frame in an image according to an embodiment of the present invention. Fig. 6A , the controller 160 can determine that the display frame B11 in the image Img11 is complete. Therefore, based on the position information of the center point of the display frame B11 in the image Img11 (e.g., the pixel coordinates of the center point C11), the controller 160 can estimate the X coordinate component and the Y coordinate component of the stylus 100 relative to the display 200.

[0070] If the judgment in step S451 is no, in step S452, in response to the incomplete display frame in the image, the controller 160 determines the first coordinate component and the second coordinate component according to the distance and the number of pixels occupied by the display frame in the image. In some embodiments, in response to the incomplete display frame in the image, the controller 160 determines the first coordinate component and the second coordinate component according to the tilt state, the distance and the number of pixels. In some embodiments, the controller 160 may perform a table lookup according to the tilt state, the distance and the number of pixels occupied by the display frame in the image to obtain the first coordinate component and the second coordinate component of the stylus 100 in the world coordinate system.

[0071] Figure 6B is a schematic diagram of a display frame in an image according to an embodiment of the present invention. Figure 6B , the controller 160 may determine that the display frame B12 in the image Img12 is incomplete. Therefore, the controller 160 performs a table lookup according to the tilt state of the stylus 100, the distance, and the number of pixels occupied by the display frame B12 to determine the first coordinate component and the second coordinate component.

[0072] For example, the controller 160 may determine the X-coordinate component and the Y-coordinate component of the stylus pen 100 in the world coordinate system according to the following Table 1. Table 1 may be established based on previous tests and experiments.

[0073] distance Number of pixels in the display border Tilt state X-coordinate component Y coordinate component D1(cm) A θ1 x1 y1 D2(cm) B θ1 x2 y2 D3(cm) C θ1 x3 y3 D4(cm) D θ1 x4 y4 D1(cm) E θ2 x5 y5 D2(cm) F -θ2 x6 y6 D3(cm) G θ3 x7 y7 D4(cm) H -θ4 x8 y8

[0074] Table 1

[0075] In step S460, the controller 160 determines a third coordinate component on the third axis according to the distance. Specifically, the controller 160 can determine the distance between the stylus 100 and the display 100 according to the depth sensing data provided by the ranging sensor 120. Therefore, the controller 160 can determine the Z coordinate component of the stylus 100 in the world coordinate system according to the distance.

[0076] In step S470, the controller 160 maps the spatial position of the stylus pen 100 to the screen coordinates according to the spatial position and the tilt state of the stylus pen 100. In some embodiments, after obtaining the spatial position of the stylus pen 100 in the world coordinate system and the tilt state of the stylus pen 100, the controller 160 may project the spatial position of the stylus pen 100 onto the display plane along the direction of the stylus pen 100 to obtain a screen coordinate, so that the stylus pen 100 can interact with the electronic device including the display 200 according to the screen coordinate. For example, a 3D object model displayed on the display 200 can be dragged or rotated according to the screen coordinate.

[0077] However, the implementation of the present invention is not limited to the above description, and the contents of the above embodiments may be modified according to actual needs. For example, in some embodiments of the present invention, the spatial position of the stylus 100 may be determined based on sensing data from multiple sensors to further improve estimation accuracy.

[0078] Figure 7 is a schematic diagram of determining the spatial position of a stylus according to an embodiment of the present invention. Figure 7 In some embodiments, the image acquisition device 100 may acquire an image Img1 toward the display 100. The controller 160 may determine the first reference position information RP1 of the stylus 100 relative to the display 200 according to the display frame in the image. The first reference position information RP1 may include the three-dimensional coordinate position of the stylus 100 in the world coordinate system. Further, according to the position information and size information of the display frame in the image Img1, the controller 160 may estimate the three-dimensional coordinate position of the stylus 100 in the world coordinate system.

[0079] On the other hand, the distance measuring sensor 120 is a multi-region TOF distance measuring sensor, which can output depth sensing data D_d1. The controller 160 can determine the second reference position information RP2 of the stylus 100 relative to the display 200 according to the depth sensing data D_d1 of the multi-region TOF distance measuring sensor. The second reference position information RP2 may include the distance between the stylus 100 and the display 200. In addition, the second reference position information RP2 may include the tilt state of the stylus 100. In addition, since the depth sensing data D_d1 can be used to determine whether the distance measuring sensor 120 detects the display frame, the second reference position information RP2 may include the X coordinate component and the Y coordinate component of the stylus 100 in the world coordinate system.

[0080] In addition, the inertial sensor 130 can output inertial sensing data M_d1. The controller 160 can determine third reference position information RP3 of the stylus pen 100 relative to the display 200 according to the inertial sensing data M_d1 of the inertial sensor 130. The third reference position information RP31 includes the three-dimensional coordinate position of the stylus pen 100 in the world coordinate system and the tilt state of the stylus pen 100.

[0081] Based on this, the controller 160 can determine the spatial position FP of the stylus pen 100 relative to the display 200 according to the first reference position information RP1, the second reference position information RP2, and the third reference position information RP3. In detail, the controller 160 can perform data fusion and prediction related processing according to the first reference position information RP1, the second reference position information RP2, and the third reference position information RP3 to obtain the accurate spatial position FP. For example, the controller 160 can give different weight values ​​to the first reference position information RP1, the second reference position information RP2, and the third reference position information RP3, and perform weighted sum processing on the first reference position information RP1, the second reference position information RP2, and the third reference position information RP3 to perform data fusion to obtain the spatial position FP of the stylus pen 100 relative to the display 200. Alternatively, the controller 160 can use the second reference position information RP2 and the third reference position information RP3 to adjust the first reference position information RP1 to obtain the spatial position FP of the stylus pen 100 relative to the display 200. Alternatively, when the frame acquisition frequency of the image acquisition device 100 is low, the spatial position FP of the stylus 100 relative to the display 200 may be updated by the inertial sensing data M_d1 of the inertial sensor 130 .

[0082] In summary, in an embodiment of the present invention, an image acquisition device disposed on the stylus can acquire an image toward a display, and a distance sensor can detect the distance between the stylus and the display. Based on the display frame of the display in the image and the above distance, the spatial position of the stylus relative to the display can be estimated. Thus, the spatial position of the stylus relative to the display can be used to perform control functions to increase the diversification of the use of the stylus. In addition, by using the sensing data of multiple sensors on the stylus, the estimation accuracy of the spatial position of the stylus relative to the display can be further improved.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating a stylus position, characterized in that: include: Acquiring an image toward a display through an image acquisition device disposed on the stylus; Detecting the distance between the stylus pen and the display by a distance measuring sensor disposed on the stylus pen; detecting a display frame of the display in the image; as well as The spatial position of the stylus relative to the display is determined according to the display frame in the image and the distance.

2. The method for estimating the position of a stylus pen according to claim 1, characterized in that: The spatial position includes a first coordinate component, a second coordinate component, and a third coordinate component, and the step of determining the spatial position of the stylus relative to the display according to the display frame and the distance in the image includes: Determining the first coordinate component along a first axis and the second coordinate component along a second axis according to the display frame in the image; and The third coordinate component on a third axis is determined according to the distance, wherein the third axis is a normal direction of a display plane of the display, and the first axis, the second axis and the third axis are perpendicular to each other.

3. The method for estimating the position of a stylus pen according to claim 2, wherein: The step of determining the first coordinate component along the first axis and the second coordinate component along the second axis according to the display frame in the image comprises: Determining whether the display frame in the image is complete; and In response to the display frame in the image being incomplete, the first coordinate component along a first axis and the second coordinate component along a second axis are determined according to the distance and the number of pixels occupied by the display frame in the image.

4. The method for estimating the position of a stylus pen according to claim 3, characterized in that: The step of determining the first coordinate component along the first axis and the second coordinate component along the second axis according to the display frame in the image further includes: In response to the display frame in the image being complete, the first coordinate component along a first axis and the second coordinate component along a second axis are determined according to position information of the display frame in the image.

5. The method for estimating the position of a stylus pen according to claim 3, wherein: Also includes: detecting a tilt state of the stylus pen, In response to the incomplete display frame in the image, the step of determining the first coordinate component along the first axis and the second coordinate component along the second axis according to the distance and the number of pixels occupied by the display frame in the image comprises: In response to the incomplete display frame in the image, the first coordinate component and the second coordinate component are determined according to the tilt state, the distance, and the number of pixels.

6. The method for estimating the position of a stylus pen according to claim 5, characterized in that: The inertial sensor is disposed on the stylus pen and is used to detect the tilt state of the stylus pen.

7. The method for estimating the position of a stylus pen according to claim 5, characterized in that: The distance measuring sensor includes a multi-region time-of-flight distance measuring sensor, and the multi-region time-of-flight distance measuring sensor is used to detect the tilt state of the touch pen.

8. The method for estimating the position of a stylus pen according to claim 1, wherein: The image acquisition device includes a fisheye lens, the image includes a fisheye image, and the step of detecting the display frame of the display in the image includes: Performing image correction processing on the fisheye image according to the distance to obtain a corrected image; and The display frame of the display is detected in the corrected image.

9. The method for estimating the position of a stylus pen according to claim 1, characterized in that: The ranging sensor includes a multi-region time-of-flight ranging sensor, an inertial sensor is disposed on the stylus, and according to the display frame in the image and the distance, determining the spatial position of the stylus relative to the display includes: Determining first reference position information of the stylus relative to the display according to the display frame in the image; determining second reference position information of the stylus relative to the display according to the depth sensing data of the multi-region time-of-flight sensor; Determining third reference position information of the stylus relative to the display according to the inertial sensing data of the inertial sensor; and The spatial position of the stylus relative to the display is determined according to the first reference position information, the second reference position information, and the third reference position information.

10. A touch pen, characterized in that: include: Image acquisition device; Distance measuring sensor; as well as A controller, coupled to the image acquisition device and the distance measuring sensor, is configured to: Acquiring an image toward a display through the image acquisition device disposed on the stylus; Detecting the distance between the stylus pen and the display by the distance measuring sensor disposed on the stylus pen; detecting a display frame of the display in the image; as well as The spatial position of the stylus relative to the display is determined according to the display frame in the image and the distance.

Images