Method, device, medium and smart pen for solving image coordinates of smart pen tip

Abstract

The present invention belongs to the field of image processing technology, and specifically provides a method, device, medium and smart pen for solving the pen tip image coordinates of a smart pen. The method includes: obtaining N frames of first images; determining the corresponding first perspective transformation matrix according to each frame of the first image; constructing a pen tip image coordinate equation group according to the N first perspective transformation matrices; solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group. The embodiment of the present invention obtains the first images respectively when the smart pen is in different spatial postures, and finally constructs the pen tip image coordinate equation group with multiple first images. Therefore, the pen tip image coordinates obtained based on the pen tip image coordinate equation group take into account the decoding error when the smart pen is in different spatial postures, so that the solved pen tip image coordinates have higher accuracy, and can restore the writing trajectory under different pen holding postures with high accuracy to meet the pen holding habits of different users.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, medium and smart pen for solving the coordinates of an image at the tip of a smart pen. Background Art

[0002] How to perform real-time and high-precision digital acquisition of handwriting on paper is a long-standing research topic in the field of image processing technology. Especially in recent years, with the rise of remote conferencing and distance learning, business people, teachers and students need a "smart pen" that can write on traditional writing media and upload the handwriting content to the cloud or other electronic devices in real time.

[0003] At present, there are two main technical solutions in the existing technology to realize the function of smart pen. The first solution is to lay ordinary paper on the digital coordinate sensing board and write with a special electronic pen. The electronic pen leaves handwriting on the paper during the writing process. At the same time, the digital coordinate sensing board collects the coordinate signal of the pen tip contact point and realizes the digital collection of handwriting coordinates synchronously. Since the digital coordinates of this technical solution are generated by the digital coordinate sensing board and have nothing to do with the paper on which writing is done, if the displacement of the paper relative to the digital coordinate sensing board changes during the writing process, the actual handwriting on the paper and the handwriting restored to the screen after collection will be offset. For example, the handwriting correctly filled in the form on the paper will be displayed as filled outside the form on the computer, which brings inconvenience to practical application.

[0004] The second solution is to print or print a coordinate dot code (also known as code points) that is almost invisible to the human eye on ordinary writing paper, and use it to write with a smart pen that integrates a lens, an image sensor, and a dot code image recognition and coordinate decoding module. The smart pen leaves handwriting on the paper during the writing process, and the image sensor takes a picture of the coordinate dot code near the pen tip to obtain a digital image. The dot code image recognition and coordinate decoding module processes the digital image to obtain the coordinate value (also known as decoding), thereby realizing the digital collection of handwriting coordinates.

[0005] like Figure 1 The figure shows the working principle of the smart pen. Figure 1The housing, structural support frame and circuit board of the smart pen are omitted. 1001 is a paper with a coordinate dot matrix code printed on it; 1002 is the refill of the smart pen; 1003 is a lens; 1004 is an image sensor; 1005 is a pixel array of the image sensor; 1006 is the imaging target area of ​​the pixel array 1005 on the paper through the lens 1003; 1007 is the center origin of the pixel array, which is located on the optical axis of the lens 1003; 1008 is the target point on the paper corresponding to 1007; 1009 is the contact point between the refill and the paper (i.e., the pen tip); 1010 is the image of the contact point 1009 between the refill and the paper, which is virtually imaged on the image sensor 1004 through the lens 1003.

[0006] The paper 1001 is printed with a coordinate dot matrix code, and the corresponding coordinates can be obtained after the coordinate dot matrix code is decoded, so it can be considered that the paper 1001 has its own coordinate information. However, due to the spatial structure constraints of the pen core 1002 and the lens 1003, the center point 1008 of the lens's field of view on the paper cannot coincide with the pen tip 1009, and different pen holding postures will cause the pen body's tilt angle, pen body's rotation angle, and paper's rotation angle to change, that is, the plane angle and plane relative rotation angle of the image sensor 1004 and the paper 1001 are changed. In this way, even if the pen tip 1009 is fixed on the paper and does not move, the target field of view of the image sensor can change within a certain range, so the coordinate value obtained by decoding also changes within a certain range.

[0007] To explain the problem more intuitively, see Figure 2 : Fix the pen tip 1009 on the paper, keep the angle between the pen core 1002 and the paper unchanged, and move the pen body 360° along the circular trajectory shown in 2001. The corresponding running trajectory 2002 of the center of the lens field of view 1008 on the paper must be a circle with a radius greater than 0. Therefore, the original handwriting coordinates collected by the smart pen must also be a circle with a radius greater than 0. However, the handwriting left by the pen core on the paper is just a point. Furthermore, different people hold the pen in different postures, which makes the angle between the pen core 1002 and the paper 1001 different. The distribution of the trajectory 2002 falls within the annular surface with the pen tip 1009 as the center. As shown in 2003, the larger the angle between the pen core and the paper, the smaller the trajectory radius. As shown in 2004, the smaller the angle between the pen core and the paper, the larger the trajectory radius.

[0008] To further illustrate the problem, please see Figure 3A , Figure 3A The figure shows two tangent arcs drawn by the smart pen in two different spatial postures, namely arc ab and arc cd. In fact, the arc trajectories "seen" by the center of the pixel matrix 1007 are a'b' and c'd'. Ideally, the graphics reconstructed on the computer screen should be as follows Figure 3BAs shown in (1) in , however, a non-optimal design often leads to Figure 3B (2) or Figure 3B The result shown in (3) is that the tangent arcs drawn on paper are displayed as non-intersecting arcs on the screen.

[0009] For the above technical problems, CN201611188735.7 discloses a method for solving the position of the pen tip, which obtains the first image taken when the user clicks the first point on the paper to obtain the first transformation function, and then obtains the pen tip image coordinates based on the first transformation function and the paper coordinates of the first point. This technical solution obtains the pen tip image coordinates by clicking a point. Although it has the advantage of less calculation, it does not take into account the influence of the pen tip image coordinates when the user writes with different pen holding postures (i.e., different spatial postures of the smart pen). Therefore, when the user's pen holding posture when actually writing is different from the pen holding posture when clicking the first point on the paper, there will be a large error between the predetermined pen tip image coordinates and the coordinates of the pen tip actually imaged on the image sensor 1004, resulting in poor restoration accuracy of the handwriting. At the same time, when clicking the first point, it is difficult to ensure that the pen tip and the first point are perfectly matched. Therefore, there is a large sampling error in collecting the first image only once for the first point, which also leads to a large error in the predetermined pen tip image coordinates. Furthermore, during the writing process, the pen holding posture often changes, which also leads to poor accuracy in restoring the handwriting. Summary of the invention

[0010] In view of this, the embodiments of the present invention provide a method, device, medium and smart pen for solving the coordinates of the pen tip image of a smart pen, so as to solve the technical problem in the prior art that the error of the determined pen tip image coordinates is large when the pen holding posture changes.

[0011] In a first aspect, an embodiment of the present invention provides a method for solving the coordinates of an image of a smart pen tip, wherein the smart pen includes an image sensor, and the method includes the following steps:

[0012] S1: Control the image sensor to acquire N frames of first images, where N is a positive integer greater than or equal to 2; wherein when acquiring each frame of the first image, the spatial posture of the smart pen is different and the pen tip is located at the same preset point of the writing medium;

[0013] S2: determining the first perspective transformation matrix corresponding to each frame of the first image respectively, so as to determine N first perspective transformation matrices;

[0014] S3: constructing a pen tip image coordinate equation group according to the N first perspective transformation matrices;

[0015] S4: Solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group.

[0016] The embodiment of the present invention places the pen tip at a preset point and collects N frames of first images, and the spatial posture of the smart pen is different when acquiring each frame of the first image, and then determines the corresponding first perspective transformation matrix according to each frame of the first image and constructs a pen tip image coordinate equation group. Therefore, the pen tip image coordinates obtained by solving the pen tip image coordinate equation group fully consider the influence of the smart pen in different spatial postures, that is, the pen tip image coordinates better restore the coordinates of the pen tip actually imaged on the image sensor 1004 under different pen holding postures. In this way, even if the user writes with different pen holding postures or different users (pen holding postures often have very different) write, the embodiment of the present invention can better restore the coordinates of the pen tip actually imaged on the image sensor 1004, thereby effectively solving the technical problem in the prior art that the error of the determined pen tip image coordinates is large when the pen holding posture changes.

[0017] Preferably, the writing medium includes first code points arranged in an array, the first image includes second code points corresponding to the first code points, and in S2: determining the first perspective transformation matrix corresponding to each frame of the first image respectively, including:

[0018] Constructing a pairing relationship between the first code point and the second code point to determine at least 4 groups of pairing points;

[0019] Determine a first perspective transformation matrix corresponding to the first image according to the at least four groups of paired points;

[0020] The pairing points include first code points arranged along a first direction and first code points arranged along a second direction in the writing medium, and the first direction and the second direction are orthogonal.

[0021] As mentioned above, a dot matrix coordinate code, i.e., a first code point, is printed on paper 1001 (i.e., the writing medium referred to in the present invention). The first code point includes first code points arranged in an array, specifically including first code points arranged along a first direction and first code points arranged along a second direction in the writing medium. In the prior art, a certain offset processing is usually performed on the first code point to encode coordinate information, which results in errors between the actual paper coordinates and the nominal paper coordinates of the first code point. If the first perspective transformation matrix is ​​solved using these offset-processed first code points, the solution accuracy of the first perspective transformation matrix will be reduced. Therefore, the embodiment of the present invention solves the first perspective transformation matrix based on the first code points of the writing medium that have not been offset (i.e., the first code points arranged along the first direction and the first code points arranged along the second direction), which can ensure the solution accuracy.

[0022] Preferably, the pen tip image coordinate equation group is an overdetermined equation group, and in S4: solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group, it includes: solving the optimal solution of the pen tip image coordinates according to the least squares method.

[0023] The more first images are acquired, the more first perspective transformation matrices are determined. Correspondingly, the more perspective transformation equations are included in the pen tip image coordinate equation group. Among them, every two corresponding perspective transformation equations can determine the pen tip image coordinates (the unknowns of each perspective transformation equation are the horizontal and vertical coordinates of the pen tip image coordinates). Therefore, when the number of perspective transformation equations in the pen tip image coordinate equation group is greater than 2, the pen tip image coordinate equation group is an overdetermined equation group, and the optimal solution of the pen tip image coordinates can be obtained according to the least squares method, that is, the error of the optimal solution relative to various pen holding postures is small. Therefore, by adopting the technical solution of the embodiment of the present invention, no matter what pen holding posture is adopted, the pen tip image coordinates can well restore the coordinates of the pen tip actually imaged on the image sensor 1004.

[0024] Preferably, a three-dimensional space coordinate system is defined, in which the coordinate axis of one dimension is perpendicular to the writing medium, the angle between the smart pen and the first dimension is denoted as the first angle, the angle between the smart pen and the second dimension is denoted as the second angle, and the angle between the smart pen and the third dimension is denoted as the third angle, and the spatial posture includes the first angle, the second angle and the third angle.

[0025] The embodiment of the present invention defines the spatial posture using a first angle, a second angle and a third angle, which can better simulate the user's holding posture of the smart pen, so that when the pen holding posture changes, the determined pen tip image coordinates can well restore the coordinates of the pen tip actually imaged on the image sensor 1004.

[0026] Preferably, in S1, it includes: among the acquired N frames of the first images, at least two adjacent frames of the first images have different at least one of the first angle, the second angle and the third angle when acquired.

[0027] The embodiment of the present invention changes the spatial posture of the smart pen by changing at least one of the first angle, the second angle, and the third angle. In specific implementation, at least one of the first angle, the second angle, and the third angle can be adjusted by user operation, or at least one of the first angle, the second angle, and the third angle can be adjusted by a device for adjusting the above angles. When the device is used for adjustment, the values ​​of each angle can be sent to the corresponding driving mechanism to achieve adjustment of the spatial posture.

[0028] Preferably, in S4: solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group, it includes:

[0029] S41: Calculating the error of the pen tip image coordinates relative to the preset coordinates;

[0030] S42: Determine the magnitude relationship between the error and a threshold value to determine the calibration status of the smart pen.

[0031] When designing a smart pen, there will be a design value (i.e., preset coordinate) for the pen tip image coordinates. However, there are processing tolerances and / or assembly errors in optical and structural components. For products with the same design specifications, there must be an error between the design value and the actual value of each pen. Therefore, the embodiment of the present invention calculates the error of the pen tip image coordinates relative to the preset coordinates. If the error is less than a threshold, the calibration state of the smart pen is determined to be passed; if the error is greater than the threshold, the calibration state of the smart pen is determined to be failed. For a smart pen with a failed calibration state, steps S1 to S4 can be repeated to eliminate false detections. Alternatively, a smart pen with a failed calibration state can be determined to be a defective product to prevent it from entering the market.

[0032] Preferably, after S4, the method further includes:

[0033] S5: controlling the image sensor to acquire a second image;

[0034] S6: Acquire a second perspective transformation matrix according to the second image;

[0035] S7: Solve the pen tip paper coordinates according to the pen tip image coordinates and the second perspective transformation matrix.

[0036] After the pen tip image coordinates are determined, the smart pen can be used for writing. During the writing process, the image sensor is controlled to acquire the second image and determine the corresponding second perspective transformation matrix, and then the pen tip paper coordinates can be solved based on the second perspective transformation matrix and the pen tip image coordinates. During the continuous writing process, multiple frames of the second image can be continuously collected to determine multiple pen tip paper coordinates, and the handwriting can be restored based on the chronological order. Even if the pen holding posture changes during the continuous writing process, the pen tip image coordinates of the embodiment of the present invention can be well adapted to ensure the accuracy of handwriting restoration.

[0037] Preferably, the method further comprises:

[0038] Calculate the decoding success rate of the pen tip paper coordinates under different spatial postures;

[0039] The effective gesture working range of the smart pen is determined according to the decoding success rate.

[0040] Due to image distortion and other reasons, it is difficult for a smart pen to achieve a 100% decoding success rate for the pen tip paper coordinates. Therefore, the embodiment of the present invention calculates the decoding success rate of the pen tip paper coordinates in different spatial postures, and determines the effective posture working range of the smart pen according to a pre-set threshold. Thus, the smart pens whose effective posture working range does not meet the design standards are tested and screened to prevent defective products from entering the market.

[0041] In a second aspect, an embodiment of the present invention further provides a device for solving the coordinates of an image of a smart pen tip, wherein the smart pen includes an image sensor, and the device includes:

[0042] A control module, configured to control the image sensor to acquire N frames of first images, where N is a positive integer greater than or equal to 2; wherein when acquiring each frame of the first image, the spatial posture of the smart pen is different and the pen tip is located at the same preset point of the writing medium;

[0043] A first perspective matrix determination module, used to determine the first perspective transformation matrix corresponding to each frame of the first image, so as to determine N first perspective transformation matrices;

[0044] An equation group construction module, used for constructing a pen tip image coordinate equation group according to N first perspective transformation matrices;

[0045] A decoding module is used to solve the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group.

[0046] In a third aspect, an embodiment of the present invention provides a storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the method of the first aspect in the above-mentioned embodiment.

[0047] In a fourth aspect, an embodiment of the present invention provides a smart pen, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, and when the computer program instructions are executed by the processor, the method of the first aspect in the above embodiment is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0049] Figure 1 It is a schematic diagram of the working principle of a smart pen in the prior art.

[0050] Figure 2 It is a schematic diagram of decoding coordinates when the spatial posture of the smart pen changes in the prior art.

[0051] Figure 3A It is a schematic diagram of two tangent arcs drawn by a smart pen in the prior art.

[0052] Figure 3B Yes Figure 3A Schematic diagram of the restoration effect of the two tangent arcs.

[0053] Figure 4 It is a flowchart of a method for solving the coordinates of a smart pen tip image provided by an embodiment of the present invention.

[0054] Figure 5 The figure is a schematic diagram of imaging of a pen tip on an image sensor provided by an embodiment of the present invention.

[0055] Fig. 6A It is a schematic diagram of the effect of restoring handwriting using existing technology.

[0056] Figure 6B It is a schematic diagram of the effect of restoring handwriting using an embodiment of the present invention.

[0057] Figure 7 It is a flowchart of a method for solving a first perspective transformation matrix provided by an embodiment of the present invention.

[0058] Figure 8 It is a schematic diagram of solving a first perspective transformation matrix provided by an embodiment of the present invention.

[0059] Fig. 9 It is a schematic diagram of a smart pen spatial posture provided by an embodiment of the present invention.

[0060] Fig.10 It is a flow chart of a method for determining the calibration status of a smart pen provided by an embodiment of the present invention.

[0061] Fig.11 It is a schematic flow chart of a method for solving the coordinates of a pen tip on paper provided by an embodiment of the present invention.

[0062] Fig.12 It is a flow chart of a method for determining an effective posture working range of a smart pen provided by an embodiment of the present invention.

[0063] Fig.13 It is a structural schematic diagram of a smart pen tip image coordinate solving device provided by an embodiment of the present invention.

[0064] Fig.14 It is a structural schematic diagram of a smart pen provided by an embodiment of the present invention.

[0065] Fig.15 It is a structural schematic diagram of a smart pen calibration device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0067] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0068] As mentioned above, the pen tip position solution method in the prior art lacks consideration of the spatial posture of the smart pen, resulting in that the solved pen tip image position cannot be well adapted to different spatial postures.

[0069] In view of this, an embodiment of the present invention provides a method for solving the coordinates of the pen tip image of a smart pen, see Figure 4 , is a flow chart of the method, which specifically includes the following steps.

[0070] S1: Control the image sensor to acquire, where N is a positive integer greater than or equal to 2; wherein when acquiring each frame of the first image, the spatial posture of the smart pen is different and the pen tip is located at the same preset point of the writing medium;

[0071] S2: determining the first perspective transformation matrix corresponding to each frame of the first image respectively, so as to determine N first perspective transformation matrices;

[0072] S3: constructing a pen tip image coordinate equation group according to the N first perspective transformation matrices;

[0073] S4: Solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group.

[0074] like Figure 1 As shown, because the pen core 1002, the lens 1003, and the image sensor 1004 are all fixed on a rigid structure, no matter what spatial posture the pen body is in, for a certain design parameter, the pen tip 1009 must have a corresponding design value (i.e., the pen tip image coordinate) on the image sensor 1004 after being virtually imaged by the lens. Figure 5 As shown, a plane rectangular coordinate system is constructed on the image sensor 1004 with 1007 as the origin. It is assumed that the horizontal distance of the image 1010 formed by the pen tip from the origin is h microns, and the vertical distance is v microns. The horizontal side length of a single pixel of the image sensor 1004 is s microns, and the vertical side length is t microns. Then the design value of the pen tip image coordinates is (x, y) = (h / s, v / t).

[0075] In one embodiment of the present invention, the parameter design of the lens 1003 ensures that the pen tip 1009 is not imaged in the image sensor pixel array 1005, otherwise the pixel utilization of the image sensor will be reduced, or even not imaged in the entire image sensor 1004, but this does not affect the implementation of the present invention because the present invention does not actually use the optical image of the pen tip 1009.

[0076] In one embodiment of the present invention, the image sensor may use a high frame rate image sensor to increase the image acquisition speed.

[0077] The writing medium can be any plane such as paper, ceramic, plastic, display screen, etc. The writing medium is printed or printed with first code points that are nearly invisible to the human eye. The specific arrangement method is not specifically limited. The first code points all correspond to corresponding coordinates. The preset point is the first code point, so the coordinates of the preset point are known, and are recorded as (C0, R0) for ease of description.

[0078] Therefore, the embodiment of the present invention places the tip of the smart pen on a preset point (C0, R0), and continuously adjusts the spatial posture of the smart pen, and controls the image sensor to acquire the first image under different spatial postures, and finally acquires N frames of the first image. The first code point is imaged as a second code point in the image sensor, so the first image includes the second code point corresponding to the first code point.

[0079] Due to the angular relationship between the image sensor and the writing medium, the image captured by the image sensor has certain distortion and needs to be corrected. The correction can be performed based on the perspective transformation principle. Specifically, the perspective transformation equation is used for correction. The perspective transformation equation is:

[0080]

[0081] is equivalent to:

[0082]

[0083] Where C represents the horizontal coordinate of the first code point on the writing medium, and R represents the vertical coordinate of the first code point on the writing medium. x represents the horizontal coordinate of the second code point on the first image, and y represents the vertical coordinate of the second code point on the first image. m0, m1, m2, m3, m4, m5, m6, m7, and m8 constitute a first perspective transformation matrix M with 3 rows and 3 columns. Where,

[0084] As mentioned above, there is a one-to-one correspondence between the second code point and the first code point. Therefore, when the coordinates of the first code point and the second code point are known, the first perspective transformation matrix M can be obtained according to the above formula (2). Therefore, N first perspective transformation matrices M can be obtained according to N frames of the first image. Each first perspective transformation matrix M can be solved by the following technical solution:

[0085] At least 4 second code points are selected in the first image, and the first code points corresponding to the 4 second code points are determined. The coordinates of the second code points can be obtained by analyzing or measuring the first image, while the coordinates of the first code points are known. For example, the number of selected second code points is 4, and the coordinates of these 8 points (4 first code points and 4 second code points) are substituted into the above formula (2), and a system of equations including 8 perspective transformation equations can be obtained, and the number of unknowns is 9. At this time, all perspective transformation equations can be divided by the same unknown number, for example, all divided by m8, that is, the system of equations only includes 8 unknown numbers, and the first perspective transformation matrix M can be obtained according to the system of equations. It should be particularly noted that the more the number of selected second code points, the more stable the statistical information contained in the data, and the higher the accuracy of solving the system of equations. The optimal solution of the linear equation system is the basic theoretical knowledge of technicians in the field of image processing technology, which will not be repeated here. Therefore, the first images of N frames are solved according to the above technical solution, and N first perspective transformation matrices can be determined.

[0086] After obtaining N first perspective transformation matrices, the pen tip image coordinates can be solved. The specific solution still uses the perspective transformation principle. After the first perspective transformation matrix M is determined, since the coordinates of the preset point (C0, R0) are known, the pen tip image coordinates can be determined according to the above perspective transformation equation. It is obvious that the pen tip image coordinates can be obtained according to a first perspective transformation matrix M, but only using one first perspective transformation matrix M for solution, the error under different spatial postures is not taken into account, and the solved pen tip image coordinates are difficult to achieve high-precision restoration of handwriting under different spatial postures. Therefore, the embodiment of the present invention constructs a pen tip image coordinate equation group based on N first perspective transformation matrices M to obtain the optimal solution of the pen tip image coordinates under different spatial postures. If the pen tip image coordinate equation group is an overdetermined equation group, the optimal solution of the pen tip image coordinates is solved according to the least squares method.

[0087] Specifically, the above formula (2) is equivalent to:

[0088]

[0089] Wherein, let: a00=(m0-m6*C);

[0090] a01=(m1-m7*C);

[0091] a10=(m3-m6*R);

[0092] a11=(m4-m7*R);

[0093] b0=-(m2-m8*C);

[0094] b1=-(m5-m8*R);

[0095] Then formula (3) is equivalent to:

[0096]

[0097] There are N first perspective transformation matrices, that is, N groups of a00, a01, a10 and a11. Therefore, substituting into formula (4) can obtain a pen tip image coordinate equation group including 2N equations, specifically:

[0098]

[0099] Therefore, the least square method is used to solve formula (5) to obtain the optimal solution of the pen tip image coordinates (x, y).

[0100] In summary, the embodiment of the present invention places the pen tip at a preset point and collects N frames of first images, and the spatial posture of the smart pen is different when acquiring each frame of the first image, and then determines the corresponding first perspective transformation matrix of each frame of the first image and constructs a pen tip image coordinate equation group. Therefore, the pen tip image coordinates obtained based on the solution of the pen tip image coordinate equation group fully consider the influence of the smart pen in different spatial postures, that is, the pen tip image coordinates better restore the coordinates of the pen tip actually imaged on the image sensor 1004 under different pen holding postures. In this way, even if the user writes with different pen holding postures or different users (pen holding postures often have very different) write, the embodiment of the present invention can better restore the coordinates of the pen tip actually imaged on the image sensor 1004, thereby effectively solving the technical problem in the prior art that the error of the determined pen tip image coordinates is large when the pen holding posture changes. Specifically, for a more intuitive explanation of the technical effect of the present invention, please refer to Fig. 6A and Figure 6B . Fig. 6A In order to restore the handwriting using the designed value as the pen tip image coordinate (i.e. without considering the error of the smart pen in different spatial postures), Fig. 6A It can be seen intuitively that the handwriting written on the paper is basically located inside the table, while part of the restored handwriting is located outside the table, and the restoration effect is poor. Figure 6B This is a schematic diagram of the restoration effect of restoring handwriting by solving the handwriting image coordinates according to the embodiment of the present invention. The restored handwriting is located inside the table, and the restoration effect is obviously better than Fig. 6A .

[0101] See also Figure 7 The embodiment of the present invention also provides another method for solving the first perspective transformation matrix, which specifically includes the following steps:

[0102] S21: construct a pairing relationship between the first code point and the second code point to determine at least 4 groups of pairing points;

[0103] S22: Determine a first perspective transformation matrix corresponding to the first image according to the at least four groups of paired points;

[0104] The pairing points include first code points arranged along a first direction and first code points arranged along a second direction in the writing medium, and the first direction and the second direction are orthogonal.

[0105] For easier understanding, see Figure 8, the plane coordinate system formed by the x-axis and the y-axis is the image coordinate system, in which each code point is the second code point, and the plane coordinate system formed by the C-axis and the R-axis is the paper coordinate system, in which each code point is the first code point. When constructing a pairing relationship, at least 4 groups of pairing points still need to be determined. In the embodiment of the present invention, the selection of pairing points is as follows: select the dotted line C i The second code point and the dotted line R i The second code point passed through and its corresponding first code point are used as pairing points. It is obvious that the selected pairing points include the first code points arranged along the C-axis direction and the first code points arranged along the R-axis direction.

[0106] like Figure 8 As shown, the pairing points include 25 first code points and second code points corresponding to the first code points. Each group of pairing points is substituted into formula (2) to obtain a set of 50 equations to obtain the optimal solution of the first perspective transformation matrix.

[0107] See also Fig. 9 , which is a schematic diagram of a spatial posture provided by the implementation of the present invention. A three-dimensional spatial coordinate system is defined, wherein the coordinate axis of one dimension of the three-dimensional spatial coordinate system is perpendicular to the writing medium, the angle between the smart pen and the first dimension is denoted as the first angle, the angle between the smart pen and the second dimension is denoted as the second angle, and the angle between the smart pen and the third dimension is denoted as the third angle, and the spatial posture includes the first angle, the second angle, and the third angle.

[0108] like Fig. 9 As shown, a three-dimensional coordinate space is defined with the row direction of the paper plane as the b-axis, the column direction as the a-axis, and the vertical upward direction of the paper plane as the e-axis. The origin O is located at the coordinate (C0, R0), and the angle between the pen core and the a-axis is defined as α, the angle between the pen core and the b-axis is defined as β, and the angle between the pen core and the e-axis is defined as γ. That is, (α, β, γ) completely defines the spatial posture of the pen body relative to the coordinate paper, which can better simulate the user's holding posture of the smart pen, so that when the pen holding posture changes, the determined pen tip image coordinates can well restore the coordinates of the pen tip actually imaged on the image sensor 1004.

[0109] In one embodiment of the present invention, in S1, it includes: among the acquired N frames of the first images, at least two adjacent frames of the first images have different at least one of the first angle, the second angle and the third angle when acquired.

[0110] Specifically, two adjacent first image frames refer to two first image frames acquired continuously. In a specific implementation, at least one of the first angle, the second angle, and the third angle can be adjusted manually, or at least one of the first angle, the second angle, and the third angle can be adjusted by an automated device. Therefore, an embodiment of the present invention further provides a smart pen calibration device as shown in 15, which can be used to adjust the first angle, the second angle, and the third angle. For the specific implementation of the smart pen calibration device, please refer to the description below.

[0111] See also Fig.10 In one embodiment of the present invention, a method for determining the calibration status of a smart pen is also provided, comprising the following steps.

[0112] S41: Calculating the error of the pen tip image coordinates relative to the preset coordinates;

[0113] S42: Determine the magnitude relationship between the error and a threshold value to determine the calibration status of the smart pen.

[0114] As mentioned above, each smart pen has a design value of the pen tip image coordinates during design. However, due to the processing tolerances and assembly errors of optical and structural components, there must be errors between the actual value and the design value of each pen for the same design specifications. Therefore, it is necessary to solve the actual value of the pen tip image coordinates. Figure 4 The method described above has obtained the true value of the pen tip image coordinates. At this point, the error of the pen tip image coordinates relative to the preset coordinates can be calculated, and it can be determined whether the error is within the permitted range. If so, the smart pen can be considered to be a qualified product and the calibration has passed. If not, the calibration is considered to have failed, and it needs to be calibrated again using steps S1 to S4 or determined to be a defective product, and reassembly or other processes are performed.

[0115] See also Fig.11 , is a flow chart of a method for solving the coordinates of a pen tip on paper provided by an embodiment of the present invention, comprising the following steps.

[0116] S5: controlling the image sensor to acquire a second image;

[0117] S6: Acquire a second perspective transformation matrix according to the second image;

[0118] S7: Solve the pen tip paper coordinates according to the pen tip image coordinates and the second perspective transformation matrix.

[0119] Specifically, by combining Figure 4The described method has determined the pen tip image coordinates, and the pen tip paper coordinates can now be solved to restore the handwriting. When the pen tip is placed at any point on the writing medium, the image sensor is controlled to acquire a second image, and a second perspective transformation matrix is ​​acquired based on the second image. The method for acquiring the second perspective transformation matrix can be found in the aforementioned embodiment and will not be repeated here. Then, according to the aforementioned formula (1), since the pen tip image coordinates and the second perspective transformation matrix are both known, the pen tip paper coordinates can be obtained. Furthermore, when writing continuously, the image sensor is controlled to acquire multiple frames of the second image, and the corresponding pen tip paper coordinates are solved according to the method of this embodiment, and the handwriting can be restored based on each pen tip paper coordinate.

[0120] See also Fig.12 , is a flow chart of a method for determining an effective posture working range of a smart pen provided by an embodiment of the present invention. The method for determining an effective posture working range of a smart pen includes the following steps.

[0121] S8: Calculate the decoding success rate of the pen tip paper coordinates under different spatial postures;

[0122] S9: Determine the effective posture working range of the smart pen according to the decoding success rate.

[0123] Specifically, the decoding success rate of the pen tip paper coordinates by the smart pen is not always 100%, especially when the spatial posture is an extreme value. Therefore, the embodiment of the present invention obtains the decoding success rate of the pen tip paper coordinates under different spatial postures to determine the effective posture working range of the smart pen. For example, the decoding success rate is set to be above 90% as the effective working range, and the value combination of α, β, and γ is traversed in an orthogonal experimental design to determine the value of the decoding success rate above 90%, and the corresponding spatial posture range is the effective posture working range.

[0124] It should also be noted that formula (1), formula (2), formula (3) and formula (4) in the present invention are equivalent, and are only deformed in form for the sake of ease of understanding. Obviously, those skilled in the art can also make other equivalent deformations, which should all be regarded as the same technical solutions as the present invention.

[0125] See also Fig.13 , is a schematic diagram of a structure of a device for solving the coordinates of a smart pen tip image provided by an embodiment of the present invention, the device comprising:

[0126] A control module, configured to control the image sensor to acquire N frames of first images, where N is a positive integer greater than or equal to 2; wherein when acquiring each frame of the first image, the spatial posture of the smart pen is different and the pen tip is located at the same preset point of the writing medium;

[0127] A first perspective matrix determination module, used to determine the first perspective transformation matrix corresponding to each frame of the first image, so as to determine N first perspective transformation matrices;

[0128] An equation group construction module, used for constructing a pen tip image coordinate equation group according to N first perspective transformation matrices;

[0129] A decoding module is used to solve the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group.

[0130] In addition, the method for solving the coordinates of the smart pen tip image according to the embodiment of the present invention can be implemented by the smart pen. Fig.14 A schematic diagram of the hardware structure of a smart pen provided in an embodiment of the present invention is shown.

[0131] The smart pen may include a processor and a memory storing computer program instructions.

[0132] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.

[0133] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside a data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0134] The processor implements any one of the methods for solving the coordinates of the smart pen tip image in the above embodiments by reading and executing the computer program instructions stored in the memory.

[0135] In one example, the smart pen may also include a communication interface and a bus. Fig.14As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.

[0136] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0137] The bus includes hardware, software, or both, coupling the components of the smart pen to each other. By way of example and not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or other suitable bus or a combination of two or more of the above. Where appropriate, the bus may include one or more buses. Although embodiments of the present invention describe and illustrate specific buses, the present invention contemplates any suitable bus or interconnect.

[0138] In addition, in combination with the method for solving the coordinates of the image of the pen tip of the smart pen in the above embodiment, the embodiment of the present invention can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any method for solving the coordinates of the image of the pen tip of the smart pen in the above embodiment is implemented.

[0139] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0140] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0141] like Fig.15 , which is a schematic diagram of the structure of a smart pen calibration device provided by an embodiment of the present invention, the device comprises a rotating member 10, a first driving mechanism 20, a second driving mechanism 30, a third driving mechanism 40 and a supporting member 50: the rotating member 10 is provided with a first code point of preset row and column coordinates and a fixing structure for fixing the pen tip of the smart pen 100; the output end of the first driving mechanism 20 is transmission-connected with the rotating member 10, and the first driving mechanism 20 is used to drive the rotating member 10 to rotate around a third direction; the second driving mechanism 30 is transmission-connected with the smart pen 100, and drives the smart pen 100 to rotate around a fourth direction; the output end of the third driving mechanism 40 is transmission-connected with the second driving mechanism 30, and the third driving mechanism 40 is used to drive the second driving mechanism 30 to move along an arc centered on a preset axis; the supporting member 50 is used to support the first driving mechanism 20 and the third driving mechanism 40.

[0142] The first driving mechanism 20 may be a motor. For ease of description, the motor of the first driving mechanism 20 is referred to as the first motor in this article. The rotating member 10 may be in the form of a turntable. The stator portion of the first motor is fixed to the aforementioned support member 50, and the rotor portion is connected to the turntable. Specifically, the output shaft of the first motor may be fixedly connected to the turntable. In this way, the rotation of the rotor of the first motor can drive the turntable to rotate around the axis (i.e., the third direction) of the rotation of the shaft rotor. The first code point is printed on the side of the turntable that needs to contact the smart pen 100. The smart pen 100 obtains the position of the pen tip relative to the turntable by identifying the first code point.

[0143] Similarly, the second drive mechanism 30 and the third drive mechanism 40 may also use motors, wherein the motor used by the second drive mechanism 30 is a second motor, and the motor used by the third drive mechanism 40 is a third motor, and the stator part of the third motor may be fixed on the aforementioned support member 50.

[0144] The rotor of the second motor forms a transmission connection with the smart pen 100, and the rotor of the second motor transmits power to the smart pen 100, driving the smart pen 100 to rotate. The fourth direction is the direction of the axis of rotation of the rotor.

[0145] The third driving mechanism 40 drives the second driving mechanism 30 to move along a certain set arc, thereby driving the smart pen 100 to move together.

[0146] In order to facilitate the third drive mechanism 40 to drive the second drive mechanism 30 to move along the circular arc, the device of this embodiment also includes a swing member 60, one end of the swing member 60 is connected to the third drive mechanism 40 in a transmission manner, and the other end is connected to the second drive mechanism 30. The swing member 60 swings with a preset axis as the swing center under the drive of the third drive mechanism 40. The swing member 60 can be a swing arm, one end of which is connected to the rotor of the third motor. The other end is connected to the stator part of the second motor through an intermediate connecting member 80 such as a bolt. Specifically, one end of the bolt can be fixedly connected to the swing arm away from the movement of the third motor, and the other end is fixed to the stator of the second motor. The length of the swing arm and the positional relationship with the second motor and the third motor meet the requirements that when the third motor drives the swing arm to rotate, the second motor rotates along a predetermined circular arc driven by the swing arm, and the center of the circular arc coincides with the rotation axis of the third motor rotor. In this way, when the third motor rotor rotates, the swing arm is driven to swing around the axis of the third motor rotor, thereby driving the second motor to move along an arc centered on the axis of the third motor rotor.

[0147] The motion range of the first motor is 360°. In this embodiment, the first driving mechanism 20 is used to drive the turntable to rotate to simulate all the rotation angles of the paper plane relative to the smart pen 100 within the range of 360° when a person holds the smart pen 100 to write. The motion range of the second motor is ±180° reciprocating rotation or rotation at any angle. When the smart pen 100 is connected to the controller via a wired method, ±180° reciprocating rotation is adopted. When the smart pen 100 is connected to the controller via a wireless method, the rotation angle is not limited. In this embodiment, the second driving mechanism 30 is used to drive the rotation of the smart pen 100 to simulate all the rotation angles of the smart pen 100 relative to the handshake. The motion mode of the third motor is ±K° reciprocating motion, where ±K° is greater than the normal working inclination angle of the smart pen 100, and the angle of the smart pen 100 perpendicular to the turntable is defined as 0°. In this embodiment, the third driving mechanism 40 is used to simulate all the angles of the smart pen 100 relative to the paper when a person holds the smart pen 100 to write.

[0148] In addition, the present embodiment also includes a controller, which is electrically connected to the first drive mechanism 20, the second drive mechanism 30 and the third drive mechanism 40 respectively. The controller controls the first drive mechanism 20, the second drive mechanism 30 and the third drive mechanism 40 to rotate respectively to a set angle and direction according to the test requirements or calibration requirements, so as to accurately control the spatial posture of the smart pen 100, so as to accurately simulate the posture of the smart pen 100 relative to the handshake and writing paper when a person holds the pen to write during the test and calibration process of the smart pen 100. When calibrating and testing the smart pen 100, the pen tip can be fixed at a pre-set position on the turntable by using a fixed structure arranged on the turntable. The fixed structure on the aforementioned rotating member 10 can be a concave point arranged on the surface of the rotating member 10.

[0149] In another embodiment, Fig.15 As shown, the device further includes a guide rail 70 whose constraint trajectory is an arc, and the guide rail 70 is used to constrain the second drive mechanism 30 to move along an arc centered on a preset axis. The guide rail 70 is provided with an arc-shaped guide groove 71, one end of the intermediate connecting member 80 is connected to the second drive mechanism 30, and the other end thereof passes through the arc-shaped guide groove 71 and is connected to the end of the swing member 60 away from the third drive mechanism 40. In order to improve the reliability and accuracy of the smart pen 100 when making an arc motion during calibration and testing, the present invention is provided with a guide rail 70, so that the second drive mechanism 30 can accurately move along an arc centered on a set axis under the constraint of the guide rail 70, so that the calibration and testing of the smart pen 100 by the device is more accurate. The guide rail 70 of this embodiment can be a flat plate-shaped component mounted on the support member 50, and a circular arc-shaped guide groove 71 is provided on one end of the component as the guide rail 70. The center of the circular arc guide groove 71 coincides with the position of the output shaft of the third motor, and the middle section of the intermediate connector 80 used to connect the second motor and the swing arm is stuck in the circular arc guide groove 71. When the third motor drives the swing arm to swing, due to the restraining effect of the guide groove 71 on the intermediate connector 80, the intermediate connector 80 moves accurately along the ideal circular arc trajectory under the drive of the swing arm, and finally drives the smart pen 100 to be tested or calibrated to make accurate circular arc motion. The intermediate connector 80 can be a component with a cylindrical outer surface, such as a bolt, and the width of the guide groove 71 matches the diameter of the cylinder, and the axis of the cylinder is parallel to the axis of the output shaft of the third motor.

[0150] In another embodiment, Fig.15As shown, in this embodiment, the output end of the second driving mechanism 30 is further provided with a clamping member 31, and the clamping member 31 is used to clamp the smart pen 100. The device of this embodiment can be provided with a clip, and the clip is connected to the output shaft of the second motor. The additional clip can be used to clamp the tail of the smart pen 100, so that the second driving mechanism 30 can stably drive the smart pen 100 to move. In addition, the clamping member 31 that can both loosen and clamp the smart pen 100 can be used to conveniently load and unload the smart pen 100, greatly improving the efficiency of testing and calibrating the smart pen 100.

[0151] In this embodiment, the axis of the output shaft of the third motor and the axis of the output shaft of the first motor can be installed in a mutually perpendicular manner. In this regard, the support member 50 in this embodiment includes a first fixed surface and a second fixed surface that are perpendicular to each other, the first drive mechanism 20 is connected to the first fixed surface, and the third drive mechanism 40 is connected to the second fixed surface. The device of this embodiment can set the support member 50 to an "L" shape, so that the support member 50 has two mutually perpendicular surfaces, namely the first fixed surface and the second fixed surface, and then the first motor and the third motor are installed on the first fixed surface and the second fixed surface respectively, so that the axis of the output shaft of the third motor and the axis of the output shaft of the first motor are perpendicular to each other.

[0152] As another implementation, the support member 50 may also be a combination of a base 52 and an "L"-shaped bracket 51, wherein one surface of the "L"-shaped bracket 51 is connected to the base 52, and the other surface is perpendicular to the upper surface of the base 52, and the third drive mechanism 40 is mounted on the surface of the "L"-shaped bracket 51 perpendicular to the upper surface of the base 52. The base 52 may be a flat-plate-shaped platform having a horizontal surface, and the stator portion of the first motor is mounted on the horizontal surface. The output shaft of the first motor is vertically upward and connected to the rotating member 10. At the same time, a bracket of an "L"-shaped bracket 51 is mounted on the horizontal surface of the base 52, wherein one surface of the bracket is horizontally placed and fixed on the horizontal surface of the base 52, and the other surface is vertically arranged, and the stator portion of the third motor is mounted on the vertically arranged surface.

[0153] It should be noted that Fig.15 Some structures in the embodiment can be omitted or replaced by other structures during implementation to achieve the same technical effect.

[0154] based on Fig.15 The smart pen calibration device shown in the figure, the embodiment of the present invention also provides a method for adjusting the spatial posture of the smart pen, comprising the following steps:

[0155] Obtain a first angle value, a second angle value, and a third angle value;

[0156] The first angle value, the second angle value and the third angle value are sent to the first drive mechanism, the second drive mechanism and the third drive mechanism respectively; wherein the first angle value is used to control the parking position of the first drive mechanism, the second angle value is used to control the parking position of the second drive mechanism, and the third angle value is used to control the parking position of the third drive mechanism.

[0157] Therefore, the embodiment of the present invention realizes motion control of the first driving mechanism, the second driving mechanism and the third mechanism through the first angle value, the second angle value and the third angle value, and further realizes control of the first angle, the second angle and the third angle.

[0158] Furthermore, the first angle value, the second angle value and the third angle value may be obtained from a preset angle table.

[0159] In one embodiment of the present invention, the controller may be implemented by a host computer, and the host computer and the smart pen calibration device constitute a smart pen calibration system.

[0160] The first driving mechanism, the second driving mechanism and the third driving mechanism receive angle values ​​sent by the host computer to control the spatial posture of the smart pen. The angle values ​​include the first angle value, the second angle value and the third angle value.

[0161] The host computer and the first driving mechanism, the second driving mechanism and the third driving mechanism may be connected by wire or communicate wirelessly.

[0162] The technical solution described in this embodiment can accurately control the spatial posture of the smart pen, making it easier to perform combined Figure 4 The technical solution or combination described Fig.12 The technical solution described is used to automatically solve the pen tip image coordinates or automatically determine the effective posture working range of the smart pen.

[0163] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0164] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A method for solving the coordinates of an image at a tip of a smart pen, wherein the smart pen includes an image sensor, characterized in that: The method comprises the following steps: S1: Control the image sensor to acquire N frames of first images, where N is a positive integer greater than or equal to 2; wherein when acquiring each frame of the first image, the spatial posture of the smart pen is different and the pen tip is located at the same preset point of the writing medium; S2: determining the first perspective transformation matrix corresponding to each frame of the first image respectively, so as to determine N first perspective transformation matrices; S3: constructing a pen tip image coordinate equation group according to the N first perspective transformation matrices; S4: solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group; S5: controlling the image sensor to acquire a second image; S6: Acquire a second perspective transformation matrix according to the second image; S7: solving the pen tip paper coordinates according to the pen tip image coordinates and the second perspective transformation matrix; S8: Calculate the decoding success rate of the pen tip paper coordinates under different spatial postures; S9: Determine the effective posture working range of the smart pen according to the decoding success rate; The writing medium includes first code points arranged in an array, the first image includes second code points corresponding to the first code points, and in S2: first perspective transformation matrices corresponding to the first images of each frame are determined respectively, including: Constructing a pairing relationship between the first code point and the second code point to determine at least 4 groups of pairing points; Determine a first perspective transformation matrix corresponding to the first image according to the at least four groups of paired points; Wherein, the pairing points include a first code point arranged along a first direction in the writing medium and a first code point arranged along a second direction, the first direction and the second direction are orthogonal, the pen tip image coordinate equation group is an overdetermined equation group, and in S4: solving the pen tip image coordinates according to the coordinates of the preset points and the pen tip image coordinate equation group, it includes: solving the optimal solution of the pen tip image coordinates according to the least squares method.

2. The method according to claim 1, characterized in that A three-dimensional spatial coordinate system is defined, in which the coordinate axis of one dimension is perpendicular to the writing medium. The angle between the smart pen and the first dimension is denoted as the first angle, the angle between the smart pen and the second dimension is denoted as the second angle, and the angle between the smart pen and the third dimension is denoted as the third angle. The spatial posture includes the first angle, the second angle and the third angle.

3. The method according to claim 2, characterized in that In S1, it includes: among the acquired N frames of the first images, at least two adjacent frames of the first images have at least one of the first angle, the second angle and the third angle different when they are acquired.

4. The method according to claim 1, characterized in that In S4: solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group, the method includes: S41: Calculating the error of the pen tip image coordinates relative to the preset coordinates; S42: Determine the magnitude relationship between the error and a threshold value to determine the calibration status of the smart pen.

5. A device for solving the coordinates of an image of a smart pen tip, wherein the smart pen includes an image sensor, characterized in that: The device comprises: A control module, configured to control the image sensor to acquire N frames of first images, where N is a positive integer greater than or equal to 2; wherein when acquiring each frame of the first image, the spatial posture of the smart pen is different and the pen tip is located at the same preset point of the writing medium; A first perspective matrix determination module, used to determine the first perspective transformation matrix corresponding to each frame of the first image, so as to determine N first perspective transformation matrices; An equation group construction module, used for constructing a pen tip image coordinate equation group according to N first perspective transformation matrices; A decoding module, configured to solve the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group, wherein the writing medium includes first code points arranged in an array, the first image includes second code points corresponding to the first code points, and the first perspective transformation matrix corresponding to each frame of the first image is determined respectively, including: Constructing a pairing relationship between the first code point and the second code point to determine at least 4 groups of pairing points; Determine a first perspective transformation matrix corresponding to the first image according to the at least four groups of paired points; Wherein, the pairing points include first code points arranged along a first direction and first code points arranged along a second direction in the writing medium, the first direction and the second direction are orthogonal, the pen tip image coordinate equation group is an overdetermined equation group, and in S4: solving the pen tip image coordinates according to the coordinates of the preset points and the pen tip image coordinate equation group, the method includes: solving the optimal solution of the pen tip image coordinates according to the least squares method; The device is also used for: controlling the image sensor to acquire a second image; Acquire a second perspective transformation matrix according to the second image; Solve the pen tip paper coordinates according to the pen tip image coordinates and the second perspective transformation matrix; Calculate the decoding success rate of the pen tip paper coordinates under different spatial postures; The effective gesture working range of the smart pen is determined according to the decoding success rate.

6. A smart pen, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 4.

7. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images