Smart pen nib positioning method and device and smart pen

Abstract

The invention relates to the technical field of image processing, particularly provides an intelligent pen tip positioning method and device and an intelligent pen, and is used for solving the technical problem that the error of the determined coordinates of the paper surface of the pen tip is large when the intelligent pen is in different posture changes in the prior art. The method comprises the following steps: determining a first perspective transformation matrix corresponding to N frames of first images according to the N frames of first images; constructing a nib image coordinate equation set according to the first perspective transformation matrix; solving nib image coordinates according to the coordinates of the preset point and the nib image coordinate equation set; according to a second image acquired by the image sensor, determining a second perspective transformation matrix corresponding to the second image; and solving according to the nib image coordinates and the second perspective transformation matrix, and determining nib paper coordinates. The precision of the determined pen point paper surface coordinates is higher, the writing track height of the intelligent pen in different posture changes can be restored, and the pen holding habits of different users are met.

Description

[0001] The present application is a divisional application of the invention patent application with the application number of 2020110491683, which is filed on September 29, 2020, and the invention name of “Solution method, device, medium and intelligent pen for image coordinates of intelligent pen nib”. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, and in particular to an intelligent pen nib positioning method, device and intelligent pen. BACKGROUND

[0003] How to collect the handwriting on paper in real time with high precision is a research topic in the field of image processing technology that has been around for a long time. In recent years, with the rise of the demand for remote meetings and remote teaching, business people, teachers and students need an “intelligent pen” that can not only write on traditional writing media, but also upload the written content to the cloud or other electronic devices in real time.

[0004] There are currently two technical solutions to achieve the function of the intelligent pen. The first solution is to place ordinary paper on a digital coordinate sensing board and write with a special electronic pen. The electronic pen leaves a handwriting on the paper during the writing process, and the digital coordinate sensing board collects the coordinate signals of the nib contact point at the same time, thereby realizing the digital collection of handwriting coordinates. This technical solution is based on the digital coordinate sensing board, which has nothing to do with the paper. If the displacement of the paper relative to the digital coordinate sensing board changes during the writing process, the actual handwriting on the paper will be offset from the handwriting restored and displayed on the screen after collection, for example, the handwriting filled in the table on the paper will be displayed outside the table on the computer, which is inconvenient for practical application.

[0005] The second solution is to print or print coordinate dot array codes (also known as code points) on ordinary writing paper, which are invisible to the human eye, and write with an intelligent pen that integrates a lens, an image sensor, and a dot array code image recognition and coordinate decoding module. The intelligent pen leaves a handwriting on the paper during the writing process, and the image sensor takes a digital image of the coordinate dot array code near the nib. The dot array code image recognition and coordinate decoding module processes the digital image to obtain the coordinate value (also known as decoding), thereby realizing the digital collection of handwriting coordinates.

[0006] As shown in FIG. 1, which is a working schematic diagram of an intelligent pen, in order to facilitate the explanation of the working principle of the intelligent pen, Figure 1 Figure 1 ​The shell, structural support frame and circuit board of the smart pen are omitted. Among them, 1001 is a paper printed with a coordinate dot array code; 1002 is a pen refill of the smart pen; 1003 is a lens; 1004 is an image sensor; 1005 is a pixel array of the image sensor; 1006 is an imaging target area of the pixel array 1005 on the paper surface through the lens 1003; 1007 is a center origin of the pixel array, which is located on the optical axis of the lens 1003; 1008 is a target point on the paper surface corresponding to 1007; 1009 is a contact point of the pen refill and the paper surface (i.e. the pen tip); and 1010 is an image of the contact point 1009 of the pen refill and the paper surface virtually imaged on the image sensor 1004 through the lens 1003.

[0007] The paper 1001 is printed with a coordinate dot array code, which can obtain corresponding coordinates after decoding, so the paper 1001 can be considered as having coordinate information. However, due to the restriction of the spatial structure of the pen refill 1002 and the lens 1003, the center point 1008 of the field of view of the lens on the paper surface cannot coincide with the pen tip 1009, and different holding postures of the pen will cause the inclination angle and rotation angle of the pen body and the rotation angle of the paper to change, that is, the angle between the plane of the image sensor 1004 and the plane of the paper 1001 and the relative rotation angle between the planes are changing. In this way, even if the pen tip 1009 is fixed on the paper and does not move, the target field of view of the image sensor can change within a certain range, and thus the decoded coordinate values will also change within a certain range.

[0008] In order to more intuitively explain the problem, please refer to Figure 2 : The pen tip 1009 is fixed on the paper, the angle between the pen refill 1002 and the paper is kept unchanged, and the pen body moves along the circular trajectory shown by 2001 for 360°. The running trajectory 2002 of the corresponding lens field of view center 1008 on the paper surface must be a circle with a radius greater than 0, so the original handwriting coordinates collected by the smart pen must also be a circle with a radius greater than 0. However, the handwriting left by the pen refill on the paper is only a point. Further, different people have different holding postures of the pen, which makes the angle between the pen refill 1002 and the paper 1001 different, and the distribution of the trajectory 2002 falls within the annular surface with the pen tip 1009 as the center, as shown by 2003. The larger the angle between the pen refill and the paper, the smaller the radius of the trajectory, as shown by 2004, and the smaller the angle between the pen refill and the paper, the larger the radius of the trajectory.

[0009] In order to more intuitively and visually explain the problem, please see Figure 3A , Figure 3A , which shows two tangent arcs drawn by the smart pen in two different spatial postures, i.e. arc ab and arc cd. In fact, the arc trajectory "seen" by the pixel matrix center 1007 is a'b' and c'd'. Under ideal conditions, the graphics reconstructed on the computer screen should be as shown in Figure 3Bas shown in (1) in FIG. 1, however, one non-optimized design often leads to the results as shown in (2) or (3) in FIG. 1: the tangential arc drawn on paper is displayed as non-intersecting arcs on the screen. Figure 3B as shown in (2) in FIG. 1, or Figure 3B as shown in (3) in FIG. 1: the tangential arc drawn on paper is displayed as non-intersecting arcs on the screen.

[0010] For the above technical problems, CN201611188735.7 discloses a pen tip position solving method, which obtains a first image by shooting when a user clicks a first point on paper, obtains a first transformation function, and then obtains a pen tip image coordinate according to the first transformation function and a paper coordinate of the first point. This technical solution obtains the pen tip image coordinate by clicking one point, and has the advantage of small calculation amount, but does not consider the influence of different pen holding postures (i.e. different spatial postures of the smart pen) on the pen tip image coordinate when the user writes. Therefore, when the pen holding posture of the user in actual writing is different from the pen holding posture when the user clicks the first point on paper, there will be a large error between the pre-determined pen tip image coordinate and the coordinate of the actual imaging of the pen tip on the image sensor 1004, resulting in an error of the pen tip paper coordinate and poor accuracy of the restoration of the handwriting. SUMMARY

[0011] Therefore, the embodiments of the present application provide a smart pen tip positioning method, device and smart pen to solve the technical problem of large error of the determined pen tip paper coordinate when the pen holding posture changes in the prior art.

[0012] In a first aspect, the embodiments of the present application provide a smart pen tip positioning method, and the smart pen comprises an image sensor, and the method comprises the following steps:

[0013] determining a corresponding first perspective transformation matrix of each of N frames of first images obtained by the image sensor, to determine N first perspective transformation matrices, wherein N is a positive integer greater than or equal to 2, the spatial postures of the smart pen in each frame of the first images are all different, and the pen tips of the smart pen are all located on a same preset point of a writing medium;

[0014] constructing a pen tip image coordinate equation set according to the N first perspective transformation matrices;

[0015] solving the pen tip image coordinate according to the coordinate of the preset point and the pen tip image coordinate equation set;

[0016] determining a corresponding second perspective transformation matrix of each of N frames of second images obtained by the image sensor;

[0017] determining the pen tip positioning according to the pen tip image coordinate and the second perspective transformation matrix.

[0018] Preferably, the writing medium comprises first code points arranged in an array, the first image comprises second code points corresponding to the first code points, and in the step of determining the corresponding first perspective transformation matrix of each frame of the first image, the step comprises:

[0019] building a pairing relationship between the first code points and the second code points to determine at least 4 groups of paired points;

[0020] determining the first perspective transformation matrix corresponding to the first image according to the at least 4 groups of paired points;

[0021] wherein the paired points comprise first code points arranged along a first direction and first code points arranged along a second direction in the writing medium, and the first direction and the second direction are orthogonal;

[0022] neither the first code points arranged along the first direction nor the first code points arranged along the second direction are subjected to offset processing.

[0023] Preferably, a three-dimensional space coordinate system is defined, a coordinate axis of one dimension of the three-dimensional space coordinate system is perpendicular to the writing medium, the first angle between the smart pen and the first dimension is denoted as a first angle, the second angle between the smart pen and the second dimension is denoted as a second angle, and the third angle between the smart pen and the third dimension is denoted as a third angle, and the spatial pose comprises the first angle, the second angle, and the third angle.

[0024] Preferably, the identified N frames of the first image at least include at least one of the first angle, the second angle, and the third angle at the time of acquisition of adjacent 2 frames of the first image;

[0025] by changing at least one of the first angle, the second angle, and the third angle, thereby changing the spatial pose of the smart pen.

[0026] Preferably, in the step of correcting the second image acquired by the image sensor and determining the corresponding second perspective transformation matrix, the second image is a plurality of frames of second images acquired continuously based on the time sequence in a continuous writing process.

[0027] Preferably, in the step of solving the pen tip image coordinates according to the coordinates of the preset points and the pen tip image coordinate equation set, the step comprises:

[0028] calculating the error of the pen tip image coordinates relative to the preset coordinates;

[0029] judging the size relationship between the error and a threshold value to determine the correction state of the smart pen.

[0030] Preferably, if the error is less than a threshold value, the correction state of the smart pen is determined as passed; if the error is greater than the threshold value, the correction state of the smart pen is determined as failed, and for the smart pen with the correction state of failed, the previous steps of repeating the step are performed, or for the smart pen with the correction state of failed, the smart pen is determined as a defective product.

[0031] Preferably, after the pen tip paper coordinate is determined according to the pen tip image coordinate and the second perspective transformation matrix, the method further comprises: calculating a decoding success rate of the pen tip paper coordinate under different spatial postures, and determining a corresponding spatial posture range as an effective posture working range by traversing value combinations of the first included angle, the second included angle and the third included angle in an orthogonal experimental design manner according to the decoding success rate.

[0032] In a second aspect, an embodiment of the present application further provides a device for solving a pen tip paper coordinate of a smart pen, the smart pen comprising an image sensor, and the device comprising:

[0033] a first perspective transformation matrix obtaining module, configured to correct N frames of first images obtained by the image sensor by using a perspective transformation equation, and specifically comprising: determining a first perspective transformation matrix corresponding to each of the N frames of first images, so as to determine N first perspective transformation matrices, wherein N is a positive integer greater than or equal to 2, the spatial postures of the smart pen in each of the first images are different, and the pen tips of the smart pen are located at a same preset point on a writing medium;

[0034] an image coordinate equation set constructing module, configured to construct a pen tip image coordinate equation set according to the N first perspective transformation matrices, wherein the pen tip image coordinate equation set is an over-determined equation set;

[0035] a pen tip image coordinate solving module, configured to solve an optimal solution of the pen tip image coordinate according to the coordinates of the preset point and the pen tip image coordinate equation set by using a least square method;

[0036] a second perspective transformation matrix obtaining module, configured to correct a second image obtained by the image sensor by using the perspective transformation equation and determine a second perspective transformation matrix corresponding to the second image;

[0037] a pen tip paper coordinate determining module, configured to determine a pen tip paper coordinate according to the pen tip image coordinate and the second perspective transformation matrix.

[0038] In a third aspect, an embodiment of the present application provides a smart pen, the smart pen comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, the computer program instructions realize the method of the first aspect in the above-mentioned embodiment.

[0039] In summary, the beneficial effects of the present application are as follows:

[0040] The present application provides a smart pen tip positioning method, the smart pen comprises an image sensor, the method comprises: determining a corresponding first perspective transformation matrix of each of N frames of first images obtained by the image sensor, to determine N first perspective transformation matrices, wherein N is a positive integer greater than or equal to 2, the spatial poses of the smart pen in each frame of the first images are all different, and the smart pen tips are all located on a same preset point of a writing medium; constructing a pen tip image coordinate equation set according to the N first perspective transformation matrices; solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation set; determining a corresponding second perspective transformation matrix of each of N frames of second images obtained by the image sensor; and determining the pen tip positioning according to the pen tip image coordinates and the second perspective transformation matrix.

[0041] The present application corrects N frames of first images obtained by the image sensor according to the perspective transformation principle, processes the distortion caused by the angle relationship between the image sensor and the writing medium, and makes the restoration of the image position coordinates more accurate; the spatial poses of the smart pen in each frame of the first images obtained by the image sensor are all different, and the smart pen tips are all located on a same preset point of a writing medium, then a corresponding first perspective transformation matrix of each of the first images is determined, and a pen tip image coordinate equation set is constructed, so that the pen tip image coordinates obtained based on the pen tip image coordinate equation set fully consider the influence of the smart pen in different spatial poses;

[0042] After the pen tip image coordinates are determined, the smart pen can be used for writing, in the writing process, the image sensor is controlled to obtain second images and determine corresponding second perspective transformation matrices, then the pen tip paper coordinates can be solved based on the second perspective transformation matrices and the pen tip image coordinates, in the continuous writing process, multiple second images can be continuously collected, so that multiple pen tip paper coordinates are determined, and the handwriting is restored based on the time sequence, even if the pen holding posture changes in the continuous writing process, the pen tip paper coordinate restoration method of the present application can be well adapted, the technical problem of large pen tip paper coordinate error is solved, and the accuracy of handwriting restoration is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments of the present application will be briefly introduced as follows, and for those skilled in the art, other drawings can also be obtained without creative labor on the premise of not paying creative labor, and these are within the protection scope of the present application.

[0044] Figure 1is a schematic diagram of the working principle of the smart pen in the prior art.

[0045] Figure 2 is a schematic diagram of decoding coordinates when the spatial pose of the smart pen changes in the prior art.

[0046] Figure 3A is a schematic diagram of two tangent arcs drawn by the smart pen in the prior art.

[0047] Figure 3B is a schematic diagram of the restoration effect of the two tangent arcs in Figure 3A

[0048] Figure 4 is a schematic diagram of the flow of the method for solving the image coordinates of the nib of the smart pen provided by the embodiment of the application.

[0049] Figure 5 is a schematic diagram of the imaging of the nib on the image sensor provided by the embodiment of the application.

[0050] Figure 6A is a schematic diagram of the effect of restoring handwriting using the prior art.

[0051] Figure 6B is a schematic diagram of the effect of restoring handwriting using the embodiment of the application.

[0052] Figure 7 is a schematic diagram of the flow of the method for solving the first perspective transformation matrix provided by the embodiment of the application.

[0053] Figure 8 is a schematic diagram of the method for solving the first perspective transformation matrix provided by the embodiment of the application.

[0054] Figure 9 is a schematic diagram of the spatial pose of the smart pen provided by the embodiment of the application.

[0055] Figure 10 is a schematic diagram of the flow of the method for judging the correction state of the smart pen provided by the embodiment of the application.

[0056] Figure 11 is a schematic diagram of the flow of the method for solving the paper coordinates of the nib provided by the embodiment of the application.

[0057] Figure 12 is a schematic diagram of the flow of the method for determining the effective pose working range of the smart pen provided by the embodiment of the application.

[0058] Figure 13 is a schematic diagram of the structure of the device for solving the image coordinates of the nib of the smart pen provided by the embodiment of the application.

[0059] Figure 14 ​is a structural schematic diagram of a smart pen provided by an embodiment of the present application.

[0060] Figure 15 is a structural schematic diagram of a smart pen correction device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0061] The features and exemplary embodiments of various aspects of the present application will be described in detail below with reference to the drawings and embodiments, so that the purposes, technical solutions and advantages of the present application are more clearly apparent. It should be understood that the specific embodiments described herein are configured only to explain the present application and are not configured to limit the present application. The present application can be implemented without some of the specific details by those skilled in the art. The following description of the embodiments is merely to provide a better understanding of the present application by showing examples of the present application.

[0062] It should be noted that, in this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... " does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the stated elements.

[0063] As described above, the existing pen tip position solving method lacks consideration of the spatial pose of the smart pen, resulting in that the solved pen tip image position cannot be well adapted to different spatial poses.

[0064] Therefore, an embodiment of the present application provides a method for solving a smart pen tip image coordinate, please refer to Figure 4 is a flowchart of the method, and specifically includes the following steps.

[0065] S1: controlling the image sensor to acquire, N is a positive integer greater than or equal to 2; wherein when acquiring each frame of the first image, the spatial pose of the smart pen is different and the pen tip is located on the same preset point of the writing medium;

[0066] S2: determining a corresponding first perspective transformation matrix for each frame of the first image, to determine N first perspective transformation matrices;

[0067] S3: constructing a pen tip image coordinate equation group according to the N first perspective transformation matrices;

[0068] S4: Solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group.

[0069] like Figure 1 As shown, because the pen core 1002, lens 1003, and image sensor 1004 are all fixed on a rigid structure, no matter what spatial posture the pen body is in, for a given design parameter, the pen tip 1009 must also have a corresponding design value (i.e., pen tip image coordinate) on the image sensor 1004 after being virtually imaged by the lens. Figure 5 As shown, a plane rectangular coordinate system is constructed on the image sensor 1004 with 1007 as the origin. Assume that the horizontal distance of the image 1010 formed by the pen tip from the origin is h microns and the vertical distance is v microns. The horizontal side length of a single pixel of the image sensor 1004 is s microns and the vertical side length is t microns. Then the design value of the pen tip image coordinate is (x, y) = (h / s, v / t).

[0070] In one embodiment of the present invention, the parameter design of the lens 1003 ensures that the pen tip 1009 is not imaged in the image sensor pixel array 1005, otherwise the pixel utilization of the image sensor will be reduced, or even the entire image sensor 1004 will not be imaged. However, this does not affect the implementation of the present invention because the present invention does not actually use the optical image of the pen tip 1009.

[0071] In one embodiment of the present invention, the image sensor may be a high frame rate image sensor to increase the image acquisition speed.

[0072] The writing medium can be any flat surface, such as paper, ceramic, plastic, or a display screen. First code points, which are substantially invisible to the human eye, are printed or imprinted on the writing medium. The specific arrangement is not particularly limited. Each of the first code points has corresponding coordinates. The predetermined point is the first code point, so its coordinates are known and are denoted as (C0, R0) for ease of description.

[0073] Therefore, in an embodiment of the present invention, the tip of the smart pen is placed at a preset point (C0, R0), and the spatial posture of the smart pen is continuously adjusted. Under different spatial postures, the image sensor is controlled to capture a first image, ultimately acquiring N frames of first images. The first code point is imaged as a second code point on the image sensor, and thus the first image includes the second code point corresponding to the first code point.

[0074] Limited by the angle relationship between the image sensor and the writing medium, the image collected by the image sensor has certain distortion, which needs to be corrected. The correction can be performed according to the principle of perspective transformation. Specifically, the correction is performed by using a perspective transformation equation, which is:

[0075]

[0076] which is equivalent to:

[0077]

[0078] where C represents the horizontal coordinate of the first code point on the writing medium, and R represents the vertical coordinate of the first code point on the writing medium. x represents the horizontal coordinate of the second code point on the first image, and y represents the vertical coordinate of the second code point on the first image. m0, m1, m2, m3, m4, m5, m6, m7, and m8 constitute a first perspective transformation matrix M of 3 rows and 3 columns. Among them,

[0079] As described above, the second code point and the first code point have a one-to-one correspondence, so in the case where the coordinates of the first code point and the second code point are known, the first perspective transformation matrix M can be obtained according to the above formula (2). Therefore, N first perspective transformation matrices M can be obtained according to N first images. Each first perspective transformation matrix M can be solved by the following technical solution:

[0080] At least 4 second code points are selected in the first image, and the first code points corresponding to the 4 second code points are determined. The coordinates of the second code points can be obtained by analyzing or measuring the first image, and the coordinates of the first code points are known. For example, the number of selected second code points is 4, and the coordinates of the 8 points (4 first code points and 4 second code points) are substituted into the above formula (2) to obtain an equation group including 8 perspective transformation equations, and the unknowns are 9. At this time, all perspective transformation equations can be divided by the same unknown, for example, divided by m8, that is, the equation group only includes 8 unknowns, and the first perspective transformation matrix M can be obtained according to the equation group. It needs to be specially pointed out that the more the number of selected second code points is, the more stable the statistical information contained in the data is, and the higher the accuracy of the equation group solution is. The optimal solution of the linear equation group is the basic theoretical knowledge of the image processing technology field, which will not be repeated here. Therefore, N first perspective transformation matrices can be determined by solving N first images according to the above technical solution.

[0081] After obtaining the N first perspective transformation matrices, the pen tip image coordinates can be solved. Specifically, after the first perspective transformation matrix M is determined, the pen tip image coordinates can be determined according to the above perspective transformation equation, since the coordinates (C0, R0) of the preset point are known. It is obvious that the pen tip image coordinates can be solved according to one first perspective transformation matrix M, but the solution obtained by using only one first perspective transformation matrix M does not take into account the error in different spatial postures, and it is difficult to realize high-precision restoration of handwriting in different spatial postures. Therefore, according to the N first perspective transformation matrices M, the pen tip image coordinate equation group is constructed to obtain the optimal solution of the pen tip image coordinates in different spatial postures. If the pen tip image coordinate equation group is an overdetermined equation group, the optimal solution of the pen tip image coordinates is solved according to the least square method.

[0082] Specifically, the above formula (2) is equivalent to:

[0083]

[0084] wherein a00=(m0-m6*C);

[0085] a01=(m1-m7*C);

[0086] a10=(m3-m6*R);

[0087] a11=(m4-m7*R);

[0088] b0=-(m2-m8*C);

[0089] b1=-(m5-m8*R);

[0090] Then formula (3) is equivalent to:

[0091]

[0092] There are N first perspective transformation matrices, that is, there are N groups of a00, a01, a10 and a11, therefore, the pen tip image coordinate equation group including 2N equations can be obtained by substituting formula (4), and the specific equation is:

[0093]

[0094] Therefore, the optimal solution of the pen tip image coordinates (x, y) is obtained by solving formula (5) according to the least square method.

[0095] In summary, the embodiment of the present application places the pen tip at a preset point and collects N frames of first images, and the spatial poses of the smart pen are different when acquiring each frame of the first images, then determines the corresponding first perspective transformation matrix of each frame of the first images and constructs a pen tip image coordinate equation set. Therefore, the pen tip image coordinates obtained by solving the pen tip image coordinate equation set fully consider the influence of the smart pen in different spatial poses, that is, the pen tip image coordinates better restore the coordinates of the actual imaging of the pen tip on the image sensor 1004 under different pen holding poses. In this way, even if the user writes with different pen holding poses or different users (the pen holding poses often differ greatly) write, the embodiment of the present application can better restore the coordinates of the actual imaging of the pen tip on the image sensor 1004, thereby effectively solving the technical problem that the error of the determined pen tip image coordinates is large when the pen holding pose changes. Specifically, to more intuitively illustrate the technical effect of the present application, please refer to Figure 6A and Figure 6B . Figure 6A The restoration effect of using the design value as the pen tip image coordinates to restore the handwriting (that is, without considering the error of the smart pen in different spatial poses) is shown in Figure 6A It can be seen directly that the handwriting written on the paper is basically located in the table, and part of the restored handwriting is located outside the table, and the restoration effect is poor. Figure 6B The restoration effect of using the pen tip image coordinates obtained by the embodiment of the present application to restore the handwriting is shown in the restoration effect diagram, and the restored handwriting is located in the table, and the restoration effect is obviously better than Figure 6A .

[0096] Please refer to Figure 7 , the embodiment of the present application further provides another method for solving the first perspective transformation matrix, which specifically includes the following steps:

[0097] S21: Construct a pairing relationship of the first code points and the second code points to determine at least 4 groups of paired points;

[0098] S22: Determine the first perspective transformation matrix corresponding to the first image according to the at least 4 groups of paired points;

[0099] Wherein, the paired points include first code points arranged along a first direction and first code points arranged along a second direction in the writing medium, and the first direction and the second direction are orthogonal.

[0100] For ease of understanding, please refer to Figure 8The plane coordinate system formed by the x-axis and the y-axis is the image coordinate system, in which each code point is the second code point. The plane coordinate system formed by the C-axis and the R-axis is the paper coordinate system, in which each code point is the first code point. When establishing a pairing relationship, at least four pairs of pairing points must still be determined. In the embodiment of the present invention, the pairing points are selected as follows: select the dotted line C i The second code point and the dotted line R i The second code point passed through and its corresponding first code point are used as pairing points. Obviously, the selected pairing points include the first code points arranged along the C-axis direction and the first code points arranged along the R-axis direction.

[0101] like Figure 8 As shown, the pairing points include 25 first code points and the second code points corresponding to the first code points. Each set of pairing points is substituted into formula (2) to obtain a set of 50 equations to obtain the optimal solution of the first perspective transformation matrix.

[0102] See Figure 9 , which is a schematic diagram of a spatial posture provided by an embodiment of the present invention. A three-dimensional spatial coordinate system is defined, where one axis of one dimension of the three-dimensional spatial coordinate system is perpendicular to the writing medium. The angle between the smart pen and the first dimension is denoted as a first angle, the angle between the smart pen and the second dimension is denoted as a second angle, and the angle between the smart pen and the third dimension is denoted as a third angle. The spatial posture includes the first angle, the second angle, and the third angle.

[0103] like Figure 9 As shown, a three-dimensional coordinate space is defined with the row direction of the paper plane as the b-axis, the column direction as the a-axis, and the vertical upward direction of the paper as the e-axis. The origin O is located at the coordinate (C0, R0). The angle between the pen core and the a-axis is defined as α, the angle between the pen core and the b-axis is defined as β, and the angle between the pen core and the e-axis is defined as γ. That is, (α, β, γ) completely defines the spatial posture of the pen body relative to the coordinate paper, which can better simulate the user's holding posture of the smart pen, so that when the pen holding posture changes, the determined pen tip image coordinates can well restore the coordinates of the pen tip actually imaged on the image sensor 1004.

[0104] In one embodiment of the present invention, in S1, it includes: among the N frames of the first images acquired, at least two adjacent frames of the first images have different at least one of the first angle, the second angle and the third angle when acquired.

[0105] Specifically, the two adjacent first images refer to two first images acquired successively. In a specific implementation, at least one of the first angle, the second angle and the third angle can be adjusted in a manual manner, or at least one of the first angle, the second angle and the third angle can be adjusted by an automatic device. Therefore, the embodiment of the present application further provides an intelligent pen correction device as shown in FIG. 15, which can be used to adjust the first angle, the second angle and the third angle. The specific implementation of the intelligent pen correction device will be described hereinafter.

[0106] Referring to Figure 10 In an embodiment of the present application, a method for judging the correction state of the intelligent pen is further provided, which comprises the following steps.

[0107] S41: calculating the error of the stylus image coordinate relative to a preset coordinate;

[0108] S42: judging the size relationship between the error and a threshold value to judge the correction state of the intelligent pen.

[0109] As described above, each intelligent pen has a design value of stylus image coordinate when designed, but because of the machining tolerance and assembly error of optical and structural components, the real value of each pen of the same design specification product must be different from the design value, and therefore the real value of the stylus image coordinate needs to be solved. The described method has solved the real value of the stylus image coordinate. Figure 4 At this time, the error of the stylus image coordinate relative to a preset coordinate can be calculated, and it is judged whether the error is within the permissible range. If yes, it is considered that the intelligent pen is a qualified product and the correction is passed, if not, it is considered that the correction is not passed, and the steps of S1 to S4 need to be performed again for correction or it is determined that it is a defective product, and reassembly or other processes are performed.

[0110] Referring to Figure 11 is a method flow diagram for solving the stylus paper coordinate provided by the embodiment of the present application, which comprises the following steps.

[0111] S5: controlling the image sensor to acquire a second image;

[0112] S6: acquiring a second perspective transformation matrix according to the second image;

[0113] S7: solving the stylus paper coordinate according to the stylus image coordinate and the second perspective transformation matrix.

[0114] Specifically, the method for solving the stylus paper coordinate is combined with Figure 4The method described has determined the pen tip image coordinates, at which time the pen tip paper coordinates can be solved to restore the handwriting. Wherein, the pen tip is placed at any point of the writing medium, the second image is acquired by controlling the image sensor, and the second perspective transformation matrix is acquired according to the second image. The acquisition method of the second perspective transformation matrix can refer to the foregoing embodiments, which will not be described here. Then, according to the foregoing formula (1), since the pen tip image coordinates and the second perspective transformation matrix are both known, the pen tip paper coordinates can be solved. Further, when continuous writing is performed, a plurality of second images are acquired by controlling the image sensor, the corresponding pen tip paper coordinates are solved according to the method of the embodiment, and the handwriting can be restored according to the pen tip paper coordinates.

[0115] Please refer to Figure 12 , which is a method flow diagram for determining the effective gesture working range of the smart pen provided by the embodiment of the application. The method for determining the effective gesture working range of the smart pen comprises the following steps.

[0116] S8: calculating the decoding success rate of the pen tip paper coordinates under different spatial gestures;

[0117] S9: determining the effective gesture working range of the smart pen according to the decoding success rate.

[0118] Specifically, the decoding success rate of the smart pen for the pen tip paper coordinates is not always 100%, especially in the case of extreme values of the spatial gesture. Therefore, the embodiment of the application determines the effective gesture working range of the smart pen by acquiring the decoding success rate of the pen tip paper coordinates under different spatial gestures. For example, the decoding success rate above 90% is set as the effective working range, and the value corresponding to the decoding success rate above 90% is determined by traversing the value combination of α, β and γ in the orthogonal experimental design manner, so that the spatial gesture range corresponding to the decoding success rate above 90% is the effective gesture working range.

[0119] It also needs to be particularly pointed out that the formula (1), formula (2), formula (3) and formula (4) in the application are equivalent, and are only transformed in form for the convenience of understanding. Obviously, other equivalent transformations can also be performed by those skilled in the art, which should be regarded as the same technical solutions as the application.

[0120] Please refer to Figure 13 , which is a structure diagram of a device for solving the pen tip image coordinates of the smart pen provided by the embodiment of the application. The device comprises:

[0121] A control module is configured to control the image sensor to acquire N frames of first images, where N is a positive integer greater than or equal to 2; wherein when each frame of the first images is acquired, the spatial gesture of the smart pen is different and the pen tip is located on the same preset point of the writing medium;

[0122] a first perspective matrix determination module, configured to determine a corresponding first perspective transformation matrix of each frame of the first images respectively according to the first images, to determine N first perspective transformation matrices;

[0123] an equation set construction module, configured to construct a pen tip image coordinate equation set according to the N first perspective transformation matrices;

[0124] a decoding module, configured to solve the pen tip image coordinate according to the coordinates of the preset points and the pen tip image coordinate equation set.

[0125] In addition, the method for solving the pen tip image coordinate of the smart pen according to the embodiments of the present application can be implemented by the smart pen. Figure 14 A hardware structure schematic diagram of the smart pen according to the embodiments of the present application is shown.

[0126] The smart pen can include a processor and a memory storing computer program instructions.

[0127] Specifically, the processor can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0128] The memory can include a mass storage for data or instructions. By way of example and not limitation, the memory can include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory can include removable or non-removable (or fixed) media. Where appropriate, the memory can be internal or external to the data processing device. In certain embodiments, the memory is a nonvolatile solid-state memory. In certain embodiments, the memory includes read-only memory (ROM). Where appropriate, this ROM can be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0129] The processor reads and executes the computer program instructions stored in the memory to implement any one of the methods for solving the pen tip image coordinate of the smart pen in the above embodiments.

[0130] In one example, the smart pen can further include a communication interface and a bus. Wherein, as Figure 14As shown, the processor, the memory, the communication interface are connected through the bus and complete the communication between each other.

[0131] The communication interface is mainly used for realizing the communication between the modules, devices, units and / or equipment in the embodiments of the application.

[0132] The bus includes hardware, software or both, coupling the components of the smart pen to each other. By way of example, and not limitation, the bus can include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an infiniband interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standards association local (VLB) bus, or another suitable bus or combination of two or more of these. Where suitable, the bus can include one or more buses. Although particular buses are described and shown in the embodiments of the application, the application contemplates any suitable bus or interconnect.

[0133] In addition, in combination with the smart pen tip image coordinate solving method in the above-mentioned embodiments, the embodiments of the application can provide a computer readable storage medium for implementation. The computer readable storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement any one of the smart pen tip image coordinate solving methods in the above-mentioned embodiments.

[0134] It needs to be clear that the application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of simplicity, detailed descriptions of known methods are omitted here. In the above-mentioned embodiments, several specific steps are described and shown as examples. However, the method process of the application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the application.

[0135] The functional blocks shown in the above described structural block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, functional cards, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. A "machine-readable medium" includes any medium that can store or transfer information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments can be downloaded via a computer network such as the Internet, an intranet, and the like.

[0136] As shown in Figure 15 Fig. 1 is a structural schematic diagram of a smart pen correction device according to an embodiment of the present application. The device comprises a rotating member 10, a first driving mechanism 20, a second driving mechanism 30, a third driving mechanism 40, and a support member 50. The rotating member 10 is provided with a first code point of a preset row-column coordinate and a fixing structure for fixing a pen tip of a smart pen 100. An output end of the first driving mechanism 20 is in transmission connection with the rotating member 10. The first driving mechanism 20 is used to drive the rotating member 10 to rotate around a third direction. The second driving mechanism 30 is used to be in transmission connection with the smart pen 100 and drive the smart pen 100 to rotate around a fourth direction. An output end of the third driving mechanism 40 is in transmission connection with the second driving mechanism 30. The third driving mechanism 40 is used to drive the second driving mechanism 30 to move along a circular arc with a preset axis as the center. The support member 50 is used to support the first driving mechanism 20 and the third driving mechanism 40.

[0137] The first driving mechanism 20 can adopt a motor. For the convenience of description, the motor of the first driving mechanism 20 is referred to as a first motor. The rotating member 10 can adopt the form of a rotating disc. A stator part of the first motor is fixed on the aforementioned support member 50. A rotor part is connected with the rotating disc. Specifically, an output shaft of the first motor can be fixedly connected with the rotating disc. In this way, the rotation of the rotor of the first motor can drive the rotating disc to rotate around the axis of the rotation of the rotor (i.e., the third direction). A side of the rotating disc that needs to be in contact with the smart pen 100 is printed with the first code point. The smart pen 100 can acquire the position of the pen tip relative to the rotating disc by identifying the first code point.

[0138] Similarly, the second driving mechanism 30 and the third driving mechanism 40 can also adopt motors. The motor adopted by the second driving mechanism 30 is referred to as a second motor. The motor adopted by the third driving mechanism 40 is referred to as a third motor. A stator part of the third motor can be fixed on the aforementioned support member 50.

[0139] The rotor of the second motor is in transmission connection with the smart pen 100, and the rotor of the second motor transmits power to the smart pen 100 to drive the smart pen 100 to rotate. The fourth direction is the direction of the axis of the rotor.

[0140] The third driving mechanism 40 drives the second driving mechanism 30 to move along a certain set circular arc, thereby driving the smart pen 100 to move.

[0141] In order to facilitate the third driving mechanism 40 to drive the second driving mechanism 30 to move along a circular arc, the device of the embodiment further comprises a swing member 60, one end of the swing member 60 is in transmission connection with the third driving mechanism 40, and the opposite end is connected with the second driving mechanism 30, and the swing member 60 swings around a preset axis as the swing center under the driving of the third driving mechanism 40. The swing member 60 can be a swing arm, one end of the swing arm is connected with the rotor of the third motor. The other end is connected with the stator part of the second motor through an intermediate connecting member 80 such as a bolt. Specifically, one end of the bolt is fixedly connected with the swing arm away from the third motor, and the other end is fixed with the stator of the second motor. The length of the swing arm and the positional relationship with the second motor and the third motor satisfy that when the third motor drives the swing arm to rotate, the second motor is driven by the swing arm to rotate along a predetermined circular arc, and the center of the circular arc coincides with the rotation axis of the rotor of the third motor. In this way, when the rotor of the third motor rotates, the swing arm swings around the axis of the rotor of the third motor, thereby driving the second motor to move along a circular arc with the axis of the rotor of the third motor as the center.

[0142] The movement range of the first motor is 360°, and the first driving mechanism 20 is used to drive the rotating disc to rotate to simulate all rotation angles of the paper plane relative to the smart pen 100 in the range of 360° when a person holds the smart pen 100 to write. The movement range of the second motor is ±180° reciprocating rotation or rotation of any angle. When the smart pen 100 is connected with the controller in a wired manner, the ±180° reciprocating rotation is adopted, and when the smart pen 100 is connected with the controller in a wireless manner, the angle of rotation can not be limited. The second driving mechanism 30 is used to drive the rotation of the smart pen 100 to simulate all rotation angles of the smart pen 100 relative to the hand. The movement mode of the third motor is ±K° reciprocating movement, wherein ±K° is greater than the normal working inclination angle of the smart pen 100, and the angle of the smart pen 100 perpendicular to the rotating disc is defined as 0°. The third driving mechanism 40 is used to simulate all angles of the smart pen 100 relative to the paper when a person holds the smart pen 100 to write.

[0143] In addition, this embodiment includes a controller electrically connected to the first drive mechanism 20, the second drive mechanism 30, and the third drive mechanism 40. Based on test or calibration requirements, the controller controls the first drive mechanism 20, the second drive mechanism 30, and the third drive mechanism 40 to rotate at predetermined angles and directions, precisely controlling the spatial posture of the smart pen 100. This accurately simulates the posture of the smart pen 100 relative to the hand and writing paper during testing and calibration of the smart pen 100. During calibration and testing of the smart pen 100, a fixing structure provided on the turntable can be used to secure the pen tip to a predetermined position on the turntable. The fixing structure on the rotating member 10 can be a concave recessed point on the surface of the rotating member 10.

[0144] In another embodiment, Figure 15 As shown, the device also includes a guide rail 70 with a circular arc-shaped trajectory. The guide rail 70 is used to constrain the second drive mechanism 30 to move along a circular arc centered on a preset axis. The guide rail 70 is provided with an arc-shaped guide groove 71. One end of the intermediate connector 80 is connected to the second drive mechanism 30, while the other end, after passing through the arc-shaped guide groove 71, is connected to the end of the swing member 60 away from the third drive mechanism 40. To improve the reliability and accuracy of the circular motion of the smart pen 100 during calibration and testing, the present invention provides the guide rail 70. This allows the second drive mechanism 30 to accurately move along a circular arc centered on the preset axis under the constraint of the guide rail 70, thereby enhancing the accuracy of calibration and testing of the smart pen 100. In this embodiment, the guide rail 70 can be a flat plate mounted on the support member 50, with a circular arc-shaped guide groove 71 defined at one end. The center of the arcuate guide groove 71 coincides with the position of the third motor's output shaft. A central section of the intermediate connector 80, which connects the second motor and the swing arm, is located within the arcuate guide groove 71. When the third motor drives the swing arm to swing, the guide groove 71 constrains the intermediate connector 80, allowing it to accurately follow a desired arc trajectory, ultimately driving the smart pen 100 under test or calibration to perform precise arc motion. The intermediate connector 80 can be a component with a cylindrical outer surface, such as a bolt. The width of the guide groove 71 matches the diameter of the cylinder, and the axis of the cylinder is parallel to the axis of the third motor's output shaft.

[0145] In another embodiment, Figure 15As shown, in this embodiment, the output end of the second drive mechanism 30 is further provided with a clamp 31 for clamping the smart pen 100. The device of this embodiment can be provided with a clip connected to the output shaft of the second motor. This additional clip can be used to clamp the tail of the smart pen 100, allowing the second drive mechanism 30 to stably drive the smart pen 100. Furthermore, the use of the clamp 31, which can both release and tighten the smart pen 100, facilitates loading and unloading of the smart pen 100, significantly improving the efficiency of testing and calibration of the smart pen 100.

[0146] In this embodiment, the axis of the output shaft of the third motor and the axis of the output shaft of the first motor can be installed in a mutually perpendicular manner. To this end, the support member 50 in this embodiment includes a first fixed surface and a second fixed surface that are perpendicular to each other, the first drive mechanism 20 is connected to the first fixed surface, and the third drive mechanism 40 is connected to the second fixed surface. The device of this embodiment can set the support member 50 to be "L"-shaped, so that the support member 50 has two mutually perpendicular surfaces, namely the first fixed surface and the second fixed surface, and then the first motor and the third motor are respectively installed on the first fixed surface and the second fixed surface, so that the axis of the output shaft of the third motor and the axis of the output shaft of the first motor are perpendicular to each other.

[0147] As another implementation method, the support member 50 can also adopt a combination of a base 52 and an "L"-shaped bracket 51, one surface of the "L"-shaped bracket 51 is connected to the base 52, and the other surface is perpendicular to the upper surface of the base 52, and the third drive mechanism 40 is installed on the surface of the "L"-shaped bracket 51 that is perpendicular to the upper surface of the base 52. The base 52 can adopt a flat-plate-shaped platform, which has a horizontal surface, and the stator part of the first motor is installed on the horizontal surface. The output shaft of the first motor is connected to the rotating member 10 vertically upward. At the same time, a bracket of an "L"-shaped bracket 51 is installed on the horizontal surface of the base 52, one surface of the bracket is horizontally placed and fixed on the horizontal surface of the base 52, and the other surface is vertically set, and the stator part of the third motor is installed on the vertically set surface.

[0148] It should be noted that Figure 15 Some structures in the embodiment can be omitted or replaced by other structures during implementation to achieve the same technical effect.

[0149] based on Figure 15 The embodiment of the present invention further provides a method for adjusting the spatial posture of a smart pen, including the following steps:

[0150] Obtaining a first angle value, a second angle value, and a third angle value;

[0151] The first angle value, the second angle value and the third angle value are respectively sent to the first driving mechanism, the second driving mechanism and the third driving mechanism, wherein the first angle value is used for controlling the parking position of the first driving mechanism, the second angle value is used for controlling the parking position of the second driving mechanism, and the third angle value is used for controlling the parking position of the third driving mechanism.

[0152] Therefore, the first angle value, the second angle value and the third angle value are used to realize the movement control of the first driving mechanism, the second driving mechanism and the third driving mechanism, and further realize the control of the first included angle, the second included angle and the third included angle.

[0153] Further, the first angle value, the second angle value and the third angle value can be obtained from a pre-set angle table.

[0154] In an embodiment of the present application, the controller can be realized by a host computer, and the host computer and the intelligent pen correction device form an intelligent pen correction system.

[0155] The first driving mechanism, the second driving mechanism and the third driving mechanism receive the angle value sent by the host computer to control the spatial posture of the intelligent pen. The angle value includes the first angle value, the second angle value and the third angle value.

[0156] The host computer and the first driving mechanism, the second driving mechanism and the third driving mechanism can be wired connection or wireless communication.

[0157] The technical solution of the embodiment can accurately control the spatial posture of the intelligent pen, and is convenient for performing the technical solution described in combination Figure 4 with the technical solution described in combination Figure 12 to realize the automatic solving of the pen tip image coordinates or the automatic determination of the effective attitude working range of the intelligent pen.

[0158] It should be further noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps are executed simultaneously.

[0159] The above merely describes specific implementation of the present application, and those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, module and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A method for locating a tip of a smart pen, wherein the smart pen includes an image sensor, characterized in that: The method comprises the following steps: Determining first perspective transformation matrices corresponding to N frames of first images acquired by the image sensor, respectively, to determine N first perspective transformation matrices, where N is a positive integer greater than or equal to 2, the spatial posture of the smart pen in each frame of the first image is different, and the tip of the smart pen is located at the same preset point on the writing medium; Constructing a pen tip image coordinate equation group according to the N first perspective transformation matrices; Solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group; Obtain N frames of second images from the image sensor and determine the corresponding second perspective transformation matrices respectively; The pen tip location is determined based on the pen tip image coordinates and the second perspective transformation matrix.

2. The method according to claim 1, characterized in that The writing medium includes first code points arranged in an array, the first image includes second code points corresponding to the first code points, and the step of determining the first perspective transformation matrix corresponding to each frame of the first image includes: Constructing a pairing relationship between the first code point and the second code point to determine at least four sets of pairing points; Determining a first perspective transformation matrix corresponding to the first image according to the at least four groups of paired points; The paired points include first code points arranged along a first direction and first code points arranged along a second direction in the writing medium, and the first direction and the second direction are orthogonal; The first code points arranged along the first direction and the first code points arranged along the second direction are not subjected to offset processing.

3. The method according to claim 1, characterized in that A three-dimensional spatial coordinate system is defined, in which the coordinate axis of one dimension of the three-dimensional spatial coordinate system is perpendicular to the writing medium. The angle between the smart pen and the first dimension is denoted as a first angle, the angle between the smart pen and the second dimension is denoted as a second angle, and the angle between the smart pen and the third dimension is denoted as a third angle. The spatial posture includes the first angle, the second angle, and the third angle.

4. The method according to claim 3, wherein: The N identified first images include at least two adjacent first images having at least one difference among the first angle, the second angle, and the third angle when acquired; and the spatial posture of the smart pen is changed by changing at least one of the first angle, the second angle, and the third angle.

5. The method according to claim 1, wherein The second image acquired by the image sensor is corrected using a perspective transformation equation and its corresponding second perspective transformation matrix is ​​determined. The second image is a plurality of frames of second images acquired continuously in a time sequence during a continuous writing process.

6. The method according to claim 1, characterized in that Solving the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group includes: Calculating the error of the pen tip image coordinates relative to the preset coordinates; The magnitude relationship between the error and a threshold is determined to determine the calibration status of the smart pen.

7. The method according to claim 6, characterized in that If the error is smaller than a threshold, the calibration status of the smart pen is determined to be passed; if the error is larger than the threshold, the calibration status of the smart pen is determined to be failed. For the smart pen with a failed calibration status, the preceding steps of this step are repeated, or the smart pen with a failed calibration status is determined to be a defective product.

8. The method according to claim 4, characterized in that After determining the pen tip paper coordinates based on the pen tip image coordinates and the second perspective transformation matrix, it also includes: calculating the decoding success rate of the pen tip paper coordinates under different spatial postures, and based on the decoding success rate, traversing the value combinations of the first angle, the second angle and the third angle in an orthogonal experimental design manner, thereby determining that the corresponding spatial posture range is the effective posture working range.

9. A device for determining the paper coordinates of a smart pen tip, the smart pen comprising an image sensor, characterized in that: The device comprises: a first perspective transformation matrix acquisition module, configured to determine first perspective transformation matrices corresponding to N frames of first images acquired by the image sensor, thereby determining N first perspective transformation matrices, where N is a positive integer greater than or equal to 2, and the spatial posture of the smart pen in each frame of the first image is different, and the tip of the smart pen is located at the same preset point on the writing medium; An image coordinate equation group construction module, which constructs a pen tip image coordinate equation group according to N first perspective transformation matrices; a pen tip image coordinate solving module, which solves the pen tip image coordinates according to the coordinates of the preset point and the pen tip image coordinate equation group; A second perspective transformation matrix acquisition module is configured to acquire N frames of second images from the image sensor and determine the corresponding second perspective transformation matrices thereof; The pen tip paper coordinate determination module determines the pen tip location based on the pen tip image coordinates and the second perspective transformation matrix.

10. A smart pen, characterized in that: include: At least one processor, at least one memory, and computer program instructions stored in the memory, which implement the method according to any one of claims 1 to 8 when the computer program instructions are executed by the processor.

Images