Handwriting pen optical image positioning device and method based on retroreflection

Abstract

The invention discloses a handwriting pen optical image positioning device and method based on retroreflection. The method comprises the following steps: proposing a handwriting pen optical image positioning neural network model based on retroreflection; designing a handwriting pen structure meeting the positioning requirement; setting a retro-reflection material calibration point according to the actual screen size, and calculating to obtain an inverse perspective transformation matrix for calculating transformation coordinates; preprocessing such as gray level conversion and Gaussian filtering is carried out on the collected image; performing subsequent operations such as threshold segmentation, convex hull algorithm shaping and centroid extraction on the preprocessed image to obtain an image coordinate of the pen point of the handwriting pen; inputting the nib image coordinates and the conversion coordinates into a positioning neural network, and outputting the accurate position of the handwriting pen through the positioning neural network; the passive positioning function of the writing pen can be achieved based on the retroreflection technology, implantation of an electronic circuit is omitted, and multi-point positioning is achieved only through optics and machinery.

Description

Technical Field

[0001] The present invention relates to the technical field of image positioning, and in particular to a stylus optical image positioning device and method based on retroreflection. Background Art

[0002] Stylus positioning is a technology that converts the actual position of a stylus into a digital signal so that it can be written, drawn or interacted on an electronic screen. With the development of digital technology, stylus positioning technology has been widely used in education, design, conference recording and other fields. Among the existing stylus positioning technologies, electromagnetic induction, ultrasonic positioning, optical positioning and other methods are mainly used. Among them, electromagnetic induction technology realizes the positioning of the stylus through the interaction between the electromagnetic induction element built into the stylus and the electromagnetic induction board on the screen; ultrasonic positioning technology determines the position of the stylus by emitting ultrasonic waves and receiving reflected signals; optical positioning technology uses a camera to capture the movement trajectory of the stylus on the screen and realizes positioning through image processing technology. Both optical positioning and ultrasonic positioning technologies may be affected by environmental factors such as light and sound, resulting in reduced positioning accuracy or failure, slow positioning time, and thus affecting the user's writing experience. High-resolution cameras can provide high-precision optical positioning, but the processing time is long; in contrast, infrared positioning is faster, but the accuracy may be relatively low.

[0003] In view of the defects of the prior art, this patent proposes a method and device for optical image positioning of a stylus based on retroreflection, including: proposing an optical image positioning model of a stylus based on retroreflection; designing a stylus structure that meets the positioning requirements; setting retroreflective material calibration points according to the actual screen size, and calculating the inverse perspective transformation matrix; constructing a receiving power model; performing grayscale conversion, Gaussian filtering and other preprocessing on the collected image; performing subsequent operations such as threshold segmentation, convex hull algorithm shaping, centroid extraction and other subsequent operations on the preprocessed image to obtain the image coordinates of the stylus positioning area; performing inverse perspective transformation on the image coordinates to obtain the conversion coordinates, and then fitting the coordinates using a neural network to obtain the precise position of the stylus on the screen. The present invention designs a regular polygonal multi-segment prism structure of the stylus tip according to the retroreflective properties of the retroreflective material. The stylus adopts a passive design and omits the implantation of electronic circuits; proposes an optical positioning system layout and algorithm that uses a camera to collect the highlighted retroreflective image of the pen tip, which reduces the equipment complexity and positioning time of the positioning system, and achieves multi-point positioning only by optics and mechanics. Summary of the invention

[0004] In response to the problems that optical positioning technology may be affected by light, resulting in positioning errors, a large number of devices, and long calculation time, this patent proposes a handwriting pen optical image positioning method based on retroreflection, which avoids the influence of ambient light as much as possible, achieving the beneficial effects of streamlined equipment, accurate positioning, and fast calculation.

[0005] The technical solution of the present invention is as follows:

[0006] A retroreflection-based stylus optical image positioning device comprises a stylus and a screen; the stylus adopts a passive form, and the tip of the stylus is covered with retroreflective material, and the retroreflective material is used to reflect light to form a highlight spot on an image for positioning; the tip of the stylus adopts a multi-segment prism structure to ensure that the pen tip can perform highlight retroreflection at any posture angle (highlight retroreflection means that the average reflectivity of the retroreflective pen tip observed in all directions is greater than or equal to 70%, which is used to ensure that the reflected light can form a highlight spot of the pen tip on the image); a camera is arranged obliquely above the screen to capture the position information of the pen tip; the pen tip position information is input into a positioning neural network model, and the precise position of the stylus is output through the positioning neural network model.

[0007] Furthermore, the cross section of the pen tip is a regular polygon or a circle to ensure that the pen tip has high-brightness retroreflection in all directions in the horizontal direction.

[0008] Furthermore, the pen tip is multi-segmented in the longitudinal direction to ensure that the pen tip has high-brightness retroreflection in all directions in the longitudinal direction.

[0009] Furthermore, the nib is preferably a regular decagonal, four-segment nib.

[0010] Furthermore, a push-type telescopic structure is provided above the pen tip, which will extend the pen tip for positioning when pressed, and retract the pen tip for non-positioning and protection of the pen tip when not pressed.

[0011] Furthermore, retroreflective materials are placed at four boundary points of the screen to locate the screen area and to perform inverse perspective transformation matrix calculation.

[0012] Furthermore, the camera adopts an infrared camera with an infrared LED, the camera points to the front of the screen, and the camera is set at a predetermined distance from the edge of the screen to ensure that the entire screen can be captured by the camera.

[0013] A positioning method for a stylus optical image positioning device based on retroreflection, comprising the following steps:

[0014] Step 1) Image input:

[0015] Use the camera to collect the screen image with the highlight spot of the pen tip;

[0016] Step 2) Preprocessing:

[0017] The camera is calibrated to obtain the intrinsic and extrinsic parameters of the camera; the collected image is corrected using the intrinsic and extrinsic parameters obtained by the camera calibration, and the grayscale image containing the three RGB color information after correction is converted into a grayscale image with only brightness information; the image processing methods such as Gaussian filtering, morphological corrosion, threshold segmentation, mask extraction, and convex hull algorithm shaping are used for preprocessing;

[0018] Step 3) Extract the centroid to get the image coordinates, and perform inverse perspective transformation to get the transformed coordinates:

[0019] The centroid of the highlighted spot after shaping is extracted to obtain the centroid image coordinates of the pen tip; the centroid image coordinates of the highlighted spot at the four vertices of the screen after binarization are obtained in the image, and then the actual size of the screen to be mapped is specified, and the inverse perspective transformation matrix is ​​obtained using the four-point perspective transformation method; the centroid image coordinates of the pen tip are subjected to inverse perspective transformation using the perspective transformation matrix obtained by screen calibration, and the tilted screen image is converted into a front view of the screen to obtain the transformation coordinates;

[0020] Step 4) Train the localization neural network:

[0021] Use the coordinates of the pen tip image and the transformed coordinates after inverse perspective transformation as input data, the actual coordinates of the screen as output data, and use both as training data. By using a large amount of data from different positions for training, the optimal positioning neural network model is obtained;

[0022] Step 5) Locate the neural network output fitting coordinates, and map the coordinates to get the pixel coordinates:

[0023] The pen tip image coordinates and conversion coordinates calculated in real time are input into the trained optimal positioning neural network model for fitting, and the precise fitting coordinates of the stylus are output; the fitting coordinates are mapped according to the displayed pixel size to obtain the displayed pixel coordinates.

[0024] Furthermore, the positioning neural network model includes an input layer, three hidden layers and an output layer; the features during model training are the pen tip image coordinates and the conversion coordinates, and the label is the pen tip real coordinates; the Euclidean distance between the output coordinates and the real coordinates is used as the evaluation function, and the model with the smallest Euclidean distance is saved as the optimal model.

[0025] The design principle of the present invention is as follows: the infrared LED lights around the infrared camera are used to actively illuminate the positioning screen area. The light is retroreflected by the retroreflective material covered on the stylus tip. The screen image with the highlighted light spot of the stylus tip is collected by the infrared camera. Image processing and neural network fitting are performed on it to quickly and accurately obtain the actual position of the stylus.

[0026] Design idea of ​​the present invention:

[0027] 1. Propose a stylus optical image positioning model based on retroreflection

[0028] A camera with an LED light source is placed diagonally above the positioning screen to collect images of the screen area including the retroreflective pen tip. The retroreflective pen tip will reflect the highlight on the image to form a highlight spot area. The pen tip image coordinates and conversion coordinates after image processing are input into the positioning neural network, and the precise position of the stylus is output through the positioning neural network.

[0029] 2. Design a stylus structure that meets positioning requirements

[0030] The stylus adopts a passive design. The retroreflective pen tip is covered with retroreflective material and adopts a multi-segment prism structure. The cross-section shape is preferably a regular polygon to ensure that the pen tip can be highly retroreflective at any posture angle. In addition, the pen tip structure has a telescopic function to realize the switch of the pen tip retroreflection.

[0031] 3. Set the retroreflective material calibration points according to the actual screen size, and calculate the inverse perspective transformation matrix for calculating the conversion coordinates.

[0032] 4. Perform preprocessing such as image correction, grayscale conversion, Gaussian filtering, and morphological corrosion on the collected images.

[0033] 5. Perform subsequent operations such as threshold segmentation, mask extraction, highlight shaping, and centroid extraction on the preprocessed image to obtain the image coordinates of the stylus.

[0034] 6. Convert image coordinates to real-world coordinates, that is, perform inverse perspective transformation to obtain the converted coordinates of the stylus on the actual screen. Train the positioning neural network model, use the trained optimal model to perform positioning and output accurate fitting coordinates, and then convert the fitting coordinates to obtain the screen display coordinates.

[0035] Preferably, the tip of the stylus pen used is covered with retroreflective material, and the tip adopts a four-section prism design, and its cross-sectional shape is a regular decagon, and a press-type telescopic structure is arranged on the top.

[0036] Preferably, optical image processing such as threshold segmentation and inverse perspective transformation and neural network fitting methods are used to quickly and accurately obtain the precise position of the stylus.

[0037] The present invention is particularly suitable for scenes such as interactive whiteboards, interactive sandboxes and virtual meetings, and can achieve precise stylus positioning. In the field of education, this technology can be applied to real-time writing and drawing interactions in interactive whiteboards and smart classrooms to achieve real-time display and recording of handwritten content. In the design industry, this technology helps to create and review digital sketches and improve the work efficiency of designers. In virtual conference systems, the present invention can be used as a precise input and control device to enhance the remote collaboration experience. This technology can also be extended to smart office equipment, augmented reality (AR) systems and virtual reality (VR) interactions, providing precise positioning solutions in a variety of scenarios.

[0038] In summary, the present invention has the following beneficial effects:

[0039] 1) Retroreflective material is applied to the stylus pen, and a multi-section prism structure of the pen tip is designed to ensure that the stylus pen tip can be highly retroreflective in all directions in the horizontal and vertical directions for positioning. A push-type telescopic structure is used to realize the retroreflective switch and protect the pen tip. Based on the retroreflective technology, the passive positioning function of the writing pen can be realized, the implantation of electronic circuits is omitted, and multi-point positioning is achieved only by optics and mechanics.

[0040] 2) The camera captures the screen image with the highlight spot of the pen tip, inputs the pen tip image coordinates and conversion coordinates into the positioning neural network, and outputs the precise position of the stylus through the positioning neural network, achieving multi-point positioning only by optics and mechanics. The average positioning error of the positioning system is 2.24mm. If hundreds of megahertz edge computing processing is used, the average positioning time can be as low as a few milliseconds (3ms). BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 1 is a diagram of a retroreflective stylus optical image positioning model in an implementation method; 1 is a passive stylus, 2-1 is a positioning screen area, 2-2 is a retroreflective material calibration point, 3-1 is an infrared camera module, 3-2 is an infrared fill light module, and 3-3 is edge computing processing;

[0042] Figure 2 is a flow chart of a method for positioning a stylus optical image based on retroreflection in an implementation manner;

[0043] Figure 3 A schematic diagram of a module of a passive stylus in an implementation scheme, particularly identifying a pen head structure 1-1 with a four-section prism design and a regular decagonal cross-sectional shape, a press-type telescopic structure 1-2, and a pen body 1-3;

[0044] Figure 4 A polar coordinate distribution diagram of the longitudinal incident angle-reflectivity of the retroreflective pen tip in the embodiment;

[0045] Figure 5 A polar coordinate distribution diagram of the lateral incident angle-reflectivity of the retroreflective pen tip in the embodiment;

[0046] Figure 6 Positioning a neural network model diagram in an implementation manner;

[0047] Figure 7 A distribution diagram of the real coordinates and the fitted coordinates of the positioning points during positioning error analysis in the implementation manner;

[0048] Figure 8 It is a cumulative distribution function diagram of the average positioning error during the positioning error analysis in the implementation manner. DETAILED DESCRIPTION

[0049] The method of the present invention will be further described in detail below by means of simple examples in conjunction with the accompanying drawings.

[0050] A stylus optical image positioning device based on retroreflection:

[0051] Retroreflective pen tip structure design:

[0052] Retroreflection, also known as reflection, retroreflection, retroreflection, retroreflection, directional reflection or reverse reflection, is a kind of reflection in which the reflected light returns from the opposite direction of the incident light. This property can be maintained when the direction of the incident light changes within a large range. However, when the incident light is incident at a large angle, very little light is reflected in the opposite direction.

[0053] Figure 3 Schematic diagram of a passive stylus module according to an embodiment of the present invention. The pen tip is covered with retroreflective material, and the rest of the surface is black. A push-type telescopic structure is arranged above the stylus tip, and the push area is connected to the pen tip through a spring (it can also be other push-type telescopic structures, which is just one embodiment of the present invention). When pressed, the pen tip will be extended for positioning, and when not pressed, the pen tip will be retracted without positioning and protect the pen tip.

[0054] The pen tip uses a four-section prism structure with a regular decagonal cross section. There is a certain angle between the prisms, and the surfaces of the prisms are covered with retroreflective materials. The angle setting is based on the maximum light incident angle of the retroreflective material that can be retroreflected brightly obtained from the experiment. The angle measured in this embodiment is 40°, so the preferred setting angle is 144° to ensure that the pen tip can still be retroreflected brightly through the retroreflective material on one section of the structure at any longitudinal tilt angle. The regular decagonal cross section ensures that the pen tip can still be retroreflected brightly through the retroreflective material on one section of the structure at any lateral rotation angle.

[0055] like Figure 4 and Figure 5As shown, according to the properties of the retroreflective material, the longitudinal and transverse incident angle-reflectivity polar coordinate distribution diagrams are drawn for pen tip design. The preferred regular decagonal four-segment pen tip has an average reflectivity of about 70% in the longitudinal 180° full range and an average reflectivity of about 79% in the transverse 360° full range.

[0056] A stylus optical image positioning method based on retroreflection:

[0057] S100: Image Input

[0058] Figure 2 It is a side model diagram of an embodiment of the present invention. An infrared camera with an infrared LED is installed obliquely above the periphery of a screen positioning area with a size of 810mm×550mm. The camera points to the front of the screen and is a certain distance away from the edge of the screen to ensure that the entire screen can be captured by the camera. Retroreflective materials of appropriate size are placed at the four boundary points of the positioning area to locate the screen area and perform inverse perspective transformation matrix calculation. The number of pixels of the infrared camera used is 1 million, and the image size is 1280×720. Depending on the size of the screen, a camera with millions of pixels or higher resolution is generally used to ensure image quality. The infrared LED provides active lighting, and the light is retroreflected by the retroreflective material on the tip of the passive stylus pen, enters the camera and is captured, and a high-brightness light spot area that is clearly separated from the dark background will appear on the image.

[0059] The solar spectrum is an absorption spectrum of different wavelengths. It is divided into two parts: visible light and invisible light. The intensity of visible light in sunlight decreases as the wavelength increases, and the near-infrared band of invisible light has a large trough at 940nm, that is, a local minimum. Collecting images in the 940nm infrared band can minimize the interference of sunlight or ambient light, so this embodiment uses a 940nm infrared narrowband camera.

[0060] S110: grayscale conversion, Gaussian filtering, morphological corrosion and other preprocessing

[0061] The camera is calibrated to obtain the intrinsic and extrinsic parameters of the camera. The captured image is corrected using the intrinsic and extrinsic parameters obtained by the camera calibration, and then the grayscale image containing the three RGB color information after correction is converted into a grayscale image with only brightness information. If a distortion-free camera is used to capture images, image correction has little effect on the positioning result. This embodiment does not perform correction to reduce calculation time.

[0062] Gaussian filtering of grayscale images can smooth the image and remove noise from the image, making the image clearer and smoother, and relatively well retaining the edge information of the image. The Gaussian filter is in the form of a 3×3 matrix with a standard deviation of 2, and its representation is:

[0063]

[0064] S120: threshold segmentation, mask extraction, convex hull algorithm shaping

[0065] The obtained image is segmented by threshold, and the threshold is selected according to the actual minimum grayscale value of the highlight spot. In general, the average grayscale value of the spot area of ​​the retroreflective material calibration point farthest from the camera can be selected as the minimum grayscale value of the highlight spot. In this embodiment, the grayscale value 220 is selected as the segmentation threshold. The background noise is removed by mask extraction, and the highlight spot area of ​​the pen tip is shaped using the convex hull shaping algorithm to describe the contour of the spot.

[0066] S130: Extract the centroid to obtain the image coordinates, and perform inverse perspective transformation to obtain the conversion coordinates

[0067] The centroid of the highlighted spot after shaping is extracted to obtain the centroid image coordinates of the pen tip. The centroid extraction formula is:

[0068]

[0069] The display screens or projection screens seen on the market are all rectangular in shape, so retro-reflective materials of appropriate sizes are placed at the four vertices of the rectangular screen. The centroid image coordinates of the high-brightness spots at the four vertices of the screen are obtained in the image after binarization, and then the actual size of the screen to be mapped is specified, and the inverse perspective transformation matrix is ​​obtained using the four-point perspective transformation method. The four-point perspective transformation is as follows:

[0070]

[0071] In the above formula (x s ,y s ) is the image coordinate of the centroid of the boundary point, (x d ,y d ) is the actual screen size coordinate. The inverse perspective transformation is a homogeneous coordinate transformation that can normalize the matrix parameters. The ninth element M in the matrix 33 can be fixed to 1. Therefore, there are 8 independent unknowns in the perspective transformation matrix. In order to determine these 8 unknowns, we need at least 4 pairs of source points and target points. The source point is the centroid image coordinates of the four vertices, and the target point is the real coordinate corresponding to the actual size of the screen. The screen size selected in this implementation is 810mm×550mm, and the four target points are selected as [0,0], [810,0], [0,550], [810,550]. Each pair of source and target points can provide two independent equations, so 4 pairs of points can provide a total of 8 equations to meet the needs of solving these 8 unknowns.

[0072] The coordinates of the pen tip centroid image are inversely transformed using the perspective transformation matrix obtained by screen calibration, from the tilted screen image to the front view of the screen, to obtain the transformed coordinates. The inverse perspective transformation formula is:

[0073]

[0074] S140: Positioning neural network output fitting coordinates, coordinate mapping to obtain pixel coordinates

[0075] Positioning neural network model such as Figure 6 As shown, (the positioning neural network of this embodiment uses a fully connected neural network based on the BP algorithm). The pen tip image coordinates and the transformed coordinates after the inverse perspective transformation are used as input data, with a total of 4 elements, and the actual coordinates of the screen are used as output data, with a total of 2 elements. Both are used as training data. By using a large amount of data from different positions for training, the model has good generalization ability. The model consists of five layers of neural units, including an input layer, three hidden layers, and an output layer. The activation function uses tanh, and the expression of tanh is:

[0076]

[0077] During the model training process, the Adam optimizer was used, the batch size was set to 16, the initial learning rate was set to 0.001, and the learning rate gradient descent was adopted. The learning rate was reduced to 10% of the original value every 500 training times. After about 1100 training times, the training set loss and test set loss were 0.00016384 and 0.00016908, respectively. In the prediction stage, the Euclidean distance between the output data and the real data was used as the evaluation function. The smaller the Euclidean distance, the better the model training. The model with the smallest Euclidean distance was saved as the optimal model.

[0078] The pen tip image coordinates and conversion coordinates calculated in real time are input into the trained optimal positioning neural network model for fitting, and the precise fitting coordinates (x f ,y f ).

[0079] If the pixel size for display is a×b, the fitting coordinates need to be converted to pixel coordinates. Assuming the actual screen size is c×d, the conversion formula is:

[0080]

[0081]

[0082] For x in the above formula p and p Round off to get the final display pixel coordinates (x p ,yp ).

[0083] Error analysis: The stylus is positioned on a screen of 810mm×550mm in size. Starting from 30mm, a positioning point is set every 60mm, with a total of 14×9 positioning points. The error between the actual coordinates of the reference point obtained by the above positioning method and the fitted screen coordinates is as follows: Figure 7 The positioning error is judged by the Euclidean distance between the real coordinates and the fitted screen coordinates. The average positioning error of 14×9 positioning points is 2.24mm. Figure 8 It can be seen from the cumulative distribution function of the average positioning error that the positioning error of more than 75% of the positioning points can be within 2mm.

[0084] If a 150MHz edge computing processor is used in this embodiment, the average positioning time is several milliseconds (3ms). The present invention satisfies the accuracy and real-time performance of the stylus optical positioning well.

[0085] This patent solution is not only applicable to infrared optical systems, but can also be extended to other optical bands to achieve wider device adaptation. At the same time, the stylus technology is applicable to a variety of interactive devices, including but not limited to interactive electronic whiteboards, interactive sandboxes, virtual conferences, augmented reality (AR) systems and virtual reality (VR) interactions, etc., which will open up newer and broader application scenarios for interactive technology.

Claims

1. A retroreflection-based stylus optical image positioning device, comprising a stylus, a screen, and a processor; characterized in that: The stylus pen is passive, and the tip of the stylus pen is covered with retroreflective material, which is used to reflect light to form a highlight spot on the image for positioning; the tip of the stylus pen adopts a multi-segment prism structure to ensure that the pen tip can perform highlight retroreflection at any posture angle, wherein highlight retroreflection refers to an average reflectivity of the retroreflective pen tip observed in all directions greater than or equal to 70%, which is used to ensure that the reflected light can form a highlight spot of the pen tip on the image; a camera is arranged obliquely above the screen to capture image data of the pen tip; the processor processes the image data to obtain position information of the pen tip, inputs the position information of the pen tip into a positioning neural network model, and outputs the precise position of the stylus pen through the positioning neural network model.

2. The retroreflection-based stylus optical image positioning device according to claim 1, characterized in that: The cross section of the pen tip is a regular polygon or a circle to ensure that the pen tip has high-brightness retroreflection in all directions in the horizontal direction.

3. The retroreflection-based stylus optical image positioning device according to claim 2, characterized in that: The pen head adopts a multi-segment type in the longitudinal direction to ensure that the pen tip has high-brightness retroreflection in all directions in the longitudinal direction.

4. The retroreflection-based stylus optical image positioning device according to claim 3, characterized in that: The nib is preferably a regular decagonal, four-section nib.

5. The retroreflection-based stylus optical image positioning device according to claim 1, characterized in that: A push-type telescopic structure is arranged above the pen tip, which will extend the pen tip for positioning when pressed, and retract the pen tip without positioning and protect the pen tip when not pressed.

6. The retroreflection-based stylus optical image positioning device according to claim 1, characterized in that: Retroreflective materials are placed at the four boundary points of the screen to locate the screen area and to perform inverse perspective transformation matrix calculation.

7. The retroreflection-based stylus optical image positioning device according to claim 1, characterized in that: The camera uses an infrared camera with an infrared LED. The camera points to the front of the screen. The camera is set at a predetermined distance from the edge of the screen to ensure that the entire screen can be captured by the camera.

8. A positioning method for a stylus optical image positioning device based on retroreflection, characterized in that: The steps include: Step 1) Image input: Use the camera to collect the screen image with the highlight spot of the pen tip; Step 2) Preprocessing: The camera is calibrated to obtain the intrinsic and extrinsic parameters of the camera; the collected image is corrected using the intrinsic and extrinsic parameters obtained by the camera calibration, and the grayscale image containing the three RGB color information after correction is converted into a grayscale image with only brightness information; the image processing methods such as Gaussian filtering, morphological corrosion, threshold segmentation, mask extraction, and convex hull algorithm shaping are used for preprocessing; Step 3) Extract the centroid to get the image coordinates, and perform inverse perspective transformation to get the transformed coordinates: The centroid of the highlighted spot after shaping is extracted to obtain the centroid image coordinates of the pen tip; the centroid image coordinates of the highlighted spot at the four vertices of the screen after binarization are obtained in the image, and then the actual size of the screen to be mapped is specified, and the inverse perspective transformation matrix is ​​obtained using the four-point perspective transformation method; the centroid image coordinates of the pen tip are subjected to inverse perspective transformation using the perspective transformation matrix obtained by screen calibration, and the tilted screen image is converted into a front view of the screen to obtain the transformation coordinates; Step 4) Train the localization neural network: Use the coordinates of the pen tip image and the transformed coordinates after inverse perspective transformation as input data, the actual coordinates of the screen as output data, and use both as training data. By using a large amount of data from different positions for training, the optimal positioning neural network model is obtained; Step 5) Locate the neural network output fitting coordinates, and map the coordinates to get the pixel coordinates: The pen tip image coordinates and conversion coordinates calculated in real time are input into the trained optimal positioning neural network model for fitting, and the precise fitting coordinates of the stylus are output; the fitting coordinates are mapped according to the displayed pixel size to obtain the displayed pixel coordinates.

9. The positioning method of the retroreflection-based stylus optical image positioning device according to claim 8, characterized in that: The positioning neural network model includes an input layer, three hidden layers and an output layer; the features during model training are the pen tip image coordinates and the conversion coordinates, and the label is the pen tip real coordinates; the Euclidean distance between the output coordinates and the real coordinates is used as the evaluation function, and the model with the smallest Euclidean distance is saved as the optimal model.

Images