Handwriting projection method, system and equipment based on virtual canvas

Abstract

The invention discloses a handwriting projection method, system and equipment based on a virtual canvas. The method comprises the following steps: acquiring three-dimensional coordinate data and attitude data of an electronic pen according to a preset canvas triggering condition; constructing a two-dimensional virtual canvas according to the attitude data, the three-dimensional coordinate data and a three-dimensional coordinate system where the electronic pen is located; acquiring real-time space coordinates of the electronic pen according to a preset anchor point, and mapping the real-time space coordinates to the two-dimensional virtual canvas to obtain virtual handwriting coordinates; according to a plurality of continuous virtual handwriting coordinates, drawing virtual handwriting in the two-dimensional virtual canvas; and constructing a track change matrix of the two-dimensional virtual canvas and a display terminal, and projecting the virtual handwriting to the display terminal according to the track change matrix so as to display the handwriting of the electronic pen in the display terminal, thereby improving the degree of freedom of touch interaction.

Description

Technical Field

[0001] The present invention relates to the field of display technology, and in particular to a handwriting projection method, system and device based on a virtual canvas. Background Art

[0002] With the rapid development of touch interaction technology, electronic pens have become an important input tool in smart terminals, educational whiteboards, conference systems and other fields.

[0003] Existing touch interaction methods typically require users to write on the touch screen or in a designated area, limiting operation and interaction methods. For example, traditional electronic whiteboards and touch screens require users to write directly on the physical screen surface, and their interaction range is strictly limited to a two-dimensional plane. This design forces users to operate the device in close proximity, making it impossible to achieve natural interaction at a distance (such as in the back of a classroom) or contactless (such as writing in the air). This significantly limits the freedom of interaction, especially in large conferences or educational settings.

[0004] At the same time, in scenarios of fast writing or continuous drawing, existing technologies use sensors such as accelerometers and angular velocity meters carried by the electronic pen to calculate the trajectory of the electronic pen coordinates. This not only easily causes problems such as handwriting breakpoints and coordinate jumps, but also fails to recognize the user's real handwriting, resulting in a mismatch between the rendering effect and the input intention, thereby reducing the handwriting display effect. Summary of the Invention

[0005] To solve the above technical problems, the present invention discloses a handwriting projection method, system and device based on a virtual canvas, which are used to improve the freedom of touch interaction and the accuracy of handwriting recognition.

[0006] In order to achieve the above objectives, in a first aspect, the present invention discloses a handwriting projection method based on a virtual canvas, comprising:

[0007] Acquire the three-dimensional coordinate data and posture data of the electronic pen according to the preset canvas trigger conditions;

[0008] constructing a two-dimensional virtual canvas according to the posture data, the three-dimensional coordinate data and the three-dimensional coordinate system where the electronic pen is located;

[0009] Acquiring the real-time spatial coordinates of the electronic pen according to a preset anchor point, and mapping the real-time spatial coordinates to the two-dimensional virtual canvas to obtain virtual handwriting coordinates;

[0010] Drawing virtual handwriting on the two-dimensional virtual canvas according to a plurality of continuous virtual handwriting coordinates;

[0011] A trajectory change matrix between the two-dimensional virtual canvas and the display terminal is constructed, and the virtual handwriting is projected onto the display terminal according to the trajectory change matrix, so as to display the handwriting of the electronic pen on the display terminal.

[0012] The present invention discloses a handwriting projection method based on a virtual canvas, which overcomes the limitations of traditional touch interaction methods in terms of operating space and interaction dimensions. By constructing a virtual canvas, it enables natural long-distance or contactless interaction and increases the freedom of touch interaction. During handwriting recognition, anchor points are used to improve handwriting recognition accuracy, effectively avoiding problems such as handwriting breakpoints and coordinate jumps, and ensuring that the rendering effect closely matches the user's input intent. The virtual canvas is constructed using preset canvas trigger conditions and the electronic pen's own coordinate data, avoiding resource utilization by the electronic pen and improving handwriting recognition efficiency. Secondly, when using the virtual canvas to project virtual handwriting generated by the electronic pen onto a display terminal to display the handwriting trajectory, the three-dimensional handwriting coordinates of the electronic pen are first projected onto the virtual canvas. Then, the virtual handwriting on the virtual canvas is projected onto the display terminal using the trajectory change matrix between the virtual canvas and the display terminal, thereby improving the accuracy of handwriting projection.

[0013] As a preferred example, the step of obtaining the three-dimensional coordinate data and posture data of the electronic pen according to a preset canvas trigger condition includes:

[0014] Acquiring the pen tip pressure value of the electronic pen in real time according to a preset polling cycle;

[0015] When the pen tip pressure value changes, the three-dimensional coordinate data of the electronic pen and the tilt angle of the electronic pen are acquired.

[0016] In the above scheme, the coordinate collection is triggered by detecting the change of the pen tip pressure value through the polling cycle, ensuring that data is obtained only when the user is actually writing, reducing invalid data processing, and reducing system resource consumption. Combined with the tilt angle data, it can more accurately reflect the user's current writing posture, and then subsequently construct a virtual canvas corresponding to the writing posture to improve the accuracy of handwriting recognition.

[0017] As a preferred example, the method of obtaining the three-dimensional coordinate data and posture data of the electronic pen according to the preset canvas triggering condition further includes:

[0018] Acquiring sampling three-dimensional coordinate data corresponding to each sampling moment of the electronic pen according to a preset sampling time;

[0019] When the coordinate difference between any two adjacent sampled three-dimensional coordinate data is less than or equal to a preset threshold, the three-dimensional coordinate data of the electronic pen and the tilt angle of the electronic pen are acquired.

[0020] In this solution, the coordinate difference between adjacent sampling points is used to determine the electronic pen's movement state. This determines the triggering of the virtual canvas based on the user's current gesture, reducing the impact of the operating environment on the electronic pen's touch control and increasing the freedom of touch interaction. Furthermore, utilizing the coordinate difference between adjacent sampling points helps reduce unnecessary computing resource consumption and improves the efficiency of handwriting projection.

[0021] As a preferred example, the method of obtaining the three-dimensional coordinate data and posture data of the electronic pen according to the preset canvas triggering condition further includes:

[0022] Obtaining the acceleration value corresponding to the electronic pen at each sampling moment according to a preset sampling time;

[0023] When there are at least two acceleration values greater than or equal to a preset acceleration threshold, the three-dimensional coordinate data of the electronic pen and the tilt angle of the electronic pen are acquired.

[0024] In the above scheme, by monitoring the changes in the acceleration value of the electronic pen, its movement status can be assisted in judging, especially in fast-moving or writing scenarios. This can significantly improve the accuracy and real-time performance of handwriting capture, thereby triggering the generation of a virtual canvas in a timely manner and improving the accuracy of handwriting recognition.

[0025] As a preferred example, the constructing of a two-dimensional virtual canvas according to the posture data, the three-dimensional coordinate data, and the three-dimensional coordinate system where the electronic pen is located includes:

[0026] When a tip pressure value of the electronic pen is detected or a tilt angle of the electronic pen is detected to be greater than a preset angle threshold, constructing a first two-dimensional coordinate system according to the first coordinate axis and the second coordinate axis in the three-dimensional coordinate system;

[0027] When it is detected that the tilt angle of the electronic pen is less than or equal to the angle threshold, constructing a second two-dimensional coordinate system according to the first coordinate axis and the third coordinate axis in the three-dimensional coordinate system;

[0028] The three-dimensional coordinate data is used as the origin of the coordinate system to construct the two-dimensional virtual canvas according to a preset canvas size, a spatial scaling ratio, the first two-dimensional coordinate system or the second two-dimensional coordinate system.

[0029] In this solution, the two-dimensional coordinate system (first or second coordinate axis combination) is dynamically switched based on the electronic pen's tilt angle to accommodate different writing postures (such as vertical or sideways writing). This ensures that the projection plane always aligns with the user's actual writing direction, preventing handwriting deformation or offset due to posture changes, enhancing adaptability in multiple scenarios, and improving the naturalness and precision of interaction.

[0030] As a preferred example, the step of acquiring the real-time spatial coordinates of the electronic pen according to a preset anchor point and mapping the real-time spatial coordinates to the two-dimensional virtual canvas to obtain virtual handwriting coordinates includes:

[0031] Acquiring the real-time three-dimensional spatial coordinates of the electronic pen according to a preset anchor point;

[0032] Projecting the real-time three-dimensional space coordinates onto the plane where the two-dimensional virtual canvas is located according to the normal vector of the two-dimensional virtual canvas to obtain the two-dimensional projection coordinates of the projection point;

[0033] A vector difference between the two-dimensional projection coordinates and the origin of the coordinate system is obtained, and virtual handwriting coordinates of the real-time three-dimensional space coordinates in the two-dimensional virtual canvas are obtained according to the vector difference and the spatial scaling ratio.

[0034] In the above scheme, the virtual handwriting coordinates are calculated through normal vector projection and spatial scaling ratio, and the three-dimensional spatial coordinates are accurately mapped to the two-dimensional plane, solving the problem of handwriting proportion imbalance caused by projection distortion or improper scaling, ensuring that the handwriting on the display terminal is consistent with the actual writing trajectory, and improving the accuracy of handwriting projection.

[0035] As a preferred example, drawing virtual handwriting on the two-dimensional virtual canvas according to a plurality of continuous virtual handwriting coordinates includes:

[0036] performing time series filtering analysis on the plurality of virtual handwriting coordinates to remove discrete virtual handwriting coordinates from the plurality of virtual handwriting coordinates to obtain a plurality of first virtual handwriting coordinates;

[0037] Obtaining trajectory segments corresponding to any two of the first virtual handwriting coordinates, and calculating the curvature of each of the trajectory segments, so as to eliminate jump virtual handwriting coordinates from a plurality of the first virtual handwriting coordinates according to the curvature, and obtain a plurality of second virtual handwriting coordinates;

[0038] A first virtual handwriting is formed according to a plurality of the second virtual handwriting coordinates, and the first virtual handwriting is smoothed to obtain a virtual handwriting.

[0039] In the above scheme, methods such as time series filtering and curvature analysis are used to optimize the continuous handwriting coordinates, effectively eliminating discrete and jump points, significantly improving the smoothness and accuracy of the handwriting, making the final displayed handwriting more natural, smooth, and closer to the user's actual input intention, thereby improving the accuracy of handwriting projection.

[0040] As a preferred example, the step of constructing a trajectory change matrix of the two-dimensional virtual canvas and the display terminal includes:

[0041] Mapping the origin of the coordinate system of the two-dimensional virtual canvas to the origin of the terminal coordinate system of the display terminal;

[0042] The terminal size information of the display terminal is acquired, so as to construct a trajectory change matrix of the two-dimensional virtual canvas and the display terminal according to the size information and the canvas size.

[0043] In the above solution, a trajectory change matrix is constructed through the mapping relationship between the coordinate system origin and the canvas size to ensure that the virtual canvas is adapted to the size of the display terminal, avoid handwriting stretching and compression problems caused by resolution or scale differences, and achieve accurate projection across devices.

[0044] In a second aspect, the present invention discloses a handwriting projection system based on a virtual canvas, comprising a canvas trigger module, a canvas virtual module, a coordinate recognition module, a handwriting drawing module and a handwriting projection module;

[0045] The canvas trigger module is used to obtain the three-dimensional coordinate data and posture data of the electronic pen according to the preset canvas trigger conditions;

[0046] The canvas virtual module is used to construct a two-dimensional virtual canvas according to the posture data, the three-dimensional coordinate data and the three-dimensional coordinate system where the electronic pen is located;

[0047] The coordinate recognition module is used to obtain the real-time spatial coordinates of the electronic pen according to a preset anchor point, and map the real-time spatial coordinates to the two-dimensional virtual canvas to obtain virtual handwriting coordinates;

[0048] The handwriting drawing module is used to draw virtual handwriting on the two-dimensional virtual canvas according to a plurality of continuous virtual handwriting coordinates;

[0049] The handwriting projection module is used to construct a trajectory change matrix between the two-dimensional virtual canvas and the display terminal, and project the virtual handwriting to the display terminal according to the trajectory change matrix to display the handwriting of the electronic pen on the display terminal.

[0050] The present invention discloses a handwriting projection system based on a virtual canvas, which overcomes the limitations of traditional touch interaction methods in terms of operating space and interaction dimensions. By constructing a virtual canvas, it enables natural long-distance or contactless interaction and increases the freedom of touch interaction. During handwriting recognition, anchor points are used to improve handwriting recognition accuracy, effectively avoiding problems such as handwriting breakpoints and coordinate jumps, and ensuring that the rendering effect closely matches the user's input intent. The virtual canvas is constructed using preset canvas trigger conditions and the electronic pen's own coordinate data, avoiding resource utilization by the electronic pen and improving handwriting recognition efficiency. Secondly, when using the virtual canvas to project virtual handwriting generated by the electronic pen onto a display terminal to display the handwriting trajectory, the electronic pen's three-dimensional handwriting coordinates are first projected onto the virtual canvas. Then, the virtual handwriting on the virtual canvas is projected onto the display terminal using the trajectory change matrix between the virtual canvas and the display terminal, thereby improving the accuracy of handwriting projection.

[0051] In a third aspect, the present invention discloses a terminal device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a handwriting projection method based on a virtual canvas as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the implementation. Obviously, the drawings described below are only some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 This is a flowchart of a handwriting projection method based on a virtual canvas disclosed in one embodiment of the present invention;

[0054] Figure 2 This is a structural diagram of a handwriting projection system based on a virtual canvas disclosed in one embodiment of the present invention;

[0055] Figure 3 This is a flowchart of a handwriting projection method based on a virtual canvas disclosed in another embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0058] In the description of the embodiments of this application, the technical terms "first," "second," etc. are used only to distinguish different objects and should not be understood to indicate or imply relative importance or to implicitly indicate the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise specifically defined.

[0059] References to "embodiments" herein mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0061] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0062] In the description of the embodiments of the present application, unless otherwise expressly specified or limited, technical terms such as "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components or interactions between two components. Those skilled in the art can understand the specific meanings of the above terms in the embodiments of the present application based on specific circumstances.

[0063] See also Figure 1 To solve the technical problem of low degree of freedom of touch interaction in the prior art, an embodiment of the present invention provides a handwriting projection method based on a virtual canvas, comprising:

[0064] Step 101: Acquire the three-dimensional coordinate data and posture data of the electronic pen according to a preset canvas trigger condition.

[0065] In this embodiment, this step mainly includes: obtaining the pen tip pressure value of the electronic pen in real time according to a preset polling cycle; when the pen tip pressure value changes, obtaining the three-dimensional coordinate data of the electronic pen and the tilt angle of the electronic pen.

[0066] Or obtain the sampled three-dimensional coordinate data corresponding to the electronic pen at each sampling moment according to a preset sampling time; when the coordinate difference between any two adjacent sampled three-dimensional coordinate data is less than or equal to a preset threshold, obtain the three-dimensional coordinate data of the electronic pen and the tilt angle of the electronic pen.

[0067] Or obtain the acceleration value corresponding to the electronic pen at each sampling moment according to a preset sampling duration; when there are at least two acceleration values greater than or equal to a preset acceleration threshold, obtain the three-dimensional coordinate data of the electronic pen and the tilt angle of the electronic pen.

[0068] In this embodiment, the above steps trigger coordinate acquisition by detecting changes in the pen tip pressure value through a polling cycle, or use the changes in the coordinate difference between adjacent sampling points to determine the movement state of the electronic pen, or monitor the changes in the acceleration value of the electronic pen to assist in determining its movement state, so as to ensure that data is obtained only when the user is actually writing, reduce invalid data processing, reduce system resource consumption, and combine with the tilt angle data to more accurately reflect the user's current writing posture, and then subsequently construct a virtual canvas corresponding to the writing posture to improve the accuracy of handwriting recognition.

[0069] Step 102: constructing a two-dimensional virtual canvas according to the posture data, the three-dimensional coordinate data and the three-dimensional coordinate system where the electronic pen is located.

[0070] In this embodiment, this step mainly includes: when the tip pressure value of the electronic pen is detected or the tilt angle of the electronic pen is detected to be greater than a preset angle threshold, constructing a first two-dimensional coordinate system according to the first coordinate axis and the second coordinate axis in the three-dimensional coordinate system; when the tilt angle of the electronic pen is detected to be less than or equal to the angle threshold, constructing a second two-dimensional coordinate system according to the first coordinate axis and the third coordinate axis in the three-dimensional coordinate system; using the three-dimensional coordinate data as the origin of the coordinate system to construct the two-dimensional virtual canvas according to the preset canvas size, space scaling ratio, the first two-dimensional coordinate system or the second two-dimensional coordinate system.

[0071] In this embodiment, the above steps dynamically switch the two-dimensional coordinate system (the first or second coordinate axis combination) according to the tilt angle of the electronic pen, adapting to different writing postures (such as vertical writing or writing with the tip of the pen sideways), ensuring that the projection plane always aligns with the user's actual writing direction, avoiding handwriting deformation or offset caused by posture changes, and enhancing adaptability to multiple scenarios. This improves the naturalness and accuracy of interaction.

[0072] Step 103: obtaining the real-time spatial coordinates of the electronic pen according to the preset anchor point, and mapping the real-time spatial coordinates to the two-dimensional virtual canvas to obtain virtual handwriting coordinates.

[0073] In this embodiment, this step mainly includes: obtaining the real-time three-dimensional spatial coordinates of the electronic pen based on a preset anchor point; projecting the real-time three-dimensional spatial coordinates to the plane where the two-dimensional virtual canvas is located according to the normal vector of the two-dimensional virtual canvas to obtain the two-dimensional projection coordinates of the projection point; obtaining the vector difference between the two-dimensional projection coordinates and the origin of the coordinate system, and obtaining the virtual handwriting coordinates of the real-time three-dimensional spatial coordinates in the two-dimensional virtual canvas based on the vector difference and the spatial scaling ratio.

[0074] In this embodiment, the above steps calculate the virtual handwriting coordinates through normal vector projection and spatial scaling ratio, accurately mapping the three-dimensional spatial coordinates to the two-dimensional plane, solving the handwriting proportion imbalance problem caused by projection distortion or improper scaling, ensuring that the handwriting on the display terminal is consistent with the actual writing trajectory, and improving the accuracy of handwriting projection.

[0075] Step 104: Draw virtual handwriting on the two-dimensional virtual canvas according to a plurality of continuous virtual handwriting coordinates.

[0076] In this embodiment, this step mainly includes: performing time-series filtering analysis on multiple virtual handwriting coordinates to eliminate discrete virtual handwriting coordinates from the multiple virtual handwriting coordinates to obtain multiple first virtual handwriting coordinates; obtaining trajectory segments corresponding to any two of the first virtual handwriting coordinates, and calculating the curvature of each of the trajectory segments to eliminate jump virtual handwriting coordinates from the multiple first virtual handwriting coordinates according to the curvature to obtain multiple second virtual handwriting coordinates; forming a first virtual handwriting based on the multiple second virtual handwriting coordinates, and smoothing the first virtual handwriting to obtain virtual handwriting.

[0077] In this embodiment, the above steps use methods such as time series filtering and curvature analysis to optimize the continuous handwriting coordinates, effectively eliminate discrete and jump points, and significantly improve the smoothness and accuracy of the handwriting, so that the final displayed handwriting is more natural, smooth, and closer to the user's actual input intention, thereby improving the accuracy of handwriting projection.

[0078] Step 105: constructing a trajectory change matrix between the two-dimensional virtual canvas and the display terminal, and projecting the virtual handwriting to the display terminal according to the trajectory change matrix, so as to display the handwriting of the electronic pen on the display terminal.

[0079] In this embodiment, this step mainly includes: mapping the origin of the coordinate system of the two-dimensional virtual canvas to the origin of the terminal coordinate system of the display terminal; obtaining the terminal size information of the display terminal to construct a trajectory change matrix of the two-dimensional virtual canvas and the display terminal based on the size information and the canvas size.

[0080] In this embodiment, the above steps construct a trajectory change matrix through the mapping relationship between the coordinate system origin and the canvas size, ensuring that the virtual canvas is adapted to the size of the display terminal, avoiding handwriting stretching and compression problems caused by resolution or scale differences, and achieving accurate projection across devices.

[0081] like Figure 2 As shown, based on the above method embodiment, a corresponding device embodiment is provided; an embodiment of the present invention provides a handwriting projection system based on a virtual canvas, including a canvas trigger module 201, a canvas virtual module 202, a coordinate recognition module 203, a handwriting drawing module 204 and a handwriting projection module 205.

[0082] The canvas triggering module 201 is used to obtain the three-dimensional coordinate data and posture data of the electronic pen according to a preset canvas triggering condition.

[0083] The canvas virtualization module 202 is configured to construct a two-dimensional virtual canvas according to the posture data, the three-dimensional coordinate data, and the three-dimensional coordinate system where the electronic pen is located.

[0084] The coordinate recognition module 203 is used to obtain the real-time spatial coordinates of the electronic pen according to a preset anchor point, and map the real-time spatial coordinates to the two-dimensional virtual canvas to obtain virtual handwriting coordinates.

[0085] The handwriting drawing module 204 is configured to draw virtual handwriting on the two-dimensional virtual canvas according to a plurality of continuous virtual handwriting coordinates.

[0086] The handwriting projection module 205 is used to construct a trajectory change matrix between the two-dimensional virtual canvas and the display terminal, and project the virtual handwriting to the display terminal according to the trajectory change matrix to display the handwriting of the electronic pen on the display terminal.

[0087] It can be understood that the above-mentioned device embodiment corresponds to the method embodiment of the present invention, and can implement a handwriting projection method based on a virtual canvas provided by any of the above-mentioned method embodiments of the present invention.

[0088] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Furthermore, in the drawings of the device embodiments provided by the present invention, the connection relationship between modules indicates that they have a communication connection, which may be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement the present invention without inventive effort.

[0089] Based on the above-mentioned embodiment of a handwriting projection method based on a virtual canvas, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, a handwriting projection method based on a virtual canvas according to any embodiment of the present invention is implemented.

[0090] For example, in this embodiment, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more module elements may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0091] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0092] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, connecting various parts of the entire terminal device using various interfaces and lines.

[0093] The present embodiment discloses a handwriting projection method, system, and device based on a virtual canvas, which overcomes the limitations of traditional touch interaction methods in terms of operating space and interaction dimensions. By constructing a virtual canvas, it enables natural long-distance or contactless interaction, increasing the freedom of touch interaction. During handwriting recognition, anchor points are used to improve handwriting recognition accuracy, effectively avoiding problems such as handwriting breakpoints and coordinate jumps, and ensuring that the rendering effect closely matches the user's input intent. The virtual canvas is constructed using preset canvas trigger conditions and the electronic pen's own coordinate data, avoiding resource utilization and improving handwriting recognition efficiency. Secondly, when using the virtual canvas to project virtual handwriting generated by the electronic pen onto a display terminal to display the handwriting trajectory, the three-dimensional handwriting coordinates of the electronic pen are first projected onto the virtual canvas. Then, the virtual handwriting on the virtual canvas is projected onto the display terminal using the trajectory change matrix between the virtual canvas and the display terminal, thereby improving the accuracy of handwriting projection.

[0094] Example 2

[0095] Existing electronic whiteboard interaction methods typically require users to write on the touchscreen or within a designated area, limiting operation and interaction options. For example, traditional electronic whiteboards require users to write directly on the screen, which reduces user interaction freedom. Furthermore, existing technologies rely solely on the IMU sensor within the electronic pen for trajectory estimation, which is prone to drift and leads to accumulated handwriting errors. Furthermore, the low latency of trajectory data transmission between the electronic pen and the electronic whiteboard makes it impossible to guarantee the accuracy of the electronic whiteboard's trajectory.

[0096] In order to solve the above technical problems, refer to Figure 3 This embodiment discloses a handwriting projection method based on a virtual canvas, comprising:

[0097] Step 301: Detecting the tip pressure data, acceleration data, or coordinate movement data of the electronic pen in real time to determine whether to trigger the generation of a virtual canvas.

[0098] In this embodiment, whether to trigger the generation of a virtual canvas is determined based on preset canvas trigger conditions. These trigger conditions include a change in the tip pressure of the electronic pen, i.e., a "touch" event between the electronic pen and an operating area such as the desktop; multiple jumps in the acceleration of the electronic pen within a preset time window, i.e., a "double-click" event; or a "hover" event within an area for a certain duration, i.e., the coordinates of the electronic pen remain consistent or undergo slight changes.

[0099] Specifically, when the trigger mode is pen tip contact, that is, when the electronic pen collides with an operating area, such as a desktop, the creation of the virtual canvas is determined. The electronic pen determines whether it is near the operating area, such as the desktop, and a "touch event" has occurred based on the change in the UWB Z-axis coordinate. When the Z value quickly decreases to a threshold (e.g., 5mm), it is determined that the "pen tip is close to the desktop."

[0100] When the trigger mode is hovering, that is, when the electronic pen is determined to have hovered above the desktop for more than a set time (e.g., 2 seconds), the creation of the virtual canvas is automatically triggered. The electronic pen obtains its coordinates using a preset positioning system, such as a UWB positioning system, in conjunction with a preset anchor point. The electronic pen is determined to have "hovered" when the coordinates remain unchanged or when the displacement between two adjacent coordinates is less than or equal to a preset threshold, indicating slight movement of the electronic pen.

[0101] When the trigger mode is gesture triggering, that is, when the user opens the virtual canvas by double-clicking the desktop or the pen button, the electronic pen determines whether a double-click occurs by changing the acceleration. Specifically, when two obvious acceleration peak events (such as >2g) are detected within a short time window (such as within 500ms) or the interval between the two peaks is within the set threshold range (such as 80ms to 300ms) and the direction is in the same direction, the electronic pen is determined to have a "double-click" event. Among them, a high-pass filter is used to remove low-frequency hand shake interference and extract short-term impact signals. Among them, the electronic pen has built-in sensor equipment such as accelerometer, gyroscope, magnetometer, etc. to measure the movement state of the pen itself.

[0102] When it is detected that the electronic pen is in one of the pen tip contact triggering mode, the hovering triggering mode, and the gesture triggering mode, it is determined to generate a virtual canvas.

[0103] Step 302: When it is determined that the generation of the virtual canvas is triggered, the three-dimensional coordinate data and posture data of the electronic pen are obtained to construct a two-dimensional virtual canvas.

[0104] In this embodiment, when a trigger is determined to generate a virtual canvas, the three-dimensional spatial coordinates of the electronic pen are determined using a preset coordinate positioning system, such as a UWB positioning system, using the coordinates of a preset anchor point. Preferably, the UWB system constructs a three-dimensional coordinate system using the anchor point, and the electronic pen reports its spatial coordinates (X, Y, Z) in real time.

[0105] Specifically, the electronic pen captures its real-time 3D position within a global 3D coordinate system, such as the coordinate system defined by the UWB Anchor. When determining to create a virtual canvas, the electronic pen's spatial coordinates are determined using a positioning system constructed using the electronic pen and multiple anchor points. After deploying the anchor points, the system performs initial calibration using known reference points (such as the four corners of a desktop). It compares the UWB-measured coordinates with the actual reference coordinates, constructs an error model (such as an affine error matrix), and updates the anchor point parameters or coordinate inversion formula to improve subsequent positioning accuracy.

[0106] In one implementation of this embodiment, when the UWB system acquires the coordinate data of the electronic pen, it uses preset anchor points as the origin and reference point of the spatial coordinate system to form a global three-dimensional coordinate framework. The electronic pen measures the distance to at least 3 to 4 anchor points by sending / receiving UWB pulses, obtains the relative distance, and uses the known coordinates of multiple anchor points and the measured distances to reversely infer the position of the electronic pen using multilateral measurement to obtain the three-dimensional coordinate data of the electronic pen. Among them, the anchor point refers to a UWB communication node installed in a fixed position with known three-dimensional coordinates. When positioning the electronic pen, it supports an automatic calibration mechanism for the anchor point position; a multi-point constraint solving algorithm (such as the least squares method) can be used to calculate the position of each anchor point in a virtual coordinate system. The specific implementation logic of using anchor points to position the electronic pen is as follows:

[0107] (1) There are 4 anchor points, the coordinates are (x i ,y i , z i );

[0108] (2) The electronic pen measures the distance d from each anchor point i , the coordinates (x, y, z) of the tip of the electronic pen are obtained by solving the following set of equations; wherein the set of equations is:

[0109]

[0110] Here, 1, 2, 3, and 4 represent the order corresponding to different anchor points.

[0111] After obtaining the three-dimensional coordinate data of the electronic pen, the three-dimensional coordinate data is used as the origin or center of the virtual canvas, and the tilt angle of the electronic pen is obtained to determine the horizontal plane of the virtual canvas according to the tilt angle.

[0112] In one embodiment, because the virtual canvas is essentially a coordinate projection mapping area that exists within the display space of the electronic whiteboard, but its coordinates are defined by the user's operating area (e.g., the horizontal plane of the desktop or a horizontal plane perpendicular to the desktop), to improve the accuracy of electronic pen handwriting mapping, a sensor device such as a gyroscope built into the electronic pen detects the angle between the electronic pen body and the horizontal plane of the desktop, and uses this angle as the electronic pen's tilt angle. A smaller angle indicates that the electronic pen body is more parallel to the horizontal plane of the desktop. In this case, to improve the accuracy of electronic pen handwriting mapping, the normal vector of the virtual canvas is defined as perpendicular to the horizontal plane of the desktop, i.e., the horizontal plane perpendicular to the desktop is defined as the horizontal plane of the virtual canvas. Conversely, a larger angle indicates that the electronic pen body is more perpendicular to the horizontal plane of the desktop. In this case, to improve the accuracy of electronic pen handwriting mapping, the normal vector of the virtual canvas is defined as parallel to the horizontal plane of the desktop, i.e., the horizontal plane of the desktop is defined as the horizontal plane of the virtual canvas. The desktop serves only as the starting point for interaction, and the virtual canvas is a two-dimensional interactive projection surface above it or in the spatial extension area; the two can physically overlap, but the virtual canvas has an independent and scalable logical meaning.

[0113] Then, a virtual canvas is constructed according to the three-dimensional coordinate data, a normal vector corresponding to the virtual canvas, a preset canvas size, and a scaling ratio from the space coordinates to the canvas coordinates.

[0114] Step 303: Acquire a plurality of continuous real-time three-dimensional coordinates of the electronic pen, and project the real-time three-dimensional coordinates onto the two-dimensional virtual canvas to obtain a plurality of two-dimensional virtual coordinates.

[0115] In this embodiment, for any of the real-time 3D coordinates, the real-time 3D coordinate is projected onto the 2D virtual canvas according to the normal vector of the 2D virtual canvas to obtain the projection point of the real-time 3D coordinate on the 2D virtual canvas.

[0116] An offset vector from the projection point to the origin of the two-dimensional virtual canvas is calculated, and the offset is expanded on the coordinate system of the two-dimensional virtual canvas to determine the two-dimensional virtual coordinates of the projection point on the canvas according to a scaling ratio from the spatial coordinates to the canvas coordinates.

[0117] In one embodiment of the present invention, the two-dimensional virtual coordinates can be converted into two-dimensional virtual pixel coordinates. For example, if the canvas size is A3 (420mm*297mm), the two-dimensional virtual coordinates can be converted into two-dimensional virtual pixel coordinates by setting the pixel density.

[0118] Step 304: Identify abnormal virtual coordinates among the two-dimensional virtual coordinates to obtain a number of standard two-dimensional virtual coordinates.

[0119] In this embodiment, after obtaining a plurality of the two-dimensional virtual coordinates, abnormal virtual coordinates in the two-dimensional virtual coordinates can be identified through timing filtering, jump detection, etc., and the abnormal virtual coordinates are processed to obtain a plurality of standard two-dimensional virtual coordinates.

[0120] Specifically, the abnormal virtual coordinates are extracted from the abnormal virtual coordinates in the two-dimensional virtual coordinates by utilizing the preset velocity smoothing constraint, i.e., the velocity field continuity. For example, the state quantities corresponding to several consecutive two-dimensional virtual coordinates including position p, velocity v, acceleration a, etc. are identified based on Kalman filtering or extended Kalman filtering (EKF), so as to predict the next virtual two-dimensional coordinate, such as Pt+1=Pt+Vt·△t. When the error between the next measured two-dimensional virtual coordinate and the UWB position is large, the predicted trajectory is used instead.

[0121] In one implementation of this embodiment, instantaneous jump judgment is performed, that is, the curvature corresponding to the trajectory segment composed of two adjacent two-dimensional virtual coordinates is obtained, and the curvature difference between the two adjacent trajectory segments is obtained. When the curvature difference is greater than a preset threshold, the two-dimensional virtual coordinates existing in both trajectory segments are determined to be jump virtual coordinates, and the jump virtual coordinates are eliminated.

[0122] Step 305: projecting a plurality of the standard two-dimensional virtual coordinates onto the display terminal according to the trajectory change matrix between the two-dimensional virtual canvas and the display terminal, so as to display the handwriting of the electronic pen on the display terminal.

[0123] In this embodiment, to avoid data delays between the electronic pen and the display terminal, data is collected from the electronic pen at fixed time intervals (e.g., 10ms) and preliminary filtering is performed. A timestamp synchronization mechanism is used to ensure that data from different sensors are fused under the same time reference, and then the obtained standard two-dimensional virtual coordinates are sent to the display terminal using different data transmission methods. The data transmission methods include BLE (Bluetooth Low Energy): suitable for short-range, low-power data synchronization. Wi-Fi Direct: suitable for long-distance, high-throughput handwriting transmission. UWB communication: UWB two-way communication can be used to directly transmit handwriting data to improve real-time performance.

[0124] When projecting the standard two-dimensional virtual coordinates onto the display terminal, the mapping relationship across the device coordinate system can be determined based on the size information of the display terminal, the screen resolution, the size information of the virtual canvas, the pixel density and other information, so as to obtain the display coordinates of each standard two-dimensional virtual coordinate on the display terminal according to the mapping relationship.

[0125] In one implementation of this embodiment, a coordinate system mapping relationship between the virtual canvas and the display terminal may be constructed based on the three-dimensional space coordinates corresponding to each of the four corners of the canvas and the three-dimensional space coordinates corresponding to each of the four corners of the display terminal.

[0126] When the coordinate data is sent to the display terminal, the projection position of each coordinate on the screen of the display terminal is calculated through affine transformation or perspective transformation and the coordinate system mapping relationship.

[0127] Based on the curve formed by several projection positions, the handwriting trajectory is smoothed using Bezier curve fitting or B-spline interpolation methods to avoid jitter. Smoothing segments are inserted between the trajectory transition points or breakpoints to ensure a natural and continuous trajectory. The display terminal's GPU is then controlled to accelerate rendering, ensuring low-latency display of the handwriting.

[0128] This embodiment discloses a handwriting projection method based on a virtual canvas, which overcomes the limitations of traditional touch interaction methods in terms of operating space and interaction dimensions. By constructing a virtual canvas, it enables natural, long-distance or contactless interaction, increasing the freedom of touch interaction. During handwriting recognition, anchor points are used to improve accuracy, effectively avoiding issues such as handwriting breakpoints and coordinate jumps, and ensuring that the rendering effect closely matches the user's input intent.

[0129] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A handwriting projection method based on a virtual canvas, characterized in that: include: Acquire the three-dimensional coordinate data and posture data of the electronic pen according to the preset canvas trigger conditions; constructing a two-dimensional virtual canvas according to the posture data, the three-dimensional coordinate data and the three-dimensional coordinate system where the electronic pen is located; Acquiring the real-time spatial coordinates of the electronic pen according to a preset anchor point, and mapping the real-time spatial coordinates to the two-dimensional virtual canvas to obtain virtual handwriting coordinates; Drawing virtual handwriting on the two-dimensional virtual canvas according to a plurality of continuous virtual handwriting coordinates; A trajectory change matrix between the two-dimensional virtual canvas and the display terminal is constructed, and the virtual handwriting is projected onto the display terminal according to the trajectory change matrix, so as to display the handwriting of the electronic pen on the display terminal.

2. The handwriting projection method based on a virtual canvas according to claim 1, characterized in that: The step of obtaining the three-dimensional coordinate data and posture data of the electronic pen according to the preset canvas triggering condition includes: Acquiring the pen tip pressure value of the electronic pen in real time according to a preset polling cycle; When the pen tip pressure value changes, the three-dimensional coordinate data of the electronic pen and the tilt angle of the electronic pen are acquired.

3. The handwriting projection method based on a virtual canvas according to claim 1, characterized in that: The method of obtaining the three-dimensional coordinate data and posture data of the electronic pen according to the preset canvas triggering condition further includes: Acquiring sampling three-dimensional coordinate data corresponding to each sampling moment of the electronic pen according to a preset sampling time; When the coordinate difference between any two adjacent sampled three-dimensional coordinate data is less than or equal to a preset threshold, the three-dimensional coordinate data of the electronic pen and the tilt angle of the electronic pen are acquired.

4. The handwriting projection method based on a virtual canvas according to claim 1, characterized in that: The method of obtaining the three-dimensional coordinate data and posture data of the electronic pen according to the preset canvas triggering condition further includes: Obtaining the acceleration value corresponding to the electronic pen at each sampling moment according to a preset sampling time; When there are at least two acceleration values greater than or equal to a preset acceleration threshold, the three-dimensional coordinate data of the electronic pen and the tilt angle of the electronic pen are acquired.

5. A handwriting projection method based on a virtual canvas according to any one of claims 1 to 4, characterized in that: The constructing of a two-dimensional virtual canvas according to the posture data, the three-dimensional coordinate data, and the three-dimensional coordinate system where the electronic pen is located includes: When a tip pressure value of the electronic pen is detected or a tilt angle of the electronic pen is detected to be greater than a preset angle threshold, constructing a first two-dimensional coordinate system according to the first coordinate axis and the second coordinate axis in the three-dimensional coordinate system; When it is detected that the tilt angle of the electronic pen is less than or equal to the angle threshold, constructing a second two-dimensional coordinate system according to the first coordinate axis and the third coordinate axis in the three-dimensional coordinate system; The three-dimensional coordinate data is used as the origin of the coordinate system to construct the two-dimensional virtual canvas according to a preset canvas size, a spatial scaling ratio, the first two-dimensional coordinate system or the second two-dimensional coordinate system.

6. The handwriting projection method based on a virtual canvas according to claim 5, characterized in that: The step of acquiring the real-time spatial coordinates of the electronic pen according to a preset anchor point and mapping the real-time spatial coordinates to the two-dimensional virtual canvas to obtain virtual handwriting coordinates includes: Acquiring the real-time three-dimensional spatial coordinates of the electronic pen according to a preset anchor point; Projecting the real-time three-dimensional space coordinates onto the plane where the two-dimensional virtual canvas is located according to the normal vector of the two-dimensional virtual canvas to obtain the two-dimensional projection coordinates of the projection point; A vector difference between the two-dimensional projection coordinates and the origin of the coordinate system is obtained, and virtual handwriting coordinates of the real-time three-dimensional space coordinates in the two-dimensional virtual canvas are obtained according to the vector difference and the spatial scaling ratio.

7. The handwriting projection method based on a virtual canvas according to claim 1, characterized in that: Drawing a virtual handwriting on the two-dimensional virtual canvas according to a plurality of continuous virtual handwriting coordinates includes: performing time series filtering analysis on the plurality of virtual handwriting coordinates to remove discrete virtual handwriting coordinates from the plurality of virtual handwriting coordinates to obtain a plurality of first virtual handwriting coordinates; Obtaining trajectory segments corresponding to any two of the first virtual handwriting coordinates, and calculating the curvature of each of the trajectory segments, so as to eliminate jump virtual handwriting coordinates from a plurality of the first virtual handwriting coordinates according to the curvature, and obtain a plurality of second virtual handwriting coordinates; A first virtual handwriting is formed according to a plurality of the second virtual handwriting coordinates, and the first virtual handwriting is smoothed to obtain a virtual handwriting.

8. The handwriting projection method based on a virtual canvas according to claim 5, characterized in that: The constructing of the trajectory change matrix of the two-dimensional virtual canvas and the display terminal includes: Mapping the origin of the coordinate system of the two-dimensional virtual canvas to the origin of the terminal coordinate system of the display terminal; The terminal size information of the display terminal is acquired, so as to construct a trajectory change matrix of the two-dimensional virtual canvas and the display terminal according to the size information and the canvas size.

9. A handwriting projection system based on a virtual canvas, characterized in that: It includes canvas trigger module, canvas virtual module, coordinate recognition module, handwriting drawing module and handwriting projection module; The canvas trigger module is used to obtain the three-dimensional coordinate data and posture data of the electronic pen according to the preset canvas trigger conditions; The canvas virtual module is used to construct a two-dimensional virtual canvas according to the posture data, the three-dimensional coordinate data and the three-dimensional coordinate system where the electronic pen is located; The coordinate recognition module is used to obtain the real-time spatial coordinates of the electronic pen according to a preset anchor point, and map the real-time spatial coordinates to the two-dimensional virtual canvas to obtain virtual handwriting coordinates; The handwriting drawing module is used to draw virtual handwriting on the two-dimensional virtual canvas according to a plurality of continuous virtual handwriting coordinates; The handwriting projection module is used to construct a trajectory change matrix between the two-dimensional virtual canvas and the display terminal, and project the virtual handwriting to the display terminal according to the trajectory change matrix to display the handwriting of the electronic pen on the display terminal.

10. A terminal device, characterized in that: The invention comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the handwriting projection method based on a virtual canvas according to any one of claims 1 to 8 is implemented.

Images