Handwriting rendering method, device, equipment, medium and program product

Abstract

Embodiments of the present application provide a handwriting rendering method, device, equipment, medium and program product. The method comprises: obtaining initial touch data of a touch object; determining a sampling density of handwriting track point resampling and a track prediction mode based on the initial touch data; generating target touch data based on the sampling density and the track prediction mode; determining a submission time window of the target touch data based on a start time of a target screen refresh cycle and an estimated value of handwriting display delay; the estimated value is used to represent a time length between a submission time of the target touch data and a display time of handwriting corresponding to the display of the target touch data; and submitting the target touch data to a rendering thread for handwriting rendering within the submission time window. The method is used to improve the real-time performance and smoothness of handwriting display.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to a handwriting rendering method and device, equipment, medium and program product. BACKGROUND

[0002] In a mobile device (such as a tablet computer, a smart phone), when a user writes or draws by using a stylus, the real-time performance, smoothness and visual quality of the handwriting directly affect the user experience.

[0003] In the prior art, in a fast writing or high-curvature handwriting scenario, there are problems such as device perception delay and handwriting breakage, which seriously affect the interaction smoothness.

[0004] Therefore, how to improve the real-time performance, smoothness and visual quality of handwriting in a stylus interaction scenario of a mobile device has become a key problem to be solved for optimizing the user experience. SUMMARY

[0005] Embodiments of the present application provide a handwriting rendering method, device, equipment, medium and program product to improve the real-time performance and smoothness of handwriting display.

[0006] In a first aspect, embodiments of the present application provide a handwriting rendering method, comprising:

[0007] obtaining initial touch data of a touch object;

[0008] determining a sampling density of handwriting trajectory point resampling and a trajectory prediction mode based on the initial touch data;

[0009] generating target touch data based on the sampling density and the trajectory prediction mode;

[0010] determining a submission time window of the target touch data based on a start time of a target screen refresh period and an estimated value of handwriting display delay; the estimated value is used to represent a time length between a submission time of the target touch data and a display time of handwriting corresponding to the target touch data;

[0011] submitting the target touch data to a rendering thread for handwriting rendering within the submission time window.

[0012] In a possible implementation, the determining of the sampling density of handwriting trajectory point resampling and the trajectory prediction mode based on the initial touch data comprises:

[0013] determining a moving speed of the touch object and a curvature of the handwriting trajectory based on the initial touch data;

[0014] in response to the moving speed being less than or equal to a preset speed threshold and the curvature being higher than a preset curvature threshold, determining the sampling density as a first sampling density and the trajectory prediction manner as a first trajectory prediction manner;

[0015] in response to the moving speed being greater than the preset speed threshold and / or the curvature being less than or equal to the preset curvature threshold, determining the sampling density as a second sampling density and the trajectory prediction manner as a second trajectory prediction manner;

[0016] the first sampling density is greater than the second sampling density;

[0017] the first trajectory prediction manner is curve fitting, and the second trajectory prediction manner is a trajectory prediction manner based on Kalman prediction and Bayesian mixture.

[0018] In a possible implementation, before the target touch data submission time window is determined based on the start time of the target screen refresh period and the estimated value of the handwriting display delay, the method further includes:

[0019] obtaining a target time length between a start time of a first screen refresh period after a current time and the current time;

[0020] in response to the target time length being greater than or equal to a sum of the estimated value of the handwriting display delay and a safety time length, taking the start time of the first screen refresh period after the current time as the start time of the target screen refresh period.

[0021] In a possible implementation, a start time of the target touch data submission time window is the current time;

[0022] an end time of the target touch data submission time window is a target time before the start time of the target screen refresh period, and a time interval between the target time and the start time of the target screen refresh period is a sum of the estimated value and the safety time length.

[0023] In a possible implementation, before the start time of the target screen refresh period is taken as the start time of the target screen refresh period in response to the target time length being greater than or equal to the sum of the estimated value of the handwriting display delay and the safety time length, the method further includes:

[0024] obtaining a confidence degree of the target touch data;

[0025] determining the safety time length based on the confidence degree and a variance of the handwriting display delay.

[0026] In a possible implementation, the method further includes:

[0027] determining a rendering quality parameter based on a utilization rate of a real-time graphics processing unit (GPU) and a frame time of the GPU;

[0028] performing handwriting rendering by the rendering thread based on the rendering quality parameter and the target touch data.

[0029] In a second aspect, an embodiment of the present application provides a handwriting rendering apparatus, including:

[0030] a obtaining module configured to obtain initial touch data of a touch object;

[0031] a first determining module configured to determine a sampling density of handwriting trajectory point resampling and a trajectory prediction manner based on the initial touch data;

[0032] a generating module configured to generate target touch data based on the sampling density and the trajectory prediction manner;

[0033] a second determining module configured to determine a submission time window of the target touch data based on a start time of a target screen refresh cycle and an estimated value of handwriting display delay; the estimated value is used to represent a time length between a submission time of the target touch data and a display time of handwriting corresponding to the target touch data;

[0034] a rendering module configured to submit the target touch data to a rendering thread for handwriting rendering within the submission time window.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor.

[0036] The memory stores computer execution instructions.

[0037] The processor executes the computer execution instructions stored in the memory, so that the processor performs the method in any one of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method in any one of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and the computer program is executed by a processor to implement the method in any one of the first aspect.

[0040] The handwriting rendering method, device, equipment, medium and program product provided by the embodiments of the present application can dynamically determine the resampling density of handwriting track points and the track prediction mode through initial touch data, and then determine the submission time window based on the target screen refresh period and the handwriting display delay estimation value, so that the target touch data can be submitted to the rendering thread at the best time. Compared with the device sensing delay, handwriting breakage and other problems in the prior art, the present application can reduce the delay of handwriting rendering, improve the real-time performance and smoothness of handwriting display, reduce visual lag caused by improper submission time, and improve the user's writing experience while ensuring the quality of handwriting. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the present application.

[0042] Figure 1 A handwriting rendering process schematic diagram provided by the embodiments of the present application;

[0043] Figure 2 A specific handwriting rendering method flowchart provided by the embodiments of the present application;

[0044] Figure 3 A handwriting rendering device structure schematic diagram provided by the present application;

[0045] Figure 4 A structure schematic diagram of an electronic device provided by the present application.

[0046] The above-described drawings have shown the specific embodiments of the present application, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the present application concept in any way, but to illustrate the present application concept to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] The exemplary embodiments will be described in detail herein with reference to the drawings. Unless otherwise specified, the same numbers in different drawings indicate the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present application.

[0048] In the present application, the term "comprising" and its variants can refer to non-limiting inclusion; the term "or" and its variants can refer to "and / or". In the present application, the terms "first", "second", etc. are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. In the present application, "multiple" refers to two or more. "And / or", which describes the relationship between the associated objects, means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the front and rear associated objects.

[0049] In a mobile device (such as a tablet computer, a smart phone), when a user writes or draws by a stylus, the real-time, smoothness and visual quality of the handwriting directly affect the user experience. There is a contradiction between the delay and quality of the handwriting in the prior art.

[0050] For example, the prior art adopts an interpolation-based smoothing method, which fits the original touch points by a Bezier curve, a Catmull-Rom spline or a Hermite interpolation to generate a continuous handwriting trajectory. Such a method can improve visual smoothness in a regular writing scenario, but has weak prediction ability for high-curvature or fast-turning strokes, and is prone to problems such as redundant extension at the end of the handwriting or interruption of local continuity.

[0051] For another example, the prior art can use linear extrapolation or polynomial fitting to make a short-time prediction of the current stylus motion trend. Such a method performs well in a stable straight-line writing scenario, but has significant errors in a stylus direction mutation or complex trajectory, resulting in a large deviation between the predicted position and the actual trajectory.

[0052] Secondly, the prior art lacks timing perception and does not grasp the real submission or scanning window of the synthesizer and the display, resulting in that the application submits a frame at the wrong time, so that even if the correct predicted position is rendered, it will be displayed with lag due to missing VSync or GPU queuing.

[0053] Therefore, the embodiments of the present application dynamically determine the resampling density of the handwriting trajectory points and the trajectory prediction method based on the initial touch data, and then, based on the target screen refresh period and the handwriting display delay estimation value, the submission time window is determined, so that the target touch data can be submitted to the rendering thread at the best time. Compared with the device perception delay, handwriting breakage and other problems in the prior art, the present application can reduce the delay of handwriting rendering, improve the real-time and smoothness of handwriting display, and reduce the visual lag caused by improper submission time, while ensuring the quality of handwriting and improving the user's writing experience.

[0054] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0055] It should be noted that the execution subject of this application embodiment can be any touch device with processing capabilities, such as a user terminal or a server, for example, a tablet computer.

[0056] Figure 1 This is a schematic diagram of a handwriting rendering process provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0057] S101. Obtain the initial touch data of the touch object.

[0058] Optionally, the touch object can be an object that interacts with a touch device (such as a touch screen), for example, a stylus. The initial touch data can be data collected by the touch device during the contact or movement of the touch object with the touch device, and may include any one or more of the following: touch position coordinates, touch pressure, tilt, azimuth angle, tool type, touch timestamp, monotonic clock, etc.

[0059] For example, the initial touch data can be represented as TouchSample (initial touch data) = {t_event (monotonic clock), x (touch position horizontal coordinate), y (touch position vertical coordinate), pressure (touch pressure), tilt (tilt), azimuth (azimuth angle), toolType (tool type), rawTimestamp (touch timestamp)}.

[0060] Optionally, the touch device can obtain the initial touch data of the touch object according to its own calling interface, which can be called by the input dispatch layer or received and converted by the application layer.

[0061] In one implementation, the touch device processes the initial touch data. For example, this may include: dejittering the initial touch data using short-time median filtering; performing coordinate transformation through physical-to-logical dots per inch (DPI) normalization; converting the initial touch data into a touch timestamp based on the system monotonic clock reference; and further processing the data by... computing initial instantaneous velocity of initial touch data; discarding abnormal sample points (e.g. noise with long time pause) and pushing valid sample points into a sample buffer queue, e.g. circular buffer.

[0062] S102, determining a sampling density of trajectory point resampling and a trajectory prediction method based on the initial touch data.

[0063] Optionally, the trajectory points can be a set of points formed by a touch object moving on a touch device. The sampling density can be a number of trajectory points sampled per unit length or per unit time. The trajectory prediction method can be an algorithm to predict the position and feature of a subsequent trajectory point. For example, the trajectory prediction method can be any one or more of curve fitting, Kalman prediction, polynomial extrapolation, etc.

[0064] Optionally, the touch device can determine the sampling density of trajectory point resampling and the trajectory prediction method based on a mapping relationship between the initial touch data and the sampling density of trajectory point resampling, and based on a mapping relationship between the initial touch data and the trajectory prediction method. The mapping relationship between the initial touch data and the sampling density of trajectory point resampling, and the mapping relationship between the initial touch data and the trajectory prediction method can be pre-stored in the touch device.

[0065] For example, the touch device can determine a velocity between adjacent trajectory points based on the initial touch data, and if the velocity between the adjacent trajectory points is less than a preset velocity threshold, a high resampling density is used to capture details. The touch device can determine a curvature between adjacent trajectory points based on the initial touch data, and if the curvature between the adjacent trajectory points is greater than a preset curvature threshold, a trajectory prediction method with higher accuracy is used, e.g. a method of combining Kalman filtering and Bayesian to predict the trajectory.

[0066] S103, generating target touch data based on the sampling density and the trajectory prediction method.

[0067] Optionally, the target touch data can be touch data used to represent handwriting. Optionally, the touch device can process and calculate the initial touch data based on a method of generating target touch data based on a sampling density and a method of generating target touch data based on a trajectory prediction method in the prior art, to generate the target touch data.

[0068] Exemplarily, the touch device can dynamically select a prediction model at different writing stages based on the sampling density and the trajectory prediction manner. A Kalman filter combined with linear interpolation is used to correct a short-time local trajectory in real time, and a lightweight Long Short-Term Memory (LSTM) network is called to predict the future stroke direction for a long-time global trend. The touch device can fuse the resampling points based on the sampling density and the predicted trajectory according to timestamps to generate target touch data containing spatial coordinates, pressure values, and continuity.

[0069] In S104, a submission time window of the target touch data is determined based on a start time of the target screen refresh period and an estimated value of the handwriting display delay. The estimated value is used to represent a time length between a submission time of the target touch data and a display time of handwriting corresponding to the target touch data.

[0070] Optionally, the target screen refresh period can be a fixed time interval of image refresh operation of the screen. The estimated value of the handwriting display delay can be a time length of the target touch data from being submitted to the touch device to being displayed on the touch screen.

[0071] Exemplarily, the estimated value of the handwriting display delay can include any one or more of an average delay from inputting initial touch data to application processing, an average delay between application processing completion and Graphic Processing Unit (GPU) start processing, and an average delay between GPU processing completion and synthesizer start synthesizing.

[0072] Optionally, the estimated value of the handwriting display delay can estimate each segment of delay and variance using weighted linear regression or exponential smoothing of a sliding window. Exemplarily, the expected value and / or the variance may be used to represent. Optionally, the touch device can inject a micro-bench or record a timestamp of a real frame from being submitted to being synthesized and displayed to update the estimated value of the handwriting display delay online. For example, the touch device can perform a quick probe at the first start or installation to measure the delay of each stage from input to display, and save the delay to a device database as an initial value, and update the delay in real time during runtime.

[0073] Optionally, the submission time window of the target touch data can be a time range, for example, the submission time window can include a start time and an end time. Submitting the target touch data in the submission time window can ensure that the handwriting can be displayed on the screen on time.

[0074] Optionally, the touch device can determine the submission time window of the target touch data based on the start time of the target screen refresh cycle and the estimated value of the handwriting display delay according to a preset rule. For example, the touch device can determine the start time of the target screen refresh cycle and the sum of the estimated value of the handwriting display delay as the start time of the submission time window.

[0075] S105, submitting the target touch data to the rendering thread for handwriting rendering within the submission time window.

[0076] Optionally, the rendering thread can be an independent execution thread in the touch device responsible for handwriting rendering. The handwriting rendering can be a process of drawing corresponding handwriting on the touch screen through a graphic algorithm and display technology.

[0077] The embodiments of the present application can dynamically determine the resampling density of the handwriting trajectory points and the trajectory prediction mode based on the initial touch data, and then determine the submission time window based on the target screen refresh cycle and the estimated value of the handwriting display delay, so that the target touch data can be submitted to the rendering thread at the best time. Compared with the device sensing delay and handwriting breakage in the prior art, the present application can reduce the delay of handwriting rendering, improve the real-time performance and smoothness of handwriting display, and reduce the visual lag caused by improper submission time, thereby improving the user's writing experience while ensuring the quality of handwriting.

[0078] The following will describe in detail how to determine the sampling density of handwriting trajectory point resampling and the trajectory prediction mode based on the initial touch data.

[0079] Firstly, the touch device can determine the moving speed of the touch object and the curvature of the handwriting trajectory based on the initial touch data. Optionally, the moving speed of the touch object can represent the distance moved by the touch object per unit time, which can reflect the speed of the touch object. For example, the moving speed of the touch object can be represented by The curvature of the handwriting trajectory can represent the bending degree of the handwriting trajectory at a certain point. The greater the curvature of the handwriting trajectory, the more intense the bending degree of the handwriting trajectory at the point; the smaller the curvature of the handwriting trajectory, the closer the handwriting trajectory to a straight line. For example, the curvature of the handwriting trajectory can be represented by

[0080] The touch device can determine the sampling density as the first sampling density and the trajectory prediction mode as the first trajectory prediction mode when the moving speed is less than or equal to a preset speed threshold and the curvature is higher than a preset curvature threshold.

[0081] For example, the preset speed threshold can be represented by and the preset curvature threshold can be represented by ​Characterization. The preset speed threshold and the preset curvature threshold can be set according to actual needs. For example, the moving speed is less than or equal to the preset speed threshold, and the curvature is higher than the preset curvature threshold, i.e. , and , it can be characterized that the touch object is in a slow writing stage at this time, and the bending change of the handwriting trajectory is more obvious, which may be in the process of drawing more fine and complex strokes.

[0082] In this case, the touch device can determine the sampling density to be a first sampling density, for example, high-density resampling. Through high-density resampling, the touch device can capture small actions such as stroke turning and pressure change.

[0083] The touch device can also determine the trajectory prediction mode to be a first trajectory prediction mode, i.e. curve fitting. For example, the touch device can globally segment the initial touch data to perform multi-spline or segmented Bezier curve fitting, and locally use three or four points to construct a tangent line, for example, use Hermite interpolation method or Catmull-Rom spline interpolation method to generate a smooth vector description to ensure the continuity of the control point sequence, i.e. the handwriting trajectory.

[0084] The touch device can determine the sampling density to be a second sampling density and the trajectory prediction mode to be a second trajectory prediction mode when the moving speed is greater than the preset speed threshold and / or the curvature is less than or equal to the preset curvature threshold. The first sampling density is greater than the second sampling density.

[0085] For example, in the case where the moving speed is greater than the preset speed threshold and / or the curvature is less than or equal to the preset curvature threshold, i.e. , and / or, , it can be characterized that the touch object is in a fast writing stage at this time, and the bending change of the handwriting trajectory is relatively smooth.

[0086] In this case, the touch device can determine the sampling density to be a second sampling density, for example, low-density resampling. When writing fast, the distance between touch points naturally increases due to high speed, and low-density resampling can avoid excessive insertion of redundant points, directly use original points or a small number of interpolation points, and significantly reduce the data size of subsequent processing.

[0087] The touch device can also determine the trajectory prediction mode to be a second trajectory prediction mode, i.e. a trajectory prediction mode based on Kalman prediction and Bayesian mixture. Kalman filtering provides fast and low-cost preliminary prediction, and Bayesian mixture optimizes the results through post-processing, which ensures real-time performance and improves the accuracy of long-term prediction. The trajectory prediction mode based on Kalman prediction and Bayesian mixture is described in detail below.

[0088] Exemplarily, the touch device can determine its state vector according to the initial touch data, which can be characterized as wherein, may represent the horizontal coordinate of the touch position, may represent the vertical coordinate of the touch position, may represent the velocity component of the touch position in the horizontal coordinate, may represent the velocity component of the touch position in the vertical coordinate, may represent the acceleration component of the touch position in the horizontal coordinate, may represent the acceleration component of the touch position in the vertical coordinate.

[0089] The touch device can employ a constant acceleration or constant velocity model for the handwriting trajectory prediction, which can be characterized as wherein, may be the predicted value of the handwriting trajectory, may be a state transition matrix, which can be related to the acceleration and constant velocity. may represent the process noise, which is subject to a zero-mean normal distribution , i.e. . is the process noise covariance matrix, which physically represents the disturbance (such as external interference, model approximation error) that is not captured by the model, the larger the value, the higher the uncertainty of the model prediction. It should be noted that may be updated in real time during the running process.

[0090] The touch device can map the state space to the observable measurement space by the formula to establish the relationship between the state and the measurement value. may represent the current latest resampling position. may represent the observation matrix, may represent the state vector, may represent the measurement noise, which is subject to a zero-mean normal distribution , i.e. . is the measurement noise covariance matrix, which can reflect the accuracy of the sensor, the smaller the value, the more reliable the measurement. It should be noted that may be updated in real time during the running process.

[0091] The touch device can employ adaptive noise covariance adjustment, when a larger measurement residual or high curvature is detected, increase to reduce the prediction weight; when writing a stable straight line, reduce to improve the prediction accuracy and suppress the measurement noise interference.

[0092] The touch device can simultaneously run short-time polynomial extrapolation (2nd order) and Kalman prediction, and according to fitting error and curvature, the two prediction results are weighted by a Bayesian mixture method to obtain the final handwriting trajectory prediction.

[0093] The embodiment of the present application can determine the corresponding sampling density and trajectory prediction mode based on the moving speed of the touch object and the handwriting trajectory curvature, can provide high-density and continuous curve fitting in the slow and fine writing stage to ensure the stroke details and stroke effect, and can reduce the sampling density and assist the high-robustness prediction in the fast writing stage. Compared with the interpolation based on the historical points in the prior art, the embodiment of the present application can adapt to both fast and slow writing situations, and improve the real-time and fluency of handwriting display.

[0094] Before the above-mentioned target screen refresh cycle start time and the estimated value of handwriting display delay are used to determine the submission time window of the target touch data, the touch device can first obtain the target time length between the start time of the first screen refresh cycle after the current time and the current time.

[0095] Optionally, the touch device can obtain the start time of the first screen refresh cycle after the current time by using any one or more of Choreographer (a tool of an Android development environment), a display timing application programming interface (Display Timing API), vertical synchronization callbacks (VSync callbacks), etc. The touch device can calculate the difference between the current time and the start time of the first screen refresh cycle as the target time length. Exemplarily, the start time of the first screen refresh cycle can be represented as .

[0096] Secondly, the touch device can take the start time of the first screen refresh cycle after the current time as the start time of the target screen refresh cycle when the target time length is greater than or equal to the sum of the estimated value of handwriting display delay and the safety time length.

[0097] Optionally, the safety time length can be a period of time additionally set to ensure the handwriting display effect, which can act as a buffer, and can be represented as Exemplarily, the target time length can be represented as As described above, the estimated value of handwriting display delay can be represented as the expected ​, the target screen refresh period is the first screen refresh period after the current time. , the target screen refresh period is the first screen refresh period after the current time.

[0098] Optionally, the touch device can dynamically read the current refresh period and adjust the target screen refresh period if the touch device is a variable refresh rate (VRR) display. , the target screen refresh period is the first screen refresh period after the current time.

[0099] In an embodiment, the touch device can first obtain the confidence of the target touch data before determining the safety time period in response to the target time period being greater than or equal to the sum of the estimated value of the handwriting display delay and the safety time period, and then determine the safety time period based on the confidence and the variance of the handwriting display delay.

[0100] Optionally, the confidence of the target touch data can be used to measure the reliability of the target touch data, which can reflect the confidence of the accuracy and authenticity of the target touch data. High confidence can indicate that the target touch data is relatively accurate and less likely to have errors. Low confidence can indicate that the target touch data may have large errors. The variance of the handwriting display delay can reflect the fluctuation of the handwriting display delay time. Large variance indicates that the delay time changes greatly and is unstable. Small variance indicates that the delay time is relatively stable.

[0101] The touch device can determine the safety time period based on the mapping relationship between the confidence and the variance of the handwriting display delay and the safety time period. For example, if the confidence is high and / or the variance of the handwriting display delay is small, it can be characterized that the target touch data is relatively accurate and / or the handwriting delay time is relatively stable, so a shorter safety time period can be selected. If the confidence is low and / or the variance of the handwriting display delay is large, it can be characterized that the target touch data has large errors and / or the handwriting delay time changes greatly, so a longer safety time period can be selected.

[0102] The embodiments of the present application can determine the safety time period by considering the confidence of the touch data and the variance of the handwriting display delay, thereby enhancing the tolerance of the touch device to uncertainty and improving the stability and reliability of handwriting rendering. Even in the case of large data fluctuations, the rendering effect can also be maintained.

[0103] The embodiments of the present application can ensure that the handwriting data is submitted within an optimal time window by accurately calculating and selecting a suitable screen refresh period, further reducing display delay, reducing visual lag, and improving the synchronization of handwriting and screen refresh.

[0104] In an embodiment, the start time of the submission time window is the current time, and the end time of the submission time window can be a target time before the start time of the target screen refresh period. The time interval between the target time and the start time of the target screen refresh period is the sum of the estimated value and the safety time length.

[0105] Alternatively, the end time of the submission time window can be the time point at which the submission time window ends. There is a time interval between the target time and the start time of the target screen refresh period, and the length of the time interval is the sum of the estimated value and the safety time length, which can enable the handwriting to be rendered before the screen refresh, avoiding rendering failure due to time misalignment.

[0106] The embodiments of the present application define the start and end times of the submission time window, provide an accurate time range for the submission of handwriting data, effectively avoid display lag caused by improper submission time, and ensure the timeliness and accuracy of handwriting rendering.

[0107] Based on the above embodiments, the touch device of the embodiments of the present application can also determine the rendering quality parameter based on the real-time GPU utilization and the frame time of the GPU.

[0108] Alternatively, the GPU utilization can represent the degree to which the GPU is used at the current time, usually in the form of a percentage. For example, if the GPU utilization is 80%, it means that 80% of the computing resources of the GPU are being occupied for graphics processing tasks. The GPU utilization reflects the workload of the GPU, and when the utilization is close to 100%, it means that the GPU is in a high-load running state, which can affect the performance and efficiency of graphics rendering.

[0109] The frame time of the GPU can be the time required for the GPU to complete one frame of graphics rendering. The rendering quality parameter can be used to control the quality and effect of graphics rendering. For example, the rendering quality parameter can include any one or more of texture resolution, shadow quality, shader, anti-aliasing, lighting effect, etc.

[0110] Optionally, the touch device can determine the rendering quality parameter based on the GPU utilization rate and the mapping relationship between the frame time of the GPU and the rendering quality parameter. For example, if the GPU utilization rate is low and the frame time is short, it indicates that the GPU has sufficient computing resources, and a higher rendering quality parameter can be determined to obtain better graphics effects; on the contrary, if the GPU utilization rate is high and the frame time is close to or exceeds the target value, the rendering quality parameter needs to be reduced to ensure the smoothness of graphics rendering.

[0111] For example, the touch device can implement anti-aliasing and pressure simulation based on a signed-distance-field (SDF) or a stroke mesh scheme with width deformation. The fragment shader calculates coverage instead of a large number of oversampling, thereby reducing the filling pressure. For fast writing, switch to a simplified shader (low sampling, approximate width), and for slow refinement, switch back to a high-quality shader.

[0112] After determining the rendering quality parameter, the touch device can perform handwriting rendering based on the rendering quality parameter and the target touch data through a rendering thread. In an embodiment, the touch device can store the vector description (spline / Bézier) of the stabilized historical trajectory based on a hierarchical idea, using a vector buffer method at the bottom layer, and reconstruct a local mesh only when the handwriting is determined or needs to be updated as a whole. At the top layer, a dynamic mask or a fast raster layer method is used, and the top layer is used to display the predicted tail and the latest unstable handwriting segment, and a low-overhead incremental rasterization is used to reduce the redraw area.

[0113] In an embodiment, the touch device can use a strategy combination when high input rate and GPU bottleneck are detected, merge the intermediate points (for example, keep the first / last point and the maximum pressure point) and reduce the resampling density or the shader quality on the premise of ensuring the key endpoint and pressure peak information.

[0114] The touch device can also use a dynamic Quality of Service (QoS) method to automatically downgrade or upgrade the rendering quality based on real-time GPU utilization and frame time. The shader and resampling strategy are controlled through an interface. If the device is overloaded for a long time, the throttling logic (limiting the sampling rate and using a coarser interpolation) is started, and the fallback is smoothed after the load is restored.

[0115] The embodiment of the present application can dynamically adjust the rendering quality parameter to adapt to the real-time load of the GPU, and balance the rendering performance and quality. Compared with the prior art, in order to ensure smoothness, the rendering resolution is increased as a cost, thereby causing the problems of increased GPU or memory pressure, frame drop and significant power consumption of low-end devices under high sampling rate. The embodiment of the present application not only ensures the smoothness of handwriting rendering, but also avoids the problems of stuttering or frame drop caused by GPU overload, and improves the overall rendering efficiency.

[0116] The handwriting rendering method is described below with one specific embodiment, Figure 2 A specific handwriting rendering method provided by the embodiment of the present application is shown in a flowchart as Figure 2 The flowchart includes:

[0117] 1. Receive touch or handwriting pen original data.

[0118] 2. Detect touch sampling rate and device refresh rate.

[0119] 3. Analyze timestamp and system delay.

[0120] 4. Event de-bouncing and merging processing.

[0121] 5. Calculate the writing speed and determine whether the writing speed is less than the speed threshold; if yes, execute step 6; if no, execute step 7.

[0122] 6. Use high-density sampling and curve fitting for prediction.

[0123] 7. Use low-density sampling and trajectory prediction based on Kalman prediction and Bayesian mixture for prediction.

[0124] 8. Obtain display pipeline timing information.

[0125] 9. Calculate the best submission time.

[0126] 10. Prepare rendering using layered rendering architecture.

[0127] 11. Generate bottom layer vector cache.

[0128] 12. Apply top layer dynamic mask and anti-aliasing shader.

[0129] 13. Submit frame to GPU rendering queue.

[0130] 14. Wait for the start time signal of the target screen refresh period and display.

[0131] The above is the method embodiment provided by the present application, and the device provided by the present application is described below.

[0132] The embodiments of the present application can maintain smoothness and fine balance on multi-rate and multi-topology devices, dynamically compensate display and synthesis delay, and reduce rendering load. Through collaborative optimization at the levels of sampling, prediction, timing perception and rendering, various beneficial effects are achieved, as follows.

[0133] First, through speed adaptive resampling combined with segmented Bezier / spline interpolation, high-density and continuous curve fitting can be provided in the slow and fine writing stage to ensure stroke details and stroke effect. In the fast writing stage, the sampling density is reduced and high-robustness prediction (Kalman / Bayesian hybrid) is supplemented, which significantly reduces device perception delay and handwriting breakage, and improves user perception of handwriting immediacy.

[0134] Second, display pipeline timing modeling and optimal submission scheduling based on VSync are introduced, which can align the visible time of application rendering with the display scanning time, reduce the redundant delay caused by improper submission time, and alleviate the device perception delay problem.

[0135] Third, through layered rendering (vector bottom layer + dynamic raster top layer) and SDF-based anti-aliasing strategy, the stroke edge quality can be guaranteed, and the filling area and shader complexity of each frame can be minimized, which significantly reduces the GPU filling pressure and avoids frame drop and high power consumption of low-end devices.

[0136] Fourth, runtime device adaptation, including delay detection, dynamic process noise covariance matrix and measurement noise covariance matrix adjustment, and quality grading, so that the same algorithm can adaptively perform on devices with different performance levels, reducing the workload of developers for manual parameter tuning for different models.

[0137] In summary, the present application brings significant improvement in subjective interaction experience, objective delay indicators, rendering resource consumption and cross-device consistency, thereby having good practical application value in commercial handwriting pen input, drawing applications and education writing scenarios.

[0138] The embodiments of the present application can also test and verify the handwriting rendering method, for example, by using a robot pen or a high-precision touch console to record different speed, curvature and pressure situations through automation; statistical perception delay (submission to visible time), breakage rate (breakpoint detection), jitter (path and ground truth error). Run on low-end / mid-end / high-end representative models and record graphics processor / central processing unit (GPU / CPU) utilization, frame loss rate and power consumption curve, and combine user experience test to do subjective scoring.

[0139] Figure 3 A structure diagram of a handwriting rendering device provided by the present application is shown in FIG. 1.Figure 3 As shown, the handwriting rendering apparatus 200 provided by the embodiment comprises an acquisition module 201, a first determination module 202, a generation module 203, a second determination module 204, and a rendering module 205. Optionally, the handwriting rendering apparatus 200 can further comprise a processing module 206.

[0140] The acquisition module 201 is configured to acquire initial touch data of a touch object.

[0141] The first determination module 202 is configured to determine, based on the initial touch data, a sampling density of handwriting trajectory point resampling and a trajectory prediction manner.

[0142] The generation module 203 is configured to generate target touch data based on the sampling density and the trajectory prediction manner.

[0143] The second determination module 204 is configured to determine, based on a start time of a target screen refresh period and an estimated value of handwriting display delay, a submission time window of the target touch data. The estimated value is used to represent a time length between a submission time of the target touch data and a display time of handwriting corresponding to the display of the target touch data.

[0144] The rendering module 205 is configured to submit the target touch data to a rendering thread for handwriting rendering within the submission time window.

[0145] Optionally, the first determination module 202 is specifically configured to determine, based on the initial touch data, a moving speed of the touch object and a curvature of the handwriting trajectory. In response to the moving speed being less than or equal to a preset speed threshold and the curvature being higher than a preset curvature threshold, the sampling density is determined as a first sampling density and the trajectory prediction manner is determined as a first trajectory prediction manner. In response to the moving speed being greater than the preset speed threshold and / or the curvature being less than or equal to the preset curvature threshold, the sampling density is determined as a second sampling density and the trajectory prediction manner is determined as a second trajectory prediction manner. The first sampling density is greater than the second sampling density. The first trajectory prediction manner is curve fitting. The second trajectory prediction manner is a trajectory prediction manner based on Kalman prediction and Bayesian mixture.

[0146] Optionally, the processing module 206 is configured to acquire a target time length between a start time of a first screen refresh period after a current time and the current time. In response to the target time length being greater than or equal to a sum of the estimated value of the handwriting display delay and a safety time length, the start time of the first screen refresh period after the current time is taken as the start time of the target screen refresh period.

[0147] Optionally, the start time of the submission time window is the current time. The end time of the submission time window is a target time before the start time of the target screen refresh period; the time interval between the target time and the start time of the target screen refresh period is the sum of the estimated value and the safety time length.

[0148] Optionally, in response to the target time length being greater than or equal to the sum of the estimated value of the handwriting display delay and the safety time length, the processing module 206 is further configured to obtain a confidence level of the target touch data, based on the confidence level and a variance of the handwriting display delay, determine the safety time length.

[0149] Optionally, the processing module 206 is further configured to determine a rendering quality parameter based on a utilization rate of a real-time graphics processing unit (GPU) and a frame time of the GPU, and perform handwriting rendering based on the rendering quality parameter and the target touch data through a rendering thread.

[0150] The handwriting rendering apparatus provided in the embodiment can perform the method provided in any of the above method embodiments, and has similar implementation principles and technical effects, which will not be described here again in the embodiment.

[0151] Figure 4 A structural schematic diagram of an electronic device provided in the present application is shown in FIG. 1. Figure 4 As shown in FIG. 1, the electronic device 500 provided in the embodiment includes at least one processor 501 and a memory 502. Optionally, the device 500 further includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected through a bus 504.

[0152] In the specific implementation process, the at least one processor 501 executes the computer execution instructions stored in the memory 502, so that the at least one processor 501 performs the above method.

[0153] The specific implementation process of the processor 501 can refer to the above method embodiments, and has similar implementation principles and technical effects, which will not be described here again in the embodiment.

[0154] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0155] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0156] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0157] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above method.

[0158] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the above method is implemented.

[0159] The above readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0160] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. Of course, the readable storage medium can also be a part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0161] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0162] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0163] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0164] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0165] Those skilled in the art can understand that all or part of the steps of the foregoing method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the foregoing method embodiments; and the foregoing storage medium includes various storage media that can store program codes, such as ROM, RAM, magnetic disk, or optical disk.

[0166] Finally, it should be noted that other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the present application disclosed herein. The present application is intended to include all such variations and modifications in keeping with the principles of the present application and including such steps in addition to those which are nec¬ essary to practice the application. The application is not limited to the precise construction and combinations of parts and steps disclosed herein and as such can vary from those disclosed without departing from the scope of the application.

Claims

1. A handwriting rendering method, characterized in that, The method includes: Obtain the initial touch data of the touch object; Based on the initial touch data, the moving speed of the touch object and the curvature of the handwriting trajectory are determined; In response to the movement speed being less than or equal to a preset speed threshold and the curvature being higher than a preset curvature threshold, the sampling density of the handwriting trajectory point resampling is determined to be a first sampling density, and the trajectory preset method is a first trajectory prediction method; the first trajectory prediction method is curve fitting. In response to the moving speed being greater than the preset speed threshold, and / or the curvature being less than or equal to the preset curvature threshold, the sampling density is determined to be a second sampling density, and the trajectory preset method is a second trajectory prediction method; the first sampling density is greater than the second sampling density; the second trajectory prediction method is a trajectory prediction method based on a hybrid of Kalman prediction and Bayesian prediction. Based on the sampling density and the trajectory prediction method, target touch data is generated; Obtain the start time of the first screen refresh cycle after the current moment, and the target duration between the current moment and the current moment; Obtain the confidence level of the target touch data; based on the confidence level and the variance of the handwriting display delay, determine the safe duration; In response to a target duration, which is greater than or equal to the sum of the estimated handwriting display delay and the safety duration, the start time of the first screen refresh cycle after the current moment is taken as the start time of the target screen refresh cycle. Based on the start time of the target screen refresh cycle and the estimated value of the handwriting display delay, the submission time window of the target touch data is determined; the estimated value is used to characterize the duration between the submission time of the target touch data and the display time of the handwriting corresponding to the target touch data. Within the submission time window, the target touch data is submitted to the rendering thread for handwriting rendering.

2. The method according to claim 1, characterized in that, The starting time of the submission time window is the current time; The end time of the submission time window is the target time before the start time of the target screen refresh cycle; the time interval between the target time and the start time of the target screen refresh cycle is the sum of the estimated value and the safety duration.

3. The method according to claim 1, characterized in that, The method further includes: Rendering quality parameters are determined based on the utilization of the real-time graphics processing unit (GPU) and the frame time of the GPU. Handwriting rendering is performed using the rendering thread, based on the rendering quality parameters and the target touch data.

4. A handwriting rendering device, characterized in that, include: The acquisition module is used to acquire the initial touch data of the touch object; The first determining module is used to determine the moving speed of the touch object and the curvature of the handwriting trajectory based on the initial touch data; In response to the movement speed being less than or equal to a preset speed threshold and the curvature being higher than a preset curvature threshold, the sampling density of the handwriting trajectory point resampling is determined to be a first sampling density, and the trajectory preset method is a first trajectory prediction method; the first trajectory prediction method is curve fitting; in response to the movement speed being greater than the preset speed threshold, and / or the curvature being less than or equal to the preset curvature threshold, the sampling density is determined to be a second sampling density, and the trajectory preset method is a second trajectory prediction method; the first sampling density is greater than the second sampling density; the second trajectory prediction method is a trajectory prediction method based on a hybrid of Kalman prediction and Bayesian prediction; The generation module is used to generate target touch data based on the sampling density and the trajectory prediction method; The processing module is used to obtain the start time of the first screen refresh cycle after the current time and the target duration between the current time; Obtain the confidence level of the target touch data; Based on the confidence level and the variance of the handwriting display delay, the safe duration is determined; In response to a target duration, which is greater than or equal to the sum of the estimated handwriting display delay and the safety duration, the start time of the first screen refresh cycle after the current moment is taken as the start time of the target screen refresh cycle. The second determining module is used to determine the submission time window of the target touch data based on the start time of the target screen refresh cycle and the estimated value of the handwriting display delay. The estimated value is used to characterize the duration between the submission time of the target touch data and the display time of the handwriting corresponding to the target touch data; The rendering module is used to submit the target touch data to the rendering thread for handwriting rendering within the submission time window.

5. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-3.

7. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-3.

Images