Method for generating drawing on basis of touch input, and electronic device for performing same

Abstract

This method by which an electronic device predicts a drawing point may comprise the steps of: detecting a first touch input; estimating a first time, which is the time from the point at which a first touch input is detected to the point at which a screen is refreshed; estimating a second time, which is the time required to render a first drawing including a first drawing point corresponding to the first touch input; selecting, on the basis of the first time and the second time, a first prediction model for predicting the first drawing point; predicting the first drawing point from the first touch input by using the selected first prediction model; and displaying, on a screen, the first drawing including the first drawing point.

Description

Method for generating a drawing based on touch input and an electronic device for performing the same

[0001] The present disclosure relates to an electronic device, method, and system for generating a drawing based on touch input using a prediction model.

[0002] Prediction Artificial Intelligence (Prediction AI) is an artificial intelligence technology used to predict future events or outcomes based on past data. It utilizes various algorithms and machine learning techniques to analyze data patterns and, based on this, predicts results that have not yet occurred. For example, a prediction model can be used to generate or enhance lines or shapes by predicting outcomes likely to occur next based on user input or data.

[0003] Screen refresh time can generally be defined by FPS (Frames per second) or frequency (Hz). For example, a monitor with a refresh rate of 60Hz may have a screen refresh time of 16.67ms. Display devices such as monitors, TVs, and projectors continuously update the images displayed on the screen, and screen refresh time can indicate how often this update occurs.

[0004] A whiteboard is a type of electronic device equipped with a display and a touchscreen, which can be a device that combines digital technology with a traditional physical whiteboard. Whiteboards can recognize electronic input and provide collaboration features that allow written content to be saved digitally or shared with others in real time. When the screen is touched, input data is transferred to a processor, which can interpret the data and request the Graphic Processor Unit (GPU) to draw specific elements on a virtual canvas displayed on the screen.

[0005] Some aspects will be presented in this disclosure, some will become apparent in this disclosure, and some may be learned through the embodiments.

[0006] According to one aspect of the present disclosure, a method for an electronic device to predict a drawing point may include: detecting a first touch input; estimating a first time, which is the time from when the first touch input is detected until the screen is refreshed; estimating a second time, which is the time required to render a first drawing including a first drawing point corresponding to the first touch input; selecting a first prediction model for predicting a first drawing point based on the first time and the second time; predicting a first drawing point from the first touch input using the selected first prediction model; and displaying a first drawing including a first drawing point on a screen.

[0007] A method for an electronic device to predict a drawing point may further include: detecting a second touch input; estimating a third time, which is the time from when the second touch input is detected until the screen is refreshed; estimating a fourth time, which is the time required to render a second drawing including a second drawing point corresponding to the second touch input; determining whether to perform a prediction for the second touch input based on the third time and the fourth time; selecting a second prediction model for predicting the second drawing point based on the third time and the fourth time based on the decision to perform a prediction for the second touch input; predicting a second drawing point from the second touch input using the selected second prediction model; and displaying a second drawing including the second drawing point on a screen.

[0008] A method for an electronic device to predict a drawing point may further include the step of performing a prediction for a second touch input after the screen has been refreshed, based on the decision not to perform a prediction for a second touch input.

[0009] The first prediction model is a neural network model trained to predict a first drawing point from a first touch input using past touch inputs as training data, and the second prediction model may be a neural network model trained to predict a second drawing point from a second touch input using past touch inputs as training data.

[0010] A method for an electronic device to predict a drawing point may further include: a step of estimating a fifth time, which is a time at which the refresh of the screen is delayed; a step of predicting a third drawing point from a first touch input based on a first time, a second time, and a fifth time; and a step of displaying a third drawing including the first drawing point and the third drawing point on the screen.

[0011] A method for an electronic device to predict a drawing point may further include: identifying a sixth time, which is a time when the refresh of the screen is delayed; predicting a fourth drawing point from a first touch input based on the second time and the sixth time; and displaying a fourth drawing including the first drawing point and the fourth drawing point on the screen.

[0012] The step of displaying a first drawing including a first drawing point on a screen may include: selecting a first drawing point from at least one prediction result based on a first time, a second time, accuracy, and error probability; and displaying a first drawing including the selected first drawing point on a screen. According to one aspect of the present disclosure, an electronic device for predicting a drawing point includes an input interface; an output interface; a memory for storing at least one instruction; and at least one processor for executing at least one instruction stored in the memory. The electronic device may detect a first touch input, estimate a first time which is the time from when the first touch input is detected until the screen is refreshed, estimate a second time which is the time required to render a first drawing including a first drawing point corresponding to the first touch input, select a first prediction model for predicting a first drawing point based on the first time and the second time, predict a first drawing point from the first touch input using the selected first prediction model, and display a first drawing including the first drawing point on a screen.

[0013] An electronic device for predicting a drawing point detects a second touch input, estimates a third time which is the time from when the second touch input is detected until the screen is refreshed, estimates a fourth time which is the time required to render a second drawing including a second drawing point corresponding to the second touch input, determines whether to perform a prediction for the second touch input based on the third time and the fourth time, selects a second prediction model for predicting the second drawing point based on the third time and the fourth time based on the decision to perform a prediction for the second touch input, predicts a second drawing point from the second touch input using the selected second prediction model, and can display a second drawing including the second drawing point on the screen.

[0014] An electronic device for predicting drawing points can perform a prediction for a second touch input after the screen is refreshed, based on the decision not to perform a prediction for a second touch input.

[0015] The first prediction model is a neural network model trained to predict a first drawing point from a first touch input using past touch inputs as training data, and the second prediction model may be a neural network model trained to predict a second drawing point from a second touch input using past touch inputs as training data.

[0016] An electronic device for predicting a drawing point can estimate a fifth time, which is a time when the refresh of the screen is delayed, and predict a third drawing point from a first touch input based on a first time, a second time, and a fifth time, and display a third drawing including the first drawing point and the third drawing point on the screen.

[0017] An electronic device for predicting a drawing point can identify a sixth time, which is a time when the refresh of the screen is delayed, and based on the second time and the sixth time, predict a fourth drawing point from a first touch input and display a fourth drawing including the first drawing point and the fourth drawing point on the screen.

[0018] An electronic device for predicting a drawing point may select a first drawing point from at least one prediction result based on a first time, a second time, accuracy, and error probability, and display a first drawing including the selected first drawing point on a screen. According to one aspect of the present disclosure, a computer-readable recording medium having a program may execute a method on a computer, and the method may include the steps of: detecting a first touch input; estimating a first time which is the time from when the first touch input is detected until the screen is refreshed; estimating a second time which is the time required to render a first drawing including a first drawing point corresponding to the first touch input; selecting a first prediction model for predicting a first drawing point based on the first time and the second time; predicting a first drawing point from the first touch input using the selected first prediction model; and displaying a first drawing including the first drawing point on a screen.

[0019] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description together with the accompanying drawings.

[0020] FIG. 1 is a drawing for illustrating an example of performing drawing on an electronic device using a prediction model according to one embodiment of the present disclosure.

[0021] FIG. 2 is a diagram illustrating a time delay occurring in the processing of a touch input according to one embodiment of the present disclosure.

[0022] FIG. 3 is a flowchart illustrating a method for processing a single touch input according to one embodiment of the present disclosure.

[0023] FIG. 4 is a diagram for illustrating, in a timeline, the process of processing a single touch input according to one embodiment of the present disclosure.

[0024] FIG. 5 is a flowchart illustrating a method for processing two or more touch inputs according to one embodiment of the present disclosure.

[0025] FIG. 6 is a diagram for illustrating, in a timeline, the process of processing two or more touch inputs according to one embodiment of the present disclosure.

[0026] FIG. 7 is a flowchart illustrating a method for processing touch input based on a delay of screen refresh according to one embodiment of the present disclosure.

[0027] FIG. 8 is a diagram illustrating, in timeline form, a process of processing touch input based on a delay in screen refresh according to one embodiment of the present disclosure.

[0028] FIG. 9 is a flowchart illustrating a method for processing touch input when screen refresh is delayed according to one embodiment of the present disclosure.

[0029] FIG. 10 is a flowchart illustrating an operation for predicting a drawing point according to one embodiment of the present disclosure.

[0030] FIG. 11 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0031] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail. In the present disclosure, the expression "at least one of a, b, or c" may refer to "a," "b," "c," "a and b," "a and c," "b and c," "all of a, b, and c," or variations thereof.

[0032] The terms used in this disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant explanatory sections. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0033] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art as described in this specification. Additionally, terms including ordinal numbers, such as "first" or "second," used in this specification may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another.

[0034] Throughout the specification, when a part is described as "include" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Similarly, when a part is described as "have" a certain component or "comprise" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but includes additional components. Furthermore, terms such as "part" or "module" as used in the specification refer to a unit that handles at least one function or operation, which may be implemented in hardware or software, or as a combination of hardware and software.

[0035] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0036] The present disclosure will be described below with reference to the attached drawings.

[0037] FIG. 1 is a drawing for explaining an example of performing a drawing on an electronic device (1100) using a prediction model according to one embodiment of the present disclosure.

[0038] The electronic device (1100) may include an input interface and an output interface. The input interface and the output interface may be separate components or may be integrated to operate as a single component. Referring to FIG. 1, for example, the electronic device (1100) may include a display and a touchscreen. The display may be represented as an output interface, and the touchscreen may be represented as an input interface. In one embodiment, when a touch input is detected on the touchscreen of the electronic device (1100), the electronic device (1100) may display a dot or line corresponding to the touch input on the display.

[0039] In one embodiment, touch input may refer to an input method in which a user directly presses and operates the screen of an electronic device (1100) using an input device (100), such as a finger or a stylus pen. However, the present disclosure is not limited thereto. For example, in one embodiment, touch input may refer to the user's finger or input device (100) being detected on another surface or device without direct physical contact with the electronic device (1100). For illustrative purposes, an embodiment in which a user directly presses and operates the screen of an electronic device is primarily disclosed, but this is merely an example and the present disclosure is not limited thereto.

[0040] In one embodiment, when a user moves over the touchscreen of an electronic device (1100) using an input device (100) or a hand, the electronic device (1100) may display a dot or a line on the display according to the movement.

[0041] In one embodiment, the electronic device (1100) may require time to process data for touch input. In one embodiment, when drawing a line using the input device (100), the current position of the input device (100) and the endpoint of the line displayed on the display may differ due to the data processing for touch input of the electronic device (1100).

[0042] In one embodiment, the touchscreen of the electronic device (1100) may require time to accumulate data depending on resistance and capacitance. The touchscreen may require time to transmit the data to a processor. For example, the processor may include a CPU (Central Processing Unit). The processor may require time to process the data and transmit it to a GPU (Graphic Processing Unit).

[0043] In one embodiment, the GPU may require time to process data. After the GPU processes the data, the GPU may display content in frames. After the GPU processes the data, the GPU may display content at screen refresh times. For example, with a refresh rate of 60Hz, the screen refresh time may be 16.67ms.

[0044] In one embodiment, when a line is drawn on the touchscreen of the electronic device (1100), the electronic device (1100) may experience a delay due to the processing time of the touchscreen, CPU, and GPU. Assuming the total processing time of the touchscreen, CPU, and GPU is 16ms, there may be a delay of 16ms between the line displayed on the display and the actual position of the finger (or input device (100)).

[0045] In one embodiment, assuming that the input device (100) (or finger) is moved from left to right by 1 cm every 16 ms, the operation method of the electronic device (1100) is as follows. The electronic device (1100) detects a touch input from the input device (100) (or finger) at the 0 cm position. The touchscreen of the electronic device (1100) begins to process the touch input, and the input device (100) (or finger) moves to the 0.1 cm position. The touchscreen processes the touch input and transmits it to the CPU, and the input device (100) (or finger) moves to the 0.2 cm position. The CPU copies memory to the GPU and requests the GPU to draw a dot or line for the touch input. The input device (100) (or finger) is now at the 0.3 cm position.

[0046] In one embodiment, when the screen refresh time is set to 60Hz, it is refreshed every 16.67ms, so the GPU cannot display data on the display until the screen refresh time arrives, even after processing the data. The input device (100) (or finger) continues to move until the screen refresh time arrives. The GPU displays a dot or line for the touch input on the display according to the screen refresh time.

[0047] In one embodiment, the first line (110) may be a line that displays a drawing generated based on past touch inputs on a display. The first line (110) may be a line that displays a drawing data corresponding to past touch inputs on a display.

[0048] In one embodiment, drawing data may refer to data generated by an input device (100) (or finger) drawing a dot or a line on a touchscreen of an electronic device (1100). Drawing data may include a history of generating drawings based on past touch inputs. According to one embodiment of the present disclosure, a prediction model may be learned based on drawing data.

[0049] In one embodiment, the circular point (130) may refer to a point displayed to indicate the current position of the input device (100) (or finger). In one embodiment, the circular point (130) may not be the same as the endpoint of the first line (110). In one embodiment, the circular point (130) may not be the same as the endpoint of the first line (110) due to a time delay caused by data processing of the electronic device (1100).

[0050] In one embodiment, the second line (120) may be a line displaying a drawing on the screen based on a drawing point predicted from drawing data using a prediction model. In other words, if a prediction model learned using drawing data that includes a history of creating drawings based on past touch inputs predicts a drawing point corresponding to a touch input of the input device (100) (or finger), the line displaying the drawing on the screen based on the predicted drawing point may be the second line (120). In one embodiment, due to the imperfection of the prediction model, the circular point (130) may not be the same as the endpoint of the second line (120). The electronic device (1100) may use various prediction models to adjust the position of the prediction result and the actual input device (100) (or finger) so that they are the same.

[0051] In one embodiment, the electronic device (1100) can provide a fast and smooth response by using a prediction model to predict the location of the next point (drawing point) to be drawn based on the current movement pattern. In one embodiment, the electronic device (1100) can enable the drawing of smooth lines by using a prediction model to predict the flow of curves or straight lines in advance. In one embodiment, the electronic device (1100) can provide a natural response without delay by using a prediction model to predict the drawing point, maintain the smoothness and consistency of the line, and correct unnecessary lines or provide improved results.

[0052] In one embodiment, a prediction model may refer to an artificial intelligence (AI) model used to predict future results based on past data. The prediction model may refer to a neural network model that uses neural networks to analyze data and learn patterns to perform predictions. By learning the patterns and structures of training data, the prediction model can predict new data corresponding to the input data based on the characteristics of the input data.

[0053] In one embodiment, the electronic device (1100) may be a device of various types that processes touch input. For example, the electronic device (1100) may be implemented as an electronic device of various types and forms including an input interface and an output interface. The electronic device (1100) may include devices capable of displaying touch input via a touchscreen through a display, such as a smart TV, smartphone, tablet PC, laptop PC, glasses-type display, head-mounted display (HMD), and projected display, but is not limited thereto. For example, the electronic device (1100) may be implemented as an electronic device of various types and forms capable of wired / wireless connection with a display and a touchscreen. Additionally, the electronic device (1100) may be a fixed electronic device placed at a fixed location or a mobile electronic device having a portable form, and may be a digital broadcast receiver capable of receiving digital broadcasts. However, it is not limited thereto.

[0054] In one embodiment, the input device (100) may be a device of various types that interacts with the electronic device (1100) to create a drawing on the display of the electronic device (1100). The input device (100) may be implemented as an electronic device of various types and forms and may be referred to as an electronic pen, a digital pen, a stylus, or a stylus pen. For example, the input device (100) may include devices that interact with the electronic device (1100), such as an S Pen, a Wacom stylus, an Apple Pencil, a stylus, or a stylus pen, but is not limited thereto.

[0055] In one embodiment, the means for creating a drawing on the display of the electronic device (1100) is not limited to a pen-shaped input device and may include cases where a drawing is created using a finger or a mouse, etc.

[0056] FIG. 2 is a diagram illustrating a time delay occurring in the processing of a touch input according to one embodiment of the present disclosure.

[0057] Referring to FIG. 2, it is assumed that the screen refresh time is 60Hz, i.e., 16.67ms, and that the user is drawing a line on the touchscreen of the electronic device (1100) using an input device (100) (or finger).

[0058] Referring to 201, touch input can be detected at regular time intervals. The time at which touch input is detected can be constant.

[0059] In one embodiment, the electronic device (1100) may require time between the occurrence of an actual touch input and the start of processing for the touch input. In one embodiment, the time at which the actual touch input is detected and the time at which processing for the touch input begins may be different. In one embodiment, the time at which the actual touch input is received and the time at which processing for the touch input begins may be different.

[0060] Referring to 202, the time at which the processing of the touch input begins may not be constant. Since there is a delay in the time until the processing of the touch input begins, the electronic device (1100) can perform the processing of the touch input in batches.

[0061] Referring to 203, the electronic device (1100) may output the result of the touch input to the display after a considerable amount of time has elapsed since the touch input was detected. There may be a difference between the detection of the touch input and the display time.

[0062] Referring to 204, the electronic device (1100) may require time to display a touch input on the screen. In one embodiment, the time at which processing for the touch input begins and the time at which the touch input is displayed on the screen may differ.

[0063] In one embodiment, the electronic device (1100) may be unable to process the touch input for a significant period of time due to other background tasks while receiving the touch input from the touchscreen, and as a result, the processing speed for the touch input may be slowed. In one embodiment, the electronic device (1100) may experience a delay in processing the touch input depending on CPU performance and the current load.

[0064] In one embodiment, data for touch inputs is placed in a queue so that the CPU can receive multiple data to be processed in batches. In one embodiment, the electronic device (1100) can detect data for many touch inputs in time slots between screen refresh times. If the electronic device (1100) decides to process all touch inputs, the load to be processed increases, and the screen refresh time may be delayed.

[0065] In one embodiment, if the electronic device (1100) detects data for a touch input within a time insufficient to process within the screen refresh time, the electronic device (1100) may not be able to display the result for the touch input on the screen. Accordingly, the data for the touch input may be transmitted to the next screen refresh time slot.

[0066] When data for a large number of touch inputs is processed in batches, screen refresh is delayed; furthermore, if touch input data is detected too late for processing, there is a problem where it cannot be displayed on the screen within the allotted time even if processed. Such delays can hinder the performance of electronic devices.

[0067] Through the specific embodiments of the present disclosure described below, the processing of touch input can be optimized and adjusted to match the screen refresh time, thereby reducing the load on the CPU and GPU. Furthermore, by simultaneously reducing CPU and GPU usage, better prediction of touch input can be performed, more CPU and GPU power can be secured for other processes, and adaptation to patterns of touch input time differences can be achieved. Additionally, delays caused when the screen refresh time changes due to high CPU and GPU loads can be compensated for, and many algorithms can be utilized within a given time constraint until content is drawn on the screen.

[0068] FIG. 3 is a flowchart illustrating a method for processing a single touch input according to one embodiment of the present disclosure.

[0069] Referring to FIG. 3, when a single touch input is detected, the electronic device (1100) can process the single touch input and display a drawing on the screen by performing the following operation.

[0070] In operation S301, the electronic device (1100) can detect a first touch input. In one embodiment, the first touch input can be detected on a touchscreen of the electronic device (1100). In one embodiment, the first touch input can be detected on a display of the electronic device (1100).

[0071] In one embodiment, the electronic device (1100) can detect a first touch input input using an input device (100) (or a finger). In one embodiment, the first touch input can be detected on a touchscreen or display of the electronic device (1100).

[0072] In one embodiment, the input device (100) (or finger) may interact with the electronic device (1100) using methods such as pressure, electromagnetic induction, or Bluetooth. In one embodiment, the input device (100) (or finger) may be used to draw a dot or a line on the touchscreen or display of the electronic device (1100).

[0073] In one embodiment, the electronic device (1100) can detect a first touch input corresponding to a drawing of the input device (100) (or finger). In one embodiment, the electronic device (1100) can detect a first touch input based on a drawing of the input device (100) (or finger).

[0074] In one embodiment, the detection of the first touch input may be a concept that includes both the detection of the first touch input and the processing of the first touch input. In one embodiment, the detection of the first touch input may be a concept that includes both the detection of the first touch input and the preprocessing of the first touch input.

[0075] In one embodiment, the electronic device (1100) can process the detected first touch input. In one embodiment, the electronic device (1100) can preprocess the detected first touch input. In one embodiment, a CPU included in the electronic device (1100) can process the detected first touch input.

[0076] In one embodiment, the electronic device (1100) may require time to process the detected first touch input. In one embodiment, the CPU included in the electronic device (1100) may require time to process the detected first touch input.

[0077] In one embodiment, the electronic device (1100) can detect the three-dimensional coordinates of the first touch input. In one embodiment, the electronic device (1100) can interact with the input device (100) to detect the three-dimensional coordinates indicated by the pen tip of the input device (100). In one embodiment, the electronic device (1100) can display two-dimensional coordinates on a display based on the three-dimensional coordinates of the first touch input.

[0078] In one embodiment, the electronic device (1100) can analyze the time interval between touch inputs. The electronic device (1100) can analyze the pattern of the time interval between touch inputs. The electronic device (1100) can identify that the time interval between touch inputs is not constant and is variable. The electronic device (1100) can analyze the time pattern and trend between touch inputs.

[0079] In one embodiment, the electronic device (1100) can estimate a seventh time when a second touch input is detected based on the pattern of time intervals analyzed. The electronic device (1100) can estimate whether the second touch input will be detected within a short time or after a long time based on the pattern of time intervals between touch inputs.

[0080] In one embodiment, the electronic device (1100) can perform a prediction based on the time interval and pattern between touch inputs.

[0081] In operation S302, the electronic device (1100) can estimate a first time, which is the time from when the first touch input is detected until the screen is refreshed. For example, if the display has a screen refresh frequency of 60Hz and the time at which the first touch input is detected is 1ms, the first time, which is the time from when the first touch input is detected until the screen is refreshed, may be 15.57ms.

[0082] In one embodiment, the electronic device (1100) can calculate the time for the screen to be refreshed based on the screen refresh rate of the display. In one embodiment, the screen refresh rate may be expressed as a screen refresh rate, screen refresh frequency, number of screen refresh frequencies, or screen refresh rate, but is not limited thereto. For example, if the screen refresh rate is 60Hz, the refresh time may be 16.67ms.

[0083] In one embodiment, the electronic device (1100) can estimate a first time, which is the time from when the processing of the first touch input is completed until the screen is refreshed. In one embodiment, the electronic device (1100) can estimate a first time, which is the time from when the processing of the first touch input is completed in the CPU until the screen is refreshed.

[0084] In one embodiment, the electronic device (1100) can estimate a first time, which is the time from when the detection and processing of the first touch input is completed until the screen is refreshed. In one embodiment, the electronic device (1100) can estimate a first time, which is the time from when the detection and processing of the first touch input is completed in the CPU until the screen is refreshed.

[0085] In one embodiment, the electronic device (1100) can estimate a first time, which is the time from when the preprocessing of the first touch input is completed until the screen is refreshed. In one embodiment, the electronic device (1100) can estimate a first time, which is the time from when the preprocessing of the first touch input is completed in the CPU until the screen is refreshed.

[0086] In one embodiment, the electronic device (1100) can estimate a first time, which is the time from when the detection and preprocessing of the first touch input is completed until the screen is refreshed. In one embodiment, the electronic device (1100) can estimate a first time, which is the time from when the detection and preprocessing of the first touch input is completed in the CPU until the screen is refreshed.

[0087] In operation S303, the electronic device (1100) can estimate a second time, which is the time required to render a first drawing including a first drawing point corresponding to a first touch input. The electronic device (1100) can estimate a second time, which is the time required to render the first drawing based on the first drawing point corresponding to the first touch input. In other words, the electronic device (1100) can estimate a second time, which is the time required to render the first drawing based on the first drawing point corresponding to the first touch input.

[0088] In one embodiment, the second time may refer to the time required for the CPU to generate and prepare the first drawing and transmit data for the first drawing to the GPU after the prediction model is executed for the first touch input. In one embodiment, the second time may refer to the time required to display the first drawing on the screen.

[0089] In one embodiment, the first drawing point may refer to a point where the user predicts where to draw a line or how the drawing will proceed based on the drawing drawn so far, past drawing history, drawing data corresponding to past touch inputs, and the first touch input. In addition, in one embodiment, the first drawing including the first drawing point may refer to a drawing that proceeds to the predicted first drawing point. For example, if the first drawing point is predicted while the user is drawing a line, the line extended to the predicted first drawing point may correspond to the first drawing. In one embodiment, rendering may refer to a process of processing data or commands to generate a visual result that is finally output to the screen.

[0090] In one embodiment, the electronic device (1100) may estimate a time interval excluding a second time from a first time. In one embodiment, this time interval may mean a time interval from when a first touch input is detected until the creation of a first drawing begins. In one embodiment, this time interval may mean a time interval from when a first touch input is detected and processed until the preparation or creation of a first drawing begins.

[0091] In one embodiment, the time interval excluding the second time from the first time may represent a substantial time during which a prediction for the first touch input can be performed. By using this estimated time interval, the electronic device (1100) can select a prediction model capable of performing a prediction for the first touch input.

[0092] In operation S304, the electronic device (1100) may select at least one first prediction model based on a refresh time and a first drawing time. In one embodiment, the electronic device may select at least one first prediction model that can be performed within a time constraint.

[0093] In one embodiment, the electronic device (1100) may select a first prediction model for predicting a first drawing point based on a first time and a second time. In one embodiment, the electronic device (1100) may select at least one first prediction model that can be performed within a time interval excluding the first time and the second time.

[0094] In one embodiment, the first prediction model may be a neural network model trained to predict a first drawing point corresponding to the first touch input using past touch inputs as training data. The first prediction model may be trained using drawing data corresponding to past touch inputs as training data. The first prediction model may receive the first touch input as input, predict the first drawing point, and output it. In one embodiment, the prediction model may be referred to as a prediction algorithm, a prediction neural network model, a prediction artificial intelligence, etc., but is not limited thereto.

[0095] In one embodiment, the first prediction model can predict the location of the next point to be drawn by learning the user's past drawing patterns or user behavior data. In one embodiment, the first prediction model can provide a fast and smooth response without delay by predicting the location of the next point to be drawn based on the current movement pattern.

[0096] In operation S305, the electronic device (1100) can predict a first drawing point from a first touch input using a selected first prediction model. In one embodiment, the drawing data may include the user's past drawing patterns, past drawing history, user behavior data, and user movement patterns.

[0097] In one embodiment, the electronic device (1100) can perform a prediction for a first touch input using at least one first prediction model. In one embodiment, the electronic device (1100) can perform a prediction for a first touch input using at least one selected first prediction model. In one embodiment, the prediction models may have different processing times and accuracy for each prediction model.

[0098] In one embodiment, the electronic device (1100) can perform a prediction for a first touch input using two or more first prediction models. In one embodiment, the electronic device (1100) can predict a first drawing point for a first touch input using two or more first prediction models.

[0099] In one embodiment, predicting a first drawing point corresponding to a first touch input may mean predicting the position of a point to be drawn after the first touch input. In one embodiment, the electronic device (1100) may calculate the position of a point to be drawn after the first touch input using at least one first prediction model based on the first touch input and past touch inputs.

[0100] In one embodiment, the first prediction model can output a first drawing point using a first touch input as input data. The first prediction model can output a first drawing point predicted based on the first touch input.

[0101] In one embodiment, the electronic device (1100) can obtain a first prediction result for a first drawing point predicted from drawing data using a selected first prediction model. In one embodiment, the first prediction result may include information about the location of a point to be drawn after a first touch input.

[0102] In one embodiment, the electronic device (1100) can obtain at least one first prediction result regarding the position of a point to be drawn after at least the first touch input using a first prediction model, based on the first touch input as well as past touch inputs, the user's past drawing pattern, or user behavior data.

[0103] In one embodiment, the electronic device (1100) may select a first drawing point based on a first time, a second time, accuracy, and error probability. In one embodiment, the electronic device (1100) may select one of at least one first prediction result by considering time constraints, accuracy criteria, Key Performance Indicator (KPI) criteria, and error probability, etc.

[0104] In one embodiment, the electronic device (1100) can select the most appropriate prediction result among several prediction results by considering the time constraints excluding the second time from the first time and the accuracy and error probability according to each prediction model.

[0105] In one embodiment, the electronic device (1100) may determine whether to perform a prediction for a first touch input based on a first time, a second time, and a seventh time. The electronic device (1100) may determine whether to perform a prediction for a first touch input based on a time when a prediction can be performed for the first touch input and a time when a second touch input is expected to be detected.

[0106] In one embodiment, the electronic device (1100) may decide not to perform a prediction for the first touch input if the second touch input is expected to be detected within a short time. The electronic device (1100) may decide to perform a prediction for the first touch input if the second touch input is expected to be detected after a considerable time.

[0107] In one embodiment, if the electronic device (1100) decides not to perform a prediction for the first touch input, it may stop performing a prediction for the first touch input until the second touch input is detected. The electronic device (1100) may perform a prediction for the first touch input and the second touch input simultaneously after the second touch input is detected.

[0108] In operation S306, the electronic device (1100) can prepare a first drawing. In one embodiment, the electronic device (1100) can generate a first drawing based on a first drawing point. In one embodiment, a CPU or GPU included in the electronic device can generate a first drawing based on a first drawing point.

[0109] In one embodiment, the electronic device (1100) can generate a curve or straight line connecting the first touch input and the first drawing point. The CPU or GPU of the electronic device (1100) can generate a curve or straight line connecting the first touch input and the first drawing point.

[0110] In operation S307, the electronic device (1100) can transmit data for the first drawing to the GPU. In one embodiment, the CPU included in the electronic device (1100) can transmit data for the first drawing to the GPU.

[0111] In one embodiment, data for the first drawing may include data prepared and generated by a CPU or GPU for the first drawing. Data for the first drawing may include data necessary to display the first drawing on a screen. Data for the first drawing may include data necessary to render the first drawing.

[0112] In one embodiment, the electronic device (1100) can transmit data for the first drawing to the GPU. In one embodiment, the CPU included in the electronic device (1100) can transmit data for the first drawing to the GPU.

[0113] In one embodiment, the electronic device (1100) can display a first drawing generated based on a first drawing point on a screen. The electronic device (1100) can display a first drawing generated based on a selected first drawing point on a screen.

[0114] In one embodiment, the GPU can process to display the first drawing on the screen based on data regarding the first drawing. In one embodiment, the GPU can process to display the first drawing on the screen based on data regarding the first drawing.

[0115] In one embodiment, the electronic device (1100) can render data for the first drawing to display the first drawing on a screen. In one embodiment, the GPU of the electronic device (1100) can render data for the first drawing to display the first drawing on a screen.

[0116] In one embodiment, the electronic device (1100) can display a first drawing generated based on the three-dimensional coordinates of a first touch input on a display as two-dimensional coordinates.

[0117] In one embodiment, the electronic device (1100) can adjust the screen refresh time so that the time at which the first drawing is displayed on the screen and the time at which the screen is refreshed are synchronized.

[0118] In one embodiment, the electronic device (1100) may delay the screen refresh time if it is expected that the time for displaying the first drawing on the screen will be later than the screen refresh time. The electronic device (1100) may advance the screen refresh time if it is expected that the time for displaying the first drawing on the screen will be earlier than the screen refresh time.

[0119] FIG. 4 is a diagram for illustrating, in a timeline, the process of processing a single touch input according to one embodiment of the present disclosure.

[0120] FIG. 4 shows a time diagram including a histogram of touch input detection, processing of touch input, prediction execution, preparation for drawing, transfer of drawing data, and drawing in an electronic device (1100). The time delay from touch input to drawing can be caused by several factors. The main factors of the time delay are explained below.

[0121] Referring to 401, the electronic device (1100) can detect a first touch input. The electronic device (1100) may require time to detect the first touch input. The electronic device (1100) may experience a time delay in detecting the first touch input. For example, the time delay caused by the detection of the first touch input can be denoted as T1.

[0122] In one embodiment, the time delay (T1) resulting from the detection of the first touch input may vary depending on the hardware used. For example, the hardware may be various types and forms of electronic devices including touchscreens, and may include smart TVs, smartphones, tablet PCs, laptop PCs, glasses-type displays, head-mounted displays (HMDs), etc. The hardware may have different sensitivities and delays for each hardware. For example, a highly optimized touchscreen response may occur between 0.1ms and 1ms.

[0123] In one embodiment, the time delay (T1) resulting from the detection of the first touch input may vary depending on the input device (100) or input means used on the touchscreen of the electronic device (1100). For example, the input device (100) may include a pointer, a stylus, an electronic pen, etc., and the input means may include a finger, a whole hand, a multi-finger gesture, or a mouse.

[0124] In one embodiment, the time delay (T1) caused by the detection of the first touch input may vary depending on the window manager that reads the touch event from the driver in the electronic device (1100) and sends the touch event to the appropriate application.

[0125] In one embodiment, the electronic device (1100) can detect the coordinates where the input device (100) (or finger) presses the touchscreen. The electronic device (1100) can receive data such as pressure, tilt, and speed from the input device (100).

[0126] Referring to 402, the electronic device (1100) can process a first touch input. The electronic device (1100) may require time to process the first touch input. The electronic device (1100) may experience a time delay in processing the first touch input. For example, the time delay caused by processing the first touch input can be represented as T2.

[0127] In one embodiment, the CPU of the electronic device (1100) can process the first touch input. In one embodiment, an application within the electronic device (1100) can process the first touch input. In one embodiment, the time delay (T2) resulting from the processing of the first touch input may vary depending on the load of the CPU. The load of the CPU may include the number of parallel processes and the CPU utilization rate.

[0128] In one embodiment, the processing of the first touch input may be suspended for a significant amount of time due to CPU power consumption by a process other than the first touch input. The processing of the first touch input may further include a rendering task. The rendering task and the processing task are tasks executed on two separate threads, and additional delay may occur due to the synchronization of the two tasks.

[0129] In one embodiment, the electronic device (1100) can transmit a touch event to an operating system or application so that an appropriate processing routine is executed. The electronic device (1100) can determine how the system responds to the touch event and can check whether a touch has occurred or ended.

[0130] In one embodiment, the time difference between touch inputs may not be constant. The time interval between processing of touch inputs may not be constant. In one embodiment, the electronic device (1100) may perform a prediction by taking into account the non-constant time interval between touch inputs. The electronic device (1100) may perform a prediction by taking into account the non-constant time interval between processing of touch inputs.

[0131] In one embodiment, the electronic device (1100) can detect a time interval between touch inputs and a trend of the time intervals. In one embodiment, the electronic device (1100) can detect a time interval between touch input processing and a trend of the time intervals.

[0132] In one embodiment, the electronic device (1100) can select a prediction model and perform a prediction based on the time interval between touch inputs and the trend of the time intervals. In one embodiment, the electronic device (1100) can select a prediction model and perform a prediction based on the time interval between touch input processing and the trend of the time intervals.

[0133] In one embodiment, when the time interval between touch inputs is short, the electronic device (1100) can detect two or more touch inputs within one refresh time. The electronic device (1100) can select a prediction model that can be performed within the time from when two or more touch inputs are detected until a screen refresh occurs. The electronic device (1100) can perform predictions for two or more touch inputs simultaneously. The electronic device (1100) can perform predictions by grouping two or more touch inputs into one.

[0134] In one embodiment, the electronic device (1100) may not perform a prediction for at least one of the two or more touch inputs if at least one of the two or more touch inputs cannot be performed within the screen refresh time. The at least one touch input for which a prediction was not performed may be predicted within a new screen refresh time immediately after a screen refresh occurs.

[0135] In one embodiment, the electronic device (1100) may not process at least one of the two or more touch inputs if at least one of the two or more touch inputs cannot be performed within the screen refresh time. The at least one touch input for which no prediction was performed may be processed within a new screen refresh time immediately after the screen refresh occurs.

[0136] In one embodiment, when the time interval between touch input processing is short, the electronic device (1100) may select a prediction model that can be performed within the time until a screen refresh occurs for two or more processed touch inputs. If the prediction cannot be performed for two or more processed touch inputs within the screen refresh time, the electronic device (1100) may not perform a prediction for at least one of the two or more processed touch inputs.

[0137] In one embodiment, if the time interval between touch inputs is expected to be short, the electronic device (1100) may not perform a prediction model for one touch input until the next touch input is detected. If the time interval between touch inputs is expected to be short, the electronic device (1100) may not process one touch input until the next touch input is detected.

[0138] In one embodiment, if the time interval between touch input processing is expected to be short, the electronic device (1100) may not perform a prediction for one touch input until the next touch input is detected.

[0139] In one embodiment, when the time interval between touch inputs is long, the electronic device (1100) can perform prediction two or more times for a single touch input. By performing prediction two or more times for a single touch input, the electronic device (1100) can predict the drawing point corresponding to the touch input more accurately.

[0140] In one embodiment, if the time interval between touch inputs is long, the electronic device (1100) may not perform a prediction for at least one touch input. The at least one touch input for which a prediction was not performed may be predicted within a new screen refresh time immediately after a screen refresh occurs.

[0141] In one embodiment, if the time interval between touch inputs is long, the electronic device (1100) may not process at least one touch input. The at least one unprocessed touch input may be processed within a new screen refresh time immediately after a screen refresh occurs.

[0142] In one embodiment, when the time interval between touch inputs is expected to be long, the electronic device (1100) may perform two or more predictions for a single touch input. By performing two or more predictions for a single touch input, the electronic device (1100) can predict the drawing point corresponding to the single touch input more accurately.

[0143] Referring to 403, the electronic device (1100) can perform a prediction using a first prediction model for a first touch input. The electronic device (1100) may require time to perform a prediction for the first touch input. The electronic device (1100) may experience a time delay in performing a prediction for the first touch input. For example, the time delay caused by performing a prediction for the first touch input can be represented as T3.

[0144] In one embodiment, the electronic device (1100) can estimate a first time, which is the time from when a first touch input is detected until the screen is refreshed. The electronic device (1100) can estimate a second time, which is the time required to render a first drawing including a first drawing point corresponding to the first touch input. The electronic device (1100) can estimate a second time, which is the time required to render a first drawing based on the first drawing point corresponding to the first touch input.

[0145] In one embodiment, the electronic device (1100) may select a first prediction model for predicting a first drawing point based on a first time and a second time. The electronic device (1100) may predict a first drawing point from a first touch input using the selected first prediction model. The electronic device (1100) may calculate a first drawing point from drawing data using the selected first prediction model.

[0146] In one embodiment, the prediction using the first prediction model may be performed on a CPU, GPU, NPU (Neural Processing Unit), or FPGA (Field Programmable Gate Array) within the electronic device (1100), or may be performed in a combination of two or more of these. However, it is not limited thereto.

[0147] In one embodiment, if the number of tasks is significantly small, or the processing time is shorter than that of processing on a GPU, or is optimal from a system perspective, the prediction using the first prediction model may be performed on a CPU within the electronic device (1100). If the prediction is performed on a GPU, NPU, or FPGA, the prediction may require time for the touch event to be transmitted from the CPU to the GPU, NPU, or FPGA, and for the prediction to be performed and the prediction result to be transmitted back to the CPU.

[0148] In one embodiment, the electronic device (1100) can obtain at least one first prediction result by performing a prediction using a first prediction model. By performing a prediction, the electronic device (1100) can obtain at least one first prediction result for a first drawing point corresponding to a first touch input.

[0149] In one embodiment, the prediction model may include a prediction algorithm. The prediction model may be executed on a CPU or a GPU. In one embodiment, the electronic device (1100) may use the prediction model to predict a drawing point corresponding to a touch input. A drawing point may refer to the next point to be drawn on the screen in response to a touch input.

[0150] Next, I would like to explain four examples of prediction models.

[0151] In one embodiment, the prediction model may include a first neural network model. The first neural network model may be executed on a GPU and, for example, may output three predicted X and Y coordinates based on three input data. The first neural network model may be used when drawing starts or when the first touch input is detected, in that it requires three input data.

[0152] For example, a statistical analysis of the processing time of the first neural network model may show a maximum value of 2.213 ms, an average value of 0.493 ms, a median value of 0.445 ms, and a standard deviation of 0.1884 ms. Based on these statistics, it can be expected that there is a 95% probability that the first neural network model will be executed within 0.822 ms.

[0153] In one embodiment, the prediction model may include a second neural network model. The second neural network model may be executed on a GPU and may have a larger scale and provide better predictions compared to the first neural network model. For example, the second neural network model may output six predicted X, Y coordinates based on eight input data. The second neural network model may be slower and require more data compared to the first neural network model.

[0154] For example, a statistical analysis of the data regarding the processing time of the second neural network model may show a maximum value of 10.638ms, an average value of 2.208ms, a median value of 2.073ms, and a standard deviation of 0.554ms. Based on these statistics, it can be expected that there is a 95% probability that the second neural network model will be executed within 3.181ms.

[0155] In one embodiment, since the first neural network model and the second neural network model are executed on the GPU, touch input data may need to be received from the CPU, processed on the GPU, and then transmitted back to the CPU. Time delay may occur in the first neural network model and the second neural network model because data is transmitted and received between the CPU and the GPU.

[0156] In one embodiment, the prediction model may include a linear extrapolation model. The linear extrapolation model can be executed on a CPU, has low computational complexity, and can output fast results. The linear extrapolation model can be used when touch input generates a straight line, and accordingly, can output a prediction result in the form of a straight line.

[0157] For example, a statistical analysis of the data regarding the processing time of a linear extrapolation model may show a maximum value of 0.714 ms, an average value of 0.0078 ms, a median value of 0.006 ms, and a standard deviation of 0.0194 ms. Based on these statistics, it can be expected that there is a 95% probability that the linear extrapolation model will be completed within 0.0449 ms.

[0158] In one embodiment, the prediction model may include a Kalman filter model. A Kalman filter model may refer to an algorithm that predicts a state changing over time and modifies the prediction by comparing it with observed data. A Kalman filter model can be used to estimate results from input data containing noise or to predict future states. A Kalman filter model can be used on a CPU, and its performance may be low when acceleration or deceleration occurs or when the linear direction changes rapidly. A Kalman filter model may require more time compared to a linear extrapolation model.

[0159] For example, a statistical analysis of the data regarding the processing time of the Kalman filter model may show a maximum value of 0.242 ms, an average value of 0.005 ms, a median value of 0.0037 ms, and a standard deviation of 0.0109 ms. Based on these statistics, it can be expected that the Kalman filter model will be executed within 0.0255 ms with a 95% probability.

[0160] Prediction models are not limited to the four examples above and may include more neural network models.

[0161] In one embodiment, the electronic device (1100) can select a prediction model that can be executed within a screen refresh time and perform a prediction by considering the probability distribution of processing times of a plurality of prediction models. The electronic device (1100) can use various values ​​as thresholds, such as 90% or 98% of the probability of execution.

[0162] Referring to 404, the electronic device (1100) may prepare a first drawing. The electronic device (1100) may require time to prepare the first drawing. The electronic device (1100) may experience a time delay in preparing the first drawing. For example, the time delay caused by the preparation of the first drawing may be represented as T4.

[0163] In one embodiment, the electronic device (1100) may prepare content to be displayed on a screen based on a first drawing point. The electronic device (1100) may prepare a first drawing to be displayed on a screen. The electronic device (1100) may generate a first drawing to be displayed on a screen. The operation of preparing or generating the first drawing may be performed on a CPU or GPU included in the electronic device (1100). However, it is not limited thereto.

[0164] In one embodiment, the first drawing may refer to a line drawn from a first touch input point to a first drawing point. The first drawing may refer to a line drawn from past touch inputs to the first touch input. The first drawing may include both the line drawn from the first touch input point to the first drawing point and the line drawn from past touch inputs to the first touch input.

[0165] In one embodiment, the electronic device (1100) may generate a smoothed line using various smoothing algorithms. Generating a smoothed line may mean connecting a line drawn from a first touch input point to a first drawing point with a line drawn from past touch inputs to the first touch input point. Generating a smoothed line may mean maintaining the curvature of the line at the connection point.

[0166] In one embodiment, the operation of creating or preparing a first drawing may include the operation of creating appropriate vertices that form a triangle, the operation of transmitting data for the first drawing to a GPU, and the operations of rendering and rasterization.

[0167] In one embodiment, rendering may refer to the process of converting data of various formats, such as 3D models, images, videos, and text, into a final form that can be viewed by humans. In one embodiment, rasterization may refer to the process of converting 3D models or vector graphic data into pixel-unit images to display on a 2D screen in computer graphics.

[0168] In one embodiment, data for a first drawing may refer to data that enables the GPU to display the first drawing on a screen. Data for a first drawing may include a command to the GPU to display the first drawing on a screen. Data for a first drawing may include data necessary for rendering the first drawing.

[0169] Referring to 405, the electronic device (1100) can transmit data for the first drawing to the GPU. In one embodiment, the electronic device (1100) can transmit data for the first drawing to the GPU. The electronic device (1100) may require time to transmit data for the first drawing to the GPU. The electronic device (1100) may experience a time delay in transmitting data for the first drawing to the GPU. For example, the time delay caused by the transmission of data for the first drawing can be represented as T5.

[0170] In one embodiment, the operation of transferring data for the first drawing to the GPU may include the operation of copying data for the first drawing from the CPU to the GPU. The time delay (T5) caused by the transfer of data for the first drawing may require more time as the amount of data to be copied increases.

[0171] Referring to 406, the electronic device (1100) can display a first drawing generated based on a first drawing point on a screen. The electronic device (1100) can display the first drawing on a display. The electronic device (1100) may require time to display the first drawing on a display. The electronic device (1100) may experience a time delay in displaying the first drawing on a display. For example, the time delay caused by the screen display of the first drawing can be represented as T6.

[0172] In one embodiment, the electronic device (1100) can adjust the screen refresh time so that the time for displaying the first drawing on the screen and the time for refreshing the screen are synchronized. In one embodiment, synchronization may include both displaying the first drawing on the screen within the screen refresh time and displaying the first drawing on the screen at the exact time for refreshing the screen.

[0173] In one embodiment, the time delay (T6) caused by the display of the first drawing may include the time required to transfer data required for rendering from the CPU to the GPU and for the GPU to update its internal state to complete all drawing commands.

[0174] The present disclosure presents a technical solution for compensating for various types of delays described in FIG. 4. The technical solution of the present disclosure can be briefly introduced as follows.

[0175] Each step and process described in Fig. 4 has unpredictability when considering the touch input detection and processing time, prediction execution time, and drawing time. To determine when and which touch events should be processed and predicted, the following actions can be performed.

[0176] In one embodiment, the electronic device (1100) can calculate times T1 and T2 by taking into account the average delay time and standard deviation of the touch input. If times T1 and T2 are stable, the electronic device (1100) can replace time T1 + T2 with a single value of time T. The electronic device (1100) can calculate the time available until an operation for times T4 and T5 is performed. The electronic device (1100) can calculate a given time constraint based on the time until the screen refresh time and times T4 and T5.

[0177] In one embodiment, the electronic device (1100) can determine a prediction model available within a given time constraint. The electronic device (1100) can determine the prediction model by analyzing the time required to execute the prediction model.

[0178] In one embodiment, the electronic device (1100) may perform a prediction using one or more prediction models. If additional touch input occurs within a given time constraint, the electronic device (1100) may decide whether to recalculate the prediction with the new touch input. The electronic device (1100) may decide whether to perform a prediction for the new touch input.

[0179] In one embodiment, the electronic device (1100) can determine whether to calculate a prediction with a new touch input by analyzing the time required to perform the prediction. The electronic device (1100) can determine whether to perform a prediction for a new touch input by analyzing the time required to perform the prediction.

[0180] In one embodiment, if a touch input is detected after times T4 and T5, the electronic device (1100) may not process the touch input and may pass it to the next screen refresh time. If a new touch input is detected after times T4 and T5, the electronic device (1100) may not perform a prediction on the new touch input and may pass it to the next screen refresh time.

[0181] In one embodiment, the electronic device (1100) may select one of at least one prediction result obtained by performing a prediction. The selected prediction result may be used to generate and prepare a drawing (T4), and the drawing data may be transmitted to a GPU (T5) and displayed (T6).

[0182] In one embodiment, a screen refresh may not occur within a predicted refresh time. The electronic device (1100) may transmit drawing data to the GPU after the predicted refresh time. If a screen refresh does not occur until the drawing data is transmitted to and processed by the GPU, the drawing may be displayed on the screen. The reason a screen refresh does not occur within the predicted refresh time may be power consumption of the CPU or GPU.

[0183] Through the process described above, the delay from touch input to drawing can be compensated. Since the delay is not a fixed value but varies depending on various factors, prediction models may have different processing times and accuracy. To reduce errors that may occur on the screen of an electronic device, methods from various prediction models can be combined. Additionally, by reducing CPU and GPU usage, the load on the CPU and GPU used in the background can be decreased, thereby ensuring the stability of future predictions.

[0184] FIG. 5 is a flowchart illustrating a method for processing two or more touch inputs according to one embodiment of the present disclosure.

[0185] Referring to FIG. 5, if two or more touch inputs are detected within the same screen refresh time, the electronic device (1100) can process the touch inputs and display a drawing on the screen by performing the following operations. Content redundant with FIG. 3 and FIG. 4 is omitted.

[0186] In operation S501, the electronic device (1100) can detect a first touch input. The electronic device (1100) can detect a first touch input through a touchscreen. The electronic device (1100) can detect a first touch input through a display. The electronic device (1100) can detect a first touch input entered through an input device (100) (or a finger).

[0187] In operation S502, the electronic device (1100) can estimate a first time, which is the time from when the first touch input is detected until the screen is refreshed. The electronic device (1100) can estimate a time interval from when the first touch input is detected and processed until the earliest time when the screen is refreshed.

[0188] In operation S503, the electronic device (1100) may estimate a second time, which is the time required to render a first drawing including a first drawing point corresponding to a first touch input. The second time may include the time required to prepare and create the first drawing and to transmit data for the first drawing to the GPU.

[0189] In operation S504, the electronic device (1100) may select a first prediction model for predicting a first drawing point based on a first time and a second time. The electronic device (1100) may select at least one first prediction model among a plurality of prediction models that can perform a prediction within a time period excluding the second time from the first time.

[0190] In operation S505, the electronic device (1100) can predict a first drawing point from a first touch input using a selected first prediction model. The electronic device (1100) can predict a first drawing point based on a history of generating drawings according to past touch inputs. The electronic device (1100) can obtain at least one first drawing point using each first prediction model having different speeds, times, and accuracys.

[0191] In operation S506, the electronic device (1100) can determine whether a new touch input has been detected. It is assumed that the new touch input is detected within the same screen refresh time as the first touch input. If the new touch input is detected after the same screen refresh time as the first touch input, it is a touch input at the next screen refresh time, and in such case, the electronic device (1100) can perform the same or similar operation as the embodiment of FIG. 3.

[0192] In one embodiment, the electronic device (1100) may operate a timer. The timer may be a timer that operates until a new touch input can be processed. The timer may be a timer that expires when the time is the first time minus the second time. The timer may be a timer that expires when the last time for processing a new touch input is reached. In one embodiment, the time for processing a second touch input may mean the time remaining until the screen is refreshed minus the second time. In one embodiment, the time for processing a second touch input may be the time the first time minus the second time.

[0193] In one embodiment, when the timer expires, the electronic device (1100) may consider that there is no new touch input and display a first drawing generated based on a first drawing point on the screen. When the timer expires, the electronic device (1100) may prepare a first drawing based on a first drawing point corresponding to the first touch input, transmit data for the first drawing to the GPU, and then display the first drawing on the screen.

[0194] In one embodiment, when it is determined that a new touch input has been detected, the electronic device (1100) may process the new touch input. The new touch input may include a second touch input. The electronic device (1100) may detect the second touch input. The second touch input may be a touch input detected within the same screen refresh time as the first touch input.

[0195] In one embodiment, the first drawing point may be replaced by a second touch input. The first drawing point refers to a point predicted to be drawn after the first touch input, and since the second touch input is detected after the first touch input, the first drawing point may not be used to perform the drawing.

[0196] In operation S507, the electronic device (1100) can estimate a third time, which is the time from when the second touch input is detected until the screen is refreshed. The electronic device (1100) can estimate a time interval from when the second touch input is detected and processed until the earliest time when the screen is refreshed.

[0197] In operation S508, the electronic device (1100) may estimate a fourth time, which is the time required to render a second drawing including a second drawing point corresponding to a second touch input. The fourth time may include the time required to prepare and create the second drawing and to transmit data for the second drawing to the GPU.

[0198] In one embodiment, the second drawing point may refer to a point predicted to be drawn after the second touch input, based on drawing data corresponding to past touch inputs. In one embodiment, the second drawing may include a line connecting the second touch input to a line drawn by past touch inputs, including the first touch input, and a line connecting the second touch input to the second drawing point. The line connecting the second touch input to the second drawing point is a line predicted to be drawn and may be represented in a different color.

[0199] In operation S509, the electronic device (1100) may determine whether to perform a prediction for the second touch input based on the third time and the fourth time. The electronic device may determine whether there is a prediction model capable of performing a prediction for the second touch input within the time excluding the fourth time from the third time.

[0200] In one embodiment, if it is determined that there is a prediction model capable of performing a prediction for the second touch input within a given time, the electronic device (1100) may determine that it can perform a prediction for the second touch input. If it is determined that there is no prediction model capable of performing a prediction for the second touch input within a given time, the electronic device (1100) may decide not to perform a prediction for the second touch input.

[0201] In operation S510, if it is decided to perform a prediction for the second touch input, the electronic device (1100) may select a second prediction model for predicting the second drawing point based on the third time and the fourth time. The electronic device (1100) may select at least one second prediction model among a plurality of prediction models that can perform a prediction within the time excluding the third time and the fourth time.

[0202] In one embodiment, the second prediction model may be a neural network model trained to predict a second drawing point corresponding to the second touch input using past touch inputs as training data. The second prediction model may be trained using drawing data corresponding to past touch inputs as training data. The second prediction model may receive the second touch input as input, predict the second drawing point, and output it.

[0203] In operation S511, the electronic device (1100) can predict a second drawing point from a second touch input using a selected second prediction model. The electronic device (1100) can predict a second drawing point based on a history of generating drawings according to past touch inputs. The electronic device (1100) can obtain at least one second drawing point using each second prediction model having different speeds, times, and accuracys.

[0204] In operation S512, the electronic device (1100) can display a second drawing generated based on a second drawing point on a screen. The electronic device (1100) can render the second drawing by preparing and generating the second drawing based on the second drawing point and transmitting data for the second drawing to a GPU. The electronic device (1100) can display the rendered second drawing on a screen.

[0205] In operation S513, if it is decided not to perform a prediction for the second touch input, the electronic device (1100) may wait for the second touch input until the next screen refresh. If it is determined that even if a prediction is performed for the second touch input, it will not be completed by the screen refresh time, the electronic device (1100) may decide not to perform a prediction for the second touch input.

[0206] In one embodiment, if it is determined that even if a prediction is performed for the second touch input, it will not be completed by the time excluding the third time from the fourth time, the electronic device (1100) may decide not to perform a prediction for the second touch input. If it is determined that even if a prediction is performed for the second touch input, the second drawing will not be displayed on the screen by the screen refresh time, the electronic device (1100) may decide not to perform a prediction for the second touch input.

[0207] In one embodiment, if it is decided not to perform a prediction for a second touch input, the electronic device (1100) may perform a prediction for a second touch input after the screen has been refreshed. If it is determined that a second touch input is detected at a time too late to perform a prediction for a second touch input, the electronic device (1100) may decide not to perform a prediction for a second touch input.

[0208] In one embodiment, if it is determined that a prediction for the second touch input cannot be performed within the time period excluding the third time and the fourth time, the electronic device (1100) may decide not to perform a prediction for the second touch input. If it is determined that the electronic device (1100) will not be able to display the second drawing based on the second touch input on the screen by the third time, the electronic device may decide not to perform a prediction for the second touch input.

[0209] In one embodiment, when the next screen refresh is initiated, the electronic device (1100) can perform a prediction for a second touch input.

[0210] In operation S514, if it is determined that no new touch input is detected, the electronic device (1100) may display a first drawing generated based on a first drawing point on the screen. The electronic device (1100) may render the first drawing by preparing and generating the first drawing based on the first drawing point and transmitting data for the first drawing to the GPU. The electronic device (1100) may display the rendered first drawing on the screen.

[0211] FIG. 6 is a diagram for illustrating, in a timeline, the process of processing two or more touch inputs according to one embodiment of the present disclosure.

[0212] Referring to FIG. 6, if two or more touch inputs are detected within the same refresh time, the electronic device (1100) can process the touch inputs and display a drawing on the screen by performing the following operations. Content redundant in FIG. 3, FIG. 4, and FIG. 5 is omitted.

[0213] Referring to 601, the electronic device (1100) can detect a first touch input. The electronic device (1100) may require time to detect the first touch input. The time delay resulting from the detection of the first touch input can be denoted as T1.

[0214] Referring to 602, the electronic device (1100) can process a first touch input. The electronic device (1100) may require time to process the first touch input. The time delay resulting from processing the first touch input can be denoted as T2.

[0215] In one embodiment, the electronic device (1100) may estimate a first time, which is the time from when a first touch input is detected until the screen is refreshed. The first time may refer to the time interval from time T2 until the time when the screen refresh occurs.

[0216] In one embodiment, the electronic device (1100) may estimate a second time, which is the time required to render a first drawing including a first drawing point corresponding to a first touch input. The second time may mean the sum of a time delay T4, which is the time delay caused by the preparation of the first drawing, and a time delay T5, which is the time required to transfer data for the first drawing to the GPU.

[0217] In one embodiment, the electronic device (1100) may select a first prediction model for predicting a first drawing point based on a first time and a second time. The electronic device (1100) may calculate a time interval from after time T2 until before an operation corresponding to times T4 and T5 is performed. The electronic device (1100) may select a first prediction model that can be performed for a first touch input within this time interval.

[0218] Referring to 603, the electronic device (1100) can predict a first drawing point from a first touch input using a selected first prediction model. The electronic device (1100) may require time to perform the prediction for the first drawing point using the first prediction model. The time delay resulting from performing the prediction for the first drawing point can be represented as T3.

[0219] Referring to 604, the electronic device (1100) can detect a second touch input. The electronic device (1100) can detect the second touch input within a screen refresh time. The electronic device (1100) can detect the second touch input within a time when processing of the new touch input is possible. The time delay caused by the second touch input can be denoted as T1'.

[0220] In one embodiment, the time during which a new touch input can be processed may be the time interval from the time when a screen refresh occurs after the prediction using the first prediction model is performed, i.e., after time T3, excluding time T4 and time T5.

[0221] In one embodiment, if the second touch input is detected after a time has elapsed during which processing of the new touch input is possible, the electronic device (1100) may not perform a prediction for the second touch input. If it is decided not to perform a prediction for the second touch input, the prediction for the second touch input may be performed after the screen is refreshed.

[0222] Referring to 605, the electronic device (1100) can process a second touch input. The electronic device (1100) may require time to process the second touch input. The time delay caused by processing the second touch input can be denoted as T2'.

[0223] In one embodiment, the electronic device (1100) may estimate a third time, which is the time from when a second touch input is detected until the screen is refreshed. The third time may refer to the time interval from time T2' until the time when the screen refresh occurs.

[0224] In one embodiment, the electronic device (1100) may estimate a fourth time, which is the time required to render a second drawing including a second drawing point corresponding to a second touch input. In one embodiment, the electronic device (1100) may estimate a fourth time, which is the time required to render a second drawing based on a second drawing point corresponding to a second touch input. The fourth time may mean the sum of a time delay T5, which is the time delay caused by the preparation of the second drawing, and a time delay T6, which is the time required to transfer data for the second drawing to the GPU.

[0225] In one embodiment, the electronic device (1100) may determine whether to perform a prediction for the second touch input based on the third time and the fourth time. The electronic device (1100) may determine whether there is a second prediction model capable of performing a prediction for the second touch input during the time from after time T2' until before time T4 and time T5 are performed.

[0226] In one embodiment, if it is determined that there is no second prediction model capable of performing a prediction for the second touch input, the electronic device (1100) may not perform a prediction for the second touch input. If it is determined that there is a second prediction model capable of performing a prediction for the second touch input, the electronic device (1100) may perform the following operation.

[0227] In one embodiment, the electronic device (1100) may select a second prediction model for predicting a second drawing point based on a third time and a fourth time. The electronic device (1100) may calculate a time interval from after time T2' until an operation corresponding to times T4 and T5 is performed. The electronic device (1100) may select a second prediction model that can be performed for a second touch input within this time interval.

[0228] In one embodiment, the electronic device (1100) may select a second prediction model capable of performing a prediction faster than the last time at which an operation corresponding to time T4 can be performed. A prediction model that cannot be performed within a given time constraint on the processing time probability distribution of each prediction model may not be selected.

[0229] In one embodiment, a prediction model that can be executed within a given time constraint may produce a quick result but may have low accuracy. A prediction model with higher accuracy may produce a result too late with a non-zero probability. The electronic device (1100) may select an appropriate prediction model by taking these constraints into account. The electronic device may set a set of Key Performance Indicators (KPIs) and select a prediction model based on the set set of KPIs.

[0230] Referring to 606, the electronic device (1100) can predict a second drawing point from a second touch input using a selected second prediction model. The electronic device (1100) may require time to perform the prediction for the second drawing point using the second prediction model. The time delay resulting from performing the prediction for the second drawing point can be denoted as T3'.

[0231] Referring to 607, the electronic device (1100) may prepare a second drawing. The second drawing may include a line drawn by past touch inputs, including the first touch input, a line connecting the second touch input, and a line connecting the second touch input and the second drawing point. The time delay caused by the preparation of the second drawing may be denoted as T4.

[0232] In one embodiment, preparing the second drawing may mean generating a smoothed line using a smoothing algorithm to maintain the curvature of the line at the connection points. Additionally, preparing the second drawing may include the task of creating appropriate vertices that form triangles, the task of transmitting drawing data to the GPU, and the tasks of rendering and rasterization.

[0233] Referring to 608, the electronic device (1100) can transfer data for the second drawing to the GPU. Transferring data for the second drawing to the GPU may involve copying data for the second drawing from the CPU to the GPU. The electronic device may require time to transfer data for the second drawing to the GPU. The time delay caused by the transfer of data for the second drawing may be denoted as T5.

[0234] Referring to 609, the electronic device (1100) can display the second drawing on the screen. The electronic device (1100) may require time to display the second drawing on the display. The time delay caused by the display of the second drawing can be represented as T6.

[0235] In one embodiment, the electronic device (1100) can adjust the screen refresh time so that the time for displaying the second drawing on the screen and the screen refresh time are synchronized. In one embodiment, displaying the second drawing on the screen may include transferring data required for rendering from the CPU to the GPU, and the GPU updating its internal state to complete all drawing commands.

[0236] FIG. 7 is a flowchart illustrating a method for processing touch input based on a delay of screen refresh according to one embodiment of the present disclosure.

[0237] Referring to FIG. 7, the electronic device (1100) may perform the following operations in preparation for a case where screen refresh is delayed. Content overlapping with FIG. 3, 4, and 5 is omitted.

[0238] In operation S701, the electronic device (1100) can detect a first touch input. The electronic device (1100) can detect a first touch input through a touchscreen. The electronic device (1100) can detect a first touch input through a display. The electronic device (1100) can detect a first touch input entered through an input device (100) (or a finger).

[0239] In operation S702, the electronic device (1100) can estimate a first time, which is the time from when the first touch input is detected until the screen is refreshed. The electronic device (1100) can estimate a time interval from when the first touch input is detected and processed until the earliest time when the screen is refreshed.

[0240] In operation S703, the electronic device (1100) may estimate a second time, which is the time required to render a first drawing including a first drawing point corresponding to a first touch input. The second time may include the time required to prepare and create the first drawing and to transmit data for the first drawing to the GPU.

[0241] In operation S704, the electronic device (1100) may select a first prediction model for predicting a first drawing point based on a first time and a second time. The electronic device (1100) may select at least one first prediction model among a plurality of prediction models that can perform a prediction within a time period excluding the second time from the first time.

[0242] In operation S705, the electronic device (1100) can predict a first drawing point from a first touch input using a selected first prediction model. The electronic device (1100) can predict a first drawing point based on a history of generating drawings according to past touch inputs. The electronic device (1100) can obtain at least one first drawing point using each first prediction model having different speeds, times, and accuracys.

[0243] In operation S706, the electronic device (1100) can estimate a fifth time, which is the time during which the screen refresh is delayed. The electronic device (1100) can assume that the screen refresh will be delayed. For example, if the current screen refresh time is 16.67 ms (i.e., 60 Hz), the time during which the screen refresh is delayed may include 20 ms (i.e., 50 Hz), 25 ms (i.e., 40 Hz), 33.3 ms (i.e., 30 Hz), etc.

[0244] In operation S707, the electronic device (1100) can predict a third drawing point corresponding to a first touch input based on a first time, a second time, and a fifth time. The electronic device (1100) can predict a third drawing point corresponding to a first touch input by performing a new prediction for the first touch input based on a first time, a second time, and a fifth time. The electronic device (1100) can re-perform a prediction for the first touch input based on a first time, a second time, and a fifth time.

[0245] In one embodiment, the electronic device (1100) may select a prediction model capable of performing a prediction for a first touch input based on a refresh time delayed from the existing refresh time and the time required to render a third drawing.

[0246] In one embodiment, the electronic device (1100) can predict a third drawing point from drawing data using a selected prediction model. The selected prediction model may be a prediction model that can be executed within the time based on a delayed refresh time and a rendering time.

[0247] In operation S708, the electronic device (1100) can determine whether the refresh of the screen is delayed. The electronic device (1100) can determine that the refresh of the screen is delayed if the refresh does not occur even though the existing screen refresh time has elapsed.

[0248] In operation S709, if the refresh of the screen is delayed, the electronic device (1100) can display a third drawing generated based on the first drawing point and the third drawing point on the screen. The electronic device (1100) can obtain a more accurate prediction result based on the first drawing point and the third drawing point, for which two or more predictions have been performed for the first touch input.

[0249] In one embodiment, if the refresh of the screen is further delayed, the electronic device (1100) may perform a new prediction for the first touch input. Based on the prediction result performed, the prediction result for the first touch input may have a more accurate value.

[0250] In one embodiment, if a new touch input is detected while the screen refresh is delayed, the electronic device (1100) can perform a prediction for the new touch input. A detailed operation is implemented through the operation of FIG. 5.

[0251] In operation S710, if the screen refresh is not delayed, the electronic device (1100) can display the first drawing generated based on the first drawing point on the screen. If the screen refresh is not delayed and the refresh occurs according to the existing screen refresh time, the third drawing point may not yet be in an anticipated state. The electronic device (1100) can display the first drawing generated based on the first drawing point on the screen without the third drawing point.

[0252] FIG. 8 is a diagram illustrating, in timeline form, a process of processing touch input based on a delay in screen refresh according to one embodiment of the present disclosure.

[0253] Referring to FIG. 8, the electronic device (1100) may perform the following operations in preparation for a case where screen refresh is delayed. Content overlapping with FIG. 3, 4, 6, and 7 is omitted.

[0254] Referring to 801, the electronic device (1100) can detect a first touch input. The electronic device (1100) may require time to detect the first touch input. The time delay resulting from the detection of the first touch input can be denoted as T1.

[0255] Referring to 802, the electronic device (1100) can process a first touch input. The electronic device (1100) may require time to process the first touch input. The time delay resulting from processing the first touch input can be denoted as T2.

[0256] In one embodiment, the electronic device (1100) may estimate a first time, which is the time from when a first touch input is detected and processed until the screen is refreshed. The first time may refer to a time interval from time T2 until the time when the screen refresh occurs.

[0257] In one embodiment, the electronic device (1100) may estimate a second time, which is the time required to render a first drawing including a first drawing point corresponding to a first touch input. The second time may mean the sum of a time delay T4, which is the time delay caused by the preparation of the first drawing, and a time delay T5, which is the time required to transfer data for the first drawing to the GPU.

[0258] In one embodiment, the electronic device (1100) may select a first prediction model for predicting a first drawing point based on a first time and a second time. The electronic device (1100) may calculate a time interval from after time T2 until before an operation corresponding to times T4 and T5 is performed. The electronic device may select a first prediction model that can be performed for a first touch input within this time interval.

[0259] Referring to 803, the electronic device (1100) can predict a first drawing point from a first touch input using a selected first prediction model. The electronic device (1100) may require time to perform the prediction for the first drawing point using the first prediction model. The time delay resulting from performing the prediction for the first drawing point can be represented as T3.

[0260] In one embodiment, the electronic device (1100) may estimate a fifth time, which is the time during which the screen refresh is delayed. The electronic device may assume that the screen refresh will be delayed. For example, if the current screen refresh time is 16.67 ms (i.e., 60 Hz), the time during which the screen refresh is delayed may be 20 ms (i.e., 50 Hz).

[0261] Referring to 804, the electronic device (1100) can predict a third drawing point corresponding to a first touch input based on a first time, a second time, and a fifth time. The electronic device (1100) can predict a third drawing point corresponding to a first touch input by performing a new prediction for the first touch input based on a first time, a second time, and a fifth time. The electronic device (1100) can re-perform the prediction for the first touch input based on a first time, a second time, and a fifth time. The time required to re-perform the prediction for the first touch input can be represented as T3_2.

[0262] In one embodiment, the electronic device (1100) may select a prediction model capable of performing a prediction for a first touch input based on a refresh time delayed from the existing refresh time and the time required to render a third drawing.

[0263] In one embodiment, the time required to render the drawing may include T4_2, which is the time required to prepare the third drawing, and T5_2, which is the time required to transmit data for the third drawing to the GPU. For example, the electronic device (1100) may select a prediction model capable of performing a prediction for the first touch input within a time constraint from a delayed refresh time of 20ms excluding time T4_2 and time T5_2.

[0264] In one embodiment, the electronic device (1100) can calculate the latest time at which a prediction must be performed to obtain a result at time T6. Based on the calculated time, the electronic device (1100) can start re-performing the prediction for the first touch input.

[0265] In one embodiment, the electronic device (1100) can predict a third drawing point from a first touch input using a selected prediction model. The selected prediction model may be a prediction model that can be executed within the time based on a delayed refresh time and a rendering time.

[0266] In one embodiment, the electronic device (1100) can determine whether the refresh of the screen is delayed. The electronic device (1100) can determine that the refresh of the screen is delayed if the refresh does not occur even though the existing screen refresh time has elapsed.

[0267] In one embodiment, when the refresh of the screen is delayed, the electronic device (1100) may display a third drawing generated based on a first drawing point and a third drawing point on the screen. The electronic device (1100) may obtain a more accurate prediction result based on the first drawing point and the third drawing point, for which two or more predictions have been performed for a first touch input.

[0268] In one embodiment, if the refresh of the screen is further delayed, the electronic device (1100) may perform a new prediction for the first touch input. Based on the prediction result performed, the prediction result for the first touch input may have a more accurate value.

[0269] Screen refresh delays can occur due to excessive load on the CPU or GPU within the electronic device and screen functions. Figures 7 and 8 illustrate an example of taking measures in advance to prepare for screen refresh delays. More accurate data can be obtained when the screen refresh time is delayed, and accurate data can still be obtained even when the screen refresh time is not delayed.

[0270] FIG. 9 is a flowchart illustrating a method for processing touch input when screen refresh is delayed according to one embodiment of the present disclosure.

[0271] Referring to FIG. 9, the electronic device (1100) can perform the following operations when the screen refresh time is delayed. Content overlapping with FIG. 3 and 4 is omitted.

[0272] In operation S901, the electronic device (1100) can detect a first touch input. The electronic device (1100) can detect a first touch input through a touchscreen. The electronic device (1100) can detect a first touch input through a display. The electronic device (1100) can detect a first touch input entered through an input device (100) (or a finger).

[0273] In operation S902, the electronic device (1100) can estimate a first time, which is the time from when the first touch input is detected until the screen is refreshed. The electronic device (1100) can estimate a time interval from when the first touch input is detected and processed until the earliest time when the screen is refreshed.

[0274] In operation S903, the electronic device (1100) may estimate a second time, which is the time required to render a first drawing including a first drawing point corresponding to a first touch input. The second time may include the time required to prepare and create the first drawing and to transmit data for the first drawing to the GPU.

[0275] In operation S904, the electronic device (1100) may select a first prediction model for predicting a first drawing point based on a first time and a second time. The electronic device (1100) may select at least one first prediction model among a plurality of prediction models that can perform a prediction within a time period excluding the second time from the first time.

[0276] In operation S905, the electronic device (1100) can predict a first drawing point from a first touch input using a selected first prediction model. The electronic device (1100) can predict a first drawing point based on a history of generating drawings according to past touch inputs. The electronic device (1100) can obtain at least one first drawing point using each first prediction model having different speeds, times, and accuracys.

[0277] In operation S906, the electronic device (1100) can identify a delay in the refresh of the screen. The electronic device (1100) can determine whether the refresh of the screen is delayed. The electronic device (1100) can determine that the refresh of the screen is delayed if the refresh does not occur even though the existing screen refresh time has elapsed.

[0278] In operation S907, the electronic device (1100) may select a prediction model for predicting a fourth drawing point corresponding to a first touch input based on a sixth time and a second time, which are times when the refresh of the screen is delayed. The electronic device (1100) may select a prediction model capable of performing a prediction for the first touch input based on the delayed refresh time and the time required to render the fourth drawing.

[0279] In one embodiment, the electronic device (1100) can predict a fourth drawing point from a first touch input using a selected prediction model. The selected prediction model may be a prediction model that can be performed within the time based on a delayed refresh time and a rendering time.

[0280] In operation S908, the electronic device (1100) can predict a fourth drawing point corresponding to a first touch input based on a sixth time and a second time, which are times when the refresh of the screen is delayed. The electronic device (1100) can predict a fourth drawing point corresponding to a first touch input by performing a new prediction for the first touch input based on the sixth time and a second time, which are times when the refresh of the screen is delayed. The electronic device (1100) can re-perform the prediction for the first touch input based on the sixth time and a second time.

[0281] In operation S909, the electronic device (1100) can determine whether the screen refresh is still delayed. For example, it may include a case where the screen refresh time is delayed from 16.67 ms (i.e., 60 Hz) to 20 ms (i.e., 50 Hz), and the refresh still does not occur even after 20 ms has elapsed.

[0282] In one embodiment, if the screen refresh is still delayed, the electronic device (1100) may re-select a prediction model and re-perform a prediction for the first touch input. Operations S907 and S908 may be performed again. The electronic device (1100) may re-perform a prediction for the first touch input whenever the screen refresh is delayed. As the number of predictions increases, the electronic device (1100) can obtain more accurate prediction results.

[0283] In one embodiment, if a new touch input is detected while the screen refresh is delayed, the electronic device (1100) can perform a prediction for the new touch input. A detailed operation is implemented through the operation of FIG. 5.

[0284] In operation S910, when the screen refresh is no longer delayed, the electronic device (1100) can display a fourth drawing generated based on the first drawing point and the fourth drawing point on the screen. The electronic device (1100) can obtain a more accurate prediction result based on the first drawing point and the fourth drawing point, where two or more predictions are performed for the first touch input.

[0285] In operation S911, if the screen refresh is not delayed in the first place, the electronic device (1100) can display the first drawing generated based on the first drawing point on the screen.

[0286] Screen refresh delays can occur due to excessive load on the CPU or GPU within the electronic device and screen functions. Figure 9 illustrates the actions performed when a screen refresh delay occurs. When the screen refresh time is delayed, a decision regarding a new prediction calculation is made, and a prediction model can be selected based on available CPU / GPU usage and the prediction time for the screen refresh.

[0287] FIG. 10 is a flowchart illustrating an operation for predicting a drawing point according to one embodiment of the present disclosure.

[0288] Referring to FIG. 10, the operations for an electronic device (1100) to predict a drawing point are schematically described, and since detailed descriptions of each of the operations have been described in previous drawings, redundant content is omitted.

[0289] In operation S1010, the electronic device (1100) can detect a first touch input. In one embodiment, the electronic device (1100) can detect a second touch input. In one embodiment, the electronic device (1100) can analyze the pattern of time intervals between touch inputs.

[0290] In operation S1020, the electronic device (1100) can estimate a first time, which is the time from when the first touch input is detected until the screen is refreshed. In one embodiment, the electronic device (1100) can estimate a third time, which is the time from when the second touch input is detected until the screen is refreshed.

[0291] In one embodiment, the electronic device (1100) can estimate a fifth time, which is the time during which the refresh of the screen is delayed. In one embodiment, the electronic device (1100) can identify a delay in the refresh of the screen.

[0292] In one embodiment, the electronic device (1100) can estimate a seventh time when a second touch input is detected based on the pattern of the analyzed time intervals.

[0293] In operation S1030, the electronic device (1100) can estimate a second time, which is the time required to render a first drawing including a first drawing point corresponding to a first touch input. In one embodiment, the electronic device (1100) can estimate a fourth time, which is the time required to render a second drawing including a second drawing point corresponding to a second touch input.

[0294] In one embodiment, the electronic device (1100) may determine whether to perform a prediction for a second touch input based on a third time and a fourth time. In one embodiment, if the electronic device (1100) decides not to perform a prediction for a second touch input, it may perform a prediction for a second touch input after the screen has been refreshed.

[0295] In one embodiment, the electronic device (1100) may determine whether to perform a prediction for a first touch input based on a first time, a second time, and a seventh time. In one embodiment, if the electronic device (1100) decides not to perform a prediction for a first touch input, it may stop performing the prediction for the first touch input until the detection of a second touch input.

[0296] In operation S1040, the electronic device (1100) may select a first prediction model for predicting a first drawing point based on a first time and a second time. In one embodiment, if it is decided to perform a prediction for a second touch input, the electronic device (1100) may select a second prediction model for predicting a second drawing point based on a third time and a fourth time.

[0297] In one embodiment, the first prediction model may be a neural network model trained to predict a first drawing point corresponding to a first touch input using past touch inputs as training data. In one embodiment, the second prediction model may be a neural network model trained to predict a second drawing point corresponding to a second touch input using past touch inputs as training data.

[0298] In operation S1050, the electronic device (1100) can predict a first drawing point from a first touch input using a selected first prediction model. In one embodiment, the electronic device (1100) can predict a second drawing point from a second touch input using a selected second prediction model.

[0299] In one embodiment, the electronic device (1100) can predict a third drawing point corresponding to a first touch input based on a first time, a second time, and a fifth time. The electronic device (1100) can predict a third drawing point corresponding to a first touch input by performing a new prediction for the first touch input based on a first time, a second time, and a fifth time. In one embodiment, the electronic device (1100) can predict a fourth drawing point corresponding to a first touch input by performing a new prediction for the first touch input based on a sixth time and a second time, which are times when the refresh of the screen is delayed.

[0300] In operation S1060, the electronic device (1100) can display a first drawing generated based on a first drawing point on a screen. In one embodiment, the electronic device (1100) can display a second drawing generated based on a second drawing point on a screen.

[0301] In one embodiment, the electronic device (1100) can display a third drawing generated based on a first drawing point and a third drawing point on a screen. In one embodiment, the electronic device (1100) can display a fourth drawing generated based on a first drawing point and a fourth drawing point on a screen.

[0302] In one embodiment, the electronic device (1100) may select a first drawing point based on a first time, a second time, accuracy, and error probability. In one embodiment, the electronic device (1100) may display a first drawing generated based on the selected first drawing point on a screen.

[0303] In one embodiment, the electronic device (1100) can adjust the screen refresh time so that the time at which the screen is refreshed is synchronized with the time at which the first drawing is displayed on the screen.

[0304] The above operations can be implemented on any device equipped with a touchscreen and drawing engine software (such as drawing lines or moving a pointer using a stylus, finger, or mouse). There are no limitations on hardware or software.

[0305] FIG. 11 is a block diagram illustrating the configuration of an electronic device (1100) according to one embodiment of the present disclosure.

[0306] In one embodiment, the electronic device (1100) may include a memory (1110), an input interface (1120A), an output interface (1120B), and a processor (1130).

[0307] The memory (1110) may store instructions, data structures, and program code that can be read by the processor (1130). Operations performed by the processor (1130) may be implemented by executing the instructions or code of the program stored in the memory (1110).

[0308] The memory (1110) may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a non-volatile memory including at least one of ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk, and a volatile memory such as RAM (Random Access Memory) or SRAM (Static Random Access Memory).

[0309] The input interface (1120A) may include a touch screen, keyboard, microphone, etc., for receiving commands or information, etc., from a user, and the output interface (1120B) may include a display panel, projector, speaker, etc., for displaying the result of an operation according to a user's command or the status of the electronic device (1100). According to one embodiment of the present disclosure, the electronic device (1100) can receive input data from a user through the input interface (1120A), and when the operation is completed, it can output the result of the operation through the output interface (1120B).

[0310] The processor (1130) can control the overall operations of the electronic device (1100). For example, the processor (1130) can control the overall operations of the electronic device (1100) for predicting a drawing point by executing one or more instructions of a program stored in memory (1110). There may be one or more processors (1130).

[0311] The processor (1130) may be composed of at least one of, for example, a Central Processing Unit, a microprocessor, a Graphic Processing Unit, ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), an Application Processor, a Neural Processing Unit, or an AI-dedicated processor designed with a hardware structure specialized for processing AI models, but is not limited thereto.

[0312] Meanwhile, although not illustrated in FIG. 11, the electronic device (1100) may further include additional components to perform the operations described in the aforementioned embodiments. For example, the electronic device (1100) may further include a display, a camera, a microphone, a speaker, a touchscreen, a communication interface, etc.

[0313] In cases where a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an AI-dedicated processor). Here, the AI-dedicated processor, which is an example of the second processor, may perform operations for training / inference of an AI model. However, the embodiments of the present disclosure are not limited thereto.

[0314] One or more processors according to the present disclosure may be implemented as a single-core processor or as a multi-core processor.

[0315] In the case where a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single core or by a plurality of cores included in one or more processors.

[0316] The present disclosure relates to a method, electronic device, and system for predicting drawing points. The technical problems to be solved by the present disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description in this specification.

[0317] According to one aspect of the present disclosure, a method for an electronic device to predict a drawing point may be provided.

[0318] The above method may include a step of detecting a second touch input.

[0319] The above method may include a step of estimating a third time, which is the time from when the second touch input is detected until the screen is refreshed.

[0320] The above method may include a step of estimating a fourth time, which is the time required to render a second drawing including a second drawing point corresponding to the second touch input.

[0321] The above method may include a step of determining whether to perform a prediction for the second touch input based on the third time and the fourth time.

[0322] The above method may include the step of selecting a second prediction model for predicting the second drawing point based on the third time and the fourth time when it is decided to perform a prediction for the second touch input.

[0323] The above method may include the step of predicting the second drawing point from the second touch input using the selected second prediction model.

[0324] The above method may include the step of displaying the second drawing generated based on the second drawing point on the screen.

[0325] The above method may include the step of performing a prediction for the second touch input after the screen has been refreshed, if it is decided not to perform a prediction for the second touch input.

[0326] The first prediction model may be a neural network model trained to predict the first drawing point corresponding to the first touch input by using past touch inputs as training data.

[0327] The second prediction model may be a neural network model trained to predict the second drawing point corresponding to the second touch input by using the past touch inputs as training data.

[0328] The above method may include a step of estimating a fifth time, which is the time during which the refresh of the screen is delayed.

[0329] The above method may include the step of predicting a third drawing point corresponding to the first touch input based on the first time, the second time, and the fifth time.

[0330] The above method may include the step of displaying a third drawing generated based on the first drawing point and the third drawing point on the screen.

[0331] The above method may include a step of identifying a delay in the refresh of the screen.

[0332] The above method may include the step of predicting a fourth drawing point corresponding to the first touch input based on the sixth time, which is the time when the refresh of the screen is delayed, and the second time.

[0333] The above method may include the step of displaying a fourth drawing generated based on the first drawing point and the fourth drawing point on the screen.

[0334] The above method may include the step of selecting the first drawing point based on the first time, the second time, accuracy, and error probability.

[0335] The above method may include the step of displaying the first drawing generated based on the selected first drawing point on the screen.

[0336] The above method may include a step of adjusting the screen refresh time so that the time for displaying the first drawing on the screen and the time for refreshing the screen are synchronized.

[0337] The above method may include a step of analyzing the pattern of time intervals between touch inputs. The above method may include a step of estimating a seventh time at which the second touch input is detected based on the analyzed pattern of time intervals.

[0338] The above method may include a step of determining whether to perform a prediction for the first touch input based on the first time, the second time, and the seventh time.

[0339] The above method may include the step of stopping the prediction of the first touch input until the detection of the second touch input when it is decided not to perform the prediction for the first touch input.

[0340] According to one aspect of the present disclosure, an electronic device for predicting a drawing point may be provided.

[0341] The electronic device may include an input interface; an output interface; a memory for storing at least one instruction; and at least one processor for executing the at least one instruction stored in the memory.

[0342] The above at least one processor can detect a second touch input by executing the above at least one instruction.

[0343] The above at least one processor can estimate a third time, which is the time from when the second touch input is detected until the screen is refreshed, by executing the above at least one instruction.

[0344] The above at least one processor can estimate a fourth time, which is the time required to render a second drawing including a second drawing point corresponding to the second touch input, by executing the above at least one instruction.

[0345] The above at least one processor can determine whether to perform a prediction for the second touch input based on the third time and the fourth time by executing the above at least one instruction.

[0346] If the above at least one processor decides to perform a prediction for the second touch input by executing the above at least one instruction, it may select a second prediction model for predicting the second drawing point based on the third time and the fourth time.

[0347] The above at least one processor can predict the second drawing point from the second touch input by using the selected second prediction model by executing the above at least one instruction.

[0348] The above at least one processor can display the second drawing generated based on the second drawing point on the screen by executing the above at least one instruction.

[0349] If the above at least one processor decides not to perform a prediction for the second touch input by executing the above at least one instruction, it may perform a prediction for the second touch input after the screen has been refreshed.

[0350] The above at least one processor can estimate a fifth time, which is the time during which the refresh of the screen is delayed, by executing the above at least one instruction.

[0351] The above at least one processor can predict a third drawing point corresponding to the first touch input based on the first time, the second time, and the fifth time by executing the above at least one instruction.

[0352] The above at least one processor can display a third drawing generated based on the first drawing point and the third drawing point on the screen by executing the above at least one instruction.

[0353] The above at least one processor can identify a delay in the refresh of the screen by executing the above at least one instruction.

[0354] The above at least one processor can predict a fourth drawing point corresponding to the first touch input based on the sixth time, which is the time when the refresh of the screen is delayed, and the second time by executing the above at least one instruction.

[0355] The above at least one processor can display a fourth drawing generated based on the first drawing point and the fourth drawing point on the screen by executing the above at least one instruction.

[0356] The above at least one processor can select the first drawing point based on the first time, the second time, accuracy, and error probability by executing the above at least one instruction.

[0357] The above at least one processor can display the first drawing generated based on the selected first drawing point on the screen by executing the above at least one instruction.

[0358] The above at least one processor can adjust the screen refresh time so that the time for displaying the first drawing on the screen and the time for refreshing the screen are synchronized by executing the above at least one instruction.

[0359] The above at least one processor can analyze the pattern of time intervals between touch inputs by executing the above at least one instruction.

[0360] The above at least one processor can estimate the seventh time at which the second touch input is detected based on the pattern of the analyzed time interval by executing the above at least one instruction.

[0361] The above at least one processor can determine whether to perform a prediction for the first touch input based on the first time, the second time, and the seventh time by executing the above at least one instruction.

[0362] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include other data of modulated data signals, such as computer-readable instructions, data structures, or program modules.

[0363] Additionally, computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage media' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, 'non-transitory storage media' may include a buffer in which data is stored temporarily.

[0364] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0365] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0366] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.

Claims

1. In a method for an electronic device to predict a drawing point, A step of detecting a first touch input; A step of estimating a first time, which is the time from when the first touch input is detected until the screen is refreshed; A step of estimating a second time, which is the time required to render a first drawing including a first drawing point corresponding to the first touch input; A step of selecting a first prediction model for predicting the first drawing point based on the first time and the second time; A step of predicting the first drawing point from the first touch input using the selected first prediction model; and A method comprising the step of displaying the first drawing including the first drawing point on the screen.

2. In Paragraph 1, A step of detecting a second touch input; A step of estimating a third time, which is the time from when the second touch input is detected until the screen is refreshed; A step of estimating a fourth time, which is the time required to render a second drawing including a second drawing point corresponding to the second touch input; A step of determining whether to perform a prediction for the second touch input based on the third time and the fourth time; Based on the decision to perform a prediction for the second touch input, a step of selecting a second prediction model for predicting the second drawing point based on the third time and the fourth time; A step of predicting the second drawing point from the second touch input using the selected second prediction model; and A method further comprising the step of displaying the second drawing including the second drawing point on the screen.

3. In Paragraph 2, A method further comprising the step of performing a prediction for the second touch input after the screen has been refreshed, based on the decision not to perform a prediction for the second touch input.

4. In Paragraph 1, The first prediction model is a neural network model trained to predict the first drawing point from the first touch input using past touch inputs as training data, and A method in which the second prediction model is a neural network model trained to predict the second drawing point from the second touch input using the past touch inputs as training data.

5. In Paragraph 1, A step of estimating a fifth time, which is the time during which the refresh of the above screen is delayed; A step of predicting a third drawing point from the first touch input based on the first time, the second time, and the fifth time; and A method further comprising the step of displaying a third drawing including the first drawing point and the third drawing point on the screen.

6. In Paragraph 1, A step of identifying a sixth time, which is the time when the refresh of the above screen is delayed; Based on the second time and the sixth time, a step of predicting a fourth drawing point from the first touch input; and A method further comprising the step of displaying a fourth drawing including the first drawing point and the fourth drawing point on the screen.

7. In Paragraph 1, The step of displaying the first drawing including the first drawing point on the screen is: A step of selecting the first drawing point among at least one prediction result based on the first time, the second time, accuracy, and error probability; and A method comprising the step of displaying the first drawing, which includes the first selected drawing point, on the screen.

8. In an electronic device for predicting drawing points, Input interface; Output interface; Memory for storing at least one instruction; and It includes at least one processor that executes at least one instruction stored in the memory, The above electronic device is, Detecting the first touch input, Estimating a first time, which is the time from when the first touch input is detected until the screen is refreshed, and Estimating a second time, which is the time required to render a first drawing including a first drawing point corresponding to the first touch input, and Based on the first time and the second time, a first prediction model for predicting the first drawing point is selected, and Using the selected first prediction model, the first drawing point is predicted from the first touch input, and An electronic device that displays the first drawing including the first drawing point on the screen.

9. In Paragraph 8, The above electronic device is, Detecting a second touch input, Estimating a third time, which is the time from the point in time when the second touch input is detected until the screen is refreshed, and Estimating a fourth time, which is the time required to render a second drawing including a second drawing point corresponding to the second touch input, and Based on the above third time and the above fourth time, determine whether to perform a prediction for the above second touch input, and Based on the decision to perform a prediction for the second touch input, a second prediction model for predicting the second drawing point based on the third time and the fourth time is selected, and Using the selected second prediction model, the second drawing point is predicted from the second touch input, and An electronic device that displays the second drawing including the second drawing point on the screen.

10. In Paragraph 9, The above electronic device is, An electronic device that performs a prediction for the second touch input after the screen is refreshed, based on a decision not to perform a prediction for the second touch input.

11. In Paragraph 8, The first prediction model is a neural network model trained to predict the first drawing point from the first touch input using past touch inputs as training data, and The electronic device, wherein the second prediction model is a neural network model trained to predict the second drawing point from the second touch input using the past touch inputs as training data.

12. In Paragraph 8, The above electronic device is, Estimating the fifth time, which is the time when the refresh of the above screen is delayed, and Based on the first time, the second time, and the fifth time, a third drawing point is predicted from the first touch input, and An electronic device that displays a third drawing including the first drawing point and the third drawing point on the screen.

13. In Paragraph 8, The above electronic device is, Identify the sixth time, which is the time when the refresh of the above screen is delayed, and Based on the above second time and the above sixth time, predict a fourth drawing point from the above first touch input, and An electronic device that displays a fourth drawing including the first drawing point and the fourth drawing point on the screen.

14. In Paragraph 8, The above electronic device is, Based on the first time, the second time, accuracy, and error probability, the first drawing point is selected from at least one prediction result, and An electronic device that displays the first drawing including the selected first drawing point on the screen.

15. A computer-readable recording medium having a program for executing the method of any one of paragraphs 1 through 7 on a computer.

Images