Display device for displaying predicted trajectory and control method therefor

Abstract

This display device comprises: a memory storing one or more instructions; a display panel including a touch sensor; and at least one processor connected to the memory and the display panel and configured to execute the one or more instructions, wherein the instructions, when executed by the at least one processor, instruct the display device to: acquire a first input point corresponding to a first touch on a screen; acquire a second input point corresponding to a second touch on the screen; display a touch trajectory based on the first input point and the second input point on the display panel in a frame subsequent to the second frame; acquire a predicted trajectory on the basis of the first input point and the second input point; and in a subsequent frame, display a predicted trajectory of an adjusted length together with the touch trajectory on the display panel such that the touch trajectory and the predicted trajectory of the adjusted length form a continuous trajectory.

Description

Display device for displaying predicted trajectory and control method thereof

[0001] The present disclosure relates to a display device and a control method thereof, and more particularly, to a display device that displays a predicted trajectory by considering an area where a touch is detected, and a control method thereof.

[0002] Recently, thanks to technological advancements in the display device field, various types of display devices are being developed and distributed.

[0003] The trend is to increase productivity by using stylus pens for various tasks, such as taking notes, rather than just viewing content through display devices.

[0004] However, when performing touch operations using a stylus pen, there is a delay (input lag) between the time the display device recognizes the touch and the time it displays the corresponding touch trajectory. Input lag increases user inconvenience and degrades the user experience, as it varies depending on where the touch is detected on the display device's screen.

[0005] There has been a need for a method of maintaining a constant delay time and displaying a predicted trajectory more appropriately, regardless of the area in which a touch is detected on a display device, so as to provide an effect similar to that of no delay time occurring between the time a touch is detected and the time of displaying the touch trajectory corresponding to the touch.

[0006] According to one embodiment, a display device includes: a memory storing one or more instructions; a display panel including a touch sensor; And at least one processor connected to the memory and the display panel and configured to execute the one or more instructions; wherein the instructions, when executed by the at least one processor, cause the display device to, when a first touch on a screen of the display panel is detected through the touch sensor while a first frame among a plurality of frames included in an image is displayed on the display panel, acquire a first input point corresponding to the first touch, when a second touch on the screen is detected while a second frame among the plurality of frames is displayed on the display panel, acquire a second input point corresponding to the second touch, and in a next frame after the second frame, display a touch trajectory based on the first input point and the second input point on the display panel, acquire a predicted trajectory based on the first input point and the second input point, and in the next frame, adjust a length of the predicted trajectory based on coordinate information corresponding to the first touch within the screen, and in the next frame, predict the touch trajectory and the adjusted length. The predicted trajectory of the adjusted length is displayed on the display panel together with the touch trajectory so that the trajectory forms a continuous trajectory.

[0007] The instructions, when executed by the at least one processor, cause the display device to sequentially display each of the plurality of frames by updating the display panel from top to bottom or from left to right during a refresh rate, and to obtain a display time between an update time of the display panel for displaying the next frame and a display time of the touch trajectory in the next frame based on a ratio of the coordinate information to the size of the screen.

[0008] The instructions, when executed by the at least one processor, cause the display device to obtain the display time based on a ratio of the y-axis coordinate included in the coordinate information to the height of the screen when updating the display panel from top to bottom during the refresh rate, and to adjust the length of the predicted trajectory based on the display time in the next frame.

[0009] The instructions, when executed by the at least one processor, cause the display device to obtain the display time based on a ratio of the x-axis coordinate included in the coordinate information to the width (weight) of the screen when updating the display panel from left to right during the refresh rate, and to adjust the length of the predicted trajectory based on the display time in the next frame.

[0010] The above instructions, when executed by the at least one processor, cause the display device to obtain a buffering time between the detection time of the first touch and the update time of the display panel, and adjust the length of the predicted trajectory based on the buffering time and the display time.

[0011] The instructions, when executed by the at least one processor, cause the display device to obtain a delay time between a detection time of the first touch and a display time of the touch trajectory based on the buffering time and the display time, and to compare the delay time with a prediction time corresponding to a prediction point included in the prediction trajectory to adjust the length of the prediction trajectory.

[0012] The instructions, when executed by the at least one processor, cause the display device to adjust the length of the predicted trajectory based on the delay time and display a portion of the predicted trajectory if the sum of the refresh time corresponding to the refresh rate and the buffering time is less than the predicted time.

[0013] The instructions, when executed by the at least one processor, cause the display device to display a portion of the predicted trajectory based on the predicted time, the refresh time, and the display time, if the predicted time is less than a sum of a refresh time corresponding to the predicted time and the buffering time.

[0014] The instructions, when executed by the at least one processor, cause the display device to adjust a length of the prediction trajectory between the first prediction point and the second prediction point based on the delay time, if the delay time is included between a first prediction time corresponding to a first prediction point included in the prediction trajectory and a second prediction time corresponding to a second prediction point of the predicted trajectory.

[0015] The above instructions, when executed by the at least one processor, cause the display device to add a time required to perform an adaptive synchronization function to a buffering time when the display panel performs an adaptive synchronization function.

[0016] A method for controlling a display device according to one embodiment includes: when a first touch is detected on a screen of a display panel of the display device while displaying a first frame among a plurality of frames included in an image; when a second touch is detected on the screen while displaying a second frame among the plurality of frames; when a second touch is detected on the screen while displaying a second frame among the plurality of frames; displaying a touch trajectory based on the first input point and the second input point in a next frame after the second frame; obtaining a predicted trajectory based on the first input point and the second input point; and displaying the predicted trajectory of the adjusted length together with the touch trajectory in the next frame so that the touch trajectory and the predicted trajectory of the adjusted length form a continuous trajectory.

[0017] The control method may further include a step of sequentially displaying each of the plurality of frames by updating the display panel from top to bottom or from left to right during a refresh rate; and the step of adjusting the length of the predicted trajectory to display may include a step of obtaining a display time between an update time of the display panel for displaying the next frame and a display time of the touch trajectory in the next frame, based on a ratio of the coordinate information to the size of the screen.

[0018] The step of obtaining the display time may include a step of obtaining the display time based on a ratio of the y-axis coordinate included in the coordinate information with respect to the height of the screen when the display panel is updated from the top to the bottom during the refresh rate; and the step of adjusting the length of the predicted trajectory and displaying may include a step of adjusting the length of the predicted trajectory based on the display time in the next frame.

[0019] The step of obtaining the display time may include a step of obtaining the display time based on a ratio of the x-axis coordinate included in the coordinate information to the width (weight) of the screen when the display panel is updated from left to right during the refresh rate; and the step of adjusting the length of the predicted trajectory and displaying may include a step of adjusting the length of the predicted trajectory based on the display time in the next frame.

[0020] In one embodiment, a non-transitory computer-readable medium storing instructions that, when executed by a processor or a display device, cause the processor to execute a method, the method comprises: when a first touch is detected on a screen of a display panel of the display device while displaying a first frame among a plurality of frames included in an image, acquiring a first input point corresponding to the first touch; when a second touch is detected on the screen while displaying a second frame among the plurality of frames, acquiring a second input point corresponding to the second touch; in a next frame after the second frame, displaying a touch trajectory based on the first input point and the second input point; acquiring a predicted trajectory based on the first input point and the second input point; and in the next frame, displaying the predicted trajectory of the adjusted length together with the touch trajectory such that the touch trajectory and the predicted trajectory of the adjusted length form a continuous trajectory.

[0021] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0022] FIG. 1 is a drawing illustrating a display device and an input device according to one or more embodiments.

[0023] FIG. 2 is a block diagram illustrating a configuration of a display device according to one or more embodiments.

[0024] FIG. 3 is a drawing for explaining input lag occurring in a first area within the screen of a conventional display device.

[0025] FIG. 4 is a drawing for explaining input lag occurring in a second area within the screen of a conventional display device.

[0026] FIG. 5 is a drawing for explaining a touch trajectory according to one or more embodiments.

[0027] FIG. 6 is a diagram illustrating a predicted trajectory according to one or more embodiments.

[0028] FIG. 7 is a diagram illustrating buffering time according to one or more embodiments.

[0029] FIG. 8 is a diagram illustrating display time according to one or more embodiments.

[0030] FIG. 9 is a diagram illustrating a delay time according to one or more embodiments.

[0031] FIG. 10 is a drawing illustrating a display device for displaying a predicted trajectory with an adjusted length according to one or more embodiments.

[0032] FIG. 11 is a diagram illustrating input lag occurring in a display device that updates frames in various directions according to one or more embodiments.

[0033] FIG. 12 is a flowchart illustrating a method for controlling a display device according to one or more embodiments.

[0034] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0035] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0036] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0037] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0038] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0039] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0040] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0041] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware, software, or a combination of hardware and software. In one or more examples, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented by at least one processor (not shown), except for any "module" or "part" that needs to be implemented in specific hardware.

[0042] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0043] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0044] FIG. 1 is a drawing illustrating a display device and an input device according to one or more embodiments.

[0045] As illustrated in FIG. 1, the display device (100) may be implemented as a user terminal device, but is not limited thereto, and the display device (100) may be implemented as a TV, an Electric White Board Display, a video wall, a large format display (LFD), a digital signage, a digital information display (DID), a projector display, or various types of devices having appropriate device display functions known to those skilled in the art.

[0046] According to an embodiment, the display device (100) can detect a touch and perform an action corresponding to the touch.

[0047] For example, the display device (100) provides a handwriting function (or, handwriting input function, writing function, etc.) and can display a trajectory corresponding to a touch (hereinafter, touch trajectory).

[0048] As illustrated in FIG. 1, the display device (100) can detect a touch through an input device (200) and display a touch trajectory.

[0049] In Fig. 1, for convenience of explanation, the input device (200) is assumed to be a stylus pen. However, the stylus pen is not limited thereto, and the input device (200) may be any type of suitable device known to those skilled in the art that can touch the display device (100). In one or more examples, the user's finger may perform the function of the stylus pen.

[0050] When the display device (100) detects a touch using an electrostatic touch method, the input device (200) can be implemented as various types of objects that can change an electric signal by coming into contact with the screen on the display device (100).

[0051] Depending on the embodiment, latency may occur between the time the display device (100) detects a touch and the time it displays the touch trajectory. Depending on the embodiment, latency may be referred to as delay, input lag, etc.

[0052] According to an embodiment, the display device (100) can predict a subsequent trajectory based on a touch through the input device (200) to compensate for (or eliminate) latency. For example, the display device (100) can display both the touch trajectory and the predicted subsequent trajectory (hereinafter, predicted trajectory) to provide an effect similar to that of displaying a trajectory corresponding to the current position of the input device (200) on the screen, that is, by detecting a touch through the input device (200) and displaying a trajectory corresponding to the touch simultaneously (or within a very short time), thereby providing an effect in which the user does not perceive latency.

[0053] As will be understood by those skilled in the art, conventional display devices may exhibit an overprediction problem, in which they predict subsequent trajectories to be somewhat long, thereby displaying a trajectory longer than the current position of the input device (200). As will be understood by those skilled in the art, conventional display devices may exhibit an underprediction problem, in which they predict subsequent trajectories to be somewhat short, thereby displaying a trajectory shorter than the current position of the input device (200). Both underprediction and underprediction problems may result in conventional display devices not being able to accurately compensate for latency.

[0054] Conventional display devices have a problem of over-predicting or under-predicting subsequent trajectories depending on whether the touch was detected in a first area on the screen (e.g., the upper or left area) or a second area on the screen (e.g., the lower or right area).

[0055] According to an embodiment, the display device (100) can predict the length of subsequent trajectories to be the same or unify latency regardless of which area on the screen the touch was detected.

[0056] FIG. 2 is a block diagram illustrating a configuration of a display device according to one or more embodiments.

[0057] Referring to FIG. 2, the display device (100) includes a display panel (110) and one or more processors (120).

[0058] According to an embodiment, the display panel (110) may be implemented in various forms such as an LCD (liquid crystal display), an OLED (organic light-emitting diode), an LCoS (Liquid Crystal on Silicon), a DLP (Digital Light Processing), a QD (quantum dot) display panel, a QLED (quantum dot light-emitting diodes), a μLED (Micro light-emitting diodes), a Mini LED, etc.

[0059] According to an embodiment, the display panel (110) may be implemented as a touch screen combined with a touch sensor (111). In addition, the display panel (110) may be implemented as a flexible display, a rollable display, a 3D display, a modular display in which multiple display modules are physically connected, etc.

[0060] According to an embodiment, the touch sensor (111) can detect a touch on the display panel (110). For example, the touch sensor (111) can detect a touch in which an arbitrary object (e.g., an input device (200)) comes into contact with the display panel (110).

[0061] The touch sensor (111) can detect the touch of any object that comes into contact with the display panel (110), or can detect only the touch of objects with certain limited characteristics. The limitations on the objects that the touch sensor (111) can detect the touch of vary depending on the method by which the touch sensor (111) detects the touch.

[0062] For example, the method by which the touch sensor (111) detects a touch may be implemented by an infrared method using an IR sensor (Infrared Touch Screen), an ultrasonic method (Surface Acoustic Wave Touch Screen), a resistive method (Resistive Touch Screen) (or a pressure-sensitive method), or an electrostatic capacitive touch method (Capacitive Touch Screen). However, the present invention is not limited thereto, and the touch sensor (111) may detect a touch according to any appropriate touch detection method known to those skilled in the art. For example, the touch sensor (111) may be equipped with a camera or the like, and may detect a touch on the display panel (110) based on an image captured by the camera.

[0063] According to one embodiment, a touch sensor (111) may generate a signal corresponding to a touch and transmit the signal to one or more processors (120) when a touch is detected on the screen of a display panel (110). According to one embodiment, the signal corresponding to the touch may include location information (e.g., coordinate information) where the touch is detected on the screen, touch intensity information, or other appropriate information known to those skilled in the art in the art of predicting a touch trajectory.

[0064] According to an embodiment, one or more processors (120) control the overall operation of the display device (100).

[0065] Specifically, one or more processors (120) may be connected to each component of the display device (100) to control the overall operation of the display device (100).

[0066] One or more processors (120) can perform operations of the display device (100) according to various embodiments by executing at least one instruction stored in memory.

[0067] According to an embodiment, one or more processors (120) may include one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), a DSP (Digital Signal Processor), an NPU (Neural Processing Unit), an AI (Artificial Intelligence) processor, a hardware accelerator, or a machine learning accelerator. In addition, one or more processors (120) may be implemented as a SoC (System on Chip), an LSI (Large Scale Integration) having a built-in processing algorithm, or may be implemented in the form of an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0068] One or more processors (120) may control one or any combination of other components of the electronic device and perform operations related to communication or data processing. One or more processors (120) may execute one or more programs or instructions stored in memory. For example, one or more processors (120) may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in memory.

[0069] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0070] One or more processors (120) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (120) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0071] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0072] In one or more embodiments, a processor may mean a system on a chip (SoC) having at least one processor and other electronic components integrated therein, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited to such configurations and may include any suitable processor or system architecture known to those skilled in the art.

[0073] According to one or more embodiments, one or more processors (120) may control a display panel (110) to sequentially display a plurality of frames included in the image to display the image.

[0074] For example, one or more processors (120) can sequentially display multiple frames by updating the display panel (110) at time intervals corresponding to a refresh rate.

[0075] For example, if the refresh rate is 120 Hz, one or more processors (120) can sequentially display multiple frames by updating the display panel (110) at a time interval of 1 / 120 = 0.0083 [sec].

[0076] According to one or more embodiments, when one or more processors (120) detect a first touch on the screen of the display panel (110) through the touch sensor (111) while displaying a first frame among a plurality of frames, the one or more processors (120) may acquire a first input point corresponding to the first touch.

[0077] According to one or more embodiments, when one or more processors (120) detect a second touch on the screen of the display panel (110) through the touch sensor (111) while displaying a second frame among a plurality of frames, the one or more processors (120) may acquire a second input point corresponding to the second touch.

[0078] According to an embodiment, one or more processors (120) may control the display panel (110) to display touch trajectories based on the first input point and the second input point in subsequent frames of the second frame using a frame buffer.

[0079] According to one or more embodiments, one or more processors (120) can predict subsequent trajectories based on the first input point and the second input point.

[0080] For example, one or more processors (120) can obtain a predicted trajectory using various algorithms such as linear prediction, Taylor series, curve fitting, and NN (Neural Network)-based prediction.

[0081] One or more processors (120) according to one or more embodiments may adjust the length of the predicted trajectory based on coordinate information corresponding to the first touch within the screen in the next frame and display the same sequentially in the touch trajectory.

[0082] For example, one or more processors (120) can identify whether a first touch detected while displaying a first frame is located on the upper side (or left side) or lower side (or right side) of the display panel (110) and adjust the length of the predicted trajectory and then display it.

[0083] FIG. 3 is a drawing for explaining input lag occurring in a first area within the screen of a conventional display device.

[0084] Referring to FIG. 3, one or more processors (120) can sequentially provide multiple frames by updating the display panel (110) at time intervals corresponding to the refresh rate.

[0085] According to one or more embodiments, one or more processors (120) may update the display panel (110) in a raster scan manner during a time interval corresponding to a refresh rate.

[0086] For example, one or more processors (120) can display frames by sequentially updating from the top to the bottom of the display panel (110) or sequentially updating from the left to the right.

[0087] However, it is not limited thereto, and it is of course possible for one or more processors (120) to update from the bottom to the top of the display panel (110) or from the right to the left.

[0088] As illustrated in FIG. 3, in a conventional display device, when a touch is detected in a first area (e.g., an upper area) of a display panel, the display panel is updated from the upper side to the lower side, so the latency until displaying a touch trajectory corresponding to the touch can be relatively short.

[0089] For example, when a touch is detected in the upper area of ​​the display panel, a conventional display device displays a touch trajectory and a predicted trajectory corresponding to the detected touch in the next frame of the frame being displayed at the time the touch is detected, and since the upper area arrives at the update time quickly, it can provide an effect in which the difference between the current position of the input device (200) and the predicted trajectory is relatively small (e.g., an effect in which latency occurs short).

[0090] For example, the difference between the current position of the input device (200) and the predicted trajectory may be at most 7.85 mm and at least 5.46 mm. This is an example for convenience of explanation and is not limited to a specific number.

[0091] FIG. 4 is a drawing for explaining input lag occurring in a second area within the screen of a conventional display device.

[0092] Referring to FIGS. 3 and 4, the display device can display frames by sequentially updating the display panel from top to bottom during a time interval corresponding to a refresh rate.

[0093] As illustrated in FIG. 4, in a conventional display device, when a touch is detected in a second area (e.g., a lower area) of the display panel, the display panel is updated from the upper side to the lower side, so the latency until the touch trajectory corresponding to the touch is displayed may be relatively long.

[0094] For example, when a touch is detected in the lower area of ​​the display panel, a conventional display device displays a touch trajectory and a predicted trajectory corresponding to the detected touch in the next frame of the frame being displayed at the time the touch was detected. However, because the update time is slow, the lower area may provide an effect in which the difference between the current position of the input device (200) and the predicted trajectory is relatively large (e.g., an effect in which latency occurs for a long time).

[0095] For example, the difference between the current position of the input device (200) and the predicted trajectory may be a maximum of 9.67 mm and a minimum of 7.39 mm. This is an example for convenience of explanation and is not limited to a specific number.

[0096] As shown in FIGS. 3 and 4, conventional display devices have a problem in that latency occurs differently depending on whether a touch is detected in the first area or the second area.

[0097] FIG. 5 is a drawing for explaining a touch trajectory according to one or more embodiments.

[0098] Referring to FIG. 5, one or more processors (120) according to one or more embodiments may obtain a touch trajectory based on a plurality of input points and control a display panel (110) to display the touch trajectory.

[0099] One or more processors (120) according to one or more embodiments may obtain a predicted trajectory based on a plurality of input points, and display the touch trajectory and the predicted trajectory to provide an effect in which a trajectory corresponding to the current position of the input device (200) is displayed.

[0100] According to one or more embodiments, one or more processors (120) may adjust the length of the predicted trajectory and then display it to provide the effect of the same latency regardless of which area of ​​the display panel (110) the touch is detected in, whether the touch is detected in the upper area or the lower area.

[0101] According to one or more embodiments, when a touch is detected, one or more processors (120) can obtain a display time between an update time of the display panel (110) for displaying the next frame and a display time of the touch trajectory in the next frame based on a ratio of coordinate information of the touch to the size of the screen (ratio of coordinate information of the touch to the size of the screen).

[0102] Here, the next frame is the next frame of the frame that the display panel (110) is displaying at the time the touch is detected, and one or more processors (120) can display the touch trajectory and predicted trajectory while the next frame is being displayed.

[0103] For example, when a touch is detected on the upper side of the display panel (110), one or more processors (120) may adjust the length of the prediction trajectory to be shorter and then display it to prevent over-prediction.

[0104] For example, when a touch is detected on the lower side of the display panel (110), one or more processors (120) may adjust the length of the prediction trajectory to prevent under prediction and then display it.

[0105] According to an embodiment, one or more processors (120) can display a predicted trajectory similar to an actual touch trajectory with the same latency without overprediction or underprediction occurring, regardless of which area of ​​the display panel (110) the touch is detected in.

[0106] FIG. 6 is a diagram illustrating a predicted trajectory according to one or more embodiments.

[0107] Referring to FIG. 6, one or more processors (120) may obtain a touch trajectory based on a plurality of input points, including a first input point corresponding to a first touch, a second input point corresponding to a second touch, etc. Although FIG. 6 illustrates the plurality of input points as individual points, as will be appreciated by those skilled in the art, the first input point and the second input point may correspond to positions of the stylus pen 200 on the touch screen at different times. For example, referring to FIG. 6, the plurality of input points may correspond to a user performing a swipe motion from left to right, wherein the first input point corresponds to a position of the stylus pen 200 at time t1, and the second input point corresponds to a position of the stylus pen 200 at time t2.

[0108] In one or more embodiments, one or more processors (120) may input multiple input points into various algorithms, such as linear prediction, Taylor series, curve fitting, and NN-based prediction, to obtain a predicted trajectory. In some embodiments, the predicted trajectory may include at least one predicted point.

[0109] According to one or more embodiments, one or more processors (120) may obtain a predicted trajectory including a plurality of predicted points, and adjust the length of the predicted trajectory to correspond to the current position (or actual position) of the input device (200) at the time the predicted trajectory is displayed in the next frame of the frame being displayed at the time when the touch is detected.

[0110] For example, when one or more processors (120) update the display panel (110) from top to bottom during the refresh rate, the time at which the predicted trajectory is displayed in the next frame (hereinafter, display time) can be obtained based on the ratio of the y-axis coordinate included in the coordinate information to the height of the screen. The y-axis coordinate included in the coordinate information can include the y-axis coordinate among the coordinate information (x, y) corresponding to the first touch detected while displaying the first frame.

[0111] For example, when one or more processors (120) update the display panel (110) from left to right during the refresh rate, the time at which the predicted trajectory is displayed in the next frame (hereinafter, display time) can be obtained based on the ratio of the x-axis coordinate included in the coordinate information to the width (weight) of the screen. The x-axis coordinate included in the coordinate information can include the x-axis coordinate among the coordinate information (x, y) corresponding to the first touch detected while displaying the first frame.

[0112] One or more processors (120) can adjust the length of the predicted trajectory based on the display time and display it while the next frame is being displayed.

[0113] FIG. 7 is a diagram illustrating buffering time according to one or more embodiments.

[0114] For the convenience of explanation, the screen height is assumed to be 150 mm and the screen width is assumed to be 80 mm in one or more examples. These figures are not limiting, and the screen height and width may, of course, vary. In one or more examples, it is assumed that one or more processors (120) update the display panel (110) from top to bottom for a time corresponding to a refresh rate (hereinafter, “refresh time”) to sequentially display each of the plurality of frames.

[0115] According to one or more embodiments, when a first touch is detected while displaying a first frame, one or more processors (120) may acquire a first input point corresponding to the first touch.

[0116] According to one or more embodiments, when a second touch is detected while displaying a second frame, one or more processors (120) may acquire a second input point corresponding to the second touch.

[0117] In one or more examples, the point in time at which the display panel (110) begins updating to display the second frame (hereinafter, update point in time) (t f1 ) is assumed to be 100 ms. The specific numbers are examples for convenience of explanation, and the embodiment can be modified to use any appropriate parameters.

[0118] According to an embodiment, one or more processors (120) may update the display panel (110) during the refresh time to display the second frame. For example, if the refresh rate is 120 Hz, the refresh time is 1 / 120 = 0.0083 [sec].

[0119] According to one or more embodiments, one or more processors (120) may obtain a touch trajectory based on a first input point and a second input point, and display the touch trajectory in a frame subsequent to the second frame.

[0120] According to one or more embodiments, the next frame includes a third frame consecutive to the second frame or a fourth frame consecutive to the third frame, and for convenience of explanation, the next frame will be described as the fourth frame below.

[0121] According to one or more embodiments, one or more processors (120) detect a first touch at a time (t e-1 ) and the update time (t) of the display panel (110) for displaying the fourth frame displaying the touch trajectory based on the first input point and the second input point f3 ) can be used to obtain the buffering time between them.

[0122] For example, one or more processors (120) can obtain a buffering time based on the following mathematical expression 1.

[0123] [Mathematical Formula 1]

[0124] buffering time = t1+ t2+ t3+ t4,

[0125] In one or more examples, t1 is the difference between the times at which a touch is detected through the input device (200), i.e., the time at which the second touch is detected (t e ) - The detection point of the first touch (t e-1 ).

[0126] In one or more examples, t2 is the difference between the time when a touch is detected through the input device (200) and the time when the frame buffer is prepared to display the next frame, i.e., the update time (t) of the display panel (110) for displaying the third frame. f2 ) - The detection point of the second touch (te ).

[0127] In one or more examples, t3 is the refresh time or the time required for the frame buffer to display the next frame, i.e. the update time (t) of the display panel (110) to display the fourth frame. f3 ) - Update time (t) of the display panel (110) for displaying the third frame f2 ).

[0128] In one or more examples, t4 is the time required to perform an adaptive synchronization function (e.g., G-Sync, FreeSync) when the display panel (110) performs an adaptive synchronization function.

[0129] FIG. 8 is a diagram illustrating display time according to one or more embodiments.

[0130] Referring to FIG. 8, one or more processors (120) sequentially update the display panel (110) from the top to the bottom, so that a display time occurs between the update time of the display panel (110) and the display time of the touch trajectory, depending on the position where the touch trajectory is displayed within the screen.

[0131] For example, one or more processors (120) can obtain the display time based on the following mathematical expression 2.

[0132] [Equation 2]

[0133] display time = (PC / DS)·FTI,

[0134] In one or more examples, the parameter PC is coordinate information corresponding to the first touch.

[0135] In one or more examples, DS (Display size) is the size of the screen.

[0136] In one or more examples, the frame time interval (FTI) is the refresh time.

[0137] In one or more examples, while the fourth frame is being displayed, one or more processors (120) display the touch trajectory and update the display panel (110) for displaying the fourth frame (t f3 ) can be identified as the display time from the time when the touch trajectory is displayed (or the time when the area where the touch trajectory is located on the display panel (110) is updated).

[0138] FIG. 9 is a diagram illustrating a delay time according to one or more embodiments.

[0139] Referring to FIG. 9, one or more processors (120) can obtain a delay time between the detection time of the first touch and the display time of the touch trajectory based on the buffering time and the display time.

[0140] For example, the height of the screen may be 150 mm, the width of the screen may be 80 mm, and the coordinate information corresponding to the first touch may be (40, 75).

[0141] In one or more examples, the point in time at which the display panel (110) begins updating to display the second frame (hereinafter, update point in time) (t f1 ) is assumed to be 100ms, the update time (t) of the display panel (110) for displaying the third frame f2 ) is 108.3ms (=100ms+8.3ms), and the update time (t) of the display panel (110) for displaying the fourth frame f3 ) can be 116.6ms (=100ms+16.6ms).

[0142] In one or more examples, the time of detection of the first touch (t e-1 ) is 98ms, and the detection time of the second touch (te ) can be 106ms.

[0143] According to one or more embodiments, one or more processors (120) may obtain a buffering time based on mathematical expression 1.

[0144] t1= t e - t e-1 = 106 - 98 = 8ms,

[0145] t2= t f2 - t e = 108.3 - 106 = 2.3ms,

[0146] t3= t f3 - t f2 = 116.6 - 108.3 = 8.3ms,

[0147] t4=0 (assuming no adaptive synchronization function is performed).

[0148] buffering time = t1+ t2+ t3+ t4= 8 + 2.3 + 8.3 + 0 = 18.6ms

[0149] According to one or more embodiments, one or more processors (120) may obtain the display time based on mathematical expression 2.

[0150] display time = y-axis coordinate / height * FTI(frame time interval) = 75 / 150 * 8.33 = 4.17ms

[0151] In one or more examples, the display panel (110) may be updated from left to right rather than from top to bottom, and the display time may be obtained by calculating the x-axis coordinate / width * FTI.

[0152] According to an embodiment, one or more processors (120) can obtain a delay time using a buffering time and a display time based on the following mathematical expression 3.

[0153] delay time = min(buffering time + FTI, PT) - (FTI - display time)

[0154] In one or more examples, the parameter PT is the prediction time corresponding to the prediction point.

[0155] For example, PT detects the first touch at the time (t e-1 ) may include a prediction time (or prediction point) corresponding to the prediction point.

[0156] For example, one or more processors (120) may obtain a prediction trajectory including at least one prediction point using one of the above-described algorithms. For example, assume that one or more processors (120) obtain a prediction trajectory including three prediction points using one of the above-described algorithms.

[0157] FIG. 10 is a drawing illustrating a display device for displaying a predicted trajectory with an adjusted length according to one or more embodiments.

[0158] Referring to FIG. 10, one or more processors (120) can obtain a first prediction point, a second prediction point, and a third prediction point based on a first input point corresponding to a first touch and a second input point corresponding to a second touch.

[0159] For convenience of explanation, the prediction time (t) corresponding to the first prediction point p1 ) is 8ms, the prediction time corresponding to the second prediction point (t p2 ) is 16ms, the prediction time corresponding to the third prediction point (t p3 ) is assumed to be 29ms.

[0160] According to one or more embodiments, one or more processors (120) may compare a delay time and a prediction time corresponding to a prediction point.

[0161] As calculated in FIG. 9, one or more processors (120) can compare the sum of the buffering time and the refresh time (buffering time + FTI) (e.g., 26.9 (=18.6 + 8.3)) with the prediction time (PT).

[0162] In one or more examples, one or more processors (120) may adjust the length of the predicted trajectory based on the delay time to display a portion of the predicted trajectory if the sum of the buffering time and the refresh time is less than the predicted time.

[0163] delay time = min(buffering time + FTI, PT) - (FTI - display time) = min(26.9, prediction time corresponding to the third prediction point(t p3 )) - (8.3 - 4.17) = 26.9 -4.16 = 22.83ms

[0164] d p = (delaytime - t p2 ) / (t p3 - t p2 ) * d3= (22.83 - 16) / (29 - 16) * d3= 0.525 * d3

[0165] According to one or more embodiments, one or more processors (120) determine a prediction time (t) corresponding to a second prediction point by calculating a sum of the buffering time and the refresh time (buffering time + FTI). p2 ) is longer than the prediction time (t) corresponding to the third prediction point. p3 ) is shorter than the second prediction point, the length of the total prediction trajectory can be adjusted by adjusting the length of the prediction trajectory (d3) between the second prediction point and the third prediction point.

[0166] For example, one or more processors (120) obtain a first prediction point and a second prediction point based on a first input point corresponding to a first touch and a second input point corresponding to a second touch, and obtain a prediction time (t) corresponding to the first prediction point. p1 ) is 8ms, the prediction time corresponding to the second prediction point (t p2 ) can be assumed to be 16ms.

[0167] delay time = min(buffering time + FTI, PT) - (FTI - display time) = min(26.9, prediction time corresponding to the second prediction point (t p2 )) - (8.3 - 4.17) = 16 -4.16 = 11.87ms

[0168] d p = (delaytime - t p1 ) / (t p2 - t p1 ) * d2= (11.87 - 8) / (16 - 8) * d2= 0.48 * d2

[0169] According to one or more embodiments, one or more processors (120) determine a prediction time (t) corresponding to a first prediction point, where the delay time is p1 ) is longer than the prediction time (t) corresponding to the second prediction point. p2 ) is shorter than the first prediction point and the second prediction point, the length of the total prediction trajectory can be adjusted by adjusting the length of the prediction trajectory (d2) between the first prediction point and the second prediction point.

[0170] FIG. 11 is a diagram illustrating input lag occurring in a display device that updates frames in various directions according to one or more embodiments.

[0171] Referring to FIG. 11, the display panel (110) may be refreshed from top to bottom or from bottom to top.

[0172] According to one or more embodiments, one or more processors (120) may adjust the length of the predicted trajectory based on the time (e.g., display time) delayed until displaying the touch trajectory and predicted trajectory according to coordinate information corresponding to the first touch through the input device (200) within the screen, for example, along the y-axis coordinate, and then display the same.

[0173] According to one or more embodiments, the display panel (110) can display the predicted trajectory with the same latency whether the touch is detected in the upper region or the lower region.

[0174] In one or more examples, the display panel (110) may be refreshed from left to right or from right to left.

[0175] According to one or more embodiments, one or more processors (120) may adjust the length of the predicted trajectory based on the time delay until displaying the touch trajectory and predicted trajectory according to coordinate information corresponding to a first touch via the input device (200) within the screen, for example, along the x-axis coordinate.

[0176] According to one or more embodiments, the display panel (110) can display the predicted trajectory with the same latency whether the touch is detected in the left area or the right area.

[0177] FIG. 12 is a flowchart for explaining a method for controlling a display device according to an embodiment of the present disclosure.

[0178] Referring to FIG. 12, a method for controlling a display device acquires a first input point corresponding to the first touch when a first touch is detected on the screen while displaying a first frame among a plurality of frames included in an image (S1210).

[0179] When a second touch is detected on the screen while displaying a second frame among multiple frames, a second input point corresponding to the second touch is acquired (S1220).

[0180] In the next frame of the second frame, a touch trajectory based on the first input point and the second input point is displayed (S1230).

[0181] A predicted trajectory is obtained based on the first input point and the second input point (S1240).

[0182] In the next frame, the length of the predicted trajectory is adjusted based on the coordinate information corresponding to the first touch within the screen and displayed continuously along the touch trajectory (S1250).

[0183] A control method according to one or more embodiments further includes a step of sequentially displaying each of a plurality of frames by updating the display panel from top to bottom or from left to right during a refresh rate, and the step S1250 of adjusting the length of the predicted trajectory to display may include a step of obtaining a display time between an update time of the display panel for displaying the next frame and a display time of the touch trajectory in the next frame, based on a ratio of coordinate information to a size of the screen.

[0184] The step of obtaining a display time according to one or more embodiments includes a step of obtaining a display time based on a ratio of a y-axis coordinate included in coordinate information for a height of a screen when updating a display panel from top to bottom during a refresh rate, and the step S1250 of adjusting the length of a predicted trajectory to display may include a step of adjusting the length of the predicted trajectory based on the display time in the next frame.

[0185] The step of obtaining a display time according to one or more embodiments includes the step of obtaining a display time based on a ratio of an x-axis coordinate included in coordinate information for a width (weight) of a screen when updating a display panel from left to right during a refresh rate, and the step S1250 of adjusting the length of a predicted trajectory to display may include the step of adjusting the length of the predicted trajectory based on the display time in the next frame.

[0186] The step S1250 of adjusting and displaying the length of the predicted trajectory according to one or more embodiments may include the step of obtaining a buffering time between the detection time of the first touch and the update time, and the step of adjusting the length of the predicted trajectory based on the buffering time and the display time.

[0187] The step S1250 of adjusting and displaying the length of the predicted trajectory according to one or more embodiments may include the step of obtaining a delay time between the detection time of the first touch and the display time of the touch trajectory based on a buffering time and a display time, and the step of adjusting the length of the predicted trajectory by comparing the prediction time and the delay time corresponding to the predicted point included in the predicted trajectory.

[0188] The step S1250 of adjusting the length of a predicted trajectory according to one or more embodiments and displaying the predicted trajectory may include a step of adjusting the length of the predicted trajectory based on a delay time and displaying a portion of the predicted trajectory if the sum of a refresh time and a buffering time corresponding to a refresh rate is less than the predicted time.

[0189] The step S1250 of adjusting the length of the predicted trajectory according to one or more embodiments and displaying it may include the step of displaying a portion of the predicted trajectory based on the predicted time, the refresh time, and the display time, if the predicted time is less than the sum of the refresh time and the buffering time.

[0190] The step S1250 of adjusting and displaying the length of the prediction trajectory according to one or more embodiments may include a step of adjusting the length of the prediction trajectory between the first prediction point and the second prediction point based on the delay time, if the delay time is included between a first prediction time corresponding to a first prediction point included in the prediction trajectory and a second prediction time corresponding to a second prediction point.

[0191] A control method according to one or more embodiments may further include, when the display panel performs an adaptive synchronization function, adding a time required to perform the adaptive synchronization function to a buffering time.

[0192] It goes without saying that the various embodiments of the present disclosure can be applied not only to display devices but also to various types of electronic devices that detect touch.

[0193] Meanwhile, the various embodiments described above may be implemented in a computer-readable recording medium or similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In a software implementation, embodiments, such as the procedures and functions described herein, may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0194] Meanwhile, computer instructions for performing processing operations of a robot device according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in such a non-transitory computer-readable medium are executed by a processor of a specific device, the computer instructions cause the specific device to perform processing operations in the robot device (100) according to various embodiments described above.

[0195] A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0196] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. Memory that stores one or more instructions; a display panel including a touch sensor; and At least one processor connected to the memory and the display panel and configured to execute the one or more instructions; The instructions, when executed by the at least one processor, cause the display device to, when a first touch is detected on the screen of the display panel through the touch sensor while a first frame among a plurality of frames included in an image is displayed on the display panel, acquire a first input point corresponding to the first touch; When a second touch is detected on the screen while a second frame among the plurality of frames is displayed on the display panel, a second input point corresponding to the second touch is acquired, In the next frame after the second frame, a touch trajectory based on the first input point and the second input point is displayed on the display panel, Obtain a predicted trajectory based on the first input point and the second input point, In the next frame, the length of the predicted trajectory is adjusted based on coordinate information corresponding to the first touch within the screen, A display device that displays the predicted trajectory of the adjusted length together with the touch trajectory on the display panel so that, in the next frame, the predicted trajectory of the adjusted length and the predicted trajectory of the adjusted length form a continuous trajectory.

2. In paragraph 1, The instructions, when executed by the at least one processor, cause the display device to sequentially display each of the plurality of frames by updating the display panel from top to bottom or from left to right during a refresh rate. A display device that obtains a display time between the update time of the display panel for displaying the next frame and the display time of the touch trajectory in the next frame based on the ratio of the coordinate information to the size of the screen.

3. In paragraph 2, The instructions, when executed by the at least one processor, cause the display device to obtain the display time based on a ratio of the y-axis coordinate included in the coordinate information to the height of the screen when updating the display panel from top to bottom during the refresh rate, A display device that adjusts the length of the predicted trajectory based on the display time in the next frame.

4. In paragraph 2, The instructions, when executed by the at least one processor, cause the display device to obtain the display time based on a ratio of the x-axis coordinate included in the coordinate information to the width (weight) of the screen when updating the display panel from left to right during the refresh rate, A display device that adjusts the length of the predicted trajectory based on the display time in the next frame.

5. In paragraph 2, The above instructions, when executed by the at least one processor, cause the display device to obtain a buffering time between the detection time of the first touch and the update time of the display panel, A display device that adjusts the length of the predicted trajectory based on the buffering time and the display time.

6. In paragraph 5, The instructions, when executed by the at least one processor, cause the display device to obtain a delay time between the detection time of the first touch and the display time of the touch trajectory based on the buffering time and the display time, A display device that adjusts the length of the predicted trajectory by comparing the prediction time corresponding to the predicted point included in the predicted trajectory with the delay time.

7. In paragraph 6, The above instructions, when executed by the at least one processor, cause the display device to adjust the length of the predicted trajectory based on the delay time to display a portion of the predicted trajectory if the sum of the refresh time corresponding to the refresh rate and the buffering time is less than the predicted time.

8. In paragraph 6, A display device, wherein the instructions, when executed by the at least one processor, cause the display device to display a portion of the predicted trajectory based on the predicted time, the refresh time, and the display time, if the predicted time is less than the sum of the refresh time corresponding to the predicted time and the buffering time.

9. In paragraph 6, The instructions, when executed by the at least one processor, cause the display device to adjust the length of the prediction trajectory between the first prediction point and the second prediction point based on the delay time, if the delay time is included between a first prediction time corresponding to a first prediction point included in the prediction trajectory and a second prediction time corresponding to a second prediction point of the predicted trajectory.

10. In paragraph 1 The above instructions, when executed by the at least one processor, cause the display device, if the display panel performs an adaptive synchronization function, A display device that adds the time required to perform the above adaptive synchronization function to the buffering time.

11. In a method for controlling a display device, A step of acquiring a first input point corresponding to the first touch when a first touch is detected on a screen of a display panel of the display device while displaying a first frame among a plurality of frames included in an image; A step of acquiring a second input point corresponding to the second touch when a second touch is detected on the screen while displaying a second frame among the plurality of frames; In the next frame after the second frame, a step of displaying a touch trajectory based on the first input point and the second input point; A step of obtaining a predicted trajectory based on the first input point and the second input point; and A control method comprising: a step of displaying the predicted trajectory of the adjusted length together with the touch trajectory in the next frame so that the predicted trajectory of the adjusted length and the predicted trajectory of the adjusted length form a continuous trajectory; 12. In paragraph 11, The above control method is, Further comprising a step of sequentially displaying each of the plurality of frames by updating the display panel from top to bottom or from left to right during the refresh rate; The step of adjusting the length of the above predicted trajectory and displaying it is: A control method comprising: a step of obtaining a display time between an update time of the display panel for displaying the next frame and a display time of the touch trajectory in the next frame, based on a ratio of the coordinate information to the size of the screen; 13. In paragraph 12, The step of obtaining the above display time is: A step of obtaining the display time based on the ratio of the y-axis coordinate included in the coordinate information for the height of the screen when updating the display panel from top to bottom during the refresh rate; The step of adjusting the length of the above predicted trajectory and displaying it is: A control method comprising: a step of adjusting the length of the predicted trajectory based on the display time in the next frame; 14. In paragraph 12, The step of obtaining the above display time is: A step of obtaining the display time based on the ratio of the x-axis coordinate included in the coordinate information for the width (weight) of the screen when updating the display panel from left to right during the refresh rate; The step of adjusting the length of the above predicted trajectory and displaying it is: A control method comprising: a step of adjusting the length of the predicted trajectory based on the display time in the next frame; 15. In a non-transitory computer-readable medium storing instructions that, when executed by a processor or a display device, cause the processor to execute a method, the method comprises: A step of acquiring a first input point corresponding to the first touch when a first touch is detected on a screen of a display panel of the display device while displaying a first frame among a plurality of frames included in an image; A step of acquiring a second input point corresponding to the second touch when a second touch is detected on the screen while displaying a second frame among the plurality of frames; In the next frame after the second frame, a step of displaying a touch trajectory based on the first input point and the second input point; A step of obtaining a predicted trajectory based on the first input point and the second input point; and A non-transitory computer-readable medium comprising: a step of displaying the predicted trajectory of the adjusted length together with the touch trajectory in the next frame so that the predicted trajectory of the adjusted length and the predicted trajectory of the adjusted length form a continuous trajectory;

Images