System for touch input processing

Abstract

The invention provides a system for touch input processing. The system comprises the components of interactive hardware which comprises a display panel and a touch panel which operates touch firmware; the application processor is coupled with the interactive hardware and runs an application program and a touch driving program; wherein the application processor further converts the hardware synchronization signal of the display panel into a software synchronization signal, and generates an application synchronization signal based on the software synchronization signal; wherein expiration of the application synchronization signal triggers a user interface thread of the application program to draw one or more image layers in response to touch events collected by the touch panel; wherein the application processor further generates an input synchronization signal synchronized with the application synchronization signal; and wherein the expiration of the input synchronization signal triggers the touch control firmware to collect the touch control event, and the touch control event is transmitted to the application program through the touch control driving program.

Description

Technical Field

[0001] The present invention generally relates to display technology and communication technology used in electronic terminal devices, and in particular, to a system for touch input processing, such as an Android system. Background Art

[0002] In recent years, touch technology has become increasingly popular in electronic terminal devices, and users have increasingly higher requirements for the touch response and standby time of touch terminals.

[0003] Figure 1 is a diagram of the architecture of a prior art system 10 for touch input processing. Figure 1 As shown, the system 10 includes an application processor 11 running an application 112, and interactive hardware 12 including a display panel 120 and a touch panel 121. Generally, the touch panel 121 is used to collect touch events 150 and transmit the touch events 150 to the application processor 11. Then, the application processor 11 generates an output image 195 in response to the touch event 150 to be displayed on the display panel 120.

[0004] More details about the process from the user touching the screen to the screen presenting feedback are described as follows. When the user touches the screen, the touch firmware 122 run by the touch panel 121 collects the corresponding touch event 150. Subsequently, this touch event 150 is passed to the application processor 11. In addition to the application 112, the application processor 11 also runs the touch driver 110 and the InputFlinger 111 provided by the Android system. The touch driver 110 receives the touch event 150 from the touch firmware 122 and passes the touch event 150 to the application 112 through the InputFlinger 111. Then, the user interface thread 113 of the application 112 draws one or more image layers in response to the touch event 150. Subsequently, these image layers are rendered by the rendering thread 114 of the application 112. Then, the SurfaceFlinger thread 115 provided by the Android system combines the rendered image layers into an output image 195. Finally, according to the cycle of the hardware synchronization signal (hereinafter referred to as “HW_Vsync” in the figure) 160, the output image 195 is presented on the display panel 120, and the user can see their touch results.

[0005] Figure 2 This is a timing diagram showing the touch display delay in the early Android-based touch input processing system. Figure 2In the , each row from top to bottom shows the duration of the touch input processing, user interface thread, rendering thread, SurfaceFlinger thread, and display panel that handle touch events. The block marked with "0" represents the current touch event being processed, while the block marked with "1" represents the next touch event to be processed, and so on. "Touch input processing" refers to the touch driver and InputFlinger running on the application processor that receive and pass touch events to the application. In Figure 2 In the touch display delay, T0 is the time when the user touches the screen, and T1 is the time when the display panel responds to the user's touch by presenting corresponding visual feedback (such as pop-up windows or dialog boxes, scrolling interface content, cursor movement, or triggering game character actions). The time interval between T0 and T1 is called "touch display delay". Figure 2 In addition, the time from when the touch input processing completes the processing of the touch event to when the user interface thread actually starts drawing the image layer in response to the touch event is called the "touch response delay". Figure 2 Indicated as "L2" in the table.

[0006] like Figure 2 In the early Android-based system shown in Figure 1, the hardware synchronization signal (HW_Vsync) is used to unify the rhythm of drawing, rendering, combination and display. However, the touch display delay of the entire pipeline is relatively large. The touch display delay is a key factor affecting the smoothness of user operation. Therefore, reducing the touch display delay L1 can significantly improve the user experience. Based on this motivation, Figure 1 The system 10 shown in the figure adopts the mechanism of application synchronization signal and SurfaceFlinger synchronization signal. Figure 1 As shown, the user interface thread 113 and the SurfaceFlinger thread 115 are executed according to the cycles of the application synchronization signal (hereinafter referred to as "Vsync_APP" in the figure) 180 and the SurfaceFlinger synchronization signal (hereinafter referred to as "Vsync_SF" in the figure) 190, respectively. Specifically, the expiration of the application synchronization signal 180 triggers the user interface thread 113 to draw the image layer in response to the touch event 150, and the expiration of the SurfaceFlinger synchronization signal 190 triggers the SurfaceFlinger thread 115 to combine the image layers rendered by the rendering thread 114 into an output image 195 for display on the display panel 120. In addition, both the application synchronization signal 180 and the SurfaceFlinger synchronization signal 190 are generated based on the software synchronization signal (hereinafter referred to as "SW_Vsync" in the figure) 170, and the software synchronization signal 170 is converted from the hardware synchronization signal 160. The generation of the application synchronization signal 180 and the SurfaceFlinger synchronization signal 190 will be described in the next paragraph. Figure 3 Further explanation in.

[0007] Figure 3 is a schematic diagram illustrating the generation of application synchronization signals and SurfaceFlinger synchronization signals. Figure 3 As shown, the hardware synchronization signal 160 is converted into a software synchronization signal 170. Then, the application offset 301 and the SurfaceFlinger offset 302 are combined with the software synchronization signal 170 to generate the application synchronization signal 180 and the SurfaceFlinger synchronization signal 190 respectively. The application offset 301 refers to the time deviation between the start time of the user interface thread and the hardware synchronization signal. The SurfaceFlinger offset 302 refers to the time deviation between the start time of the SurfaceFlinger thread and the hardware synchronization signal. These offsets are designed to make the entire pipeline processing more efficient to improve system performance.

[0008] Figure 4 is a timing diagram showing the touch display delay in the prior art system 10. Figure 4 As shown, the user interface thread and rendering thread of the application start drawing and rendering according to the application synchronization signal (Vsync_App) in response to each touch event, the SurfaceFlinger thread starts to combine the image layer according to the SurfaceFlinger synchronization signal (Vsync_SF), and the display panel refreshes and displays the output image according to the hardware synchronization signal (HW_Vsync). In this way, for each touch event, rendering and combination are performed in time after the drawing is completed, resulting in visual feedback being presented on the display panel in a shorter time. Figure 4 As shown in the figure, T0' is the time when the user touches the screen, and T1' is the time when the display panel responds to the user's touch by presenting corresponding visual feedback. The time interval between T0' and T1', i.e., the touch display delay L1', is Figure 2 The touch display delay in L1 is much smaller.

[0009] In the implementation of the prior art, in order to extend the standby time of the touch terminal and reduce the system power consumption, the system 10 also involves an adaptive refresh technology to adjust the frequency of the hardware synchronization signal according to the display content. Figure 5 , which is a timing diagram of a touch display system involving adaptive refresh technology. Figure 5 As shown, the shaded area 500 indicates that when the displayed content is static or slightly changes, the system reduces the display refresh rate, that is, the frequency of the hardware synchronization signal (HW_Vsync) is reduced. When the displayed content changes dynamically, the refresh rate returns to a higher rate. In this way, the standby time of the touch terminal is extended and the power consumption is reduced.

[0010] In order to capture the user's touch actions instantly and further optimize the touch response delay L2, many terminal touch panels use high sampling frequencies, which means that the maximum sampling frequency of the touch panel is several times the refresh rate of the display panel. Figure 6 , which is a timing diagram of a touch display system with a touch panel running at a high sampling frequency. Figure 6 As shown, within a cycle of a hardware synchronization signal (HW_Vsync), the number of touch events collected and propagated by the high-frequency touch panel (3 in this case) is much larger than the number of touch events collected and propagated by the low-frequency touch panel (1 in this case). The touch response delay of the low-frequency touch panel is marked as L2, while the touch response delay of the high-frequency touch panel is marked as L2'. Obviously, L2' is much shorter than L2, indicating that the touch response delay of the high-frequency touch panel is lower. However, propagating touch events to the application processor at a high sampling frequency significantly increases power consumption.

[0011] In another prior art implementation, in order to solve the problem of high power consumption of high-frequency touch panels, the system 10 further monitors the hardware synchronization signal to determine the appropriate time to propagate the touch event. Figure 7 , which is a timing diagram of a touch display system that monitors hardware synchronization signals to determine when to propagate touch events. Figure 7 As shown in the figure, the timing of touch input processing (including the touch driver and InputFlinger that receive and propagate touch events) is scheduled to be performed immediately before the hardware synchronization signal (HW_Vsync) expires. In this way, the system does not need to propagate touch events to the application processor at a high frequency, thereby saving power consumption. However, the phase difference between the hardware synchronization signal (HW_Vsync) and the application synchronization signal (Vsync_APP) still causes a delay from the user interface thread processing the touch event to the completion of the touch event propagation, such as Figure 7 The touch response delay L2 is shown in the figure.

[0012] In addition to the problem of touch response delay, the above-mentioned system using the hardware synchronization signal monitoring mechanism may encounter additional problems when using adaptive refresh technology. Specifically, when the frequency of the application synchronization signal (Vsync_APP) is higher than the frequency of the hardware synchronization signal (HW_Vsync), if the corresponding hardware synchronization signal (HW_Vsync) is not generated in the frame and the application synchronization signal (Vsync_APP) expires, the user interface thread triggered by the application synchronization signal (Vsync_APP) will have no corresponding touch events to process. See Figure 8 , which is a timing diagram of a touch display system using adaptive refresh technology and hardware synchronization signal monitoring mechanism. Figure 8As shown in FIG. 8 , since no touch event is propagated to the application during the application synchronization signal (Vsync_APP) cycle, as shown in block 800, the user interface thread has no corresponding touch event to process during the period shown in block 801. This results in some touch events, such as Figure 8 The touch event marked as "2" in the figure may be missed, and no corresponding visual feedback may be presented to the user on the display panel.

[0013] Therefore, a touch input processing system is needed to solve the above problems. Summary of the invention

[0014] The content disclosed below is only exemplary and is not meant to be limited in any way. In addition to the illustrative aspects, embodiments and features, other aspects, embodiments and features will also be apparent by reference to the accompanying drawings and the following detailed description. That is, the content disclosed below is provided to introduce concepts, key points, benefits and novel and non-obvious technical advantages described herein. Optional but not all, the embodiments will be further described in detail as follows. Therefore, the content disclosed below is not intended to be a necessary feature of the claimed subject matter, nor is it intended to be used in determining the scope of the claimed subject matter.

[0015] One embodiment of the present invention provides a system for touch input processing. The system includes interactive hardware and an application processor. The interactive hardware includes a display panel and a touch panel running a touch firmware. The application processor is coupled to the interactive hardware and runs an application and a touch driver. The application processor further converts a hardware synchronization signal of the display panel into a software synchronization signal, and generates an application synchronization signal based on the software synchronization signal. The expiration of the application synchronization signal triggers the user interface thread of the application to draw one or more image layers in response to the touch events collected by the touch panel. In addition, the application processor further generates an input synchronization signal synchronized with the application synchronization signal. The expiration of the input synchronization signal triggers the touch firmware to collect touch events and pass the touch events to the application through the touch driver.

[0016] In one embodiment, the touch firmware includes a touch collection thread, a touch processing thread, and a control module. The touch collection thread is configured to collect a batch of touch events within a cycle of an application synchronization signal. The touch processing thread is configured to process the touch events collected by the touch collection thread, and store the processed touch events in a touch event buffer. The control module is configured to process a batch of touch events. Processing a batch of touch events is triggered by the expiration of an input synchronization signal, and includes triggering the touch collection thread and the touch processing thread to collect, process and store an instant touch event, and passing a batch of touch events including the instant touch event from the touch event buffer to the touch driver.

[0017] In one embodiment, the application processor obtains the expiration time of the application synchronization signal through a system call, sets the expiration time of the application synchronization signal as the reference time, predicts the phase time based on the system scheduling information, sets the generation time of the input synchronization signal as the sum of the reference time and the phase time, and uses the generation time of the input synchronization signal to start the timer to generate the input synchronization signal. The phase time refers to the time required for the touch driver and InputFlinger to pass the touch event to the application. The timer is configured to notify the touch driver through InputFlinger to pass the touch event to the application when the input synchronization signal expires.

[0018] In one embodiment, the application processor predicts the phase time by using a machine learning based algorithm and a de-bouncing algorithm.

[0019] In one embodiment, the application processor is directly connected to the touch panel.

[0020] In one embodiment, the application processor further runs a software timer and an interrupt service routine, and starts the software timer according to the input synchronization signal. The software timer is configured to notify the touch firmware to collect touch events through the interrupt service routine when the input synchronization signal expires, and pass the touch events to the touch driver. The software timer is further configured to notify the touch driver to pass the touch events to the application through the interrupt service routine when the input synchronization signal expires.

[0021] In one embodiment, the application processor further includes a hardware timer and an interrupt controller, and starts the hardware timer according to the input synchronization signal. The hardware timer is configured to notify the touch firmware to collect touch events when the input synchronization signal expires, and pass the touch events to the touch driver. The hardware timer is further configured to notify the touch driver to pass the touch events to the application program through the interrupt controller when the input synchronization signal expires.

[0022] In one embodiment, the system further includes a microprocessor and a shared memory. The microprocessor is directly connected to the touch panel and the application processor, includes a first interrupt controller, and runs a secondary touch driver. The expiration of the input synchronization signal triggers the first interrupt controller to send an interrupt request signal to the secondary touch driver. The shared memory can be accessed by the microprocessor and the application processor. The application processor further includes a second interrupt controller. The secondary touch driver is configured to process a touch event upon receiving the interrupt request signal. Processing the touch event includes receiving a touch event from the touch firmware, storing the touch event in the shared memory, and notifying the touch driver through the second interrupt controller to read the touch event from the shared memory.

[0023] In one embodiment, the microprocessor further includes a hardware timer, and the hardware timer is started according to the input synchronization signal. The hardware timer is configured to notify the secondary touch driver to process the touch event through the first interrupt controller when the input synchronization signal expires. In a further embodiment, the secondary touch driver is further configured to notify the touch firmware to collect touch events when receiving the interrupt request signal, and pass the touch events to the secondary touch driver. In another embodiment, the hardware timer is directly connected to the touch panel, and is further configured to notify the touch firmware to collect touch events when the input synchronization signal expires, and pass the touch events to the secondary touch driver.

[0024] In one embodiment, the application processor further includes a hardware timer directly connected to the first interrupt controller, and the hardware timer is started according to the input synchronization signal. The hardware timer is configured to notify the secondary touch driver to process the touch event through the first interrupt controller when the input synchronization signal expires. In a further embodiment, the secondary touch driver is further configured to notify the touch firmware to collect touch events when the interrupt request signal is received, and pass the touch events to the secondary touch driver. In another embodiment, the hardware timer is directly connected to the touch panel, and is further configured to notify the touch firmware to collect touch events when the input synchronization signal expires, and pass the touch events to the secondary touch driver.

[0025] In one embodiment, the application processor further generates a SurfaceFlinger synchronization signal based on the software synchronization signal. The rendering thread of the application is configured to render the image layer drawn by the user interface thread. The application processor further runs a SurfaceFlinger thread, which is triggered by the expiration of the SurfaceFlinger synchronization signal to combine the image layers rendered by the rendering thread into an output image for display on the display panel.

[0026] The system for touch input processing provided by the embodiment of the present invention can meet the user's requirements for touch response and touch terminal standby time while maintaining a high sampling rate and low power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are included in the content of the present invention to provide a further understanding of the content of the present invention and constitute a part of the content of the present invention. The accompanying drawings show embodiments of the content of the present invention and, together with the description, are used to explain the principles of the content of the present invention. It should be understood that the drawings are not necessarily drawn to scale. Due to the need to clearly illustrate the concepts of the content of the present invention, some components may be displayed out of proportion to the size in actual implementation.

[0028] Figure 1 It is an architectural diagram of the previous technical system.

[0029] Figure 2 is a timing diagram showing touch display latency in an earlier Android-based touch input processing system.

[0030] Figure 3 is a schematic diagram schematically illustrating the generation of an application synchronization signal and a SurfaceFlinger synchronization signal.

[0031] Figure 4 is a timing diagram showing touch display delay in a prior art system.

[0032] Figure 5 is a timing diagram of a display system involving adaptive refresh technology.

[0033] Figure 6 is a timing diagram of a touch display system operating at a high sampling frequency of the touch panel.

[0034] Figure 7 is a timing diagram of a touch display system that monitors hardware synchronization signals to determine when to propagate touch events.

[0035] Figure 8 It is a timing diagram of a touch display system using adaptive refresh technology and hardware synchronization signal monitoring mechanism.

[0036] Fig. 9 is an architecture diagram of an Android-based system for touch input processing according to an embodiment of the present invention.

[0037] Fig.10 is a timing diagram of an Android-based system for touch input processing according to an embodiment of the present invention.

[0038] Fig.11 is a timing diagram of an Android-based system for touch input processing using a batch propagation mechanism according to a further embodiment.

[0039] Fig.12 4 is an architecture diagram of a touch control firmware for implementing a batch propagation mechanism according to an embodiment of the present invention.

[0040] Fig.13 is a timing diagram of touch event processing in the touch firmware according to an embodiment of the present invention.

[0041] Fig.14 is a flow chart of a process of generating an input synchronization signal according to an embodiment of the present invention.

[0042] Fig.15 A first embodiment of an input synchronization signal expiration notification mechanism employed in a system for touch input processing is described.

[0043] Fig.16A second embodiment of an input synchronization signal expiration notification mechanism employed in a system for touch input processing is described.

[0044] Fig.17 A third embodiment is described in which a microprocessor is included in a system for touch input processing to implement touch event propagation.

[0045] Fig.18 A first implementation of the third embodiment of including a microprocessor in a system for touch input processing to implement touch event propagation is described.

[0046] Fig.19 A second implementation of the third embodiment of including a microprocessor in a system for touch input processing to implement touch event propagation is described.

[0047] Fig. 20 A third implementation manner of the third embodiment in which a system for touch input processing includes a microprocessor to implement touch event propagation is described.

[0048] Fig.21 A fourth implementation manner of the third embodiment in which a system for touch input processing includes a microprocessor to implement propagation of touch events is described. DETAILED DESCRIPTION

[0049] Certain words are used in the specification and claims to refer to specific components. It should be understood by those skilled in the art that hardware manufacturers may use different terms to refer to the same component. This specification and claims do not use differences in names as a way to distinguish components, but rather use differences in the functions of the components as the criteria for distinction. The term "including" mentioned throughout the specification and claims is an open-ended term and should be interpreted as "including but not limited to". In addition, the term "coupled" herein includes any direct and indirect means of electrical connection. Therefore, if the text describes a first device coupled to a second device, it means that the first device can be directly electrically connected to the second device, or indirectly electrically connected to the second device through other devices or connection means.

[0050] The following description is intended to illustrate the general principles of the invention and is not to be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims.

[0051] The following description is intended to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements. Various modifications to the embodiments and general principles and features described herein will be apparent to those skilled in the art. Therefore, the present invention is not intended to be limited to the embodiments shown, but rather to the widest scope consistent with the principles and features described herein.

[0052] Fig. 9 is an architecture diagram of an Android-based system 90 for touch input processing according to an embodiment of the present invention. Fig. 9 As shown, the system 90 includes an application processor 91 running an application 912, and interactive hardware 92 including a display panel 920 and a touch panel 921. The application processor 91 and the interactive hardware 92 are coupled to each other and can communicate with each other. Generally, the touch panel 921 is used to collect touch events 950 and pass the touch events 950 to the application processor 91. Then, the application processor 91 generates an output image 995 in response to the touch event 950 to be displayed on the display panel 920.

[0053] More details about the process from the user touching the screen to the screen presenting feedback in the system 90 will be described below. When the user touches the screen, the touch firmware 922 run by the touch panel 921 collects the corresponding touch event 950. The touch event 950 may involve the touch position (i.e., the coordinates of the touch point on the screen), the touch pressure (i.e., the amount of pressure applied to the screen), the touch size (i.e., the size of the touch area), the touch time (i.e., the timestamp of the touch event), and / or other related information to help the system accurately interpret and respond to user interactions. Subsequently, the touch event 950 is passed to the application processor 91. In addition to the application 912, the application processor 91 also runs the touch driver 910 and the InputFlinger 911 provided by the Android system. The touch driver 910 receives the touch event 950 from the touch firmware 922 and passes the touch event 950 to the application 912 through the InputFlinger 911. Then, the user interface thread 913 of the application 912 draws one or more image layers in response to the touch event 950. These image layers are then rendered by the rendering thread 914 of the application 912. The SurfaceFlinger thread 915 provided by the Android system then combines the rendered image layers into an output image 995. Finally, according to the cycle of the hardware synchronization signal 960, the output image 995 is presented on the display panel 920, and the user can see the results of their touches.

[0054] In addition, the user interface thread 913 and the SurfaceFlinger thread 915 are executed according to the cycles of the application synchronization signal 980 and the SurfaceFlinger synchronization signal 990, respectively. Specifically, the expiration of the application synchronization signal 980 triggers the user interface thread 913 to draw the image layer in response to the touch event 950, and the expiration of the SurfaceFlinger synchronization signal 990 triggers the SurfaceFlinger thread 915 to combine the image layers rendered by the rendering thread 914 into an output image 995 for display on the display panel 920. In addition, the application synchronization signal 980 and the SurfaceFlinger synchronization signal 990 are both generated based on the software synchronization signal (hereinafter referred to as "SW_Vsync" in the figure) 970, and the software synchronization signal 970 is converted from the hardware synchronization signal 960. The process of the application processor 91 generating the application synchronization signal 980 and the SurfaceFlinger synchronization signal 990 is as described above. Figure 3 shown and are not repeated here.

[0055] and Figure 1 Compared with the prior art system 10 shown in the figure, the system 90 further combines the use of an input synchronization signal (hereinafter referred to as "Vsync_TP") 985 to control the timing of the propagation of the touch event 950. Specifically, in addition to the application synchronization signal 980 and the SurfaceFlinger synchronization signal 990, the application processor 91 also generates an input synchronization signal 985 synchronized with the application synchronization signal 980. The expiration of the input synchronization signal 985 triggers the touch firmware 922 to collect the touch event 950 and pass the touch event 950 to the application 912 through the touch driver 910.

[0056] Fig.10 This is a timing diagram of an Android-based system for touch input processing according to an embodiment of the present invention. In this embodiment, the input synchronization signal (Vsync_TP) is synchronized with the application synchronization signal (Vsync_APP). In other words, each cycle of the input synchronization signal (Vsync_TP) depends on the application synchronization signal (Vsync_APP). For example, when the refresh rate of the display panel is 120 Hz, the cycle of the application synchronization signal (Vsync_APP) is approximately 8.3 milliseconds, and the cycle of the input synchronization signal (Vsync_TP) is also approximately 8.3 milliseconds. In this way, the cycle of the input synchronization signal (Vsync_TP) always starts a certain phase difference before the application synchronization signal (Vsync_APP). As Fig.10As shown, this phase difference is set to be approximately equal to the time required for the touch input processing to process a touch event (including the touch driver and InputFlinger that receive and transmit touch events). As a result, there is almost no touch response delay from the time the user interface thread processes the touch event to the time the touch event is transmitted, creating a smoother user touch experience.

[0057] As mentioned above, many terminal touch panels use high sampling frequencies. This may result in the touch panel collecting several touch events within one application synchronization signal (Vsync_APP) cycle. In order to solve this problem without increasing the touch event propagation frequency (for energy saving considerations), a further embodiment of the present invention adopts a batch propagation mechanism. See Fig.11 , which is a timing diagram of a system for touch input processing based on an Android system using a batch propagation mechanism according to this embodiment. Fig.11 As shown, within an application synchronization signal (Vsync_APP) cycle, the touch input processing propagates a batch of touch events 1100. This batch of touch events 1100 includes touch events 1101 and 1102 collected before the input synchronization signal expires, and an immediate touch event 1103 collected when the input synchronization signal expires. Subsequently, the user interface thread will draw the corresponding one or more image layers according to the content of the touch events 1101-1103.

[0058] The above batch propagation mechanism is implemented in the touch firmware running on the touch panel. Fig.12 , which is an architecture diagram of the touch firmware 922 implementing the batch propagation mechanism according to an embodiment of the present invention. Fig.12 As shown, the touch firmware 922 may include a control module 1201, a touch collection thread 1202, and a touch processing thread 1203. The touch collection thread 1202 is configured to collect a batch of touch events within a cycle of the application synchronization signal. The touch processing thread 1203 is configured to process the touch events collected by the touch collection thread 1202 and store the processed touch events in the touch event buffer 1204. The control module 1201 is configured to process the batch of touch events. The processing of the batch of touch events is triggered by the expiration of the input synchronization signal (Vsync_TP) 985. More details about processing the batch of touch events will be explained in the next paragraph.

[0059] When the input synchronization signal expires, the control module 1201 triggers the touch collection thread 1202 to collect an instant touch event. The instant touch event is then processed by the touch processing thread 1203. Processing instant touch events and other touch events may include operations of parsing, filtering, de-jittering and / or smoothing their contents, but the present invention is not limited thereto. After processing, the instant touch event is stored in the touch event buffer 1204, which may have stored other touch events accumulated within a cycle of the application synchronization signal. The touch event buffer 1204 can be implemented using a volatile memory, such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), but the present invention is not limited thereto. The control module 1201 then propagates the batch of touch events stored in the touch event buffer 1204 upward to the touch driver 910 running on the application processor. The touch driver 910 then propagates the batch of touch events upward to the application 912 (in Fig.12 not shown).

[0060] Fig.13 FIG. 9 is a timing diagram of touch event processing in the touch firmware 922 according to an embodiment of the present invention. Fig.13 In the example, the block marked with "0" represents the touch events being collected by the touch collection thread 1202 after the user touch action. The blocks marked with negative numbers such as -1 and -2 represent past touch events that have been processed by the corresponding thread and stored in the touch event buffer 1204, waiting to be propagated to the application processor. The blocks marked with positive numbers such as 1, 2 and 3 represent future touch events that will be collected and processed. If polling is performed based on the input synchronization signal 985, the delay will be longer, and at least three cycles will be required after the touch action is triggered before the touch driver 910 can receive the touch event, thereby increasing the touch response delay of the touch event. In order to further reduce the touch response delay of the touch event, the touch firmware 922 must use instant sampling technology to process and propagate the touch event to the application processor. This is why the above-mentioned instant touch events must be included in the batch of touch events being processed. When the input synchronization signal 985 expires, the touch collection thread 1202 and the touch processing thread 1203 are triggered. After completion, the touch event is propagated upward through the interruption, so that the touch driver 910 receives the touch event as soon as possible. Fig.13 As shown, L0 is the delay of the "interrupt" method, L0' is the delay of the "polling" method, and L0 is shorter than L0'.

[0061] Fig.14 FIG. 9 is a flow chart of a process for generating an input synchronization signal 985 according to an embodiment of the present invention. Fig.14As shown, the application processor 91 may generate an input synchronization signal (Vsync_TP) 985 by executing steps 1401 - 1403 .

[0062] In step 1401, the application processor 91 obtains the expiration time of the application synchronization signal (Vsync_APP) 980 through a system call, such as " / vsync_app". Then, the application processor sets the expiration time of the application synchronization signal 980 as a reference time.

[0063] In step 1402, the application processor 91 predicts the phase time based on the system scheduling information. The phase time refers to the time required for the touch driver and InputFlinger to propagate the touch event to the application. The system scheduling information may involve CPU load, process priority and / or scheduling queue, but the present invention is not limited thereto.

[0064] In one embodiment, the prediction of the phase time is achieved by using a machine learning based algorithm and a de-jittering algorithm. Specifically, a trained regression model or a neural network can be used to initially predict the phase time based on the system scheduling information. Considering that the fluctuation of the phase time value may affect the consistency of the propagation timing, the predicted phase time should be processed by a de-jittering algorithm and then updated through a system call, such as " / vsync_phase". However, it should be noted that the machine learning based algorithm is not the only way to predict the phase time. In other embodiments, the phase time prediction can be achieved using a theoretical formula, a lookup table with interpolation, or other specific algorithms.

[0065] In step 1403, the application processor 91 sets the generation time of the input synchronization signal 985 to the sum of the reference time and the phase time, and starts a timer using the generation time of the input synchronization signal 985. The timer is configured to notify the touch driver 910 to propagate the touch event to the application through InputFlinger when the input synchronization signal (Vsync_TP) expires.

[0066] It should be noted that the phase time will change dynamically with the system scheduling conditions. Therefore, the above phase time prediction mechanism takes into account the system scheduling information to ensure that the touch events are consumed by the user interface thread at the best expected timing in different system scenarios, thereby achieving the best system performance.

[0067] In some embodiments, the application processor 91 is directly connected to the touch panel 921. Therefore, the application processor 91 and the touch panel 921 can communicate via GPIO or other appropriate interface protocols. In addition, the application processor 91 can directly notify the touch firmware 922 of the expiration of the input synchronization signal 985, thereby triggering the collection and propagation of touch events. In the system 90, two embodiments of the input synchronization signal expiration notification mechanism in which the application processor 91 is directly connected to the touch panel 921 will be referred to. Fig.15 and Fig.16 .

[0068] Fig.15 An embodiment of a first input synchronization signal expiration notification mechanism used in a touch input processing system is described. Fig.15 As shown, in addition to the application 912 and the touch driver 910, the application processor 91 also runs a software timer 1501 and an interrupt service routine 1502. In addition, the application processor 91 starts the software timer 1501 according to the input synchronization signal 985. The software timer 1501 is configured to notify the touch firmware 922 to collect touch events 950 through the interrupt service routine 1502 when the input synchronization signal 985 expires, and pass the touch events 950 to the touch driver 910. At the same time, the software timer 1501 is configured to notify the touch driver 910 to pass the touch events 950 to the application 912 through the interrupt service routine 1502 when the input synchronization signal 985 expires.

[0069] Fig.16 An embodiment of a second input synchronization signal expiration notification mechanism used in a touch input processing system is described. Fig.16 As shown, the application processor 91 further includes a hardware timer 1601 and an interrupt controller 1602. The application processor 91 starts the hardware timer 1601 according to the input synchronization signal 985. The hardware timer 1601 is configured to notify the touch firmware 922 to collect touch events 950 and pass the touch events 950 to the touch driver 910 when the input synchronization signal 985 expires. At the same time, the hardware timer 1601 is also configured to notify the touch driver 910 to pass the touch events 950 to the application 912 through the interrupt controller 1602 when the input synchronization signal 985 expires.

[0070] Fig.15 and Fig.16A comparison of the two embodiments shown is as follows. In the first embodiment, software handles the timing of the input synchronization signal 985 and notifies the touch firmware 922 through the software-controlled GPIO, without the need for additional hardware components. However, the software timer 1501 and the interrupt service routine 1502 consume system computing resources and may reduce signal transmission efficiency when the system is heavily loaded. In contrast, the second embodiment directly uses interrupts to notify the touch firmware 922 and the touch driver 910, thereby enhancing system stability and improving signal transmission efficiency.

[0071] In some embodiments, the touch event 950 is not transmitted directly from the touch panel 921 to the application processor 91, but is transmitted through a microprocessor. This approach takes into account that the microprocessor consumes less power than the application processor 91 to receive the touch event data from the touch panel 921. Therefore, using a microprocessor to facilitate the propagation of the touch event 950 can further reduce system power consumption.

[0072] Fig.17 A third embodiment of a touch input processing system is described, in which a microprocessor 1700 is included to implement the propagation of touch events 950. In the third embodiment, the microprocessor 1700 is directly connected to the touch panel 921 and the application processor 91. Therefore, the microprocessor 1700 can communicate with the touch panel 921 and the application processor 91 via GPIO or other appropriate interface protocols. The microprocessor 1700 includes an interrupt controller 1701 and runs a secondary touch driver 1702. In addition, the system also includes a shared memory 1710, which can be accessed by both the microprocessor 1700 and the application processor 91. The shared memory 1710 can be implemented using volatile memory, such as dynamic random access memory (DRAM) or static random access memory (SRAM), but the content of the present invention is not limited to this. The application processor 91 also includes an interrupt controller 1715. About Fig.17 More details of the additional elements in and their interactions are explained in the next paragraph.

[0073] In a third embodiment, the expiration of the input synchronization signal 985 triggers the interrupt controller 1701 to send an interrupt request signal to the secondary touch driver 1702. The secondary touch driver 1702 is configured to process the touch event 950 after receiving the interrupt request signal. For the secondary touch driver 1702, processing the touch event includes receiving the touch event 950 propagated from the touch firmware 922, storing the touch event 950 in the shared memory 1710, and notifying the touch driver 910 to read the touch event 950 from the shared memory 1710 through the interrupt controller 1715. After receiving the notification from the secondary touch driver 1702, the touch driver 910 reads the touch event 950 from the shared memory 1710 and passes the touch event 950 to the application 912. The subsequent processes of image drawing, rendering, combination and display are as follows. Fig. 9 As shown, no further description is given here.

[0074] Different input synchronization signal expiration notification mechanisms can bring different implementations to the third embodiment. Specifically, the input synchronization signal expiration notification mechanism involves using a hardware timer to notify the secondary touch driver 1702 that the input synchronization signal 985 has expired, triggering the secondary touch driver 1702 to process the touch event 950. This hardware timer can be deployed in the application processor 91 or the microprocessor 1700. In addition, the hardware timer or the secondary touch driver 1702 can be used to notify the touch firmware 922 that the input synchronization signal 985 has expired, triggering the touch firmware 922 to collect and propagate the touch event 950. Accordingly, four implementation methods of the third embodiment will be demonstrated, see Figure 18-21 .

[0075] Fig.18 The first implementation of the third embodiment of the touch input processing system is shown, in which the system includes a microprocessor 1700 to implement the propagation of the touch event 950. In the first implementation, the microprocessor 1700 includes a hardware timer 1800. The microprocessor 1700 starts the hardware timer 1800 according to the input synchronization signal 985. The hardware timer 1800 is configured to notify the secondary touch driver 1702 to process the touch event 950 through the interrupt controller 1701 when the input synchronization signal 985 expires. For the secondary touch driver 1702, processing the touch event is as described above with reference to Fig.17 On the other hand, the secondary touch driver 1702 is further configured to notify the touch firmware 922 to collect the touch event 950 and pass the touch event 950 to the secondary touch driver 1702 after receiving the interrupt request signal from the interrupt controller 1701 .

[0076] Fig.19A second implementation of the third embodiment of the system for touch input processing is shown, wherein the system includes a microprocessor 1700 to implement the propagation of touch events 950. In the second implementation, the microprocessor 1700 includes a hardware timer 1900. In addition, the hardware timer 1900 is directly connected to the touch panel 921, so that the hardware timer 1900 can communicate with the touch panel 921 through GPIO or other appropriate interface protocols. The microprocessor 1700 starts the hardware timer 1900 according to the input synchronization signal 985. The hardware timer 1900 is configured to notify the secondary touch driver 1702 to process the touch event 950 through the interrupt controller 1701 when the input synchronization signal 985 expires. For the secondary touch driver 1702, the processing of the touch event is as follows Fig.17 On the other hand, the hardware timer 1900 is further configured to notify the touch firmware 922 to collect the touch event 950 and propagate the touch event 950 to the secondary touch driver 1702 when the input synchronization signal 985 expires.

[0077] Fig. 20 A third implementation of the third embodiment of the system for touch input processing is shown, wherein the system includes a microprocessor 1700 to implement the propagation of touch events 950. In the third implementation, the application processor 91 includes a hardware timer 2000. In addition, the hardware timer 2000 is directly connected to the interrupt controller 1701, so that the hardware timer 2000 can communicate with the interrupt controller 1701 through GPIO or other appropriate interface protocols. The application processor 91 starts the hardware timer 2000 according to the input synchronization signal 985. The hardware timer 2000 is configured to notify the secondary touch driver 1702 through the interrupt controller 1701 to process the touch event 950 when the input synchronization signal 985 expires. For the secondary touch driver 1702, the processing of the touch event is as follows Fig.17 On the other hand, the secondary touch driver 1702 is further configured to notify the touch firmware 922 to collect the touch event 950 and propagate the touch event 950 to the secondary touch driver 1702 when receiving the interrupt request signal from the interrupt controller 1701 .

[0078] Fig.21A fourth implementation of the third embodiment of the system for touch input processing is shown, wherein the system includes a microprocessor 1700 to implement the propagation of touch events 950. In the fourth implementation, the application processor 91 includes a hardware timer 2100. In addition, the hardware timer 2100 is directly connected to the interrupt controller 1701 and the touch panel 921, so that the hardware timer 2100 can communicate with the interrupt controller 1701 and the touch panel 921 through GPIO or other appropriate interface protocols. The application processor 91 starts the hardware timer 2100 according to the input synchronization signal 985. The hardware timer 2100 is configured to notify the secondary touch driver 1702 through the interrupt controller 1701 to process the touch event 950 when the input synchronization signal 985 expires. For the secondary touch driver 1702, the processing of the touch event is as follows Fig.17 On the other hand, the hardware timer 2100 is further configured to notify the touch firmware 922 to collect the touch event 950 and propagate the touch event 950 to the secondary touch driver 1702 when the input synchronization signal 985 expires.

[0079] Figure 18-21 A comparison of the four implementations shown is described below. In the scheme where the hardware timer is deployed in the microprocessor 1700 (i.e. Fig.18 and Fig.19 In the first and second implementations shown in FIG. 1 , the application processor 91 needs to send a system message to the microprocessor 1700 to start the hardware timer when the input synchronization signal 985 expires. This prolongs the notification path and the overall system touch response delay to some extent. In contrast, in the solution where the hardware timer is deployed in the application processor 91 (i.e. Fig. 20 and Fig.21 In the third and fourth implementations shown in the figure), the hardware timer can directly communicate with the interrupt controller 1701 through GPIO or other appropriate interface protocols, thereby shortening the notification path and the touch response delay of the entire system. Therefore, the third implementation is better than the first implementation, and the fourth implementation is better than the second implementation.

[0080] On the other hand, in the scheme where the hardware timer is directly connected to the touch panel 921 (i.e. Fig.19 and Fig.21 In the second and fourth implementations shown in FIG. 1 , the hardware timer can directly notify the touch firmware 922 of the expiration of the input synchronization signal 985 through GPIO or other appropriate interface protocols to trigger the collection and propagation of touch events 950. Compared with the other two solutions (i.e. Fig.18 and Fig. 20Compared with the first implementation and the third implementation shown in FIG. 1 , this method is more efficient and stable. The latter two solutions require the secondary touch driver 1702 to notify the touch firmware 922 through the software-controlled GPIO. Therefore, the second implementation is better than the first implementation, and the fourth implementation is better than the third implementation. Among these four implementations, the fourth implementation is the best implementation of the third embodiment.

[0081] In addition, it is worth mentioning that in a system that includes a microprocessor, a software timer can be configured at the application processor layer to handle signal timing and expiration notification. Using software-controlled GPIO for notification can eliminate the need for additional hardware components. However, software timers consume system computing resources, making their signal transmission performance less efficient compared to hardware timer solutions.

[0082] The Android-based touch input processing system provided by the embodiment of the present invention can meet the user's requirements for touch response and touch terminal standby time while maintaining a high sampling rate and low power consumption.

[0083] Any specific order or hierarchy of steps in the procedures disclosed herein is purely by way of example. Based on design preferences, it should be understood that any specific order or hierarchy of steps in the procedures may be rearranged within the scope of this document. The accompanying method claims present various steps in an exemplary order and therefore should not be limited to the specific order or hierarchy shown herein.

[0084] The use of ordinal numbers such as "first", "second", "third", etc. to modify components in the claims does not itself imply any priority, precedence, sequence between components, or order of steps performed by the method, but is only used as an identifier to distinguish different components with the same name (with different ordinal numbers).

[0085] Although the present disclosure has been disclosed as above by way of implementation examples, it is not intended to limit the present case. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present case shall be determined by the definition of the claims.

Claims

1. A system for touch input processing, comprising: interactive hardware, including a display panel and a touch panel running touch firmware; as well as an application processor coupled to the interactive hardware and running an application program and a touch driver; Wherein, the application processor further converts the hardware synchronization signal of the display panel into a software synchronization signal, and generates an application synchronization signal based on the software synchronization signal; wherein the expiration of the application synchronization signal triggers a user interface thread of the application to draw one or more image layers in response to touch events collected by the touch panel; Wherein, the application processor further generates an input synchronization signal synchronized with the application synchronization signal; and The expiration of the input synchronization signal triggers the touch firmware to collect the touch event, and transmits the touch event to the application program through the touch driver.

2. The system for touch input processing according to claim 1, characterized in that: The touch firmware includes: A touch collection thread, configured to collect a batch of touch events within a cycle of the application synchronization signal; a touch processing thread, configured to process the touch events collected by the touch collection thread and store the processed touch events in a touch event buffer; and A control module is configured to process the batch of touch events, wherein the processing of the batch of touch events is triggered by the expiration of the input synchronization signal and includes triggering the touch collection thread and the touch processing thread to collect, process and store an instant touch event, and pass the batch of touch events including the instant touch event from the touch event buffer to the touch driver.

3. The system for touch input processing according to claim 1, wherein: The application processor generates the input synchronization signal by performing the following steps: Obtain the expiration time of the application synchronization signal through a system call, and set the expiration time of the application synchronization signal as the reference time; Predicting a phase time based on the system scheduling information, wherein the phase time refers to a time required by the touch driver and InputFlinger to pass the touch event to the application; and The generation time of the input synchronization signal is set as the sum of the reference time and the stage time, and the generation time of the input synchronization signal is used to start the timer; When the input synchronization signal expires, the timer is configured to notify the touch driver program so as to pass the touch event to the application program through the InputFlinger.

4. The system for touch input processing according to claim 3, characterized in that: The application processor predicts this phase time by using a machine learning-based algorithm and a de-bounce algorithm.

5. The system for touch input processing according to claim 1, wherein: The application processor is directly connected to the touch panel.

6. The system for touch input processing according to claim 5, characterized in that: The application processor further runs a software timer and an interrupt service routine, and starts the software timer according to the input synchronization signal; When the input synchronization signal expires, the software timer is configured to notify the touch firmware through the interrupt service routine to collect the touch event and pass the touch event to the touch driver; and When the input synchronization signal expires, the software timer is further configured to notify the touch driver program through the interrupt service routine to pass the touch event to the application program.

7. The system for touch input processing according to claim 5, characterized in that: The application processor further includes a hardware timer and an interrupt controller, and starts the hardware timer according to the input synchronization signal; When the input synchronization signal expires, the hardware timer is configured to notify the touch firmware to collect the touch event and transmit the touch event to the touch driver; and When the input synchronization signal expires, the hardware timer is further configured to notify the touch driver program through the interrupt controller to pass the touch event to the application program.

8. The system for touch input processing according to claim 1, wherein: Further including: a microprocessor, directly connected to the touch panel and the application processor, including a first interrupt controller, and running a secondary touch driver, wherein expiration of the input synchronization signal triggers the first interrupt controller to send an interrupt request signal to the secondary touch driver; and a shared memory accessible by the microprocessor and the application processor; Wherein, the application processor further includes a second interrupt controller; When the interrupt request signal is received, the secondary touch driver is configured to process the touch event, and processing the touch event includes receiving the touch event from the touch firmware, storing the touch event in the shared memory, and notifying the touch driver through the second interrupt controller to read the touch event from the shared memory.

9. The system for touch input processing according to claim 8, characterized in that: The microprocessor further includes a hardware timer, and the hardware timer is started according to the input synchronization signal; When the input synchronization signal expires, the hardware timer is configured to notify the secondary touch driver program through the first interrupt controller to process the touch event.

10. The system for touch input processing according to claim 9, characterized in that: When the interrupt request signal is received, the secondary touch driver is further configured to notify the touch firmware to collect the touch event and transmit the touch event to the secondary touch driver.

11. The system for touch input processing according to claim 9, characterized in that: The hardware timer is directly connected to the touch panel, and when the input synchronization signal expires, the hardware timer is further configured to notify the touch firmware to collect the touch event and transmit the touch event to the secondary touch driver.

12. The system for touch input processing according to claim 8, wherein: The application processor further includes a hardware timer directly connected to the first interrupt controller, and the hardware timer is started according to the input synchronization signal; When the input synchronization signal expires, the hardware timer is configured to notify the secondary touch driver program to process the touch event through the first interrupt controller.

13. The system for touch input processing according to claim 12, wherein: When the interrupt request signal is received, the secondary touch driver is further configured to notify the touch firmware to collect the touch event and transmit the touch event to the secondary touch driver.

14. The system for touch input processing according to claim 12, wherein: The hardware timer is directly connected to the touch panel, and when the input synchronization signal expires, the hardware timer is further configured to notify the touch firmware to collect the touch event and transmit the touch event to the secondary touch driver.

15. The system for touch input processing according to claim 1, wherein: The application processor further generates a SurfaceFlinger synchronization signal based on the software synchronization signal; Wherein, configuring the rendering thread of the application to render the image layer drawn by the user interface thread; The application processor further runs a SurfaceFlinger thread, which is triggered by the expiration of the SurfaceFlinger synchronization signal to synthesize the image layer rendered by the rendering thread into an output image for display on the display panel.

Images