Screen input event processing method and device, storage medium and electronic equipment

Abstract

A screen input event processing method and apparatus, a storage medium, and an electronic device, relating to the technical field of human-computer interaction, the method comprising: capturing, through a reference transparent layer, a screen input event acting on a target application through a current panel window corresponding to the target application (S110); determining a first event occurrence position of the screen input event in the current panel window (S120); determining a current pixel value at the first event occurrence position, and determining an event processing rule of the screen input event according to the current pixel value (S130); and processing the screen input event based on the event processing rule to obtain an event processing result (S140). And the execution efficiency of the screen input event is improved.

Description

Screen input event processing method and device, storage medium, and electronic device Technical Field

[0001] The embodiments of the present disclosure relate to the field of human-computer interaction technology, and in particular, to a method for processing screen input events, a device for processing screen input events, a computer-readable storage medium, and an electronic device. Background Art

[0002] In existing event processing methods, it is impossible to determine event processing rules based on the location of the event, which leads to low event execution efficiency.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in this field.

[0004] Summary of the Invention

[0005] The purpose of the present disclosure is to provide a method for processing screen input events, a device for processing screen input events, a computer-readable storage medium, and an electronic device, thereby overcoming, at least to a certain extent, the problem of low event execution efficiency caused by the limitations and defects of related technologies.

[0006] According to one aspect of the present disclosure, a method for processing a screen input event is provided, which is configured in a display terminal, wherein the display terminal includes a current panel window and a reference transparent layer. The method for processing the screen input event includes:

[0007] capturing, through the reference transparent layer, a screen input event acting on the target application through the current panel window corresponding to the target application;

[0008] Determining a first event occurrence position of the screen input event in the current panel window;

[0009] Determining a current pixel value at the location where the first event occurs, and determining an event processing rule for the screen input event based on the current pixel value;

[0010] The screen input event is processed based on the event processing rule to obtain an event processing result.

[0011] In an exemplary embodiment of the present disclosure, capturing, through the reference transparent layer, a screen input event acting on a target application through a current panel window corresponding to the target application, includes:

[0012] Starting a first preset program in the display terminal, and using the first preset program as a main startup program to start the target application; wherein the first preset program is an embedded hidden program;

[0013] Loading a transparent full-screen container in the first preset program, and loading the current panel window of the target application and a reference transparent layer in the transparent full-screen container;

[0014] A preset event monitoring function is configured in the reference transparent layer, and based on the event monitoring function in the reference transparent layer, a screen input event acting on the target application through the current panel window corresponding to the target application is captured.

[0015] In an exemplary embodiment of the present disclosure, capturing a screen input event acting on a target application through a current panel window corresponding to the target application based on an event monitoring function in the base transparent layer includes:

[0016] Obtain the real-time refresh frequency of the twin application corresponding to the target application, and interval the real-time refresh frequency, based on the event monitoring function in the baseline transparent layer, capture the screen input events acting on the target application through the current panel window corresponding to the target application.

[0017] In an exemplary embodiment of the present disclosure, determining the first event occurrence position of the screen input event in the current panel window includes:

[0018] Obtaining the current coordinate position of the screen input event in the current panel window and the current screen resolution of the display terminal;

[0019] A first event occurrence position of the screen input event in the current panel window is determined according to the current screen resolution and the current coordinate position.

[0020] In an exemplary embodiment of the present disclosure, determining the current pixel value at the location where the first event occurs includes:

[0021] Using the first preset program as the main startup program, starting the twin application corresponding to the target application, and intercepting the twin program interface of the twin application corresponding to the target application through the first preset program;

[0022] According to the location where the first event occurs, the second event location where the screen input event occurs in the twin program interface is determined, and the current pixel value is obtained according to the pixel value at the second event location.

[0023] In an exemplary embodiment of the present disclosure, intercepting the twin program interface of the twin application program through the first preset program includes:

[0024] Based on a preset application program interface, obtain a first window handle of the target application and a second window handle of the twin application through a first preset program;

[0025] Integrating the first window handle and the second window handle into the first preset program, and monitoring screen input events acting on the target application based on the first window handle through the first preset program;

[0026] When a screen input event acting on the target application is detected, the current display interface of the target application is obtained based on the first window handle; wherein the interface shape of the current display interface is consistent with the shape of the current panel window;

[0027] Based on the second window handle, the twin application is controlled to display the current display interface, and the current display interface is intercepted from the twin application based on the second window handle through the first preset program to obtain the twin program interface.

[0028] In an exemplary embodiment of the present disclosure, obtaining the current pixel value according to the pixel value at the location where the second event occurs includes:

[0029] Draw a main display window corresponding to the display terminal and hide the main display window in the toolbar of the terminal device; wherein the main display window is a purely transparent frameless window, and the main display window is presented in the form of a modal window, and the shape of the main display window is consistent with the size of the current panel window;

[0030] intercepting the transparent display interface corresponding to the main display window, and determining a third event occurrence position of the screen input event in the transparent display interface;

[0031] Obtaining a first pixel value at the location where the third event occurs in the transparent display interface, and a second pixel value at the location where the second event occurs in the twin program interface;

[0032] The first pixel value and the second pixel value are compared, and the current pixel value is obtained according to the pixel value comparison result.

[0033] In an exemplary embodiment of the present disclosure, comparing the first pixel value and the second pixel value, and obtaining the current pixel value according to the pixel value comparison result, includes:

[0034] Obtaining a first alpha channel value in the first pixel value and a second alpha channel value in the second pixel value, and determining whether the first alpha channel value and the second alpha channel value are consistent;

[0035] If the first Alpha channel value and the second Alpha channel value are consistent, the current pixel value is determined to be zero; if the first Alpha channel value and the second Alpha channel value are inconsistent, the current pixel value is determined to be non-zero.

[0036] In an exemplary embodiment of the present disclosure, determining an event processing rule for the screen input event according to the current pixel value includes:

[0037] If the current pixel value is zero, determining that the event processing rule of the screen input event is a first event processing rule; wherein the first event processing rule is to transparently transmit the screen input event to the model rendering flow layer of the display terminal;

[0038] If the current pixel value is non-zero, it is determined that the event processing rule of the screen input event is a second event processing rule; wherein the second event processing rule is to process the screen input event through the reference transparent layer.

[0039] In an exemplary embodiment of the present disclosure, processing the screen input event based on the event processing rule to obtain an event processing result includes:

[0040] Transmitting the screen input event to the model rendering flow layer based on a first event processing rule, and processing the screen input event based on the model rendering flow layer to obtain a first event processing result; or

[0041] The screen input event is processed through the reference transparent layer based on a second event processing rule to obtain a second event processing result.

[0042] In an exemplary embodiment of the present disclosure, the screen input event is transparently transmitted to the model rendering flow layer based on the first event processing rule, and the screen input event is processed based on the model rendering flow layer to obtain a first event processing result, including:

[0043] Controlling the base transparent layer to release the control right of the screen input event, and setting the window style of the current panel window to a first preset window display style; wherein the first preset window display style includes a WS_EX_LAYERED style and a WS_EX_TRANSPARENT style;

[0044] Transmitting the screen input event to the model rendering flow layer, and obtaining the to-be-processed three-dimensional model corresponding to the screen touch event through the model rendering flow layer;

[0045] Perform interactive processing on the three-dimensional model to be processed to obtain a model interactive processing result; wherein, the interactive processing on the three-dimensional model to be processed includes: performing one or more of dragging processing, zooming processing, clicking processing and lens perspective switching processing on the three-dimensional model to be processed.

[0046] In an exemplary embodiment of the present disclosure, the method for processing a screen input event further includes:

[0047] The model interaction processing result is transmitted to the three-dimensional model interaction area in the current panel window through the model rendering flow layer via the reference transparent layer;

[0048] The model interaction processing result is displayed through the three-dimensional model interaction area.

[0049] In an exemplary embodiment of the present disclosure, processing the screen input event through the reference transparent layer based on the second event processing rule to obtain a second event processing result includes:

[0050] Controlling the base transparent layer to retain control of the screen input event, and setting the window style of the current panel window to a second preset window display style; wherein the second preset window display style includes a WS_EX_TRANSPARENT style;

[0051] Based on the control of the baseline transparent layer over the screen input event, the screen input event is interactively processed to obtain a second event processing result; wherein, the second event processing result includes at least one of pop-up window closing, sub-interface display, sub-interface closing, and attribute information display.

[0052] In an exemplary embodiment of the present disclosure, the method for processing a screen input event further includes:

[0053] The current panel window is set based on WindowChrome.

[0054] According to one aspect of the present disclosure, a device for processing screen input events is provided, which is configured in a display terminal, wherein the display terminal includes a current panel window and a reference transparent layer. The device for processing screen input events includes:

[0055] A screen input event capture module, configured to capture, through the reference transparent layer, screen input events acting on the target application via the current panel window corresponding to the target application;

[0056] A first event occurrence position determination module, configured to determine a first event occurrence position of the screen input event in the current panel window;

[0057] an event processing rule determination module, configured to determine a current pixel value at a location where the first event occurs, and determine an event processing rule for the screen input event based on the current pixel value;

[0058] The screen input event processing module is used to process the screen input event based on the event processing rule to obtain an event processing result.

[0059] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for processing a screen input event described above is implemented.

[0060] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0061] processor; and

[0062] a memory for storing executable instructions of the processor;

[0063] The processor is configured to execute any one of the above-mentioned methods for processing screen input events by executing the executable instructions.

[0064] A method for processing screen input events provided by an embodiment of the present disclosure, on the one hand, captures screen input events that act on a target application through a current panel window corresponding to the target application through a base transparent layer; then determines the first event occurrence position of the screen input event in the current panel window; then determines the current pixel value at the first event occurrence position, and determines an event processing rule for the screen input event based on the current pixel value; finally, processes the screen input event based on the event processing rule to obtain an event processing result. Since the event processing rule can be determined based on the event occurrence position, the screen input event can be processed based on the event processing rule, thereby improving the event execution efficiency; on the other hand, since the screen input event that acts on the target application through the current panel window corresponding to the target application can be captured through the base transparent layer, then the event processing rule is determined and the event processing is performed, any type of event transparent transmission is achieved, thereby avoiding the problem of poor adaptability caused by the need to enter a transparent page in Web form.

[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0067] FIG1 schematically shows a flow chart of a method for processing a screen input event according to an exemplary embodiment of the present disclosure.

[0068] FIG2 schematically shows a structural example diagram of a display terminal according to an exemplary embodiment of the present disclosure.

[0069] FIG3 schematically shows an example diagram of a region structure of a current panel window according to an exemplary embodiment of the present disclosure.

[0070] FIG4 schematically shows an example diagram of a display scene of a current panel window according to an exemplary embodiment of the present disclosure.

[0071] FIG5 schematically shows an example diagram of a display scene of a special-shaped window according to an exemplary embodiment of the present disclosure.

[0072] FIG6 schematically shows an example diagram of a display result of a model interaction processing result according to an example embodiment of the present disclosure.

[0073] FIG7 schematically shows an example diagram of a scenario in which a sub-interface is displayed in a current panel window according to an exemplary embodiment of the present disclosure.

[0074] FIG8 schematically illustrates an example scenario of displaying a pop-up display interface in a current panel window according to an exemplary embodiment of the present disclosure.

[0075] FIG9 schematically illustrates an example scenario of closing a pop-up window display interface in a current panel window according to an exemplary embodiment of the present disclosure.

[0076] FIG10 schematically illustrates an example scenario of displaying a property information interface in a current panel window according to an exemplary embodiment of the present disclosure.

[0077] FIG11 schematically shows a block diagram of a device for processing screen input events according to an exemplary embodiment of the present disclosure.

[0078] FIG12 schematically illustrates an electronic device for implementing the method for processing the screen input event according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0079] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0080] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0081] The existing IOC (Intelligent Operations Center) panel implementation scheme can be mainly achieved through the following centralized methods:

[0082] Solution 1 for implementing the IOC panel: Using materials and UMG (Universal Media Gateway) as the technical implementation solution, and then dynamically loading it through the engine; however, in actual application, this method requires rendering the UMG interface simultaneously with every frame rendered by the engine. Furthermore, while UMG is one of the engine's native UI designers, compared to traditional UI development languages, it has higher compatibility and implementation limits, performance, usage methods, and learning costs, as well as a high performance overhead.

[0083] Solution 2 for implementing the IOC panel: transparently embedding it in a web page. Although this approach can be developed using traditional front-end languages ​​and is lightweight, due to the performance limitations of the browser kernel, its adaptability and effectiveness are not ideal in high-resolution and large-screen situations.

[0084] Solution 3 for implementing the IOC panel: Embed the twin layer into an independent IOC program in the form of a stream. This method performs better than the previous two methods in terms of lightweightness and resolution. However, in some demonstration scenarios, such as no network environment, poor network or large network fluctuation environment, the latency rate is high and the picture is not clear.

[0085] Based on this, this example embodiment first provides a method for processing screen input events, which can be run on a display terminal. The display terminal can be applied to various scenarios such as conferences, teaching, and commercial use, and the smart interactive tablet has interactive functions such as touch and a large size (for example, generally larger than 50 inches); at the same time, the display terminal can also be applied to scenarios such as smart cities and smart medical care; of course, those skilled in the art can also run the method of this disclosure on other platforms (for example, servers, server clusters or cloud servers, etc.) as needed, and this exemplary embodiment does not specifically limit this.

[0086] In some possible example embodiments, in the scenario of a smart city, for example, when it is necessary to view the traffic flow of a certain lane, the street light conditions of a certain road, the flow of people in a certain area of ​​the city, or the specific conditions of a certain building, this can be achieved based on the method for processing the screen input event. For example, in the scenario of viewing the usage of the street lights in a certain lane, the current panel window can be touched, and the screen input event acting on the IOC program can be captured through the reference transparent layer; and the first event occurrence position of the screen input event in the current panel window can be determined, the event processing rules can be determined, and the screen input event can be processed to obtain the final event processing result; wherein, the specific event processing result can include but is not limited to displaying all the street lights in the lane, turning the street lights off or on, etc.; if display is required, it can be displayed in the corresponding display area (such as the three-dimensional model interaction area).

[0087] In some possible example embodiments, the method for processing screen input events described in the example embodiments of the present disclosure can be used in a scenario based on multi-EXE kneading; wherein, the multi-EXE kneading scenario refers to the ability to knead the baseline transparent layer, the current panel window, the model rendering flow layer, the target application, the twin application, and the transparent full-screen container together; based on this, it is possible to dynamically implement the process of superimposing multiple EXEs on each other, and at the same time, dynamically switch instructions and accept handles according to the user's operations without switching the modal state on the current screen, thereby achieving the purpose of timely responding to screen input events input by the user.

[0088] In some possible example embodiments, the method for processing screen input events recorded in the example embodiments of the present disclosure can be applied in the following scenarios: on the one hand, in exhibition scenarios with poor network environments, offline deployment of multiple programs can be used to complete exhibitions or displays; that is, the method for processing screen input events can be integrated into the display terminal through offline deployment, thereby achieving the purpose of processing screen input events even in scenarios with poor network environments; on the other hand, the method for processing screen input events recorded in the example embodiments of the present disclosure requires third-party integration, or a stack of programs mixed in multiple languages ​​to complete the entire system; at the same time, it can also be compatible with multi-dimensional development languages ​​without the need for wrappers (encapsulation) between multi-dimensional languages; on the other hand, the method for processing screen input events recorded in the example embodiments of the present disclosure can also be integrated into a passive (codeless, or long-lasting and difficult to maintain) client program, thereby achieving the purpose of timely responding to screen input events input by users.

[0089] It should be noted here that the method for processing screen input events recorded in the example embodiments of the present disclosure can be applied to multiple different projects. For example, in the UBP (Universal Business Park, Hengtong International Business Park) project, the processing of screen input events can be implemented based on the transparent transmission of the mouse + model layer; at the same time, in the remote equipment collaborative control simulation system, the processing of screen input events can be implemented based on the transparent transmission of mouse operations during the animation of any sidebar.

[0090] It should be further explained here that the method for processing screen input events recorded in the example embodiments of the present disclosure can be directly embedded in the corresponding client or the corresponding display terminal through program integration, without the need for control through external configuration, thereby improving the convenience of screen input events in the processing process and the efficiency of event processing; at the same time, the method can also be applied to multiple display terminals with different screen resolutions; for example, it can be applied to a variety of display terminals with different screen resolutions such as 1K to 8K; further, the method for processing screen input events can be deployed to the client or the corresponding display terminal through offline deployment when the network environment is not good; of course, when the network environment is good, it can also be deployed to the client or the corresponding display terminal through online dynamic deployment; in the actual application process, the corresponding deployment method can be selected according to the actual situation, and this example does not impose any special restrictions on this.

[0091] In an exemplary embodiment, referring to FIG1 , the method for processing a screen input event may include the following steps:

[0092] Step S110. Capturing, through the reference transparent layer, screen input events acting on the target application through the current panel window corresponding to the target application;

[0093] Step S120: Determine the first event occurrence position of the screen input event in the current panel window;

[0094] Step S130: determining a current pixel value at the location where the first event occurs, and determining an event processing rule for the screen input event based on the current pixel value;

[0095] Step S140: Process the screen input event based on the event processing rule to obtain an event processing result.

[0096] In the above-mentioned method for processing screen input events, on the one hand, the screen input events acting on the target application through the current panel window corresponding to the target application are captured through the base transparent layer; then the first event occurrence position of the screen input event in the current panel window is determined; then the current pixel value at the first event occurrence position is determined, and the event processing rule of the screen input event is determined based on the current pixel value; finally, the screen input event is processed based on the event processing rule to obtain the event processing result. Since the event processing rule can be determined according to the event occurrence position, the screen input event can be processed based on the event processing rule, thereby improving the event execution efficiency; on the other hand, since the screen input events acting on the target application through the current panel window corresponding to the target application can be captured through the base transparent layer, then the event processing rule is determined and the event processing is performed, any type of event transmission is realized, thereby avoiding the problem of poor adaptability caused by the need to enter a transparent page in Web form.

[0097] At the same time, based on the above-mentioned records, it can be known that when the method for processing screen input events recorded in the example embodiment of the present disclosure is applied in a smart city scenario, on the one hand, the event processing efficiency will be improved; on the other hand, if it is necessary to process the model object corresponding to the screen input event through the model rendering flow layer, the model object can be accurately positioned, thereby improving the analysis of the reasons for abnormal lighting of street lights, or analyzing the reasons for excessive pedestrian and vehicle traffic, thereby achieving the purpose of smart city management.

[0098] Hereinafter, the method for processing screen input events described in the exemplary embodiments of the present disclosure will be explained and illustrated in detail with reference to the accompanying drawings.

[0099] First, the terms involved in the exemplary embodiments of the present disclosure are explained and illustrated.

[0100] Digital Twin: Digital Twin is a simulation process that fully utilizes data such as physical models, sensor updates, and operating history to integrate multiple disciplines, multiple physical quantities, multiple scales, and multiple probabilities. In actual application, digital twins can complete mapping in virtual space, thereby reflecting the entire life cycle of the corresponding physical equipment. In other words, digital twins are based on the same device or system to create a digital version of the "clone" program corresponding to the target application.

[0101] IOC Data Dashboard: The digital twin IOC (Intelligent Operations Center) of a smart city is often a system built on various smart application systems. It can empower actual business applications and improve the level of operation and maintenance management by sharing and integrating business information of various industry departments. In the process of actual application, with the maturity of various technologies such as IOT (Internet of Things), 5G, cloud computing, big data and artificial intelligence, the construction of "digital twin IOC-intelligent twin" has become a trend in the digital transformation of various industries. The IOC data dashboard involved in the example embodiments of this disclosure generally refers to a visual dashboard with data as the main body.

[0102] Event transmission: Specifically, events and transmission in event transmission are two behaviors; among them, events refer to the mouse clicks, double-clicks, drags, roaming and other operations performed by users on the twin model level in non-IOC occlusion areas, for example, it can be used to refer to the screen input events recorded in the example embodiments of the present disclosure; and transmission refers to this user behavior, which occurs in the IOC data dashboard layer, but is a behavior that actually acts on the twin model layer (causing the model to change due to user operations or changes in interactive responses). For example, the screen input event can be transmitted to the baseline transparent layer through the current panel window.

[0103] High performance: The high performance here refers to a state in which IOC data display is achieved without occupying too many display resources of the GPU (Graphics Processing Unit); at the same time, the high performance recorded here is for the traditional IOC panel integrated into the digital twin program, and the additional performance consumption brought about by the engine itself rendering; in the method for processing screen input events recorded in the example embodiment of the present disclosure, a decoupling method can be used to reduce GPU utilization.

[0104] Secondly, the application scenarios of the exemplary embodiments of the present disclosure are explained and illustrated. Specifically, the method for processing screen input events recorded in the exemplary embodiments of the present disclosure provides an integration solution for native containers based on WPF (Windows Presentation Foundation, a user interface framework based on Windows), the purpose of which is to realize the superposition of multiple programs such as the IOC program panel and the twin applications corresponding to the IOC program panel in the same display terminal, thereby realizing a high-performance event processing solution for screen input events.

[0105] Next, the display terminal described in the exemplary embodiment of the present disclosure is explained and illustrated. Specifically, as shown in FIG2 , the display terminal described in the exemplary embodiment of the present disclosure may include a current panel window 210, a reference transparent layer 220, and a model rendering flow layer 230. In actual application, the current panel window is at the top level, the reference transparent layer is at the middle level, and the model rendering flow layer is at the bottom level. The specific positional relationship of each level can be specifically referred to as shown in FIG2 .

[0106] In an exemplary embodiment, the current panel window 210 may include multiple different window areas. For example, as shown in FIG3 , the current panel window may include a first window display area 301, a second window display area 302, and a three-dimensional model interaction area 303. Furthermore, as shown in FIG4 , the first window display area may include multiple first interactive controls 401, the second window display area may include multiple second interactive controls 402, and the three-dimensional model interaction area may include multiple different three-dimensional models, such as a three-dimensional building model 403 and a three-dimensional road model 404, etc. This example does not impose any special restrictions on this. In a specific application process, if the first window display area and the second window display area receive a screen input event, the information corresponding to the screen input event can be displayed. The specific display area can be suspended in the area of ​​the interactive control corresponding to the screen input event, or it can be displayed directly in the 3D model interactive area. For example, if the interactive control selected by the user is to display a map of a certain area, it can be displayed directly in the corresponding area or in the 3D model interactive area. If the user needs to zoom in on a certain part of the map, the corresponding screen input event can be continued to be input in the 3D model interactive area. Of course, it can also be implemented in other ways, and this example does not impose any special restrictions on this.

[0107] Below, the programs, windows and containers involved in the exemplary embodiments of the present disclosure will be explained and illustrated. Specifically, the target application recorded in this disclosure can be used to refer to the IOC panel program; the twin model program recorded in this disclosure refers to the twin application of the IOC panel program; the main display window recorded in this disclosure refers to the display window corresponding to the main display interface of the display terminal, and the main display window (also referred to as the main window) can be a purely transparent borderless window, which is hidden in the toolbar of the display terminal; at the same time, in the actual application process, the main display window also needs to be set as a modal window; based on this, pixel comparison can be achieved without affecting the display effect of the IOC panel program, and then the event processing rules can be determined based on the comparison results.

[0108] Furthermore, the example embodiment of the present disclosure also includes a transparent full-screen container, which can accommodate a transparent interlayer (i.e., a base transparent layer) and the current panel window corresponding to the IOC panel program; at the same time, the reason for setting up the transparent full-screen container is to be able to capture screen input events acting on the current panel window through the base transparent layer, and then determine the event processing rules based on the occurrence location of the screen input event and perform event processing, thereby avoiding the performance limitations caused by the need to embed transparent pages in the form of a Web. It should be noted here that in order to achieve the transparent transmission of screen input events to the base transparent layer through the current panel window, the current panel window needs to be set to the following style:

[0109] WS_EX_LAYERED and WS_EX_TRANSPARENT styles;

[0110] Furthermore, in order to ensure that the current panel form always has the WS_EX_LAYERED style, it is necessary to set the WS_EX_TRANSPARENT style when transparency is turned on; however, since the WPF window will automatically remove the WS_EX_LAYERED style (in the HwndTarget class) when AllowsTransparency=true is not set, if AllowsTransparency=true is set, the WPF built-in low-performance transparency implementation will be used. Therefore, it is necessary to use Hook to forcibly ensure the existence of this style without using the WPF built-in transparency implementation; in actual application, the additional properties can be rolled back at the same time after transparency is turned on to keep the screen input events transparent.

[0111] In an exemplary embodiment, the current panel window described herein can be set in the following manner: the current panel window is set based on WindowChrome. That is, in actual application, due to the particularity of the WPF window, AllowsTransparency=true is generally used to set it to transparent to achieve event transmission; however, given that the IOC window (current panel window) is usually displayed in full screen, WindowChrome can be used instead of the original AllowTransparency=true to improve performance; for example, at a resolution of 7680*4320, the video memory occupancy rate of WindowChorme is one tenth of AllowTransparency. Based on this setting method, the video memory occupancy rate of the current panel window on the display terminal can be greatly reduced, thereby achieving the purpose of high-performance event processing.

[0112] The following will further explain and illustrate the method for processing the screen input event shown in FIG1 in conjunction with FIG2 to FIG4. Specifically:

[0113] In step S110 , a screen input event acting on the target application through the current panel window corresponding to the target application is captured through the reference transparent layer.

[0114] Specifically, capturing screen input events that act on the target application through the current panel window corresponding to the target application through the baseline transparent layer can be achieved in the following way: first, starting a first preset program in the display terminal, and using the first preset program as the main startup program to start the target application; wherein, the first preset program is an embedded hidden program; secondly, loading a transparent full-screen container in the first preset program, and loading the current panel window of the target application and the baseline transparent layer in the transparent full-screen container; then, configuring a preset event monitoring function in the baseline transparent layer, and based on the event monitoring function in the baseline transparent layer, capturing screen input events that act on the target application through the current panel window corresponding to the target application.

[0115] In an example embodiment, based on the event monitoring function in the baseline transparent layer, capturing the screen input events that act on the target application through the current panel window corresponding to the target application can be achieved in the following way: obtaining the real-time refresh frequency of the twin application corresponding to the target application, and spacing the real-time refresh frequency, based on the event monitoring function in the baseline transparent layer, capturing the screen input events that act on the target application through the current panel window corresponding to the target application.

[0116] The following will further explain and illustrate the specific capture process of screen input events. Specifically, in the actual application process, first, it is necessary to configure the IOC panel program, the twin application corresponding to the IOC panel program (also known as the twin model program) and the embedded hidden program in the display terminal; secondly, when the display terminal is started, the embedded hidden program can be started first, and then the embedded hidden program can be used as the main startup program to start the IOC panel program and the twin application; further, a transparent full-screen container is loaded in the embedded hidden program, and the current panel window and the base transparent layer are recorded in the transparent full-screen container, and then the screen input event is captured based on the base transparent layer; wherein The screen input events recorded here may include single-finger input events, multi-finger input events and events input through external devices; among them, the single-finger input events recorded here, for example, refer to click events, touch events, etc. input through a single finger, and multi-finger input events may refer to zoom-in events, zoom-out events input through two fingers, or other events input through multiple fingers, etc.; events input through external devices may refer to click events, touch events, zoom-in events, and zoom-out events, etc. input through a mouse; at the same time, the external devices recorded here may also include keyboards, handles, etc., and this example does not impose any special restrictions on this.

[0117] Furthermore, in the process of capturing screen input events, it can be achieved through an event monitoring function set in the baseline transparent layer; wherein, the event monitoring function recorded here can be a mouse hook (Hook), that is, the screen input event can be captured through the mouse hook (Hook); at the same time, the capture frequency of the screen input event can be determined according to the real-time refresh frequency of the twin application; in the scheme recorded in the example embodiment of the present disclosure, the real-time refresh frequency of the twin application can be reflected by FPS (Frame Per Second, the number of frames transmitted per second); that is, when the refresh FPS of the twin application is 35Hz, the capture frequency of the screen input event is also 35Hz; in this way, it can be ensured that the IOC panel program and the twin application maintain the same frequency and the display effect is consistent, thereby achieving the purpose of improving the accuracy of the current pixel value obtained.

[0118] In step S120, the first event occurrence position of the screen input event in the current panel window is determined.

[0119] Specifically, determining the first event occurrence position of the screen input event in the current panel window can be achieved in the following manner: first, obtaining the current coordinate position of the screen input event in the current panel window and the current screen resolution of the display terminal; second, determining the first event occurrence position of the screen input event in the current panel window based on the current screen resolution and the current coordinate position. That is, in actual application, the first event occurrence position can be determined based on the current screen resolution and the current coordinate position (that is, the actual coordinate position) of the screen input event in the current panel window (or the current display interface of the display terminal); wherein the first event sound position recorded here is also the position of the screen input event in the IOC panel program. It should be supplemented here that the reason for determining the position of the screen input event in the IOC panel program is to facilitate the determination of the current pixel value, and then determine the corresponding event processing rules based on the current pixel value.

[0120] In step S130, the current pixel value at the location where the first event occurs is determined, and an event processing rule for the screen input event is determined according to the current pixel value.

[0121] In this example embodiment, first, the current pixel value at the location where the first event occurs is determined. Specifically, the specific process of determining the current pixel value can be achieved in the following manner: first, using the first preset program as the main startup program, start the twin application corresponding to the target application, and intercept the twin program interface of the twin application corresponding to the target application through the first preset program; secondly, according to the location where the first event occurs, determine the second event location of the screen input event in the twin program interface, and obtain the current pixel value based on the pixel value at the second event location. That is, in actual application, it is first necessary to use the embedded hidden program as the main startup program, and then start the twin application based on the main startup program, and then intercept the twin program interface based on the embedded hidden program; wherein, the specific display result of the twin program interface recorded here is consistent with the display result of the IOC panel program.

[0122] In an example embodiment, intercepting the twin program interface of the twin application through the first preset program can be achieved as follows: first, based on the preset application interface, the first window handle of the target application and the second window handle of the twin application are obtained through the first preset program; secondly, the first window handle and the second window handle are integrated into the first preset program, and the screen input events acting on the target application are monitored based on the first window handle through the first preset program; then, when the screen input events acting on the target application are monitored, the current display interface of the target application is obtained based on the first window handle; wherein the interface shape of the current display interface is consistent with the shape of the current panel window; finally, the twin application is controlled to display the current display interface based on the second window handle, and the current display interface is intercepted from the twin application based on the second window handle through the first preset program to obtain the twin program interface.

[0123] In an exemplary embodiment, during actual application, first, a first window handle of a target application and a second window handle of a twin application can be obtained through a preset application program interface and a first preset program; wherein the preset application program interface recorded here can be WIN32API, wherein WIN32API (Windows 32 Application Programming Interface) is an application programming interface for Microsoft 32-bit platforms; at the same time, the window handle (HWND, Handle of Window), which can be used to uniquely identify the window where the target application is located, and the window where the twin application is located; at the same time, the first window handle and the second window handle recorded here can be the same or different, and this example does not impose any special restrictions on this; secondly, the first window handle and the second window handle can be integrated into the embedded hidden program, and then the current display interface is obtained based on the embedded hidden program, and the twin program interface is intercepted based on the second window handle; and the reason why the first window handle and the second window handle need to be integrated into the embedded hidden program here is to intercept the twin program interface based on a single entry and then perform pixel comparison, and finally determine the pixel value.

[0124] In an example embodiment, the current pixel value is obtained based on the pixel value at the location where the second event occurs, which can be achieved as follows: first, a main display window corresponding to the display terminal is drawn, and the main display window is hidden in the toolbar of the terminal device; wherein the main display window is a purely transparent borderless window, and the main display window is presented in the form of a modal window, and the shape of the main display window is consistent with the size of the current panel window; secondly, the transparent display interface corresponding to the main display window is intercepted, and the third event occurrence location of the screen input event in the transparent display interface is determined; then, the first pixel value at the location where the third event occurs in the transparent display interface, and the second pixel value at the location where the second event occurs in the twin program interface are obtained; finally, the first pixel value and the second pixel value are compared, and the current pixel value is obtained based on the pixel value comparison result. That is, in the actual application process, in order to obtain the current pixel value, it is necessary to compare the twin program interface and the main display window, and then determine the current pixel value based on the comparison result; at the same time, the main display window needs to be set to a transparent borderless window to ensure that the first Alpha channel value of the main display window is always zero, and then judge whether the first Alpha channel value and the second Alpha channel value are consistent, and then determine whether the second Alpha channel value is zero based on the comparison result. Whether the current pixel value is non-zero is determined based on the comparison result.

[0125] In an example embodiment, the first pixel value and the second pixel value are compared, and the current pixel value is obtained based on the pixel value comparison result. This can be achieved as follows: first, the first Alpha channel value in the first pixel value and the second Alpha channel value in the second pixel value are obtained, and it is determined whether the first Alpha channel value and the second Alpha channel value are consistent; second, if the first Alpha channel value and the second Alpha channel value are consistent, the current pixel value is determined to be zero; if the first Alpha channel value and the second Alpha channel value are inconsistent, the current pixel value is determined to be non-zero. It should be noted here that in actual application, RGB values ​​can also be compared while comparing Alpha channel values. This example does not impose any special restrictions on this. At the same time, the reason why the first Alpha channel value and the second Alpha channel value need to be compared here is because the IOC panel program is real-time; at the same time, at certain specific moments, there will be situations where the panel floats or there is a pop-up dialog box; in this scenario, the specific display scenario of the current panel window can refer to Figure 5; at the same time, in the scenario shown in Figure 5, due to the existence of a pop-up dialog box, it is impossible to ensure that the current panel window is in a completely transparent situation; for example, the area shown by 501 is an area that allows transparent transmission, and the area shown by 502 is an area that cannot be transparent; therefore, the screen input events received by the current panel window cannot be fully transmitted to the baseline transparent layer, and thus the event processing rules cannot be directly determined based on the baseline transparent layer, so it is necessary to rely on the current pixel value at the location where the first event occurs to determine the corresponding event processing rules. That is, the reason why the method for processing screen input events described in the exemplary embodiments of the present disclosure needs to determine the event processing rules through the current pixel value rather than directly through the location where the event occurs is to avoid the problem that when a pop-up dialog box exists in the IOC panel program, a pop-up dialog box floats up in the current panel window. When the screen input event acts on the pop-up dialog box, the screen input event cannot be directly transmitted to the base transparent layer based on the current panel window, and the event processing rules cannot be directly determined through the base transparent layer.

[0126] In an exemplary embodiment, determining the event processing rule for the screen input event based on the current pixel value can be implemented as follows: if the current pixel value is zero, determining the event processing rule for the screen input event to be the first event processing rule; wherein, the first event processing rule is to transparently transmit the screen input event to the model rendering flow layer of the display terminal; if the current pixel value is non-zero, determining the event processing rule for the screen input event to be the second event processing rule; wherein, the second event processing rule is to process the screen input event through the reference transparent layer. That is, in actual application, if the current pixel value is zero, it can be determined that there is no pop-up dialog box in the current panel window; when the current pixel value is non-zero, it can be determined that there is a pop-up dialog box in the current panel window.

[0127] In step S140, the screen input event is processed based on the event processing rule to obtain an event processing result.

[0128] Specifically, processing the screen input event based on the event processing rule to obtain an event processing result can be achieved in the following ways: based on the first event processing rule, the screen input event is transmitted to the model rendering flow layer, and the screen input event is processed based on the model rendering flow layer to obtain a first event processing result; or based on the second event processing rule, the screen input event is processed through the baseline transparent layer to obtain a second event processing result.

[0129] In an example embodiment, the screen input event is transmitted to the model rendering flow layer based on a first event processing rule, and the screen input event is processed based on the model rendering flow layer to obtain a first event processing result. This can be achieved in the following manner: controlling the baseline transparent layer to release control of the screen input event, and setting the window style of the current panel window to a first preset window display style; wherein, the first preset window display style includes the WS_EX_LAYERED style and the WS_EX_TRANSPARENT style; transmitting the screen input event to the model rendering flow layer, and obtaining the three-dimensional model to be processed corresponding to the screen touch event through the model rendering flow layer; performing interactive processing on the three-dimensional model to be processed to obtain a model interactive processing result; wherein, performing interactive processing on the three-dimensional model to be processed includes: performing one or more of dragging processing, scaling processing, clicking processing, and lens perspective switching processing on the three-dimensional model to be processed. That is, in actual application, if the current pixel value is zero, the baseline transparent layer releases control of the screen input event, and sets the style of the current panel window to WS_EX_LAYERED and WS_EX_TRANSPARENT styles, and then transmits the screen input event to the model rendering flow layer, and then performs model dragging, scaling, clicking, or lens perspective switching, etc.

[0130] Furthermore, after obtaining the model interaction processing result, the screen input event processing method may further include: transmitting the model interaction processing result via the model rendering flow layer through the base transparent layer to the 3D model interaction area in the current panel window; and displaying the model interaction processing result via the 3D model interaction area. In other words, after the model rendering flow layer performs the corresponding processing, the event processing result can be directly displayed; the specific display interface diagram of the obtained event processing result can be referenced as shown in Figure 6.

[0131] In an exemplary embodiment, the screen input event is processed through the reference transparent layer based on the second event processing rule to obtain a second event processing result, which can be achieved as follows: controlling the reference transparent layer to retain the control of the screen input event and setting the window style of the current panel window to a second preset window display style; wherein the second preset window display style includes the WS_EX_TRANSPARENT style; based on the control of the screen input event by the reference transparent layer, the screen input event is interactively processed to obtain a second event processing result; wherein the second event processing result includes at least one of pop-up window closing, sub-interface display, sub-interface closing, and attribute information display. That is, if the current pixel value is non-zero, the control of the screen input event can be directly fed back to the current panel window, and then the corresponding event processing is performed. wherein, the example diagram of the sub-interface display can refer to FIG7, the pop-up window display interface can refer to FIG8, the pop-up window closing interface can refer to FIG9, and the attribute information display can refer to FIG10.

[0132] At this point, the method for processing screen input events recorded in the exemplary embodiments of the present disclosure has been fully realized. Based on the above-mentioned recorded content, it can be known that the method for processing screen input events recorded in the exemplary embodiments of the present disclosure has at least the following advantages: on the one hand, it can realize the ability of conventional operations such as mouse clicks, drags, and double-clicks to manipulate the model body in the part that is not masked by the IOC panel; on the other hand, it does not occupy too much GPU resources, and the GPU share is significantly reduced compared to traditional IOC embedded rendering; on the other hand, it can realize the transparent transmission of events of any type of IOC panel, such as pop-ups and floats at any position in the interaction process, to realize the dynamic transparent transmission of the mouse; further, the IOC panel does not rely on the development language, and only requires an independent startup program based on the Windows platform, thereby further saving the video memory consumption of the display terminal.

[0133] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0134] The exemplary embodiments of the present disclosure further provide a device for processing screen input events, configured in a display terminal, wherein the display terminal includes a current panel window and a reference transparent layer. Specifically, referring to FIG11 , the device for processing screen input events may include a screen input event capture module 1110, a first event occurrence location determination module 1120, an event processing rule determination module 1130, and a screen input event processing module 1140. Specifically:

[0135] The screen input event capturing module 1110 may be configured to capture, through the reference transparent layer, screen input events acting on the target application via the current panel window corresponding to the target application;

[0136] The first event occurrence position determining module 1120 may be configured to determine the first event occurrence position of the screen input event in the current panel window;

[0137] The event processing rule determination module 1130 may be configured to determine a current pixel value at the location where the first event occurs, and determine an event processing rule for the screen input event based on the current pixel value;

[0138] The screen input event processing module 1140 may be configured to process the screen input event based on the event processing rule to obtain an event processing result.

[0139] In an exemplary embodiment of the present disclosure, the screen input events acting on the target application through the current panel window corresponding to the target application are captured through the baseline transparent layer, including: starting a first preset program in the display terminal, and using the first preset program as the main startup program to start the target application; wherein the first preset program is an embedded hidden program; loading a transparent full-screen container in the first preset program, and loading the current panel window of the target application and the baseline transparent layer in the transparent full-screen container; configuring a preset event monitoring function in the baseline transparent layer, and based on the event monitoring function in the baseline transparent layer, capturing the screen input events acting on the target application through the current panel window corresponding to the target application.

[0140] In an exemplary embodiment of the present disclosure, based on the event monitoring function in the baseline transparent layer, screen input events acting on the target application through the current panel window corresponding to the target application are captured, including: obtaining the real-time refresh frequency of the twin application corresponding to the target application, and spacing the real-time refresh frequency, and based on the event monitoring function in the baseline transparent layer, capturing screen input events acting on the target application through the current panel window corresponding to the target application.

[0141] In an exemplary embodiment of the present disclosure, determining the first event occurrence position of the screen input event in the current panel window includes: obtaining the current coordinate position of the screen input event in the current panel window and the current screen resolution of the display terminal; and determining the first event occurrence position of the screen input event in the current panel window based on the current screen resolution and the current coordinate position.

[0142] In an exemplary embodiment of the present disclosure, determining the current pixel value at the location where the first event occurs includes: using the first preset program as the main startup program to start the twin application corresponding to the target application, and intercepting the twin program interface of the twin application corresponding to the target application through the first preset program; based on the location where the first event occurs, determining the second event location where the screen input event occurs in the twin program interface, and obtaining the current pixel value based on the pixel value at the second event location.

[0143] In an exemplary embodiment of the present disclosure, the twin program interface of the twin application is intercepted through the first preset program, including: based on the preset application interface, obtaining the first window handle of the target application and the second window handle of the twin application through the first preset program; integrating the first window handle and the second window handle into the first preset program, and monitoring the screen input events acting on the target application based on the first window handle through the first preset program; when the screen input events acting on the target application are monitored, obtaining the current display interface of the target application based on the first window handle; wherein the interface shape of the current display interface is consistent with the shape of the current panel window; controlling the twin application to display the current display interface based on the second window handle, and intercepting the current display interface from the twin application based on the second window handle through the first preset program to obtain the twin program interface.

[0144] In an exemplary embodiment of the present disclosure, the current pixel value is obtained according to the pixel value at the location where the second event occurs, including: drawing a main display window corresponding to the display terminal, and hiding the main display window in the toolbar of the terminal device; wherein the main display window is a purely transparent borderless window, and the main display window is presented in the form of a modal window, and the shape of the main display window is consistent with the size of the current panel window; intercepting the transparent display interface corresponding to the main display window, and determining the third event occurrence location of the screen input event in the transparent display interface; obtaining the first pixel value at the third event occurrence location in the transparent display interface, and the second pixel value at the second event occurrence location in the twin program interface; comparing the first pixel value and the second pixel value, and obtaining the current pixel value based on the pixel value comparison result.

[0145] In an exemplary embodiment of the present disclosure, the first pixel value and the second pixel value are compared, and the current pixel value is obtained based on the pixel value comparison result, including: obtaining a first Alpha channel value in the first pixel value and a second Alpha channel value in the second pixel value, and determining whether the first Alpha channel value and the second Alpha channel value are consistent; if the first Alpha channel value and the second Alpha channel value are consistent, determining that the current pixel value is zero; if the first Alpha channel value and the second Alpha channel value are inconsistent, determining that the current pixel value is non-zero.

[0146] In an exemplary embodiment of the present disclosure, an event processing rule for the screen input event is determined based on the current pixel value, including: if the current pixel value is zero, determining that the event processing rule for the screen input event is a first event processing rule; wherein, the first event processing rule is to transparently transmit the screen input event to the model rendering flow layer of the display terminal; if the current pixel value is non-zero, determining that the event processing rule for the screen input event is a second event processing rule; wherein, the second event processing rule is to process the screen input event through the baseline transparent layer.

[0147] In an exemplary embodiment of the present disclosure, the screen input event is processed based on the event processing rule to obtain an event processing result, including: transmitting the screen input event to the model rendering flow layer based on the first event processing rule, and processing the screen input event based on the model rendering flow layer to obtain a first event processing result; or processing the screen input event through the baseline transparent layer based on the second event processing rule to obtain a second event processing result.

[0148] In an exemplary embodiment of the present disclosure, the screen input event is transmitted to the model rendering flow layer based on the first event processing rule, and the screen input event is processed based on the model rendering flow layer to obtain a first event processing result, including: controlling the baseline transparent layer to release the control of the screen input event, and setting the window style of the current panel window to a first preset window display style; wherein, the first preset window display style includes the WS_EX_LAYERED style and the WS_EX_TRANSPARENT style; transmitting the screen input event to the model rendering flow layer, and obtaining the three-dimensional model to be processed corresponding to the screen touch event through the model rendering flow layer; performing interactive processing on the three-dimensional model to be processed to obtain a model interactive processing result; wherein, performing interactive processing on the three-dimensional model to be processed includes: performing one or more of dragging processing, scaling processing, clicking processing and lens perspective switching processing on the three-dimensional model to be processed.

[0149] In an exemplary embodiment of the present disclosure, the device for processing screen input events further includes:

[0150] A model interaction processing result transfer module can be used to transfer the model interaction processing result to the three-dimensional model interaction area in the current panel window through the model rendering flow layer and the reference transparent layer;

[0151] The model interaction processing result display module can be used to display the model interaction processing result through the three-dimensional model interaction area.

[0152] In an exemplary embodiment of the present disclosure, the screen input event is processed through the baseline transparent layer based on a second event processing rule to obtain a second event processing result, including: controlling the baseline transparent layer to retain control of the screen input event, and setting the window style of the current panel window to a second preset window display style; wherein, the second preset window display style includes the WS_EX_TRANSPARENT style; based on the control of the screen input event by the baseline transparent layer, the screen input event is interactively processed to obtain a second event processing result; wherein, the second event processing result includes at least one of pop-up window closing, sub-interface display, sub-interface closing, and attribute information display.

[0153] In an exemplary embodiment of the present disclosure, the device for processing screen input events further includes:

[0154] The current panel window setting module can be used to set the current panel window based on WindowChrome.

[0155] The specific details of each module in the above-mentioned screen input event processing device have been described in detail in the corresponding screen input event processing method, and therefore will not be repeated here.

[0156] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0157] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0158] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0159] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0160] The electronic device 1200 according to this embodiment of the present disclosure is described below with reference to Figure 12. The electronic device 1200 shown in Figure 12 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0161] As shown in FIG12 , electronic device 1200 is implemented as a general-purpose computing device. Components of electronic device 1200 may include, but are not limited to, the aforementioned at least one processing unit 1210, the aforementioned at least one storage unit 1220, a bus 1230 connecting various system components (including storage unit 1220 and processing unit 1210), and a display unit 1240.

[0162] The storage unit stores a program code, and the program code can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps described in the "Exemplary Method" section of the present specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1210 can execute step S110 as shown in Figure 1: through the reference transparent layer, capture the screen input event acting on the target application through the current panel window corresponding to the target application; step S120: determine the first event occurrence position of the screen input event in the current panel window; step S130: determine the current pixel value at the first event occurrence position, and determine the event processing rule of the screen input event based on the current pixel value; step S140: process the screen input event based on the event processing rule to obtain an event processing result.

[0163] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 12201 and / or a cache memory unit 12202 , and may further include a read-only memory unit (ROM) 12203 .

[0164] The storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0165] The bus 1230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0166] The electronic device 1200 can also communicate with one or more external devices 1300 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1200, and / or any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 1250. Furthermore, the electronic device 1200 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 1200, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0167] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0168] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present disclosure.

[0169] According to an embodiment of the present disclosure, a program product for implementing the above-mentioned method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0170] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0171] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0172] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0173] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0174] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0175] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not invented herein. The specification and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A method for processing screen input events, characterized in that: The display terminal is configured to include a current panel window and a reference transparent layer. The method for processing screen input events includes: Capturing, through the reference transparent layer, a screen input event acting on the target application through the current panel window corresponding to the target application; Determine the first event occurrence position of the screen input event in the current panel window; Determine a current pixel value at the location where the first event occurs, and determine an event processing rule for the screen input event according to the current pixel value; The screen input event is processed based on the event processing rule to obtain an event processing result.

2. The method for processing screen input events according to claim 1, characterized in that: Capturing, through the reference transparent layer, a screen input event acting on the target application through the current panel window corresponding to the target application, comprising: Starting a first preset program in the display terminal, and using the first preset program as a main startup program to start the target application; wherein the first preset program is an embedded hidden program; Loading a transparent full-screen container in the first preset program, and loading a current panel window and a reference transparent layer of the target application in the transparent full-screen container; A preset event monitoring function is configured in the reference transparent layer, and based on the event monitoring function in the reference transparent layer, a screen input event acting on the target application through the current panel window corresponding to the target application is captured.

3. The method for processing screen input events according to claim 2, characterized in that: Based on the event monitoring function in the reference transparent layer, capturing the screen input event acting on the target application through the current panel window corresponding to the target application includes: Obtain the real-time refresh frequency of the twin application corresponding to the target application, and interval the real-time refresh frequency, based on the event monitoring function in the baseline transparent layer, capture the screen input events acting on the target application through the current panel window corresponding to the target application.

4. The method for processing screen input events according to claim 1, characterized in that: Determining the first event occurrence position of the screen input event in the current panel window includes: Acquire the current coordinate position of the screen input event in the current panel window and the current screen resolution of the display terminal; A first event occurrence position of the screen input event in the current panel window is determined according to the current screen resolution and the current coordinate position.

5. The method for processing screen input events according to claim 1, characterized in that: Determining a current pixel value at a location where the first event occurs, comprising: Using the first preset program as the main startup program, starting the twin application corresponding to the target application, and Intercepting a twin program interface of a twin application corresponding to the target application through a first preset program; According to the location where the first event occurred, the location where the second event occurred of the screen input event in the twin program interface is determined, and according to the pixel value at the location where the second event occurred, the current pixel value is obtained.

6. The method for processing screen input events according to claim 5, characterized in that: Intercepting the twin program interface of the twin application program through the first preset program includes: Based on a preset application program interface, obtaining a first window handle of the target application and a second window handle of the twin application through a first preset program; Integrating the first window handle and the second window handle into the first preset program, and monitoring screen input events acting on the target application based on the first window handle through the first preset program; When a screen input event acting on the target application is detected, the current display interface of the target application is acquired based on the first window handle; wherein the interface shape of the current display interface is consistent with the shape of the current panel window; Based on the second window handle, the twin application is controlled to display the current display interface, and the current display interface is intercepted from the twin application based on the second window handle through the first preset program to obtain the twin program interface.

7. The method for processing screen input events according to claim 5, characterized in that: Obtaining the current pixel value according to the pixel value at the position where the second event occurs, including: Draw a main display window corresponding to the display terminal, and hide the main display window in the toolbar of the terminal device; wherein the main display window is a pure transparent frameless window, and the main display window is presented in the form of a modal window, and the shape of the main display window is consistent with the size of the current panel window; intercepting the transparent display interface corresponding to the main display window, and determining a third event occurrence position of the screen input event in the transparent display interface; Obtaining a first pixel value at a location where the third event occurs in the transparent display interface, and a second pixel value at a location where the second event occurs in the twin program interface; The first pixel value and the second pixel value are compared, and the current pixel value is obtained according to the pixel value comparison result.

8. The method for processing screen input events according to claim 7, characterized in that: Comparing the first pixel value and the second pixel value, and obtaining the current pixel value according to the pixel value comparison result, includes: Obtaining a first Alpha channel value in the first pixel value and a second Alpha channel value in the second pixel value, and determining whether the first Alpha channel value and the second Alpha channel value are consistent; If the first Alpha channel value and the second Alpha channel value are consistent, the current pixel value is determined to be zero; if the first Alpha channel value and the second Alpha channel value are inconsistent, the current pixel value is determined to be non-zero.

9. The method for processing screen input events according to claim 1, characterized in that: Determining an event processing rule for the screen input event according to the current pixel value includes: If the current pixel value is zero, the event processing rule of the screen input event is determined to be the first event processing rule. Rules; wherein the first event processing rule is to transparently transmit the screen input event to the model rendering flow layer of the display terminal; If the current pixel value is non-zero, it is determined that the event processing rule of the screen input event is a second event processing rule; wherein the second event processing rule is to process the screen input event through the reference transparent layer.

10. The method for processing screen input events according to claim 1, characterized in that: Processing the screen input event based on the event processing rule to obtain an event processing result includes: Based on the first event processing rule, the screen input event is transparently transmitted to the model rendering flow layer, and the screen input event is processed based on the model rendering flow layer to obtain a first event processing result; or The screen input event is processed through the reference transparent layer based on the second event processing rule to obtain a second event processing result.

11. The method for processing screen input events according to claim 10, characterized in that: The screen input event is transparently transmitted to the model rendering flow layer based on the first event processing rule, and the screen input event is processed based on the model rendering flow layer to obtain a first event processing result, including: Controlling the reference transparent layer to release the control right of the screen input event, and setting the window style of the current panel window to a first preset window display style; wherein the first preset window display style includes a WS_EX_LAYERED style and a WS_EX_TRANSPARENT style; Transmitting the screen input event to the model rendering flow layer, and obtaining the to-be-processed three-dimensional model corresponding to the screen touch event through the model rendering flow layer; The three-dimensional model to be processed is interactively processed to obtain a model interactive processing result; wherein the three-dimensional model to be processed is interactively processed including: one or more of dragging, scaling, clicking, and camera perspective switching processing on the three-dimensional model to be processed.

12. The method for processing screen input events according to claim 11, characterized in that: The method for processing the screen input event also includes: The model interaction processing result is transmitted to the three-dimensional model interaction area in the current panel window through the model rendering flow layer via the reference transparent layer; The model interaction processing result is displayed through the three-dimensional model interaction area.

13. The method for processing screen input events according to claim 10, characterized in that: Processing the screen input event through the reference transparent layer based on the second event processing rule to obtain a second event processing result includes: Controlling the reference transparent layer to retain the control right of the screen input event, and setting the window style of the current panel window to a second preset window display style; wherein the second preset window display style includes the WS_EX_TRANSPARENT style; Based on the control of the baseline transparent layer over the screen input event, the screen input event is interactively processed to obtain a second event processing result; wherein the second event processing result includes at least one of pop-up window closing, sub-interface display, sub-interface closing, and attribute information display.

14. The method for processing screen input events according to any one of claims 1 to 13, characterized in that: The method for processing the screen input event also includes: The current panel window is set based on WindowChrome.

15. A device for processing screen input events, characterized in that: Configured in a display terminal, the display terminal includes a current panel window and a reference transparent layer, and the processing device of the screen input event includes: A screen input event capture module, used to capture, through the reference transparent layer, a screen input event acting on the target application through the current panel window corresponding to the target application; A first event occurrence position determination module, used to determine the first event occurrence position of the screen input event in the current panel window; An event processing rule determination module, used to determine a current pixel value at the location where the first event occurs, and determine an event processing rule for the screen input event according to the current pixel value; The screen input event processing module is used to process the screen input event based on the event processing rule to obtain an event processing result.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for processing screen input events according to any one of claims 1 to 14 is implemented.

17. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the method for processing screen input events according to any one of claims 1 to 14 by executing the executable instructions.

Images