Annotation implementation method and device and storage medium

Abstract

The embodiment of the invention discloses an annotation implementation method and device and a storage medium, and the method comprises the steps: obtaining a first window list, and recording each visible window which is currently running in the first window list; when a first window which cannot be annotated exists in each window recorded in the first window list, the display coverage relation between the first window and the second window is determined based on the first position of the first window in the display screen, the second position of the second window in the first window list in the display screen and the display coverage relation between the first window and the second window; a first area which cannot be annotated in the display screen is determined, the number of the first windows is at least one, and the second windows are annotated windows in the first window list; receiving a touch operation of a preset touch object acting on the display screen; when the touch position of the touch operation does not fall into the first area, displaying a corresponding annotation track according to the touch operation; the technical problem that a writing application and an annotation application conflict with each other when the electronic equipment runs the writing application and the annotation application at the same time in the related technology is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an annotation implementation method, device and storage medium. Background Technology

[0002] With the development of IoT and network technologies, electronic devices are becoming increasingly feature-rich, such as annotation, drawing, and writing functions. These interactive functions bring a lot of convenience to people's work and life.

[0003] In related technologies, annotation functionality relies on annotation applications to enable instant writing. With annotation functionality, users can annotate the entire screen of their electronic device through these applications. However, if the display shows the interface of another writable application (currently referred to as the writing application), a conflict will arise between the writing application and the annotation application. Specifically, for operations performed on the writing application's interface, the electronic device will be unable to determine whether the operation is intended for writing within the writing application or for annotation within the annotation application. This hinders the proper implementation of the annotation function. Summary of the Invention

[0004] This application provides a method, device, and storage medium for implementing annotations, which can solve the technical problem in related technologies where writing applications and annotation applications conflict when electronic devices run writing applications and annotation applications simultaneously.

[0005] Firstly, one embodiment of this application provides an annotation implementation method, including:

[0006] Get the first window list, which records every currently running and visible window;

[0007] When there is an unannotable first window in each window recorded in the first window list, the first unannotable area in the display screen is determined based on the first position of the first window in the display screen, the second position of the second window in the first window list in the display screen, and the display coverage relationship between the first window and the second window. The first window is at least one, and the second window is an annotable window in the first window list.

[0008] Receives touch operations on the display screen from a preset touch object;

[0009] When the touch position of the touch operation does not fall into the first area, the corresponding annotation trajectory is displayed according to the touch operation;

[0010] When the touch position of a touch operation falls into the first area, the application corresponding to the first window where the touch position falls responds to the touch operation.

[0011] The above-described method, by acquiring a first window list containing all currently running visible windows, and when a non-annotation-friendly first window exists among all windows recorded in the first window list, determines a first non-annotation-friendly area on the display screen based on the first position of the first window on the screen, the second position of the second window in the first window list on the screen, and the display coverage relationship between the first and second windows. Then, when a touch operation is received on the display screen using a preset touch object, if the touch position does not fall into the first area, a corresponding annotation trajectory can be displayed according to the touch operation. When the touch position falls into the first area, the application corresponding to the first window where the touch position falls responds to the touch operation. This technical means solves the technical problem in related technologies where writing and annotation applications conflict when an electronic device runs both simultaneously. It enables the simultaneous existence of annotable and non-annotation-friendly windows on an electronic device. By identifying a first area that cannot be annotated, the system determines whether to perform an annotation operation based on whether the touch position of the preset touch object falls within this first area. Compared to related technologies where annotation is globally disabled as long as a first window that cannot be annotated is displayed on the screen, this embodiment disables the annotation function only in a portion of the screen (i.e., the first area). When the touch position does not fall within the first area that cannot be annotated, the annotation trajectory can be displayed normally based on the touch operation. When the touch position falls within the first area that cannot be annotated, the application corresponding to the first window responds to the touch operation. This ensures that the execution of the annotation function is not affected even when a first window that cannot be annotated appears on the screen. This allows for both annotation and other functions that cannot be annotated, such as writing or drawing within the window itself. This ensures the convenience of annotation, makes user annotations more free, and greatly increases practicality, thereby improving the user experience. Furthermore, only touch operations on the preset touch object can enable the annotation function, which can avoid false responses to the annotation function. For example, if the preset touch object is the tip of a stylus, if the touch position corresponding to the user's finger touch operation does not fall into the first area, the annotation trajectory will not be displayed, thus avoiding misidentification of the finger touch operation as an annotation operation.

[0012] In one embodiment of this application, when the touch position of the touch operation does not fall within the first area, displaying the corresponding annotation trajectory according to the touch operation includes:

[0013] When the touch position of the touch operation does not fall into the first area, the corresponding annotation trajectory is displayed according to the touch operation. When it is confirmed that there is a text input box at the touch position of the touch operation, after the touch operation ends, the annotation trajectory is converted into text content in standard font and then entered into the text input box.

[0014] As mentioned above, when a text input box exists, the annotation trajectory can be converted into a standard font and input into the text input box, which makes it easier to convert handwritten text into standard text. Especially when users need standard text as annotation content, there is no need to rely on repeatedly opening and closing the soft keyboard, which can improve the convenience of annotation and thus enhance the user experience.

[0015] In one embodiment of this application, all windows in the first window list are arranged in descending order of display hierarchy;

[0016] When there is a non-annotationable first window in each window recorded in the first window list, based on the first position of the first window on the display screen, the second position of the second window in the first window list on the display screen, and the display coverage relationship between the first window and the second window, a first non-annotationable area on the display screen is determined, including:

[0017] Iterate through each window in the first window list. During the iteration, determine whether the currently traversed window is a first window that cannot be annotated. If it is not the first window, add the currently traversed window as a second window to the second window list. If it is the first window, determine the second area in the first window that is not obscured by the second window based on the first position of the first window on the screen, the second position of each second window currently recorded in the second window list on the screen, and the display coverage relationship between the first window and the second window.

[0018] After the traversal is complete, based on each of the currently obtained second regions, determine the first region on the display screen that cannot be annotated.

[0019] As described above, when traversing each window in the first window list in descending order of display hierarchy, it can be ensured that when determining the second region, all windows that will cause occlusion (i.e., have display overlap relationships) can be considered, thus ensuring the accuracy of the determination of the second region.

[0020] In one embodiment of this application, if it is a first window, a second area in the first window that is not obscured by the second window is determined based on the first position of the first window on the display screen, the second position of each second window currently recorded in the second window list on the display screen, and the display coverage relationship between the first window and the second windows, including:

[0021] If it is the first window, then identify the second windows in the second window list that have a display overlap relationship with the first window, generate the third window list, and add each second window with a display overlap relationship to the third window list. The second windows in the third window list are arranged in order of display hierarchy from high to low.

[0022] Generate a list of regions, and add the first window as the third region to the list;

[0023] Iterate through each second window in the third window list. During the iteration, based on the third position of the third region in the region list on the screen and the second position of the currently traversed second window on the screen, determine the fourth region in the third region that is not obscured by the currently traversed second window. Use the fourth region as the updated third region in the region list so that the updated third region can be used for calculation when traversing the next second window.

[0024] After the traversal is complete, the third region in the region list is taken as the second region in the first window that is not obscured by the second window.

[0025] As mentioned above, when determining the second region, first determine the second window in the second window list that overlaps with the first window (i.e., has a display coverage relationship), which can reduce the amount of data processing when determining the second region.

[0026] In one embodiment of this application, determining a fourth region within the third region that is not obscured by the currently traversed second window, based on the third position of the third region in the region list on the display screen and the second position of the currently traversed second window on the display screen, includes:

[0027] Based on the third position of the third region in the region list on the display screen and the second position of the currently traversed second window on the display screen, determine the largest unobstructed rectangular region on the left, right, top, and bottom sides of the third region, and treat each determined largest rectangular region as a fourth region that is not obstructed by the currently traversed second window.

[0028] Based on the rectangular shape of the window, the largest rectangular area not obscured by the second window is calculated from the left, right, top, and bottom directions in the third region and used as the fourth region. This simplifies the computational complexity when calculating the unobscured area.

[0029] In one embodiment of this application, the third position includes the first coordinates of the upper left corner vertex of the corresponding third region on the display screen, the first width of the corresponding third region, and the first height of the corresponding third region; the second position includes the second coordinates of the upper left corner vertex of the corresponding second window on the display screen, the second width of the corresponding second window, and the second height of the corresponding second window.

[0030] Based on the third position of the third region in the region list on the display screen and the second position of the currently traversed second window on the display screen, determine the largest unobstructed rectangular region on the left, right, top, and bottom sides of the third region, including:

[0031] If the X-axis coordinate value in the second coordinate system is greater than the X-axis coordinate value in the first coordinate system, the first coordinate system is used as the coordinate of the top left corner vertex of the largest rectangular area that is not obscured on the left. The difference between the X-axis coordinate value in the second coordinate system and the X-axis coordinate value in the first coordinate system is used as the width of the largest rectangular area, and the first height is used as the height of the largest rectangular area.

[0032] If the sum of the X-axis coordinate value and the first width in the first coordinate system is greater than the sum of the X-axis coordinate value and the second width in the second coordinate system, the second sum is used as the X-axis coordinate value of the top-left vertex of the largest rectangular area that is not obscured on the right, the Y-axis coordinate value in the first coordinate system is used as the Y-axis coordinate value of the top-left vertex, the difference between the first sum and the second sum is used as the width of the largest rectangular area, and the first height is used as the height of the largest rectangular area.

[0033] If the Y-axis coordinate value in the second coordinate system is greater than the Y-axis coordinate value in the first coordinate system, the first coordinate system is used as the coordinate of the top left corner vertex of the largest rectangular area that is not obscured on the upper side, the first width is used as the width of the largest rectangular area, and the difference between the Y-axis coordinate value in the second coordinate system and the Y-axis coordinate value in the first coordinate system is used as the height of the largest rectangular area.

[0034] If the sum of the Y-axis coordinate value and the first height in the first coordinate system is greater than the sum of the Y-axis coordinate value and the second height in the second coordinate system, then the X-axis coordinate value in the first coordinate system is taken as the X-axis coordinate value of the top left corner vertex of the largest rectangular area that is not occluded on the lower side, the fourth sum is taken as the Y-axis coordinate value of the top left corner vertex, the first width is taken as the width of the largest rectangular area, and the difference between the third and fourth sums is taken as the height of the largest rectangular area.

[0035] As described above, by reasonably calculating the positions of the third region and the second window, the largest unobstructed rectangular region can be determined.

[0036] In one embodiment of this application, after the traversal is completed, based on each currently obtained second region, a first region on the display screen that cannot be annotated is determined, including:

[0037] After the traversal is complete, each of the obtained second regions is deduplicated, and the deduplicated regions are used as the first regions that cannot be annotated on the display screen.

[0038] As described above, after obtaining the second region, deduplication is performed on the second region to ensure that the final first region does not contain duplicate regions, thereby ensuring the accuracy of the subsequent logical judgment on whether the annotation should be responded to based on the first region.

[0039] In one embodiment of this application, determining whether the currently traversed window is the first window that cannot be annotated includes:

[0040] Determine whether the currently traversed window is included in the disabled list. The disabled list contains window identification information for each window that cannot be annotated in the current device.

[0041] If the currently traversed window is included in the disabled list, then the currently traversed window is determined to be the first window that cannot be annotated;

[0042] If the currently traversed window is not included in the disabled list, then it is determined that the currently traversed window is not the first window.

[0043] As mentioned above, the disabled list can quickly determine whether the currently traversed window is visible, which greatly improves the traversal speed, thereby improving the overall response speed of the annotation implementation and thus enhancing the user experience.

[0044] In one embodiment of this application, before determining the first region on the display screen that cannot be annotated based on each currently obtained second region after the traversal is completed, the process includes:

[0045] The iteration ends when the currently traversed window is a full-screen window; or,

[0046] The iteration ends when the currently traversed window is the last window in the first window list.

[0047] As mentioned above, when traversing the first window list, a reasonable method for determining the end of the traversal should be set. In particular, when traversing to a window that is displayed in full screen, it means that the remaining untraversed windows are all obscured by the full-screen window. Therefore, there is no need to continue traversing, which can reduce the amount of data processing required to determine the first area.

[0048] In one embodiment of this application, the annotation implementation method further includes:

[0049] If no unannotation-free first window exists in each window recorded in the first window list, determine that no first region exists on the display screen; or,

[0050] If there is no unannotation-free first window in each window recorded in the first window list, the preset edge area of ​​the display screen will be used as the first area.

[0051] As mentioned above, when the first window does not exist in the first window list, the global response annotation function of the display screen can be implemented. Alternatively, the preset edge area of ​​the display screen can be used as the first area that cannot be annotated, which can avoid the conflict between the side-sliding function and the annotation function when the display screen has a side-sliding function.

[0052] In one embodiment of this application, obtaining the first window list includes:

[0053] Get each currently running window, remove the invisible windows in each window to generate a first window list. Invisible windows include minimized windows, transparent windows, and hidden windows.

[0054] As described above, by removing invisible windows from the first window list, the influence of invisible windows on the calculation of the first region can be avoided, ensuring the accuracy of the calculation of the first region. Furthermore, recording visible windows in the first window list facilitates subsequent analysis of all visible windows to determine the first region.

[0055] Secondly, one embodiment of this application also provides an annotation implementation device, including: a display screen, one or more processors, and a memory;

[0056] Memory, used to store one or more programs;

[0057] A display screen is used to display information.

[0058] When one or more programs are executed by one or more processors, the one or more processors implement the annotation implementation method as described in the first aspect.

[0059] Thirdly, one embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the annotation implementation method as described in the first aspect.

[0060] The beneficial effects of the annotation implementation device and storage medium provided above can be referenced in relation to the beneficial effects of the annotation implementation method. Attached Figure Description

[0061] Figure 1 A schematic diagram of an annotation implementation device provided in one embodiment of this application;

[0062] Figure 2 A flowchart illustrating an annotation implementation method provided in one embodiment of this application;

[0063] Figure 3 This is a first schematic diagram of a multi-window display provided in one embodiment of this application;

[0064] Figure 4A flowchart illustrating an annotation implementation method provided in another embodiment of this application;

[0065] Figure 5 This is a second schematic diagram of a multi-window display provided in one embodiment of this application;

[0066] Figure 6 A first schematic diagram of a fourth region provided in one embodiment of this application;

[0067] Figure 7 A second schematic diagram of the fourth region provided in one embodiment of this application;

[0068] Figure 8 A schematic diagram illustrating the determination of the maximum rectangular region according to one embodiment of this application;

[0069] Figure 9 This is a third schematic diagram illustrating a multi-window display according to an embodiment of this application. Detailed Implementation

[0070] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the drawings, not the entire structure.

[0071] Annotation functionality refers to the ability for users to annotate, write comments, or draw on the display screen of electronic devices. Annotation features are widely used in education, meetings, design, and collaboration, enhancing the interactivity and convenience of users with digital content.

[0072] For touch-enabled electronic devices, the annotation function is based on annotation applications, enabling users to write annotations instantly. Once the annotation application is launched, users can write annotations on the full screen of the electronic device using touch. At this time, the display can show the desktop of the electronic device, interfaces generated by other applications installed on the device, and other content.

[0073] Generally, electronic devices may have other applications with writing capabilities besides annotation applications. For example, in drawing applications, users can draw by touch within the application's interface. Similarly, in whiteboard applications, users can write on the whiteboard by touch within the application's interface. For ease of description, we will refer to these other applications with writing capabilities as writing applications. In this case, when the annotation function is activated, if the electronic device displays the interface of a writing application, a conflict will occur. For instance, the user's touch operation might be applied to the writing application's interface, but since the annotation application is also running (i.e., the annotation function is also activated), the electronic device will be unable to distinguish whether the touch operation was for annotation or writing.

[0074] In related technologies, to avoid conflicts between annotation and writing applications, a filter list is set up in electronic devices. This filter list records applications that cannot be annotated (i.e., applications that cannot be annotated by other applications). For example, writing applications can be recorded in the filter list as applications that cannot be annotated. When the interface of an application recorded in the filter list is being displayed, the electronic device disables the annotation function, meaning the user cannot annotate.

[0075] However, the interfaces generated by applications installed on electronic devices can be displayed in the form of windows. A window is a fundamental concept in computer user interfaces, referring to the display of the computer's user interface in a window format. In a multi-window environment, windows can be displayed in a non-full-screen manner, or they can be covered by other currently displayed windows (including full or partial coverage). Windows can also be stretched or stretched by split-screen applications. In this case, when an electronic device displays a window for a writing application, this window may be a non-full-screen window, a window covered by other application windows, or a stretched window, etc. That is, in addition to the window displaying the writing application, there are other areas on the electronic device's screen. If the user wants to annotate in other areas, then, according to the technical means described in related technologies, those other areas cannot be annotated. In this case, the technical means in related technologies to avoid conflicts between annotation applications and writing applications sacrifice the convenience of annotation, restrict the user's annotation freedom, and significantly reduce practicality.

[0076] Based on this, this application provides an annotation implementation method. In this method, when the annotation function is enabled and there are both annotable and non-annotable windows on the display screen, the first non-annotable area can be determined by combining the positions and display coverage relationships of the two types of windows. Then, the annotation operation can be determined based on whether the touch position of the touch operation of the preset touch object falls into the first area. This avoids conflicts between the writing application and the annotation application when the electronic device is running the annotation application and the writing application (generating a non-annotable window) at the same time. Compared with the method of avoiding conflicts by filtering lists in related technologies, this solution can determine the first non-annotable area on the display screen when the non-annotable window is displayed and is covered by other windows or not displayed in full screen. The first area is a part of the display screen. At this time, for touch operations where the touch position does not fall into the first area, the annotation function can be responded to normally, that is, the annotation trajectory is drawn, which ensures the convenience of annotation, makes the user's annotation more free, and greatly increases the practicality, thereby improving the user experience.

[0077] The annotation implementation method provided in this application embodiment can be executed by an annotation implementation device. This annotation implementation device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. Currently, this annotation implementation device can be an electronic device capable of touch and display, such as a mobile phone, tablet computer, or interactive whiteboard.

[0078] Figure 1 This is a schematic diagram of an annotation implementation device provided in one embodiment of this application. (Reference) Figure 1 The annotation implementation device includes a processor 11, a memory 12, and a display screen 13. The processor 11, memory 12, and display screen 13 can be connected via a bus or other means.

[0079] The number of processors 11 can be one or more. Figure 1 Taking a processor 11 as an example, the processor 11 may include processing units such as an application processor (AP), a graphics processing unit (GPU), and a central processing unit (CPU).

[0080] The memory 12, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the annotation implementation method in the embodiments of this application. The memory 12 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the annotation implementation device. Furthermore, the memory 12 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 12 may further include remotely located memories 12 relative to the processor 11, which can be connected to the annotation implementation device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0081] Display screen 13 can be a liquid crystal display (LCD), an LED display, an organic light-emitting diode (OLED) display, or a flexible light-emitting diode (FLED) display, etc. Currently, display screen 13 has touch functionality. In this case, display screen 13 includes a display component and a touch component. The display panel is used to complete visual output. The touch component is used to collect the user's touch operation so that the annotation implementation device responds to the touch operation to realize the touch operation. The touch component can be a touch component that supports infrared touch, electromagnetic touch, capacitive touch, and / or resistive touch, etc.

[0082] The annotation implementation device may also include one or more communication interfaces to enable communication with other devices. Furthermore, the annotation implementation device may also include components such as a power supply, speaker, and camera; however, this embodiment does not limit the scope of the invention.

[0083] Based on the aforementioned hardware structure, the annotation implementation device has at least one operating system installed. This operating system can be Android, Windows, or Linux, among others. Currently, we will take an annotation implementation device that has at least Windows installed as an example.

[0084] Based on the installed operating system, the annotation implementation device can install at least one application. This application can be a built-in application of the operating system or an application downloaded from a backend server or a third-party device. In one embodiment, the annotation implementation device has at least one application installed for implementing annotation functions; currently, this application is referred to as the annotation application. When the annotation application starts its annotation function, it can perform pen annotations, as described in the prior art. The annotation implementation device also has applications installed for implementing non-annotation functions, such as writing or drawing applications; currently, these can all be referred to as writing applications. Writing refers to regular writing in non-annotation scenarios, and drawing refers to regular drawing in non-annotation scenarios. For example, a conferencing application (i.e., one that implements video conferencing) has its own annotation function; therefore, it can also be considered a non-annotation application.

[0085] It should be noted that, in this embodiment, the annotation function refers to adding corresponding annotations to the original displayed content. The original displayed content cannot be edited or changed. The annotation content can be written or drawn. In this case, the original displayed content can be the desktop of the annotation-implementing device or the interface of an application outside the annotation application that can be annotated. That is, the current annotation function can annotate the content displayed on the screen, and the displayed content does not necessarily have to be the interface of the annotation application. Regular writing and drawing, on the other hand, allow editing and changing the original displayed content. The original displayed content is the content within the interface of the writing application itself. That is, the writing function can only be performed within the interface of the corresponding writing application. Therefore, the underlying control logic of the two is different.

[0086] Understandably, when the annotation implementation device has touch functionality, it also has a touch driver installed. When a user performs a touch operation, the touch component can detect the touch operation and obtain touch data. This touch data includes at least the touch location and touch time, and the touch operation can be reconstructed from the touch data. The touch component then sends the touch data to the touch driver, which in turn sends the touch data outwards in response. For example, if the touch location in the touch data does not fall within the first area, the touch driver can send the touch data to the annotation application for a response; otherwise, the touch driver can send the touch data to the operating system, which then determines the application that needs to respond and sends the touch data to the appropriate application.

[0087] Based on this, Figure 2 A flowchart illustrating an annotation implementation method provided in one embodiment of this application is shown below. Figure 2 The annotation implementation method includes steps 210-250:

[0088] Step 210: Obtain the first window list, which records each currently running and visible window.

[0089] In one embodiment, when the annotation implementation device displays the interface generated by its installed applications, it displays it in the form of windows. Furthermore, the annotation implementation device supports a multi-window environment, meaning it can display multiple windows simultaneously. Each displayed window can support full-screen display, non-full-screen display, and minimized display, and the window can be stretched to adjust its size. When multiple windows are displayed, they can also overlap; for example, one window can be completely or partially covered by another window.

[0090] Currently, annotation devices can display multiple windows. To analyze each window and determine whether the annotation function is implemented, it is necessary to obtain each currently running visible window (i.e., all visible windows). A visible window is one that is displayed on the screen and can be seen by the user when there is no obstruction. For example, a window may be visible to the user when it is displayed on the screen, and it is also considered a visible window even if it is later partially or completely covered by other windows.

[0091] In one embodiment, when the annotation application in the annotation implementation device is started, the annotation application determines each visible window and records each visible window in a list to obtain a list of visible windows. Currently, this list is recorded as the first window list.

[0092] When the annotation application determines each visible window, it can first obtain all windows currently running on the annotation implementation device, then analyze and judge all windows to determine each visible window and form a first window list. At this time, this step may include: obtaining each currently running window (i.e., all windows), removing invisible windows from each window to generate a first window list, wherein invisible windows include minimized windows, transparent windows, and hidden windows.

[0093] For example, after the annotation application runs, it determines that the annotation function is enabled. At this point, the annotation application can obtain all currently running windows (i.e., each window). Alternatively, taking Windows systems as an example, the annotation application can obtain all currently running windows through the Windows Application Programming Interface (API) EnumWindows. EnumWindows is a computer function that enumerates all top-level windows. After obtaining all windows, the annotation application can remove minimized, transparent, and hidden windows; the remaining windows can be considered all visible windows.

[0094] A transparent window is a window with a transparent attribute, meaning it supports transparency. Once set to transparent, it will be invisible to the user even if displayed on the screen. A hidden window is a window whose IsWindowVisible function returns "Fail". IsWindowVisible is a function that obtains the visible state of a specified window and can return "True" or "Fail" to indicate whether the window is visible or hidden, respectively. A minimized window is a window that is minimized and not displayed on the screen, therefore it is not seen by the user. All three types of windows can be considered invisible. It's understandable that some applications run in the background and do not generate windowed interfaces displayed on the screen; these applications are not included in the statistics.

[0095] Currently, when an annotation application obtains a window, it can do so by acquiring the window information through the window handle, and then determine whether the window is visible based on this information. Optionally, the window information can include parameters such as the process name corresponding to the window, handle identifier, window class name, window title, visibility, transparency, window size, and whether the window is minimized. Visibility is used to determine if a window is hidden; for example, `IsWindowVisible` can be called to obtain information about whether it is a hidden window. Transparency is used to determine if a window is transparent; for example, the `GetWindowLong(hWnd, GWL_EXSTYLE)` function can be used to obtain information about whether it is a transparent window. `GetWindowLong` is a Windows Application Programming Interface (API) that can obtain information about a specified window; `hWnd` refers to the window handle; and `GWL_EXSTYLE` indicates the extended style to be obtained. Currently, whether a window is transparent is determined by whether the obtained extended style (`GWL_EXSTYLE`) contains `WS_EX_TRANSPARENT` (which makes the window transparent). If WS_EX_TRANSPARENT is included, the window is transparent; otherwise, it is opaque. Whether the window is minimized is used to determine if the window is minimized.

[0096] The annotation application, based on the window information of each currently running window (primarily whether it is visible and minimized), can determine whether a currently running window is invisible, and remove all identified invisible windows to obtain a list of all visible windows. Then, it records all visible windows to obtain a first window list. Optionally, the first window list records the window information of each visible window, that is, the visible windows are listed in List...<WindowI nfo> storage.

[0097] Step 220: When there is a first window that cannot be annotated in each window recorded in the first window list, based on the first position of the first window on the display screen, the second position of the second window in the first window list on the display screen, and the display coverage relationship between the first window and the second window, determine the first area on the display screen that cannot be annotated, wherein the first window is at least one, and the second window is an annotable window in the first window list.

[0098] For example, among all visible windows recorded in the first window list, there may be annotable windows and non-annotable windows. Non-annotable windows are denoted as the first window; for example, a window of a writing application (such as one used for writing or drawing) can be considered a non-annotable window. Annotation applications cannot annotate non-annotable windows. Annotable windows in the first window list are denoted as the second window; annotation should be possible on these windows, meaning the first window list includes both the first and / or second windows.

[0099] Optionally, after obtaining the first window list, the annotation application can determine whether a first window exists in the first window list. If a first window exists, it means that there may be areas on the current display screen that cannot be annotated. Therefore, the annotation application can determine the areas on the display screen that cannot be annotated based on the positions of the first window and the second window on the display screen, as well as the display coverage relationship between the first window and the second window. If a first window does not exist, it means that all areas on the current display screen can be annotated. It is understood that in touch scenarios, since there may be swipe functionality around the edges of the display screen (i.e., a corresponding function is triggered after performing a set swipe operation at the edge of the display screen), in order to avoid conflicts with the swipe functionality, in this embodiment, when all areas on the display screen can be annotated, all areas on the display screen except the edges can be considered as annotable areas.

[0100] In one embodiment, when a first window exists, the annotation application can determine the position of the first window on the display screen based on the window information (specifically, the window size) of the first window in the first window list. It is understood that each window has a corresponding position, which is its position in a pixel coordinate system. The display screen has a pixel coordinate system during display, and the origin of the pixel coordinate system is the top-left corner of the display screen. Windows displayed on the display screen are typically rectangular. In this case, the window's position can be represented by the coordinates of the top-left corner vertex of the window in the pixel coordinate system, the pixel width occupied by the window in the pixel coordinate system, and the pixel height. In practical applications, the window position can also be recorded in other ways, such as recording the window's edge coordinates. Currently, the position of the first window is recorded as the first position. Furthermore, the annotation application also determines the position of the second window on the display screen based on the window information of the second window in the first window list. Currently, the position of the second window is recorded as the second position.

[0101] Then, the annotation application can determine the non-annotation areas on the display screen based on the first and second positions and the display overlay relationship between the first and second windows. Non-annotation areas refer to areas that do not require a response from the annotation application; that is, when the user writes in these areas, the annotation application will not respond. In one embodiment, all currently running visible windows have corresponding display layers. Display layers are related to management and display order. For example, if one window covers another, then the display layer of one window is higher than that of the other. The display layer relationship can be understood as the hierarchy of display layers. The windows in the first window list can be arranged in descending order of display layer. In this case, a window with a higher display layer may cover a window with a lower display layer. Based on this, the display overlay relationship between the first and second windows can be determined according to the display layer between the first and second windows, the first position, and the second position. A display overlay relationship means that one window covers another; in this case, one window may completely or partially obscure the other window. In this embodiment, the display coverage relationship can be understood as the first window being covered by the second window. In this case, the area in the first window that is covered by the second window may be the part that can be annotated (because the second window can be annotated). Therefore, it is necessary to combine the first position and the second position as well as the display coverage relationship between the first window and the second window to determine the area in the display screen that cannot be annotated.

[0102] Currently, the area that cannot be annotated is designated as the first area. The method for determining the first area is not currently limited. For example, since the first window is the window corresponding to the application that cannot be annotated, then when the first window is not obscured by other windows, its entire area can be considered as the unannotated area. Based on this, the annotation application can determine the area in each first window on the display screen that is not covered by other windows (such as the second window) according to the first and second positions and the display overlap relationship between the first and second windows; this area can then be considered the first area. For example, Figure 3 This is a first schematic diagram of a multi-window display provided in one embodiment of this application, referring to... Figure 3 Assume that the visible windows displayed on screen A0 are window 1 and window 2, meaning that the first window list records window 1 and window 2. Window 1 is the second window that can be annotated, and window 2 is the first window that cannot be annotated. In this case, the annotation application determines the first position A10 of window 2 (i.e., the first window) on the screen and the second position A20 of window 1 (i.e., the second window). The display hierarchy of window 1 (i.e., the second window) is higher than that of window 2 (i.e., the first window). Based on the first position A10, the second position A20, and the display hierarchy relationship between window 1 (i.e., the second window) and window 2 (i.e., the first window), it is clear that window 1 (i.e., the second window) covers window 2 (i.e., the first window), indicating a display overlap relationship. Therefore, based on the first position A10, the second position A20, and the display overlap relationship between window 1 (i.e., the second window) and window 2 (i.e., the first window), the area on the screen where the first window (i.e., window 2) is not covered by other windows (currently window 1) can be determined. Figure 3 The uncovered area in the display screen A0 is area A01. Therefore, the annotation application can determine that area A01 in display screen A0 is the first area that cannot be annotated.

[0103] Optionally, all areas of the display screen other than the first area can be considered as annotable areas. Alternatively, to ensure the swipe functionality, the edge areas of the display screen can be removed from the annotable areas, or the edge areas can also be updated to the first area.

[0104] Understandably, the first region can be composed of multiple regions. For example, when there are multiple first windows, multiple regions that cannot be annotated can be identified, and these regions can form the first region.

[0105] Optionally, after the annotation application starts, a process can be created, which can then continuously execute steps 210 and 220 to continuously update the first region, ensuring that the first region is the latest region applicable to all current windows.

[0106] Optionally, after the annotation application determines the first region, it can send the first region to the touch driver through a custom interface for the touch driver to use. When sending the first region, it can send the position information of the first region on the display screen (such as the coordinate range of the first region on the display screen, or the coordinates of the top left corner vertex of the first region on the display screen, the width of the region, and the height of the region).

[0107] Step 230: Receive a touch operation on the display screen from a preset touch object. If the touch position of the touch operation does not fall into the first area, proceed to step 240; if the touch position of the touch operation falls into the first area, proceed to step 250.

[0108] In one embodiment, the preset touch object can be the tip of a stylus (also referred to as a writing pen, touch pen, etc.). It is understood that when users annotate by hand, they typically use a pen because the touch precision of a pen tip is greater than that of a finger. Therefore, in this embodiment, only touch operations of the stylus tip can achieve the annotation function, while other touch operations (such as finger touch operations) cannot achieve the annotation function; that is, the current annotation application can only respond to touch operations of the stylus tip. It should be noted that the type of stylus and the method of touch implementation are not limited in this embodiment.

[0109] Therefore, after receiving a touch operation, it is necessary to first determine whether the touch operation was performed by a preset touch object. This process can be determined by the annotation application, the operating system, or the touch driver. Here, we will describe the touch driver determination as an example.

[0110] For example, receiving a touch operation can be understood as the touch component detecting the touch operation and sending the touch data of the touch operation to the touch driver. Then, the touch driver determines whether the touch operation was performed by a preset touch object.

[0111] In one implementation, determining whether a touch operation is performed by a preset touch object is achieved by directly detecting a touch signal or by detecting specific information about the touch object using an auxiliary sensor. For example, different touch object types can be distinguished based on the size of the touch point (i.e., the touch area) when the touch object contacts the display interface (which could be a first window, a second window, or other areas of the display screen). Alternatively, different touch object types can be distinguished based on the identifiable physical signals (e.g., vibration signals) generated when the touch object contacts the display interface. Yet another example is the distinction based on the identifying wireless signals generated when the touch object contacts the display interface. The method for determining the touch object type can be determined based on the actual situation and is not specifically limited here.

[0112] In another implementation, determining whether a touch operation is performed by a preset touch object is done by receiving the type of touch object sent by the touch-sensitive display screen. For example, when the display screen uses a high-precision touch-sensitive display screen, it can not only identify the position information of the touch object on the touch-sensitive display screen, but also identify the specific shape of the touch object, thereby distinguishing the specific type of touch object.

[0113] When the touch driver determines that a touch operation is performed by a preset touch object, it can determine whether the touch operation falls within the first area based on the touch position in the touch data and the position of the first area on the display screen. It is understood that the coordinate system used by the touch component to detect touch operations may differ from the pixel coordinate system. In this case, the touch driver can first map the touch position to the pixel coordinate system before determining whether the touch position falls within the first area. If the touch position of the touch operation does not fall within the first area, step 240 is executed; if the touch position of the touch operation falls within the first area, step 250 is executed. Optionally, when the user annotates, the annotation trajectory has a certain length, and the touch operation corresponding to this annotation trajectory may have multiple touch positions. In this case, the touch driver can determine whether each touch position falls within the first area one by one.

[0114] Optionally, when the touch driver determines that the touch operation is not performed by a preset touch object, it can also determine whether the touch operation falls within the first area. If it falls within the first area, step 250 is executed. In this case, the application responding to the touch operation can also be notified that the current touch operation is performed by a non-preset touch object. If it does not fall within the first area, the response is abandoned, or the touch data of the touch operation and the identifier of the non-preset touch object are sent together to the annotation application for subsequent processing. In practical applications, the response strategy for touch operations performed by non-preset touch objects can be designed according to the actual situation, and this embodiment does not limit this.

[0115] Step 240: When the touch position of the touch operation does not fall into the first area, display the corresponding annotation trajectory according to the touch operation.

[0116] For example, when the touch position of a touch operation does not fall within the first area, the touch operation can be considered a touch operation on an annotable area. In this case, the touch driver can send the touch data to the annotation application, which will then determine whether to annotate and display the corresponding annotation trajectory based on the touch data (or the touch operation), thus performing the annotation function. Optionally, when sending touch data, the identifier of a preset touch object (which can be set according to actual conditions) can also be sent to the annotation application so that the annotation application clearly understands the touch operation performed by the preset touch object corresponding to the touch data and can respond with the annotation function.

[0117] In one embodiment, when displaying annotation traces, a writing layer (transparent layer) can be overlaid in areas other than the first area on the display screen, and corresponding annotation traces can be added and displayed in the writing layer based on touch operations.

[0118] In one embodiment, when displaying the annotation trajectory, the annotation trajectory can also be converted into text content in standard font according to the specific application scenario. In this case, this step may include: when the touch position of the touch operation does not fall into the first area, displaying the corresponding annotation trajectory according to the touch operation; when it is confirmed that there is a text input box at the touch position of the touch operation, after the touch operation ends, converting the annotation trajectory into text content in standard font and inputting it into the text input box.

[0119] A text input box (also referred to as a text box or input field) is used to enter text content. This text content can include letters, symbols, numbers, and / or Chinese characters. The text content in a text input box is typically in a standard font. A standard font can also be understood as a printed font; for example, text entered by the user via a keyboard is in a standard font. It is understood that standard fonts have various settings such as font type, size, and color, but currently, there are no restrictions on the settings for the standard font displayed in a text input box.

[0120] For example, when displaying annotation traces, it can also determine whether a text input box exists at the touch position of the touch operation corresponding to the annotation trace. Currently, the text input box can be generated and displayed by the annotation application; in practical applications, it can also be generated and notified to the annotation application by other applications. In this case, the annotation application can know the position of the text input box on the display screen (specifically, the position range of the area where the text input box is located). When the annotation application displays the annotation trace, it can determine whether a text input box exists at the touch position of the touch operation based on the touch position of the corresponding touch operation and the position of the text input box. If a text input box exists, after the touch operation ends, the annotation trace on the writing layer that repeats the position of the text input box is converted into text content in standard font and then entered into the text input box. The method for determining the end of the touch operation is not currently limited; for example, if the annotation application does not receive a touch operation performed by a preset touch object sent by the touch driver within a certain period of time (which can be set according to the actual situation) after the touch operation ends, it determines that the touch operation has ended. For example, after the annotation application displays the annotation trajectory based on the touch operation, it receives a new touch operation performed by a preset touch object sent by the touch driver. If the distance between the touch position of the new touch operation and the touch position of the previous touch operation is greater than a set distance (which can be set according to actual conditions), the previous touch operation is considered to have ended. The annotation trajectory can be converted into standard font text content through Optical Character Recognition (OCR) technology, or other methods; this embodiment does not limit this. After obtaining the standard font text content, the annotation application can simulate text input to input and display the text content in the text input box. Then, when the annotation application receives a touch operation (performed by a preset touch object) passing through the text input box, it directly edits the text content in the text input box.

[0121] Step 250: When the touch position of the touch operation falls into the first area, the application corresponding to the first window where the touch position falls responds to the touch operation.

[0122] When the touch position of a touch operation falls within the first area, the touch operation can be considered a touch operation on the first area that cannot be annotated. In this case, another application should respond to the touch operation. Specifically, the application corresponding to the first window where the touch position falls should respond to the touch operation. The first window where the touch position falls can be understood as: the coordinate range (pixel range) corresponding to the first position of the first window on the display screen contains the pixel corresponding to the touch position, and this pixel is not obstructed by other windows. That is, the touch operation is performed on the position of the first window on the display screen.

[0123] Understandably, in touch scenarios where multiple windows overlap, if there are multiple windows at the touch location of a user's touch operation, then the window with the highest display hierarchy (i.e., the topmost window at the touch location) should respond to the touch operation. In one embodiment, the first area consists of the area not covered by the first window; therefore, the touch operation should be responded to by the application corresponding to the highest-hierarchical first window. In this case, the touch area can send the touch operation (specifically, touch data) to the operating system, which then determines the application currently responding to the touch operation and distributes the touch operation to that application for response.

[0124] For example, refer to Figure 3 When the touch position falls within the first area A01, the application corresponding to window 2 responds to the touch operation. For example, assuming window 2 corresponds to a writing application, the writing application responds to the touch operation to display the corresponding handwriting or drawing strokes. When the touch position does not fall within the first area A01, the annotation application displays the corresponding annotation trajectory based on the touch operation.

[0125] Optionally, during the current operation of the annotation application, annotation functionality will be implemented in the annotable areas. In this case, the annotation application may or may not generate a windowed interface.

[0126] The above-described method, by acquiring a first window list containing all currently running visible windows, and when a non-annotation-friendly first window exists among all windows recorded in the first window list, determines a first non-annotation-friendly area on the display screen based on the first position of the first window on the screen, the second position of the second window in the first window list on the screen, and the display coverage relationship between the first and second windows. Then, when a touch operation is received on the display screen using a preset touch object, if the touch position does not fall into the first area, a corresponding annotation trajectory can be displayed according to the touch operation. When the touch position falls into the first area, the application corresponding to the first window where the touch position falls responds to the touch operation. This technical means solves the technical problem in related technologies where writing and annotation applications conflict when an electronic device runs both simultaneously. It enables the simultaneous existence of annotable and non-annotation-friendly windows on an electronic device. By identifying a first area that cannot be annotated, the system determines whether to perform an annotation operation based on whether the touch position of the preset touch object falls within this first area. Compared to related technologies where annotation is globally disabled as long as a first window that cannot be annotated is displayed on the screen, this embodiment disables the annotation function only in a portion of the screen (i.e., the first area). When the touch position does not fall within the first area that cannot be annotated, the annotation trajectory can be displayed normally based on the touch operation. When the touch position falls within the first area that cannot be annotated, the application corresponding to the first window responds to the touch operation. This ensures that the execution of the annotation function is not affected even when a first window that cannot be annotated appears on the screen. This allows for both annotation and other functions that cannot be annotated, such as writing or drawing within the window itself. This ensures the convenience of annotation, makes user annotations more free, and greatly increases practicality, thereby improving the user experience. Furthermore, annotation functionality is only enabled by touch operations on a preset touch object, preventing erroneous responses. For example, if the preset touch object is the tip of a stylus, and the touch position corresponding to the user's finger touch operation does not fall within the first area, no annotation trajectory will be displayed, avoiding misidentification of finger touch operations as annotation operations. Additionally, when a text input box exists, the annotation trajectory can be converted into standard font and input into the text input box, facilitating the conversion of handwritten text into standard characters. This is especially beneficial when users require standard text as annotation content, eliminating the need to repeatedly open and close the soft keyboard, thus improving annotation convenience and enhancing the user experience. Moreover, by removing invisible windows from the first window list, the influence of invisible windows on the calculation of the first area can be avoided, ensuring the accuracy of the first area calculation. Furthermore, recording visible windows in the first window list facilitates subsequent analysis of all visible windows to determine the first area.

[0127] Figure 4 A flowchart illustrating an annotation implementation method provided in another embodiment of this application. Figure 4 The annotation implementation method shown is in Figure 2 Based on the method shown, an exemplary description is given of how to determine the first window and how to determine the first region.

[0128] In this embodiment, all windows in the first window list are arranged in descending order of display hierarchy. All currently running and visible windows have a corresponding display hierarchy. The display hierarchy is related to management and display order. For example, if one window overlaps another, then the display hierarchy of the first window is higher than that of the second.

[0129] refer to Figure 4 The annotation implementation method includes steps 310-370:

[0130] Step 310: Obtain the first window list, which records each currently running and visible window.

[0131] Step 320: Iterate through each window in the first window list. During the iteration, determine whether the currently iterated window is the first window that cannot be annotated. If it is not the first window, proceed to step 330; if it is the first window, proceed to step 340.

[0132] Since all windows recorded in the first window list are arranged in descending order of display hierarchy, we iterate through each window in the first window list in descending order of display hierarchy. During the iteration, for each window currently being iterated, we need to determine whether the currently iterated window is the first window that cannot be annotated.

[0133] The method for determining the first window is not currently limited. In one embodiment, the method of determining the first window by setting a disabled list is taken as an example. In this case, during the traversal process, determining whether the currently traversed window is an unannotable first window includes: determining whether the currently traversed window is included in the disabled list, which contains the window identifier information of each unannotable window in the current device. If the currently traversed window is included in the disabled list, then the currently traversed window is determined to be an unannotable first window. If the currently traversed window is not included in the disabled list, then the currently traversed window is determined not to be the first window.

[0134] Each window has its own window identifier, which distinguishes different windows. The window identifier includes the window title, window class name, and the process name of the process to which the window belongs. In addition, the window information may also include other attributes that indicate whether the window supports annotations. These attributes can be manually extended and indicate whether the window supports annotations from other applications.

[0135] The method for generating the disabled list is currently unrestricted. For example, users can add the window identifiers of applications that cannot be annotated on the annotation implementation device to the disabled list. Alternatively, when an application is installed on the annotation implementation device, if the application has writing or drawing functions, the corresponding window identifiers will be automatically added to the disabled list. Furthermore, the disabled list can also be set by the annotation implementation device or the annotation application's backend server and sent to the annotation implementation device.

[0136] Currently, when the annotation application iterates through the windows in the first window list, it retrieves the window title, window class name, process name of the corresponding process, or other attributes that identify whether the window supports annotations. If the retrieved information matches the identifier of a window in the disabled list (i.e., it is included in the disabled list), then the currently iterated window is determined to be the first window that cannot be annotated. Otherwise, the currently iterated window is determined not to be the first window. By following this method, it is possible to determine whether all windows are the first windows.

[0137] Step 330: If it is not the first window, add the currently traversed window as the second window to the second window list.

[0138] If the currently traversed window is not the first window, it means that the window is annotable. In this case, the currently traversed window is designated as the second window, and then added to the second window list. Then, return to step 320 to continue traversing the next window in the first window list, and continue to determine whether the currently traversed window is the first window that is not annotable. This continues until the traversal is complete, then proceed to step 350.

[0139] Optionally, when the annotation application iterates through the windows of the first window list, it creates a new window list to record the second windows that have been iterated and can be annotated. This new window list is currently referred to as the second window list. It's understood that windows that haven't been iterated through will not appear in the second window list.

[0140] It is understandable that adding a second window to the second window list means adding the window information of the second window to the second window list.

[0141] For example, Figure 5 This is a second schematic diagram of a multi-window display provided in one embodiment of this application, referring to... Figure 5Suppose that display screen A0 shows four windows: Window 1, Window 2, Window 3, and Window 4. Window 1 is on the first layer, Window 3 on the second layer, Window 4 on the third layer, and Window 2 on the fourth layer. Therefore, the first window list contains Window 1, Window 3, Window 4, and Window 2 in that order. Window 2 is the first window; Window 1, Window 3, and Window 4 are all annotable windows. The process iterates through each window in the first window list. First, it iterates through Window 1, which is on the first layer, to determine if Window 1 is the first window. Since Window 1 is not the first window (meaning it is an annotable window), it is added to the second window list as the second window. Next, it iterates through Window 3, which is on the second layer, to determine if it is the first window. Since Window 3 is not the first window (meaning it is an annotable window), it is added to the second window list as the second window. Next, it iterates through and displays window 4, which is at the third level. It checks if window 4 is the first window. Currently, window 4 is not the first window, meaning it is an annotation-enabled window. Therefore, window 4 is added to the second window list as a second window. At this point, the second window list contains three second windows: window 1, window 3, and window 4. It should be noted that windows 1, 3, and 4 recorded in the second window list are all annotation-enabled windows.

[0142] Step 340: If it is the first window, determine the second area in the first window that is not obscured by the second window based on the first position of the first window on the display screen, the second position of each second window currently recorded in the second window list on the display screen, and the display coverage relationship between the first window and the second window.

[0143] If the currently traversed window is the first window, it proves that this window is not annotable. In this case, it's necessary to further determine whether it obscures the currently traversed second window, that is, whether the currently traversed second window covers the currently determined first window (i.e., whether there is a display overlap relationship between the first window and the second windows recorded in the second window list), before finally determining the visible non-annotable area. It's understandable that since untraversed windows have a lower display hierarchy than the currently traversed window, there won't be a situation where untraversed windows obscure the currently traversed window.

[0144] Based on this, when it is determined that the currently traversed window is the first window, the area in the first window that is not obscured by the second windows can be determined according to the first position of the first window on the display screen, the second position of each second window recorded in the aforementioned second window list on the display screen, and the display overlap relationship between the first window and the second windows recorded in the second window list. This area is currently denoted as the second region. This second region serves as the visible, non-annotation-free area in the first window.

[0145] Optionally, when determining the second region, you can first determine the second window in the second window list that has a positional overlap with the first window (it can be partially overlapping) (it can be determined by the first position of the first window and the second position of the second window), that is, determine the display coverage relationship between the first window and the second window recorded in the second window list. Then, based on the second position of the overlapping second window and the first position of the first window, you can obtain the second region that is not obscured by the second window.

[0146] For example, refer to Figure 5 After traversing window 4 and determining that it is not the first window, and adding it as the second window to the second window list, the process iterates through window 2, which is displayed at the fourth level, to determine if it is the first window. Currently, window 2 is the first window, meaning it is a non-annotation-enabled window. Therefore, based on the first position A10 of window 2 on display A0 and the second positions A20 of windows 1, 3, and 4 currently recorded in the second window list on display A0, the annotation application can determine that windows 1, 3, and 2 overlap (i.e., windows 1 and 2 have a display overlap relationship, and windows 3 and 2 also have a display overlap relationship). At this point, based on the second positions of windows 1 and 3 and the first position of window 2, the second area A011 in window 2 (i.e., the first window) that is not obscured by windows 1 and 3 can be determined. It should be noted that the second area A011 is the non-annotation-enabled area of ​​window 2 that is not obscured.

[0147] In one embodiment, this step may include steps 341-344:

[0148] Step 341: If it is the first window, then identify the second windows in the second window list that have a display overlap relationship with the first window, generate the third window list, and add each second window with a display overlap relationship to the third window list. The second windows in the third window list are arranged in descending order of display level.

[0149] Not all second windows recorded in the second window list will necessarily obscure the currently traversed first window; that is, they may not all have a display overlap relationship with the first window. In this case, to reduce the amount of data processing when determining the second region, when the currently traversed window is the first window, we can first determine the second windows in the second window list that overlap with the first window in position. It should be noted that the existence of position overlap means that the second window covers (i.e., obscures) the first window; therefore, the second window and the first window have a display overlap relationship.

[0150] Specifically, the system can determine whether the first window and the second window intersect based on their first and second positions. If there is no intersection, it means the second window and the first window do not overlap, i.e., there is no display overlap. Otherwise, it indicates that the second window and the first window overlap, i.e., there is a display overlap. Optionally, since the window shape is rectangular, the .NET-encapsulated `Rect.Intersect` method (which calculates the intersection of two rectangles) can be used to determine whether the two windows overlap.

[0151] For example, the annotation application also generates a list of windows, currently referred to as the third window list. This third window list can be understood as a list of filter rectangles (i.e., filter rectangles). This third window list is used to record second windows in the second window list that overlap with the first window. In other words, each overlapping second window is added to the third window list. The second windows in the third window list are arranged in descending order of display hierarchy. Subsequent iterations can then be performed on the second windows based on this descending hierarchy order in the third window list.

[0152] For example, refer to Figure 5 Based on the aforementioned embodiments, after determining that window 2 is the first window, the second window in the second window list (currently including window 1, window 3, and window 4) that overlaps with window 2 in position is determined, such as... Figure 5 As shown, the `Rect.Intersect` method, based on the .NET framework, can determine that both windows 1 and 3 overlap with window 2. In this case, a third window list can be generated, and both windows 1 and 3 can be added to the third window list in the order of window 1, then window 3. It should be noted that since window 4 does not overlap with window 2, it does not need to be added to the third window list.

[0153] Step 342: Generate a list of regions and add the first window as the third region to the list.

[0154] Generate a list of regions, which can be denoted as `remainingRects`. The regions recorded in this list are used in calculations with each of the second windows in the third window list to determine the regions in the list that are not obscured by any of the second windows in the third window list.

[0155] In this embodiment, after generating the region list, the window currently being traversed and identified as the first window is added to the region list as a third region. Here, "first window as third region" can be understood as the rectangular area containing the first window being considered the third region; in this case, the position of this third region on the display screen is equal to the first position of the first window. For example, Figure 5 In the middle, the first position A10 corresponding to window 2 can be added to the area list as the third area.

[0156] Step 343: Iterate through each second window in the third window list. During the iteration, based on the third position of the third region in the region list on the screen and the second position of the currently traversed second window on the screen, determine the fourth region in the third region that is not obscured by the currently traversed second window. Use the fourth region as the updated third region in the region list so that the updated third region can be used for calculation when traversing the next second window.

[0157] Currently, the position of the third area on the display screen is recorded as the third position.

[0158] For example, the third window list records second windows that overlap with the first window, and these second windows are arranged in descending order of display hierarchy. Therefore, each second window in the third window list can be traversed sequentially. During this traversal, based on the third position of the third region in the region list and the second position of the currently traversed second window on the display screen, the area in the third region not obscured by the currently traversed second window can be determined. Currently, this area in the third region not obscured by the currently traversed second window is designated as the fourth region. It is understood that there can be multiple fourth regions.

[0159] For example, when the first window is the third region, the first position of the first window is the third position. Then, based on the third position and the second position of the currently traversed second window, the fourth region in the third region that is not obscured by the second window can be determined.

[0160] Next, the fourth region is used as the updated third region in the region list, so that the updated third region (i.e., the fourth region from the previous iteration) can be used for calculations when traversing the next second window. It's understandable that, after traversing the third window list from high to low display hierarchy, the fourth region might still be obscured by second windows with lower display hierarchy in the third window list. Therefore, the newly determined unobscured fourth region needs to be assigned to the region list, replacing the third region in the region list, so that the calculations can be performed using the newly determined unobscured fourth region and the second windows that haven't been traversed yet.

[0161] After traversing all the second windows in the third window list in this way, you can obtain the second area of ​​the first window that is not obscured by any of the second windows recorded in the third window list.

[0162] For example, Figure 6 This is a first schematic diagram of the fourth region provided in one embodiment of this application. Figure 6 by Figure 5 Based on this, window 2 is then added as a third region to the region list, and the third window list includes both window 1 and window 3. (See reference...) Figure 6 At this point, the third region in the region list, located at position A30 on display A0, is the first position A10 of window 2. The process iterates through each second window in the third window list, starting with window 1. Based on the current position A30 of the third region in the region list on display A0 (i.e., the first position A10 of window 2) and the second position A20 of the currently traversed window 1 on display A0, the fourth region A0111, which is not obscured by the currently traversed window 1, is determined. This fourth region A0111 is then used as the updated third region and recorded in the region list. Figure 7 A second schematic diagram of the fourth region provided in one embodiment of this application, referring to... Figure 7 ,Will Figure 6 The fourth region A0111 in the list is used as the updated third region and recorded in the region list. Then, window 3 is traversed, and the third region in the region list is located in the third position of the display screen A0 (i.e., the third region in the region list is the third position of the third region in the display screen A0). Figure 6 The fourth region A0111 in the third region is determined by comparing the second position A20 of the currently traversed window 3 in the display screen A0 with the second position A20 of the currently traversed window 3. The fourth region A0112 in the third region is not obscured by the currently traversed window 3.

[0163] In one embodiment, to simplify the calculation process of the fourth region, the step of determining the fourth region in the third region that is not obscured by the currently traversed second window based on the third position of the third region in the region list on the display screen and the second position of the currently traversed second window on the display screen may specifically include: determining the largest unobscured rectangular region on the left, the largest unobscured rectangular region on the right, the largest unobscured rectangular region on the top, and the largest unobscured rectangular region on the bottom of the third region based on the third position of the third region in the region list on the display screen and the second position of the currently traversed second window on the display screen, and taking each determined largest rectangular region as the fourth region that is not obscured by the currently traversed second window.

[0164] Understandably, since the window is rectangular, calculating the window's position can be simplified to calculating the rectangles. In this case, each rectangle can serve as a `mainRect`, representing the coordinates of its top-left corner, width, and height. Currently, using rectangle calculations, the largest rectangular area not obscured by the currently traversed second window is calculated from the left, right, top, and bottom directions within the third region. That is, each fourth region calculated so far is a rectangular region.

[0165] Currently, we take calculating the largest rectangular area in the order of left, right, top, and bottom as an example. In practical applications, the largest rectangular area can also be calculated in other orders.

[0166] For example, first determine if the left side of the third region is obscured by the currently traversed second window. If at least one coordinate point (in pixel coordinates) of the left boundary of the third region (composed of the coordinate points of the left border of the rectangle) is contained within the second window, the left side can be considered obscured. In this case, determine the largest rectangular region on the left that is not obscured. If every coordinate point of the left boundary of the third region is not contained within the second window, then the left side is determined to be unobscured. In this case, the left edge of the third region can be used as the left border of the largest rectangular region to determine a largest rectangular region that is not obscured by the second window. This largest rectangular region is also used as a mainRect. The right border of the largest rectangular region overlaps with the left border of the second window. The largest rectangular regions on the right, top, and bottom can all refer to the method described above for determining the largest rectangular region on the left.

[0167] At this point, each determined largest rectangular region is used as the fourth region, which ensures that each fourth region is a rectangular region, that is, ensures that each third region in the region list is a rectangular region, so as to facilitate subsequent calculations.

[0168] In one embodiment, since each third region is a rectangular region, the third position of the third region may include the first coordinates of the upper left corner of the corresponding third region on the display screen, the first width of the corresponding third region, and the first height of the corresponding third region. The second window is also a rectangular region; therefore, the second position of the second window may include the second coordinates of the upper left corner of the corresponding second window on the display screen, the second width of the corresponding second window, and the second height of the corresponding second window. The aforementioned coordinates, width, and height all correspond to pixel coordinates. In this case, the largest unobstructed rectangular region on the left, the largest unobstructed rectangular region on the right, the largest unobstructed rectangular region on the top, and the largest unobstructed rectangular region on the bottom can be determined using the following four methods:

[0169] Option 1: If the X-axis coordinate value in the second coordinate system is greater than the X-axis coordinate value in the first coordinate system, then the first coordinate system is used as the coordinate of the top left corner vertex of the largest unobstructed rectangular area on the left. The difference between the X-axis coordinate values ​​in the second coordinate system and the X-axis coordinate values ​​in the first coordinate system is used as the width of the largest unobstructed rectangular area on the left. The first height is used as the height of the largest unobstructed rectangular area on the left.

[0170] If the X-axis coordinate value in the second coordinate system is greater than the X-axis coordinate value in the first coordinate system, it means that the second window being traversed does not occlude the left boundary of the third region. In this case, the top-left vertex of the third region can be used as the top-left vertex of the largest rectangular region that is not occluded on the left. The distance on the X-axis between the left boundary of the second window and the left boundary of the third region can be used as the width of the largest rectangular region. This distance is equal to the difference between the X-axis coordinate value in the second coordinate system and the X-axis coordinate value in the first coordinate system. The height of the third region (i.e., the first height) can then be used as the height of the largest rectangular region.

[0171] For example, Figure 8 A schematic diagram illustrating the determination of the maximum rectangular region according to an embodiment of this application, referring to... Figure 8 The process iterates through each second window in the third window list, taking window 1 as the currently traversed second window as an example. The third position is now window 2. The first coordinate of the top-left corner of the corresponding third area on the screen is the coordinate (X1, Y1) of the top-left corner of window 2. The first width of the corresponding third area is the width L1 of window 2, and the first height of the corresponding third area is the height H1 of window 2. Therefore, the third position includes the first coordinate (X1, Y1), the first width L1, and the first height H1. Similarly, the second coordinate of the top-left corner of the corresponding second window on the screen is the coordinate (X2, Y2) of the top-left corner of window 1. The second width of the corresponding second window is the width L2 of window 1, and the second height of the corresponding second window is the height H2 of window 1. Therefore, the second position includes the second coordinate (X2, Y2), the second width L2, and the second height H2. The origin of the pixel coordinate system on the screen is located at the top-left corner. Figure 8 It can be seen that the X-axis coordinate X2 in the second coordinate (X2, Y2) is greater than the X-axis coordinate X1 in the first coordinate (X1, Y1), indicating that window 1 does not obscure the left boundary of window 2. Therefore, the first coordinate (X1, Y1) is taken as the coordinate of the top-left vertex of the largest unobstructed rectangular area A0113 on the left. The difference between the X-axis coordinate X2 in the second coordinate (X2, Y2) and the X-axis coordinate X1 in the first coordinate (X1, Y1), which is (X2-X1), is taken as the width L3 of the largest unobstructed rectangular area A0113 on the left. The first height H1 is taken as the height of the largest unobstructed rectangular area A0113 on the left. At this point, the coordinates of the top-left vertex of the largest unobstructed rectangular area A0113 on the left are (X1, Y1), the width is L3 (i.e., X2-X1), and the height is H1.

[0172] Understandable, if Figure 8 If X2 is less than or equal to X1, it means that window 1 occludes the left boundary of window 2. Therefore, there is no largest rectangular area on the left that is not occluded.

[0173] Option 2: If the sum of the X-axis coordinate and the first width in the first coordinate system is greater than the sum of the X-axis coordinate and the second width in the second coordinate system, then the second sum is used as the X-axis coordinate of the top-left vertex of the largest unobstructed rectangular area on the right, the Y-axis coordinate of the first coordinate system is used as the Y-axis coordinate of the top-left vertex, the difference between the first and second sums is used as the width of the largest unobstructed rectangular area on the right, and the first height is used as the height of the largest unobstructed rectangular area on the right.

[0174] In this context, the sum of the X-axis coordinate value in the first coordinate system and the first width is denoted as the first sum, which can be understood as the X-axis coordinate value of the top-right corner vertex of the third position. Similarly, the sum of the X-axis coordinate value in the second coordinate system and the second width is denoted as the second sum, which can be understood as the X-axis coordinate value of the top-right corner vertex of the currently traversed second window.

[0175] If the first sum is greater than the second sum, it means that the second window being traversed does not obscure the right boundary of the third region. In this case, the coordinates of the right boundary of the second window on the X-axis (i.e., the second sum) can be used as the coordinates of the top left corner of the largest rectangular region that is not obscured on the right. The coordinates of the top boundary of the third region on the Y-axis (i.e., the Y-axis coordinates in the first coordinate) can be used as the coordinates of the top left corner of the largest rectangular region on the Y-axis. The distance between the right boundary of the second window and the right boundary of the third region on the X-axis can be used as the width of the largest rectangular region. This distance is equal to the first sum minus the second sum. The height of the third region (i.e., the first height) can be used as the height of the largest rectangular region.

[0176] Understandably, if the first sum is less than or equal to the second sum, it means the second window is occluding the right boundary of the third region. Therefore, there is no largest rectangular region on the right that is not occluded. For example, Figure 8 In the case of [the above], there is no largest rectangular area on the right that is not obscured.

[0177] Option 3: If the Y-axis coordinate value in the second coordinate system is greater than the Y-axis coordinate value in the first coordinate system, the first coordinate system is used as the coordinate of the top left corner vertex of the largest unobstructed rectangular area on the upper side, the first width is used as the width of the largest unobstructed rectangular area on the upper side, and the difference between the Y-axis coordinate value in the second coordinate system and the Y-axis coordinate value in the first coordinate system is used as the height of the largest unobstructed rectangular area on the upper side.

[0178] If the Y-axis coordinate value in the second coordinate system is greater than the Y-axis coordinate value in the first coordinate system, it means that the second window being traversed does not occlude the upper boundary of the third region. In this case, the vertex of the upper left corner of the third region can be used as the upper left corner vertex of the largest unoccluded rectangular region on the upper side. The distance on the Y-axis between the upper boundary of the second window and the upper boundary of the third region can be used as the height of the largest rectangular region. This distance is equal to the difference between the Y-axis coordinate value in the second coordinate system and the Y-axis coordinate value in the first coordinate system. The width of the third region (i.e., the first width) can then be used as the width of the largest rectangular region.

[0179] For example, refer to Figure 8 The Y-axis coordinate Y2 in the second coordinate (X2, Y2) is greater than the Y-axis coordinate Y1 in the first coordinate (X1, Y1). Therefore, the first coordinate (X1, Y1) is taken as the coordinate (X1, Y1) of the top left corner vertex of the largest unobstructed rectangular region A0114 on the upper side. The first width L1 is taken as the width of the largest unobstructed rectangular region A0114 on the upper side. The difference between the Y-axis coordinate Y2 in the second coordinate (X2, Y2) and the Y-axis coordinate Y1 in the first coordinate (X1, Y1), (Y2-Y1), is taken as the height H3 of the largest unobstructed rectangular region A0114 on the upper side. At this time, the coordinate of the top left corner vertex of the largest unobstructed rectangular region A0114 on the upper side is (X1, Y1), the width is L1, and the height is H3 (i.e., (Y2-Y1)).

[0180] Understandable, if Figure 8 If Y2 is less than or equal to Y1, it means that window 1 occludes the upper boundary of window 2. Therefore, there is no largest rectangular area that is not occluded on the upper side.

[0181] Option 4: If the third sum of the Y-axis coordinate value and the first height in the first coordinate system is greater than the fourth sum of the Y-axis coordinate value and the second height in the second coordinate system, then the X-axis coordinate value in the first coordinate system is used as the X-axis coordinate value of the top left corner vertex of the largest unobstructed rectangular area on the lower side, the fourth sum is used as the Y-axis coordinate value of the top left corner vertex, the first width is used as the width of the largest unobstructed rectangular area on the lower side, and the difference between the third and fourth sums is used as the height of the largest unobstructed rectangular area on the lower side.

[0182] In this system, the sum of the Y-axis coordinate and the first height in the first coordinate system is denoted as the third sum, which can be understood as the Y-axis coordinate of the bottom left corner vertex of the third position. The sum of the Y-axis coordinate and the second height in the second coordinate system is denoted as the fourth sum, which can be understood as the Y-axis coordinate of the bottom left corner vertex of the currently traversed second window.

[0183] If the third sum is greater than the fourth sum, it means that the currently traversed second window does not occlude the lower boundary of the third region. In this case, the X-axis coordinate value in the first coordinate system can be used as the X-axis coordinate value of the top-left vertex of the largest unoccluded rectangular region below. The Y-axis coordinate value of the lower boundary of the currently traversed second window (i.e., the fourth sum) can be used as the Y-axis coordinate value of the top-left vertex of the largest rectangular region. The first width can be used as the width of the largest rectangular region, and the distance between the lower boundary of the third region and the lower boundary of the currently traversed second window can be used as the height of the largest rectangular region. This distance is equal to the difference between the third and fourth sums.

[0184] For example, such as Figure 8 As shown, the Y-axis coordinate Y1 in the first coordinate (X1, Y1) is added to the first height H1 to obtain the third sum, which is the Y-axis coordinate Y3 of the bottom left corner vertex of the third position; the Y-axis coordinate Y2 in the second coordinate (X2, Y2) is added to the second height H2 to obtain the fourth sum, which is the Y-axis coordinate Y4 of the bottom left corner vertex of the currently traversed second window. If the third sum is greater than the fourth sum, it indicates that the bottom of window 2 is not obscured by window 1, i.e., as shown... Figure 8 Since Y3 is greater than Y4, the X-axis coordinate X1 in the first coordinate (X1, Y1) is taken as the X-axis coordinate X1 of the top-left vertex of the largest unobstructed rectangular region A0115 on the lower side. The fourth sum is taken as the Y-axis coordinate Y4 of the top-left vertex of the largest unobstructed rectangular region A0115 on the lower side. The first width L1 is taken as the width of the largest unobstructed rectangular region A0115 on the lower side. The difference between the third and fourth sums (Y3-Y4) is taken as the height H4 of the largest unobstructed rectangular region A0115 on the lower side. At this time, the coordinates of the top-left vertex of the largest unobstructed rectangular region on the lower side are (X1, Y4), the width is L1, and the height is H4 (i.e., (Y3-Y4)).

[0185] Understandable, if Figure 8 If Y3 is less than or equal to Y4, it means that window 1 occludes the lower boundary of window 2. Therefore, there is no largest rectangular area that is not occluded on the lower side.

[0186] If there is no largest rectangular region that is obscured, the largest rectangular region can also be considered empty.

[0187] After obtaining the largest rectangular area according to the four schemes mentioned above, each largest rectangular area can be used as the fourth area. Then, each fourth area is assigned to the area list so that each fourth area is used as the third area in the area list (the original third area in the area list will be deleted). At this time, there may be multiple third areas and there may be duplicates. When traversing the next second window in the third window list, the largest rectangular area that is not obscured in four directions can be determined for each third area and the currently traversed second window. Each of the obtained largest rectangular areas can be used as the fourth area. That is, repeat the above process until all the second windows in the third window list have been traversed.

[0188] For example, such as Figure 8 As shown, determine the largest unobstructed rectangular region A0113 on the left and the largest unobstructed rectangular region on the right in the third region. Figure 8 In the case of no obstruction in the middle, the largest unobstructed rectangular area A0114 on the top and the largest unobstructed rectangular area A0115 on the bottom can both be used to determine the fourth area A0111 that is not obstructed by the currently traversed window 1, and assign it to the area list. At this time, there are three third areas in the area list after assignment, and the three third areas overlap.

[0189] Optionally, after obtaining each maximum rectangular region, the currently determined maximum rectangular region can be added to newRemainingRects (a new list of regions. This list of regions is used to record the currently determined fourth region). When all fourth regions are determined during the current traversal of the second window, newRemainingRects can be assigned to the region list (i.e., remainingRects).

[0190] Step 344: After the traversal is complete, the third region in the region list is taken as the second region in the first window that is not obscured by the second window.

[0191] During the traversal of each second window in the third window list, the fourth region determined at each traversal is continuously used as the updated third region in the region list. Therefore, after the traversal is complete, the third region in the final region list (i.e., the third region obtained by assigning the final determined fourth region to the region list) can be used as the second region in the currently traversed first window that is not obscured by the already traversed second windows. It can be understood that there may be multiple second regions, and these regions can overlap. The second region is a region that cannot be annotated.

[0192] It should be noted that if there is no second window in the second window list that overlaps with the first window in position, then the entire window area of ​​the first window (i.e., the first position) can be determined as the second area that is not obscured by the second window.

[0193] After obtaining the second region, you can continue to traverse the next window in the first window list, return to step 320 to determine whether the currently traversed window is the first window that cannot be annotated, until the traversal ends.

[0194] In other words, after each execution of step 330 or step 340 (which can be understood as only executing step 330 or step 340 each time the windows in the first window list are traversed), it is determined whether the traversal has ended. If it is determined that the traversal has ended, step 350 is executed. If it is determined that the traversal has not ended, step 320 is returned to continue traversing the windows and it is determined whether the currently traversed window is the first window.

[0195] The method for determining the end of traversal of the first window list can include: determining the end of traversal when the currently traversed window is a full-screen window. Then, step 350 is executed. After processing the currently traversed window, it can be determined whether the currently traversed window is a full-screen window; if it is, then the traversal is determined to be complete. It can be understood that if there are other untraversed windows in the first window list, then these other untraversed windows will be obscured by the currently traversed full-screen window; therefore, further traversal is unnecessary, and the end of traversal can be determined.

[0196] The method for determining the end of traversal can also include: determining the end of traversal when the currently traversed window is the last window in the first window list. Then, proceed to step 350. After processing the currently traversed window, if it is the last window in the first window list, it means that all windows in the list have been traversed, therefore, the end of traversal can be determined.

[0197] Based on this, after processing the currently traversed window, determine whether the currently traversed window satisfies either being a full-screen window or the last window in the first window list. If it satisfies either one, the traversal ends; otherwise, the traversal continues.

[0198] Step 350: After the traversal is completed, based on each of the currently obtained second regions, determine the first region on the display screen that cannot be annotated.

[0199] Through steps 320-340 above, the second region of each first window that is not obscured by the second window can be obtained. After the traversal is completed, based on the currently obtained second regions, the first region of all first windows that is not obscured by all second windows can be determined, and thus the first region of the display screen that cannot be annotated can be determined.

[0200] It is understandable that when obtaining the second region based on the largest unobstructed rectangular area in four directions, there may be overlap between multiple second regions. In this case, the second regions can be deduplicated first. That is, this step may include: after traversal, deduplicating each of the currently obtained second regions, and using the deduplicated region as the first region that cannot be annotated on the display screen.

[0201] For example, deduplication can be performed as follows: if two second regions completely overlap or one second region is completely covered by the other, then the union of the two second regions is taken as the region obtained after deduplication. For partially overlapping second regions, no deduplication is performed. It can be understood that there can be multiple first regions obtained after deduplication.

[0202] Step 360: Receive a touch operation from a preset touch object on the display screen. If the touch position of the touch operation does not fall within the first area, proceed to step 370. If the touch position of the touch operation falls within the first area, proceed to step 380.

[0203] Step 370: When the touch position of the touch operation does not fall into the first area, display the corresponding annotation trajectory according to the touch operation.

[0204] Step 380: When the touch position of the touch operation falls into the first area, the application corresponding to the first window where the touch position falls responds to the touch operation.

[0205] As described above, traversing each window in the first window list sequentially according to the display hierarchy from high to low ensures that all windows that might cause occlusion are considered when determining the second region, guaranteeing the accuracy of the second region determination. Furthermore, prioritizing the second windows in the second window list that overlap with the first window (i.e., have a display coverage relationship) reduces the amount of data processing required for determining the second region. Moreover, based on the rectangular nature of windows, calculating the largest unoccluded rectangular area in the third region from the left, right, top, and bottom directions, and using this as the fourth region, simplifies the computational complexity of calculating unoccluded areas. Furthermore, by reasonably calculating the positions of the third region and the second window, the largest unoccluded rectangular areas can be determined. After obtaining the second region, deduplication ensures that the final first region does not contain duplicate areas, thus guaranteeing the accuracy of subsequent logic for determining whether annotations should respond based on the first region. Finally, the disabled list allows for quick determination of the visibility of currently traversed windows, significantly improving traversal speed and thus enhancing the overall responsiveness of annotation implementation, ultimately improving the user experience. Furthermore, when traversing the first window list, a reasonable method for determining the end of the traversal is set. In particular, when traversing to a window that is displayed in full screen, it means that the remaining untraversed windows are all obscured by the full-screen window. Therefore, there is no need to continue traversing, which can reduce the amount of data processing required to determine the first area.

[0206] In one embodiment, there is a situation where none of the windows recorded in the first window list are first windows. In this case, the method further includes: when there is no first window that cannot be annotated in each window recorded in the first window list, determining that there is no first region on the display screen; or, when there is no first window that cannot be annotated in each window recorded in the first window list, using a preset edge region of the display screen as the first region.

[0207] Optionally, when the first window list does not have a first window, it can be determined that there is no first area that cannot be annotated on the current display screen. That is, all windows (even if they overlap) can be annotated. Therefore, the annotation function of writing as soon as you pick up the pen can be implemented globally, that is, all areas on the display screen can be annotated.

[0208] Optionally, when the first window list does not contain a first window, it can be determined that all windows on the current display screen (even if they overlap) can be annotated. However, since the display screen may have edges with a sliding function, in order to avoid conflicts between the sliding function and the annotation function, other areas on the display screen outside the edge areas can be designated as annotable areas. The edge areas can be pre-set in the annotation implementation device, and may include areas in the four directions (up, down, left, and right) of the display screen, or at least one direction. The size and position of the edge areas are also set according to the actual situation. For example, Figure 9 This is a third schematic diagram of a multi-window display provided in one embodiment of this application. Area A0116 in display screen A0 can be considered as an edge area. Optionally, the edge area can also be used as the first area. In this case, when the user's touch operation is applied to the edge area, the operating system can determine which swipe function should be responded to.

[0209] As mentioned above, when the first window does not exist in the first window list, the global response annotation function of the display screen can be implemented. Alternatively, the preset edge area of ​​the display screen can be used as the first area that cannot be annotated, which can avoid the conflict between the side-sliding function and the annotation function when the display screen has a side-sliding function.

[0210] One embodiment of this application also provides an annotation implementation device, see reference. Figure 1 The annotation implementation device includes a processor 11, a memory 12, and a display screen 13. The processor 11, memory 12, and display screen 13 can be connected via a bus or other means. The display screen 13 includes a display component and a touch component; the display component is used to display information, and the touch component is used to receive touch operations. The memory 12 is used to store one or more programs. When one or more programs are executed by one or more processors 11, the one or more processors 11 implement the annotation implementation method of any of the foregoing embodiments. The relevant details of each component can be found in the foregoing description.

[0211] The aforementioned annotation implementation device is used to execute arbitrary annotation implementation methods, and has corresponding functions and beneficial effects. For specific details not described here, please refer to the relevant descriptions of the aforementioned annotation implementation methods.

[0212] One embodiment of this application also provides a storage medium containing computer-executable instructions. When executed by a processor, the computer-executable instructions are used to perform relevant operations in the annotation implementation method provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0213] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0214] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing module of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing module of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0215] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0216] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0217] Note that the above description is merely a preferred embodiment and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this application, and the scope of this application is determined by the scope of the appended claims.

Claims

1. A method for implementing annotations, characterized in that, include: Get the first window list, which records each currently running and visible window; When there is a first window that cannot be annotated in each window recorded in the first window list, the first area that cannot be annotated in the display screen is determined based on the first position of the first window in the display screen, the second position of the second window in the first window list in the display screen, and the display coverage relationship between the first window and the second window. The first window is at least one, and the second window is an annotable window in the first window list. Receives touch operations on the display screen from a preset touch object; When the touch position of the touch operation does not fall into the first area, the corresponding annotation trajectory is displayed according to the touch operation; When the touch position of the touch operation falls into the first area, the application corresponding to the first window where the touch position falls responds to the touch operation.

2. The annotation implementation method according to claim 1, characterized in that, When the touch position of the touch operation does not fall into the first area, displaying the corresponding annotation trajectory according to the touch operation includes: When the touch position of the touch operation does not fall into the first area, the corresponding annotation trajectory is displayed according to the touch operation. When it is confirmed that there is a text input box at the touch position of the touch operation, after the touch operation ends, the annotation trajectory is converted into text content in standard font and then input into the text input box.

3. The annotation implementation method according to claim 1, characterized in that, All windows in the first window list are arranged in descending order of display hierarchy; When there is a first window that cannot be annotated in each window recorded in the first window list, the first area on the display screen that cannot be annotated is determined based on the first position of the first window on the display screen, the second position of the second window in the first window list on the display screen, and the display coverage relationship between the first window and the second window, including: Each window in the first window list is traversed sequentially. During the traversal, it is determined whether the currently traversed window is a first window that cannot be annotated. If it is not a first window, the currently traversed window is added as a second window to the second window list. If it is a first window, a second area in the first window that is not obscured by the second window is determined based on the first position of the first window on the display screen, the second position of each second window currently recorded in the second window list on the display screen, and the display coverage relationship between the first window and the second window. After the traversal is completed, based on each of the currently obtained second regions, a first region in the display screen that cannot be annotated is determined.

4. The annotation implementation method according to claim 3, characterized in that, If it is a first window, then a second area in the first window that is not obscured by the second window is determined based on the first position of the first window on the display screen, the second position of each second window currently recorded in the second window list on the display screen, and the display coverage relationship between the first window and the second window, including: If it is the first window, then identify the second window in the second window list that has a display overlap relationship with the first window, generate a third window list, and add each second window that has a display overlap relationship to the third window list. The second windows in the third window list are arranged in order of display level from high to low. Generate a list of regions, and add the first window as the third region to the list of regions; Each second window in the third window list is traversed sequentially. During the traversal, based on the third position of the third region in the region list on the display screen and the second position of the currently traversed second window on the display screen, a fourth region in the third region that is not obscured by the currently traversed second window is determined. The fourth region is used as the updated third region in the region list so that the updated third region can be used for calculation when traversing the next second window. After the traversal is complete, the third region in the region list is taken as the second region in the first window that is not obscured by the second window.

5. The annotation implementation method according to claim 4, characterized in that, The step of determining a fourth region in the third region that is not obscured by the currently traversed second window based on the third position of the third region in the region list on the display screen and the second position of the currently traversed second window on the display screen includes: Based on the third position of the third region in the region list on the display screen and the second position of the currently traversed second window on the display screen, the largest unobstructed rectangular region on the left, the largest unobstructed rectangular region on the right, the largest unobstructed rectangular region on the top, and the largest unobstructed rectangular region on the bottom of the third region are determined respectively, and each determined largest rectangular region is regarded as the fourth region that is not obstructed by the currently traversed second window.

6. The annotation implementation method according to claim 5, characterized in that, The third position includes the first coordinates of the upper left corner vertex of the corresponding third region on the display screen, the first width of the corresponding third region, and the first height of the corresponding third region; the second position includes the second coordinates of the upper left corner vertex of the corresponding second window on the display screen, the second width of the corresponding second window, and the second height of the corresponding second window. The step of determining the largest unobstructed rectangular area on the left, right, top, and bottom of the third region based on the third position of the third region in the region list on the display screen and the second position of the currently traversed second window on the display screen includes: If the X-axis coordinate value in the second coordinate system is greater than the X-axis coordinate value in the first coordinate system, the first coordinate system is used as the coordinate of the top left corner vertex of the largest rectangular area that is not obscured on the left, the difference between the X-axis coordinate value in the second coordinate system and the X-axis coordinate value in the first coordinate system is used as the width of the largest rectangular area, and the first height is used as the height of the largest rectangular area. If the sum of the X-axis coordinate value and the first width in the first coordinate system is greater than the sum of the X-axis coordinate value and the second width in the second coordinate system, the second sum is used as the X-axis coordinate value of the top-left vertex of the largest rectangular area that is not obscured on the right, the Y-axis coordinate value in the first coordinate system is used as the Y-axis coordinate value of the top-left vertex, the difference between the first sum and the second sum is used as the width of the largest rectangular area, and the first height is used as the height of the largest rectangular area. If the Y-axis coordinate value in the second coordinate system is greater than the Y-axis coordinate value in the first coordinate system, the first coordinate system is used as the coordinate of the top left corner vertex of the largest rectangular area that is not obscured on the upper side, the first width is used as the width of the largest rectangular area, and the difference between the Y-axis coordinate value in the second coordinate system and the Y-axis coordinate value in the first coordinate system is used as the height of the largest rectangular area. If the third sum of the Y-axis coordinate value and the first height in the first coordinate system is greater than the fourth sum of the Y-axis coordinate value and the second height in the second coordinate system, then the X-axis coordinate value in the first coordinate system is taken as the X-axis coordinate value of the top left corner vertex of the largest rectangular area that is not obscured on the lower side, the fourth sum is taken as the Y-axis coordinate value of the top left corner vertex, the first width is taken as the width of the largest rectangular area, and the difference between the third sum and the fourth sum is taken as the height of the largest rectangular area.

7. The annotation implementation method according to claim 5 or 6, characterized in that, After the traversal is completed, based on each of the currently obtained second regions, a first region in the display screen that cannot be annotated is determined, including: After the traversal is completed, each of the second regions obtained is deduplicated, and the regions obtained after deduplication are used as the first regions that cannot be annotated on the display screen.

8. The annotation implementation method according to claim 3, characterized in that, Determining whether the currently traversed window is the first window that cannot be annotated includes: Determine whether the currently traversed window is included in the disabled list, which contains window identification information of each window that cannot be annotated in the current device; If the currently traversed window is included in the disabled list, then the currently traversed window is determined to be the first window that cannot be annotated; If the currently traversed window is not included in the disabled list, then it is determined that the currently traversed window is not the first window.

9. The annotation implementation method according to claim 3, characterized in that, Before determining the first unannotable area on the display screen based on each of the currently obtained second areas after the traversal is completed, the process includes: The traversal ends when the currently traversed window is a full-screen window; or, The traversal ends when the currently traversed window is the last window in the first window list.

10. The annotation implementation method according to claim 1, characterized in that, Also includes: If there is no first window that cannot be annotated in each window recorded in the first window list, it is determined that the first area does not exist on the display screen. or, When there is no unannotationable first window in each window recorded in the first window list, the preset edge area of ​​the display screen is taken as the first area.

11. The annotation implementation method according to claim 1, characterized in that, The process of obtaining the first window list includes: Get each currently running window, remove the invisible windows in each window to generate the first window list, wherein the invisible windows include minimized windows, transparent windows and hidden windows.

12. An annotation implementation device, characterized in that, include: Display screen, one or more processors, and memory; The memory is used to store one or more programs; The display screen is used to display information; When the one or more programs are executed by the one or more processors, the one or more processors implement the annotation implementation method as described in any one of claims 1-11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the annotation implementation method as described in any one of claims 1-11.

Images