Method for supporting selection of screen component and collection of coordinate information

Abstract

Disclosed is a method for supporting collection of information regarding a screen component, which is performed by a computing device comprising at least one processor, according to some embodiments of the present disclosure. The method for supporting collection of information regarding a screen component may comprise the steps of: according to a first user interaction for selecting one area within a screen of the computing device on which a pre-specified application is executed, determining screen coordinates corresponding to the one area; determining user interface (UI) coordinates within a UI corresponding to the screen coordinates by using hierarchical structures of a parent UI and a child UI defined for the application and a movement function corresponding to each of the hierarchical structures; determining a first node corresponding to the UI coordinates; and generating report information for displaying, in a pre-specified visualization method, an object representing the first node.

Description

How to support selection and collection of coordinate information for screen components

[0001] The present disclosure relates to a method for supporting the collection of information about a screen component, and more particularly, to a method for collecting a screen component of a remotely located computing device.

[0002] The process of deploying an app on the Android operating system can be summarized as follows:

[0003] Application Development: First, you need to develop the app that will run on the user terminal. This process may include designing the user interface, implementing the app logic, and testing.

[0004] Building the application: Once app development is complete, you need to build the app. Building can refer to the process of compiling the source code and packaging it into an APK (Android Package) file.

[0005] Signing: Apps must be signed. Adding a signature to an APK file ensures that the app is recognized as a trusted source. You can use a development debug keystore or a release keystore for signing.

[0006] Register with the Google Play Console: To distribute your app to the Google Play Store, you must register with the Google Play Console. This allows you to configure your app information, pricing, description, screenshots, version control, and more.

[0007] Upload APK: Upload your APK file to the Google Play Console. You can upload an APK file for release or set up different release tracks, such as Alpha, Beta, or Production.

[0008] Set up resources and store pages: Set up resources such as app icons, graphic materials, descriptions, and app store pages.

[0009] Pricing and Country Settings: Set your app's price and specify which countries it will be available in. You can choose whether your app is free or paid, and manage your pricing policy.

[0010] Release and Review: Release your app on the Google Play Console and it will undergo Google's review process. This review verifies the app's content, compliance with advertising policies, and more.

[0011] Application Distribution: Once review is complete, your app will be distributed to the Google Play Store. Users can search for and download your app from the store.

[0012] Managing application updates: When you need to update your app or release a new version, repeat the previous steps to upload and release the new APK file.

[0013] User Support and Feedback Management: Manage interactions with users using your app, process feedback and issue reports, and contribute to app improvement.

[0014] Marketing and Promotion: After successfully deploying your app, you'll need to engage more users and promote it through marketing and promotion activities.

[0015] Natively developed Android applications typically must follow the deployment process described above to reflect application content. The entire process of development, build, testing, and deployment consumes significant time and resources. Even using application updates, the development, build, and testing process is still necessary. Furthermore, remote resource patching methods cannot change the execution entity, limiting the scope of changes.

[0016] The present disclosure is conceived in response to the aforementioned background technology and seeks to provide a method capable of supporting the collection of information about screen components without rebuilding or redistributing the application.

[0017] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0018] According to one embodiment of the present disclosure for solving the above-described problem, a method for supporting information collection on a screen component performed by a computing device including at least one processor is disclosed. The method for supporting information collection on the screen component may include: determining screen coordinates corresponding to an area within a screen of the computing device on which a pre-specified application is executed, based on a first user interaction for selecting the area; determining UI (User Interface) coordinates within a user interface corresponding to the screen coordinates using a hierarchical structure of a parent UI and a child UI defined for the application and a movement function corresponding to each of the hierarchical structures; determining a first node corresponding to the UI coordinates; and generating report information for displaying an object representing the first node in a pre-specified visualization manner.

[0019] In addition, if there is a second user interaction for moving the user interface displayed on the screen for each of the hierarchical structures, the step of determining a movement function according to the second user interaction may further include; and the step of determining UI coordinates within the user interface corresponding to the screen coordinates may include a step of determining the UI coordinates based on the movement function and the screen coordinates.

[0020] In addition, the step of determining a movement function according to the second user interaction may include the step of determining a first matrix value related to a main area included in the user interface; the step of determining a second matrix value related to a scroll area included in the user interface; the step of determining a third matrix value related to the second user interaction; and the step of determining the movement function based on the first matrix value, the second matrix value, and the third matrix value.

[0021] Additionally, the first node may be a lowest node located at the lowest level hierarchically, and the movement function may be a function calculated for the parent node of the first node.

[0022] In addition, the method may further include: receiving a third user interaction for reselecting the above-described area; determining a second node corresponding to the UI coordinates other than the first node; and generating reporting information for displaying an object representing the second node in a pre-specified visualization manner.

[0023] The technical solutions obtainable in the present disclosure are not limited to the solutions mentioned above, and other solutions not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0024] According to some embodiments of the present disclosure, it is an object to provide a method that can support gathering information about screen components without rebuilding or redeploying an application.

[0025] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0026] Various aspects are now described with reference to the drawings, wherein like reference numerals are used to refer to similar elements generally. In the following examples, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of one or more aspects. However, it will be apparent that such aspects may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to facilitate the description of one or more aspects.

[0027] FIG. 1 is a block diagram illustrating an example of a computing device according to some embodiments of the present disclosure.

[0028] FIG. 2 is a flowchart illustrating an example of a method by which a computing device generates reporting information according to some embodiments of the present disclosure.

[0029] FIG. 3 is a flowchart illustrating an example of a method for a computing device to determine a movement function according to a second user interaction according to some embodiments of the present disclosure.

[0030] FIG. 4A is a diagram illustrating an example of a second user interaction according to some embodiments of the present disclosure.

[0031] FIG. 4b is another diagram illustrating an example of a second user interaction according to some embodiments of the present disclosure.

[0032] FIG. 5 is a drawing illustrating an example of a screen component according to some embodiments of the present disclosure.

[0033] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention. Throughout the description of each drawing, similar reference numerals have been used to designate similar components.

[0034] Terms such as "first," "second," "A," and "B" may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the "second component," and similarly, the second component may also be referred to as the "first component." The term "and / or" includes a combination of a plurality of related items described herein or any of a plurality of related items described herein.

[0035] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0036] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0038] In the present disclosure, a computing device can transmit information about screen components to a server. Specifically, the computing device can access a pre-designated library in response to the execution of a pre-designated application. The pre-designated application may be an application pre-deployed on the computing device. The pre-designated application may be an application installed in the form of a library using a Software Development Kit (SDK). The computing device may receive a user interaction selecting an area within the screen of the computing device on which the pre-designated application is executed. The computing device may determine a first node corresponding to the user interaction. Here, a node may be understood as a component constituting a user interface. For example, assume that there is an application such as "Integrated Patent Document Writer." Components constituting the user interface of the "Integrated Patent Document Writer" may include activity classes, layouts, view groups, buttons, or lists. Such activity classes, layouts, view groups, buttons, or lists may be understood as nodes. When the first node is determined, the computing device may generate reporting information for displaying an object representing the first node in a pre-designated visualization manner. The computing device can transmit report information to a server or the like, thereby causing the server to display an object representing the first node in a pre-specified visualization manner. Hereinafter, a method for a computing device according to the present disclosure to support information collection regarding screen components will be described with reference to FIGS. 1 to 5.

[0039] FIG. 1 is a block diagram illustrating an example of a computing device according to some embodiments of the present disclosure.

[0040] Referring to FIG. 1, a computing device (100) may include a control unit (110), a storage unit (120), and a communication unit (130). However, the above-described components are not essential for implementing the computing device (100), and thus the computing device (100) may have more or fewer components than the components listed above.

[0041] The computing device (100) may include any type of computer system or computer device, such as, for example, a microprocessor, a mainframe computer, a digital processor, a portable device, or a device controller.

[0042] A computing device (100) may achieve desired system performance by utilizing a combination of typical computer hardware (e.g., devices that may include a computer processor, memory, storage, input devices and output devices, and other components of conventional computing devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause the computing device to function in a particular manner).

[0043] The control unit (110) can typically process the overall operation of the computing device (100). The control unit (110) can process signals, data, information, etc. input or output through components of the computing device (100) or run application programs stored in the storage unit (120), thereby providing or processing appropriate information or functions to the user.

[0044] The control unit (110) may be composed of one or more cores and may include a processor for data analysis, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0045] In the present disclosure, the control unit (110) can determine screen coordinates corresponding to an area in response to a first user interaction selecting an area within the screen of the computing device (100). The control unit (110) can determine UI (User Interface) coordinates within the user interface corresponding to the determined screen coordinates.

[0046] Specifically, the screen coordinates may be coordinates on the display panel of the computing device (100). The display panel of the computing device (100) may have a fixed size as a physical device. Therefore, the screen coordinates may also be determined within the fixed size. On the other hand, the UI coordinates within the user interface may be changed by app functions such as scrolling or split screens within the screen. Even if the screen has coordinate values ​​from 1 to 100, the user interface may have coordinate values ​​from 1 to 1,000. Accordingly, even if the user selects the coordinates of point 1,1 within the screen, the UI coordinates may not be 1,1. Therefore, the control unit (110) may determine the UI coordinates within the user interface corresponding to the screen coordinates determined using a function. Hereinafter, an example of a method by which the control unit (110) determines the UI coordinates will be described with reference to FIGS. 2 and 3.

[0047] The storage unit (120) may include memory and / or a permanent storage medium. The memory may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0048] The communication unit (130) may include one or more modules that enable communication between the computing device (100) and a communication system, between the computing device (100) and a server, between the computing device (100) and another computing device, or between the computing device (100) and a network.

[0049] Below, a specific example of how a computing device (100) supports collecting information about screen components is described.

[0050] FIG. 2 is a flowchart illustrating an example of a method by which a computing device generates reporting information according to some embodiments of the present disclosure.

[0051] Referring to FIG. 2, the control unit (110) of the computing device (100) may determine screen coordinates corresponding to an area in response to a first user interaction of selecting an area within the screen of the computing device (100) on which a pre-designated application is executed (S110). The pre-designated application may be an application pre-distributed to the computing device. The pre-designated application may be an application in which an SDK (Software Development Kit) is installed in the form of a library. The library may be a program compiled to perform only a specific partial function and existing in the form of machine language. The screen coordinates may be coordinates on the display panel of the computing device (100).

[0052] The control unit (110) can determine UI (User Interface) coordinates within the user interface corresponding to screen coordinates by using the hierarchical structure of the parent UI and child UI defined for the application and the movement function corresponding to each hierarchical structure (S120).

[0053] A parent UI can be a user interface located at the top of the hierarchy. For example, let's assume there's an application called "Integrated Patent Document Writer." The parent UI can be assumed to be an Activity class. In this case, the home screen or login screen of the "Integrated Patent Document Writer" application can be the parent UI. Alternatively, the parent UI can be assumed to be a DecorView class. In this case, the parent UI can be the top-level view of the activity. The child UI can be a UI located below the parent UI. For example, if the parent UI is an Activity class, the child UI can be a DecorView or a LinearLayout.

[0054] A translation function can be a function that determines UI coordinates corresponding to screen coordinates. The translation function can be calculated through matrix value operations.

[0055] For example, a user interface displayed on a screen may include a main area. The main area may also be understood as a root area. The main area may be an area where movement does not occur even when an input for moving the user interface is applied. In other words, the main area may be an area where coordinates do not change according to scrolling of the screen. The control unit (110) may determine a first matrix value related to the main area. The user interface displayed on the screen may include a scroll area. The scroll area may be an area where movement occurs according to an input for moving the user interface. In other words, the scroll area may be an area where coordinates change according to scrolling of the screen. The control unit (110) may determine a second matrix value related to the scroll area. The control unit (110) may determine a translation function based on the first matrix value and the second matrix value. In addition, the control unit (110) may determine UI coordinates based on the determined translation function and screen coordinates.

[0056] The control unit (110) can determine the first node corresponding to the UI coordinates (S130).

[0057] Nodes can be understood as components that constitute a user interface. For example, let's assume that there is an application called "Integrated Patent Document Writer." Components that constitute the user interface of the "Integrated Patent Document Writer" may include activity classes, layouts, view groups, buttons, or lists. Such activity classes, layouts, view groups, buttons, or lists can be understood as nodes. The control unit (110) can determine the first node corresponding to the UI coordinates.

[0058] In one embodiment, the first node may be the lowest node, hierarchically located. Furthermore, the movement function may be a function computed for the parent node of the first node. For convenience of explanation, reference may be made to FIG. 5.

[0059] FIG. 5 is a diagram illustrating an example of a screen component according to some embodiments of the present disclosure. The screen in FIG. 5 may be a screen displayed on a server that has received the report information described below.

[0060] Referring to FIG. 5, an object (210) representing a first node may be displayed. Referring to the object, it can be confirmed that the first node is the lowest node located at the lowest level hierarchically.

[0061] Specifically, the first node may be a node related to text. There may be no other nodes below the first node. The parent node of the first node may be an activity or activity class located at the top level. The control unit (110) can determine the first node corresponding to the UI coordinates by calculating a movement function for the activity.

[0062] Referring back to FIG. 2, the control unit (110) can generate report information for displaying an object representing a first node in a pre-designated visualization manner (S140). Furthermore, the control unit (110) can transmit the generated report information to a server or other computing device via the communication unit (130). Here, the pre-designated visualization manner may be, for example, a method of highlighting an object representing the first node. Furthermore, the pre-designated manner may include a method in which multiple nodes are expressed in a tree structure.

[0063] According to the above-described configuration, the computing device (100) can determine screen coordinates based on the first user interaction. The computing device (100) can determine UI coordinates corresponding to the screen coordinates. Since the screen coordinates are determined within a fixed size, whereas the UI coordinates are determined within a flexible size, the computing device (100) can determine the UI coordinates corresponding to the screen coordinates using a movement function. In addition, the computing device (100) can generate report information for displaying an object representing a first node corresponding to the determined UI coordinates in a pre-designated visualization manner. Accordingly, the user can conveniently confirm the first node corresponding to the screen coordinates using a server or another computing device that has received the report information.

[0064] Meanwhile, according to some embodiments of the present disclosure, the user interaction may include a second user interaction that moves the user interface. Specifically, the user interaction may include a second user interaction that moves the user interface displayed on the screen for each hierarchical structure. In this case, the computing device (100) may determine a movement function according to the second user interaction. In addition, the computing device may determine UI coordinates based on the determined movement function and screen coordinates. Hereinafter, an example of a method for the computing device (100) according to the present disclosure to determine the movement function according to the second user interaction will be described with reference to FIG. 3 .

[0065] FIG. 3 is a flowchart illustrating an example of a method for a computing device to determine a movement function according to a second user interaction according to some embodiments of the present disclosure.

[0066] Referring to FIG. 3, the control unit (110) of the computing device (100) can determine a first matrix value related to the main area included in the user interface (S210).

[0067] For example, a user interface displayed on a screen may include a main area. The main area may also be understood as a root area. The main area may be an area where movement does not occur even when an input for moving the user interface is applied. In other words, the main area may be an area where coordinates do not change according to the scrolling of the screen. The control unit (110) may determine a first matrix value related to the main area. For example, in the case of a mail application, a graphic object representing a user interface for implementing a delete function, a forwarding function, a reply function, etc. of the corresponding mail may be placed in the main area. In this case, the first matrix may be implemented as a matrix that subtracts the size of the main area to define a scroll area to which a movement function is applied from the entire screen area of ​​the corresponding user terminal. In other words, the first matrix may represent a 1 by 2 matrix determined according to the size of the area where the main area is located on the screen.

[0068] The control unit (110) can determine a second matrix value related to a scroll area included in the user interface (S220).

[0069] The user interface displayed on the screen may include a scroll area. The scroll area may be an area where movement occurs in response to an input that moves the user interface. In other words, the scroll area may be an area whose coordinates change as the screen is scrolled. The control unit (110) may determine a second matrix value determined based on the size of the area where the scroll area is located on the screen.

[0070] The control unit (110) can determine a third matrix value related to the second user interaction (S230).

[0071] The second user interaction may be an interaction that moves the user interface. The second user interaction may be an interaction that moves the screen on which the user interface is displayed up and down or left and right. The control unit (110) may determine a third matrix value related to the second user interaction. For example, the control unit (110) may determine the third matrix value using the starting point where the second user interaction occurred and the ending point where the second user interaction ended.

[0072] The control unit (110) can determine a movement function based on the first matrix value, the second matrix value, and the third matrix value (S240). In addition, when a first user interaction for selecting an area occurs, the control unit (110) can determine UI coordinates within the user interface using the screen coordinates corresponding to the area and the determined movement function.

[0073] In other words, if the control unit (110) determines that the displayed screen within the user interface has changed according to the second user interaction, it can determine a movement function so that the changed UI coordinates are reflected.

[0074] For example, a screen may have coordinate values ​​from 1 to 100. A user interface may have coordinate values ​​from 1 to 1,000. A screen before a second user interaction occurs may display a user interface corresponding to coordinate values ​​from 1 to 100. A screen after a second user interaction occurs may display a user interface corresponding to coordinate values ​​from 101 to 200. If it is determined that a second user interaction for moving the user interface has occurred before the first user interaction occurs, the control unit (110) may determine a movement function using the second user interaction. In addition, the control unit (110) determines UI coordinates by reflecting a predetermined movement function according to the occurrence of the first user interaction, thereby accurately determining a node corresponding to a point selected by the user.

[0075] FIG. 4A is a diagram illustrating an example of a second user interaction according to some embodiments of the present disclosure. FIG. 4B is another diagram illustrating an example of a second user interaction according to some embodiments of the present disclosure.

[0076] Referring to FIGS. 4A and 4B , a user interface may include a main area and a scroll area. A user may initiate a second user interaction to move the user interface within the scroll area. Alternatively, the user may initiate a second user interaction to move the user interface within the scroll area and the main area. Even if the second user interaction occurs, the user interface in the main area may not move or change. In other words, the coordinates of the main area may not change according to the scrolling of the screen.

[0077] The scroll area can change the user interface displayed on the screen according to the second interaction. Comparing FIGS. 4A and 4B, the user interface displayed in the scroll area can move upward (or downward). If the control unit (110) determines that the second user interaction exists, the control unit (110) can determine a movement function using the first matrix value related to the main area, the second matrix value related to the scroll area, and the third matrix value related to the second user interaction. In addition, if the first user interaction for selecting an area occurs, the control unit (110) can determine UI coordinates within the user interface based on the movement function and the screen coordinates corresponding to the first area.

[0078] FIG. 5 is a drawing illustrating an example of a screen component according to some embodiments of the present disclosure.

[0079] The screen in Fig. 5 may be a screen displayed on a server that received the report information.

[0080] Specifically, the control unit (110) of the computing device (100) can generate report information for displaying an object representing a first node corresponding to UI coordinates in a pre-designated visualization manner. The control unit (110) can transmit the generated report information to a server or another computing device via the communication unit (130). The server that receives the report information can display the object (210) representing the first node with the object (210) highlighted. Accordingly, the user can conveniently confirm the node selected according to the first user interaction.

[0081] Meanwhile, according to some embodiments of the present disclosure, the control unit (110) may receive a third user interaction to reselect an area. In this case, the control unit (110) may determine a second node corresponding to UI coordinates other than the first node.

[0082] For convenience of explanation, Fig. 4b may be referred to. Referring to Fig. 4b, it can be assumed that the user has performed a third user interaction to reselect an area. After selecting the UI 4 button through step S110, the user can reselect the UI 4 button. The UI 4 button is displayed on the screen output through the user interface, but in reality, the UI 4 button may be in a state where multiple other UIs are overlapped. For example, UI buttons related to text, other buttons, or layouts may exist below the UI button. Here, the below can be understood as the Z-axis. For example, if the horizontal axis is assumed to be the X-axis and the vertical axis is assumed to be the Y-axis on a screen such as a display panel, the Z-axis may be an axis generated toward the front and back of the screen. Such UI buttons may exist without any particular interaction or without being selected even if an end user, such as a customer using the application, selects the UI 4 button. However, a user using a computing device (100) according to the present disclosure also needs to collect information about buttons existing below the UI 4 button. Accordingly, when a third user interaction for reselecting an area occurs, the control unit (110) can determine a second node corresponding to UI coordinates other than the first node. Then, the control unit (110) can generate report information for displaying an object representing the second node in a pre-specified visualization manner. For example, referring again to FIG. 5, the control unit (110) can generate report information for displaying an object (220) representing the second node existing in the same layer as the first node in a pre-specified visualization manner.

[0083] According to one embodiment, the control unit (110) may determine a third node hierarchically higher than the first node based on a user interaction for reselecting an area that occurs after the occurrence of the third user interaction. In addition, the control unit (110) may generate report information for displaying an object (200) representing the third node in a pre-designated visualization manner. The control unit (110) may also generate report information for displaying an object representing a parent node of the first node in a pre-designated visualization manner based on a user interaction for reselecting an area that occurs after the occurrence of the third user interaction.

[0084] According to the above-described configuration, the computing device (100) can determine nodes that are not output through the user interface, based on UI coordinates. Furthermore, the control unit (110) can generate reporting information for displaying objects representing the determined nodes in a pre-designated visualization manner. Accordingly, the user can also collect information on components that are not displayed on the screen.

[0085] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for supporting information collection on a screen component performed by a computing device including at least one processor, A step of determining screen coordinates corresponding to an area within a screen of a computing device on which a pre-designated application is executed, based on a first user interaction selecting the area; A step of determining UI (User Interface) coordinates within a user interface corresponding to the screen coordinates by using a hierarchical structure of parent UI and child UI defined for the above application and a movement function corresponding to each of the hierarchical structures; A step of determining a first node corresponding to the above UI coordinates; and A step for generating report information for displaying an object representing the first node in a pre-specified visualization manner. A method to support gathering information about screen components that include:

2. In paragraph 1, A step of determining a movement function according to a second user interaction for moving the user interface displayed on the screen for each of the above hierarchical structures; Including more, The step of determining UI coordinates within the user interface corresponding to the above screen coordinates is: A step of determining the UI coordinates based on the above movement function and the screen coordinates. A method to support gathering information about screen components that include:

3. In paragraph 2, The step of determining the movement function according to the second user interaction is: A step of determining a first matrix value related to a main area included in the above user interface; A step of determining a second matrix value related to a scroll area included in the user interface; determining a third matrix value related to the second user interaction; and A step of determining the movement function based on the first matrix value, the second matrix value, and the third matrix value. A method to support gathering information about screen components that include:

4. In paragraph 2, The above first node is the lowest node located at the lowest level hierarchically, The above movement function is a function that is operated on the parent node of the first node. How to support gathering information about screen components.

5. In paragraph 1, A step of receiving a third user interaction for reselecting the above work area; A step of determining a second node corresponding to the above UI coordinates; and A step for generating report information for displaying an object representing the second node in a pre-specified visualization manner. A method to support gathering information about a screen component that includes more than one element.

Images