Methods, systems, and computer programs for automatically capturing user interface screenshots (Automatic capture of user interface screenshots for software product documentation)

Abstract

To solve the problem in which automatic capture software often results in too many screenshots being captured.SOLUTION: Embodiments of the invention are directed to automatically capturing user interface screenshots for use in documentation of a software product. Aspects include identifying a user interface window of the software product, and creating a degree-of-completion graph for the user interface window. Aspects also include capturing a plurality of screenshots of the user interface window during use of the software product, and calculating a degree-of-completion percentage for each of the screenshots. Aspects further include identifying a subset of the plurality of screenshots to be included in the software product documentation on the basis of, the degree-of-completion percentage.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention generally relates to capturing user interface screenshots, and more specifically to automatically capturing user interface screenshots for use in software product documentation.

Background Art

[0002] Software product documentation such as user guides often includes screenshots of the user interface (UI) of the software product. By including UI screenshots, users can better follow the instructions step by step and verify their actions by comparing the UI they are experiencing with the UI screenshots. In addition, by including UI screenshots in the product documentation, users can learn about the software product via UI captures without installing the product or accessing the product. Therefore, technical writers who prepare software documentation recognize that UI captures are an important element for user-friendly product documentation and thus add exemplary UI captures.

[0003] In many cases, manually capturing UI screenshots for product documentation (such as user guides) is extremely time-consuming. Currently available software allows users to automatically capture screenshots periodically, for example, every 2-5 seconds, or when a UI event occurs (e.g., a new active window appears or a UI control is activated). However, this type of automatic capture software often results in capturing an excessive number of screenshots, requiring technical writers to sort through the captured images to identify the desired ones. [Overview of the project] [Problems that the invention aims to solve]

[0004] Automatic screen capture software often results in capturing an excessive number of screenshots. [Means for solving the problem]

[0005] Embodiments of the present invention relate to a computer implementation method for automatically capturing user interface (UI) screenshots for use in software product documentation. Non-limiting examples of the computer implementation method include identifying the user interface window of the software product and creating a completion graph for the user interface window. The method also includes capturing multiple screenshots of the user interface window while using the software product and calculating a completion percentage for each of the multiple screenshots. The method further includes identifying a subset of the multiple screenshots to be included in the software product documentation based on the completion percentages.

[0006] Embodiments of the present invention relate to a system for automatically capturing user interface (UI) screenshots for use in software product documentation. Non-limiting examples of the system include a processor communicatively coupled to memory, which is operable to identify user interface windows of a software product and to create a completion graph for user interface windows. The processor is also operable to capture multiple screenshots of user interface windows while the software product is in use and to calculate a completion percentage for each of the multiple screenshots. The processor is further operable to identify a subset of the multiple screenshots to be included in the software product documentation based on the completion percentages.

[0007] Embodiments of the present invention relate to a computer program product that automatically captures user interface (UI) screenshots for use in software product documentation, the computer program product comprising a computer-readable storage medium in which program instructions are embodied. The program instructions are executable by a processor to cause the processor to perform a method. Non-limiting examples of the method include identifying the user interface window of the software product and creating a completion graph for the user interface window. The method also includes capturing multiple screenshots of the user interface window while the software product is in use and calculating a completion percentage for each of the multiple screenshots. The method further includes identifying a subset of the multiple screenshots to be included in the software product documentation based on the completion percentages.

[0008] Additional technical features and benefits are realized through the techniques of the present invention. Embodiments and aspects of the present invention are described in detail herein and are considered to be part of the subject matter claimed. For a better understanding, refer to the detailed description and drawings. [Brief explanation of the drawing]

[0009] Details of the exclusive rights described herein are specifically indicated and expressly asserted in the claims at the end of this specification. The aforementioned and other features and advantages of embodiments of the present invention will become apparent from the following detailed description when read in conjunction with the accompanying drawings.

[0010] [Figure 1] This figure shows a cloud computing environment according to one or more embodiments of the present invention.

[0011] [Figure 2] This figure shows an abstraction model layer according to one or more embodiments of the present invention.

[0012] [Figure 3] This is a block diagram of a computer system for use in implementing one or more embodiments of the present invention.

[0013] [Figure 4] This figure shows a system for automatically capturing user interface (UI) screenshots for use in software product documentation, according to an embodiment of the present invention.

[0014] [Figure 5A] This figure shows a completion rate graph according to one or more embodiments of the present invention.

[0015] [Figure 5B] This figure shows a compressed completion graph according to one or more embodiments of the present invention.

[0016] [Figure 6] This is a flowchart illustrating a method for automatically capturing user interface (UI) screenshots for use in software product documentation, according to one or more embodiments of the present invention.

[0017] [Figure 7] This is a flowchart illustrating a method for updating software product documentation by automatically capturing user interface (UI) screenshots, according to one or more embodiments of the present invention.

[0018] The drawings shown herein are illustrative. Many variations may exist to the drawings or the actions described herein without departing from the spirit of the invention. For example, actions may be performed in a different order, and actions may be added, deleted, or modified. Furthermore, the term “coupled” and its variations describe the existence of a communication path between two elements, and do not imply a direct connection between them without an intervening element / connection between them. All of these variations are considered part of this specification. [Modes for carrying out the invention]

[0019] Various embodiments of the present invention are described herein with reference to the associated drawings. Alternative embodiments of the present invention can be devised without departing from the scope of the present invention. In the following description and the drawings, various connection relationships and positional relationships (e.g., above, below, adjacent, etc.) between elements are described. These connection or positional relationships or both can be direct or indirect, and the present invention is not intended to be limited in this regard. Thus, the coupling of entities can refer to either direct or indirect coupling, and the positional relationship between entities can be a direct or indirect positional relationship. Moreover, the various tasks and process steps described herein can be incorporated into more comprehensive procedures or processes having additional steps or functions not detailed herein.

[0020] The following definitions and abbreviations are used in the interpretation of the claims and this specification. As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," "contains," or "containing," or any other variations thereof, are intended to encompass non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, and may include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.

[0021] In addition, the term "exemplary" is used herein to mean "functioning as an example, instance or illustration". Any embodiment or design described herein as "exemplary" is not necessarily construed as being more preferred or advantageous than other embodiments or designs. The terms "at least one" and "one or more" can be understood to include any integer greater than or equal to 1, i.e., 1, 2, 3, 4, etc. The term "plural" can be understood to include any integer greater than or equal to 2, i.e., 2, 3, 4, 5, etc. The term "connection" can include both indirect "connection" and direct "connection".

[0022] The terms "about", "substantially", "approximately" and their variants are intended to include the degree of error associated with the measurement of a particular quantity based on the equipment available at the time of filing. For example, "about" can include a range of ±8%, or 5%, or 2% of a given value.

[0023] For the sake of brevity, conventional techniques related to creating and using aspects of the present invention may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs for implementing the various technical features described herein are well known. Thus, for the sake of brevity, many details of conventional implementations are only briefly mentioned herein or are completely omitted without providing details of well-known systems or processes or both.

[0024] While this disclosure includes a detailed description of cloud computing, it should be understood that the implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment that is currently known or may be developed in the future.

[0025] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0026] The characteristics are as follows:

[0027] On-demand self-service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0028] Broad network access: This capability is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0029] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. While consumers generally have no control or knowledge of the exact location of the resources provided, there is location independence in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0030] Rapid resilience: This capability allows for rapid and elastic provisioning, sometimes automatically, enabling quick scaling out and rapid release and rapid scaling in. To consumers, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0031] Services measured: Cloud systems automatically control and optimize resource usage by leveraging metric capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, thereby providing transparency to both service providers and consumers.

[0032] Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources, where consumers can deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do control the operating system, storage, and deployed applications, and in some cases have limited control over selected networking components (e.g., host firewalls).

[0033] The deployment model is as follows:

[0034] Private Cloud: This cloud infrastructure operates solely for a specific organization. The private cloud may be managed by that organization or a third party and may reside on-premises or off-premises.

[0035] Community Cloud: This cloud infrastructure is shared by several organizations and supports a specific community that shares common interests (e.g., mission, security requirements, policies, and compliance considerations). The community cloud may be managed by those organizations or third parties and may reside on-premises or off-premises.

[0036] Public Cloud: This cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0037] Hybrid Cloud: This cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain independent entities but are joined together by standard or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0038] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0039] Referring here to Figure 1, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 comprises one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof. The nodes 10 may communicate with each other. The nodes 10 may be physically or virtually grouped within one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or a combination thereof, as described above in this specification (not shown). This enables the cloud computing environment 50 to provide infrastructure, platforms, or software, or a combination thereof, as a service that does not require cloud consumers to maintain resources on their local computing devices for that purpose. The types of computing devices 54A to N shown in Figure 1 are for illustrative purposes only, and it should be understood that the computing node 10 and the cloud computing environment 50 can communicate with any type of computerized device via any type of network, or a network addressable connection (e.g., using a web browser), or both.

[0040] Referring now to Figure 2, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 1) is shown. The components, layers, and functionalities shown in Figure 2 are intended to be illustrative only, and it should be understood in advance that embodiments of the present invention are not limited to them. As illustrated, the following layers and corresponding functionalities are provided:

[0041] The hardware and software layer 60 comprises hardware components and software components. Examples of hardware components include a mainframe 61, a RISC (Reduced Instruction Set Computer) architecture-based server 62, a server 63, a blade server 64, a storage device 65, and network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0042] The virtualization layer 70 provides an abstraction layer that may provide examples of virtual entities, such as virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0043] In one example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 82 provides cost tracking as resources are used within the cloud computing environment and charges or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management to ensure that required service levels are met. Service level agreement (SLA) planning and execution 85 provides pre-arrangements and procurement of cloud computing resources where future requirements are expected to conform to the SLA.

[0044] The workload layer 90 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analysis processing 94, transaction processing 95, and capturing user interface screenshots for product documentation 96.

[0045] Referring to Figure 3, one embodiment of a processing system 300 implementing the teachings herein is shown. In this embodiment, the system 300 has one or more central processing units (processors) 21a, 21b, 21c, etc. (collectively or generally referred to as processor 21). In one or more embodiments, each processor 21 may include a reduced instruction set computer (RISC) microprocessor. The processor 21 is coupled to system memory 34 and various other components via a system bus 33. A read-only memory (ROM) 22 is coupled to the system bus 33 and may include a basic input / output system (BIOS), which controls certain basic functions of the system 300.

[0046] Figure 3 further shows an input / output (I / O) adapter 27 and a network adapter 26 coupled to the system bus 33. The I / O adapter 27 may be a Small Computer System Interface (SCSI) adapter that communicates with a hard disk 23 or a tape storage drive 25 or both, or any other similar component. The I / O adapter 27, the hard disk 23, and the tape storage device 25 are collectively referred to herein as mass storage 24. The operating system 40 running on the processing system 300 may be stored in the mass storage 24. The network adapter 26 interconnects the bus 33 to an external network 36, enabling the data processing system 300 to communicate with other such systems. A display adapter 32 connects a screen (e.g., a display monitor) 35 to the system bus 33, and the display adapter 32 may include a graphics adapter to improve the performance of graphics-intensive applications and video controllers. In one embodiment, adapters 27, 26, and 32 may be connected to one or more I / O buses connected to the system bus 33 via an intermediate bus bridge (not shown). I / O buses suitable for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as the Peripheral Component Interconnect (PCI). Additional input / output devices are shown connected to the system bus 33 via a user interface adapter 28 and a display adapter 32. The keyboard 29, mouse 30, and speaker 31 are all interconnected to the bus 33 via the user interface adapter 28, which may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit.

[0047] In an exemplary embodiment, the processing system 300 includes a graphics processing unit 41. The graphics processing unit 41 is a special electronic circuit designed to manipulate and modify memory to accelerate the creation of images in a frame buffer intended for output to a display. Generally, the graphics processing unit 41 is highly efficient in computer graphics manipulation and image processing and has a highly parallel structure that makes it more effective than a general-purpose CPU in the case of algorithms where the processing of large blocks of data is performed in parallel.

[0048] Therefore, as configured in Figure 3, the system 300 includes processing capabilities in the form of a processor 21, storage capabilities including system memory 34 and mass storage 24, input means such as a keyboard 29 and a mouse 30, and output capabilities including a speaker 31 and a display 35. In one embodiment, portions of the system memory 34 and mass storage 24 collectively store the operating system to enable the functions of the various components shown in Figure 3 to cooperate.

[0049] Now, considering an overview of the technology more specifically related to aspects of the present invention, a method, system, and computer program product are provided for automatically capturing user interface (UI) screenshots for use in software product documentation. In exemplary embodiments, the method, system, and computer program product are configured to automatically capture and identify screenshots for inclusion in product documentation based on a calculated level of completeness of the UI window shown in the screenshot. In exemplary embodiments, the level of completeness of the UI window is calculated based on a level of completeness graph, which is an undirected weighted topological graph created based on the characteristics of the UI window. In exemplary embodiments, the level of completeness graph is constructed in the backend to obtain the status of the UI controls. As used herein, the characteristics of a UI window refer to the status of whether or not the UI controls of the UI window have user input. Once created, the level of completeness graph is analyzed using a Graph Convolutional Network (GCN) to obtain the level of completeness and the level of completeness percentage of the UI window shown in the screenshot. In exemplary embodiments, the method, system, and computer program product are also configured to use machine learning to obtain a historical set of completion levels and completion percentages for all UI window categories from available systems and their associated user guides.

[0050] Referring here to a more detailed description of the embodiments of the present invention, Figure 4 shows a computing system 400 that automatically captures user interface (UI) screenshots for use in software product documentation, according to an embodiment of the present invention. The system 400 comprises a software product 402, a user interface 404, a UI analysis engine 406, product documentation 408, and optionally, historical product documentation 410. The computing system 400 can be implemented on the processing system 300 shown in Figure 3. In addition, the cloud computing system 50 can be wired or wireless electronic communication with one or all of the elements of the computing system 400. The cloud 50 can supplement, support, or replace some or all of the functions of the elements of the computing system 400. In addition, some or all of the functions of the elements of the computing system 400 can be implemented as nodes 10 of the cloud 50 (shown in Figures 1 and 2). Cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to imply any limitation on the scope of use or functionality of the embodiments of the present invention described herein.

[0051] As discussed herein, completion in a UI window is a measure of the completeness of a user action within the UI window (e.g., clicking a button, entering a value, or selecting a date). In exemplary embodiments, each completion level is independent of the sequence of execution stages and relates to which action was performed, which is closely related to the input values ​​(or states) of the UI controls within the window. For example, the input status of a typical UI control in a window, such as a user ID field, results in two different completion levels: one where a user ID has been entered, and the other where a user ID has not been entered.

[0052] In an exemplary embodiment, a completion graph is used to calculate the completion of user actions in a UI window. Figure 5A shows a schematic diagram of a completion 500 according to one embodiment. The completion graph 500 is an undirected weighted topological graph constructed based on the characteristics of the UI window. In an exemplary embodiment, the completion graph 500 is constructed starting from a window node 502, which has a node index equal to zero, and is used to represent the UI window. Then, each UI control in the UI window is assigned a unique index starting from 1. Each UI control corresponds to two nodes in the completion graph 500. One of the nodes is a no-input node 504, and the other is an input node 506. In an exemplary embodiment, the no-input node 504 and the input node 506 are placed in corresponding category groups 508, 510, which are used to group input nodes or no-input nodes of the same control type and the same value type. In one embodiment, N is the total number of UI control categories in the software world, not the number of UI control categories in a particular UI window. In the exemplary embodiment, if a UI element has an input value, a connecting edge exists between its input node 506 and the window node 502. If a UI element does not have an input value, a connecting edge exists between its no-input node 504 and the window node 502. In the exemplary embodiment, the weight of the connecting edge is always set to 1.

[0053] In an exemplary embodiment, once a completion graph 500 is created, it is compressed to create a compressed completion graph 550. Compressing the completion graph 500 simplifies its structure and reduces the number of calculations required to analyze it. In an exemplary embodiment, no-input nodes 504 from each no-input category group 508 are compressed into a virtual no-input node 514, and the connection weight between the virtual no-input node 514 and the window node 502 is the sum of the connection edges between the no-input nodes 504 from each no-input category group 508 and the window node 502. If no-input nodes 504 do not exist in the no-input category group 508, then there are no connection edges between the virtual no-input node 514 and the window node 502. Similarly, input nodes 506 from each input category group 510 are compressed into a virtual input node 516, and the connection weight between the virtual input node 516 and the window node 502 is the sum of the connection edges between the input nodes 506 from each input category group 510 and the window node 502. If no input node 506 exists in the input category group 510, then no connection edge exists between the virtual input node 516 and the window node 502.

[0054] In an exemplary embodiment, after a completion graph for a given UI window is created and compressed, the completion graph is analyzed to identify the completion level, completion percentage, or both of the screenshots of the UI window. In the exemplary embodiment, to calculate the completion level, the compressed completion graph is fed as input to a three-layer graph convolutional network (GCN). After the features of the compressed completion graph are extracted, the output of the GCN includes node embeddings of the window nodes, which are considered to be the completion level values.

[0055] In an exemplary embodiment, node embedding of nodes in a topological graph is represented by the following single-layer GCN expression.

number

number

number

number

number

[0056] In an exemplary embodiment, the completion percentage is calculated as Degree_of_completion_percentage=2-cos(window_node_embedding_1,window_node_embedding_2)) / (2-cos(window_node_embedding_3,window_node_embedding_2))*100, where window_node_embedding_1 is the node embedding of the window node in the current completion graph, window_node_embedding_2 is the node embedding of the window node in the completion graph where all non-input nodes belonging to all UI controls are connected to the window node, and window_node_embedding_3 is the node embedding of the window node in the completion graph where all input nodes belonging to all UI controls are connected to the window node.

[0057] Figure 6 shows a flowchart of a method 600 for automatically capturing user interface (UI) screenshots for use in software product documentation, according to one or more embodiments of the present invention. Method 600 comprises identifying the user interface window of the software product, as shown in block 602. Next, as shown in block 604, Method 600 comprises creating a completion graph for the user interface window. In an exemplary embodiment, the completion graph is an undirected weighted graph created by identifying whether a UI control has input and grouping the UI controls of the user interface window. Method 600 also comprises capturing multiple screenshots of the user interface window while the software product is in use, as shown in block 606. In one embodiment, capturing multiple screenshots of the user interface window while the software product is in use comprises capturing screenshots periodically while the user interface window is open. In another embodiment, capturing multiple screenshots of the user interface window while the software product is in use comprises capturing a screenshot whenever a user interface event occurs for the user interface window.

[0058] Next, as shown in block 608, method 600 comprises calculating a completion percentage for each of a plurality of screenshots. In an exemplary embodiment, calculating a completion percentage for each of a plurality of screenshots comprises analyzing a completion graph corresponding to each of the plurality of screenshots. In one embodiment, analyzing the completion graph comprises inputting the completion graph into a three-layer graph convolutional network, where the completion percentage is calculated via window_node_embedding_1, window_node_embedding_2, and window_node_embedding_3 of the corresponding completion graph. Method 600 further comprises identifying a subset of the plurality of screenshots to be included in the documentation based on the completion percentage, as shown in block 610. Also in an exemplary embodiment, the method comprises automatically inserting a subset of the plurality of screenshots into the software product documentation.

[0059] In one embodiment, the method comprises obtaining documentation for an older version of a software product and identifying screenshots included within the documentation. Once the screenshots are identified, each screenshot is categorized into a user interface window category. Next, if a newer version of the software product is being used, a user interface window category is determined for the new user interface window. Then, a completion percentage is calculated for each identified screenshot that belongs to the same user interface window category as the new user interface window. The identified screenshots in the product documentation can then be replaced with screenshots of the new user interface window that have the closest completion percentage to the identified screenshots.

[0060] In an exemplary embodiment, UI windows in software can be categorized into a limited number of UI window categories, where each category of UI window functions for the same purpose, for example, to complete the same business task, and the required UI behavior and control values ​​in those windows should be similar to achieve that purpose. Therefore, by analyzing available historical product documentation containing screenshots of all UI window categories, it is possible to determine which screenshots, at what level of completion, an experienced technical writer would conventionally choose to capture and include in the product documentation for each UI window category. As a result, a machine learning model can be trained and used to obtain a set of screenshots for all UI window categories. Thus, by analyzing historical product documentation to obtain a historical set of completion levels and completion percentages for all UI window categories, the process of automatically identifying and classifying screenshots for new versions of user guides can be performed much more efficiently.

[0061] Referring here to Figure 7, a flowchart of a method 700 for updating software product documentation by automatically capturing user interface (UI) screenshots, according to one or more embodiments of the present invention, is shown. Method 700 comprises obtaining software product documentation for an older version of the software product, as shown in block 702. Next, as shown in block 704, Method 700 comprises identifying each screenshot in the software product documentation, determining a UI window category for each screenshot, and calculating both the completion percentage and completion for each screenshot. Method 700 also comprises running a new version of the software product and capturing candidate screenshots for each UI behavior, as shown in block 706. In an exemplary embodiment, a candidate screenshot is automatically obtained if the active UI window has a completion percentage that is within a predetermined range of the identified current target completion percentage, for example, within 2%. The identified current target completion percentage is obtained from screenshots of the same UI category in the software product documentation for an older version of the software product.

[0062] Next, as shown in block 708, method 700 comprises calculating both a completion percentage and a completion percentage for each candidate screenshot and determining a UI window category for each candidate screenshot. Method 700 further comprises calculating a cosine similarity between the completion percentage of each candidate screenshot and the completion percentage of an identified screenshot having the same UI window category, as shown in block 710. Method 700 is completed in block 712 by creating new software product documentation by replacing the identified screenshot with a candidate screenshot of the same UI window category having the highest cosine similarity. In an exemplary embodiment, the cosine similarity is calculated by the completion percentage. In another embodiment, if cosine similarity is not used, the identified screenshot is replaced with a candidate screenshot of the same UI window category having the highest completion percentage.

[0063] Additional processes may be included. It should be understood that the processes shown in Figures 6 and 7 are illustrative, and that other processes may be added, or existing processes may be deleted, modified, or rearranged without departing from the scope and intent of this disclosure.

[0064] The present invention may be a system, method, or computer program product, or a combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or a set of mediums) having computer-readable program instructions that cause a processor to execute an aspect of the present invention.

[0065] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of those described above. A non-exhaustive list of more specific examples of computer-readable storage media includes, namely, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures recording instructions, and any suitable combination of those described above. When used herein, computer-readable storage media should not be interpreted as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or transient signals such as electrical signals transmitted through wires.

[0066] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers such computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0067] The computer-readable program instructions that perform the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, the one or more programming languages ​​including object-oriented programming languages ​​such as Smalltalk®, C++, etc., and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially executed on the user's computer as a standalone software package, partially executed on the user's computer and partially executed on a remote computer, or fully executed on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), and the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by executing computer-readable program instructions using state information of computer-readable program instructions in order to perform an aspect of the present invention.

[0068] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0069] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a dedicated computer, or another programmable data processing device to generate a machine, thereby creating means for implementing functions / operations specified in one or more blocks of a flowchart or block diagram, or both, through the instructions executed via the processor of the computer or other programmable data processing device. Furthermore, these computer-readable program instructions may be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, or another device, or a combination thereof, to function in a specific manner, thereby including a product containing instructions that implement modes of functions / operations specified in one or more blocks of a flowchart or block diagram, or a combination thereof.

[0070] Furthermore, computer-readable program instructions may be loaded into a computer, other programmable data processing device, or other device to execute a series of operational steps on the computer, other programmable device, or other device, thereby generating a computer implementation process in which the instructions executed on the computer, other programmable device, or other device implement functions / operations specified by one or more blocks of a flowchart or block diagram, or a combination thereof.

[0071] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions that implement a specified logical function. In some alternative implementations, the functions described in a block may be performed in an order different from the order shown in the drawings. For example, two blocks shown consecutively may actually be executed substantially simultaneously, and blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, or both, and any combination of blocks in a block diagram or flowchart, or both, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or a combination of dedicated hardware and computer instructions.

[0072] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the embodiments disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the embodiments described. The terminology used herein has been selected to best describe the principles of the embodiments, the practical applications of the technology found in the market or technical improvements thereto, or to enable other persons skilled in the art to understand the embodiments described herein.

Claims

1. A method for automatically capturing user interface screenshots for use in the documentation of software products, The user interface window identification step of the aforementioned software product, A step of creating a first completion graph for the user interface window, wherein the first completion graph includes two nodes representing each user interface control of the user interface window, The steps include capturing multiple screenshots of the user interface window while using the software product, A step of calculating a completion percentage for each of the plurality of screenshots, wherein the completion percentage for each of the plurality of screenshots is calculated by inputting the first completion graph into a three-layer graph convolutional network, and the completion percentage calculated for each of the plurality of screenshots is based on an embedding representing the current completion graph, a second completion graph having each no-input node connected to the window node, and a third completion graph having each input node belonging to all user interface controls connected to the window node, A step of identifying a subset of the plurality of screenshots to be included in the software product documentation based on the completion percentage calculated for each of the plurality of screenshots. A method that includes [a certain feature].

2. The method according to claim 1, wherein the completion graph is an undirected weighted graph created by identifying and grouping the user interface elements of the user interface window.

3. A method for automatically capturing user interface screenshots for use in the documentation of a software product, The user interface window identification step of the aforementioned software product, The steps include creating a completion graph for the aforementioned user interface window, The steps include capturing multiple screenshots of the user interface window while using the software product, The steps include calculating the completion percentage for each of the aforementioned multiple screenshots, The steps include: identifying a subset of the multiple screenshots to be included in the software product documentation based on the completion percentage; Equipped with, The completion graph is an undirected weighted graph created by identifying and grouping the user interface elements of the user interface window, according to the method.

4. The method according to claim 1, wherein the step of calculating the completion percentage for each of the plurality of screenshots includes the step of analyzing the completion graph corresponding to each of the plurality of screenshots.

5. The method according to claim 4, wherein the step of analyzing the completion graph includes the step of inputting the completion graph into a three-layer graph convolutional network, and the completion percentage is calculated via window_node_embedding_1, window_node_embedding_2, and window_node_embedding_3 of the corresponding completion graph.

6. A method for automatically capturing user interface screenshots for use in the documentation of a software product, The user interface window identification step of the aforementioned software product, The steps include creating a completion graph for the aforementioned user interface window, The steps include capturing multiple screenshots of the user interface window while using the software product, The steps include calculating the completion percentage for each of the aforementioned multiple screenshots, The steps include: identifying a subset of the multiple screenshots to be included in the software product documentation based on the completion percentage; Equipped with, The step of calculating the completion percentage for each of the plurality of screenshots includes the step of analyzing the completion graph corresponding to each of the plurality of screenshots. A method comprising the step of analyzing the completion graph, the step of inputting the completion graph into a three-layer graph convolutional network, wherein the completion percentage is calculated via window_node_embedding_1, window_node_embedding_2, and window_node_embedding_3 of the corresponding completion graph.

7. The method according to claim 1, wherein the step of capturing the plurality of screenshots of the user interface window during the use of the software product comprises the step of periodically capturing screenshots while the user interface window is open.

8. The method according to claim 1, wherein the step of capturing the plurality of screenshots of the user interface window during the use of the software product comprises the step of capturing a screenshot each time a user interface event occurs with respect to the user interface window.

9. The steps include obtaining documentation for older versions of the software product and identifying screenshots to be included in the documentation, The steps include classifying each of the identified screenshots into a user interface window category, The steps include determining the user interface window category for the aforementioned user interface window, For each of the identified screenshots belonging to the same user interface window category as the aforementioned user interface window, the step of calculating the completion percentage is as follows: The method according to any one of claims 1 to 8, further comprising:

10. The method according to claim 9, wherein the subset of the plurality of screenshots included in the documentation comprises the plurality of screenshots having the highest cosine similarity for each of the identified screenshots.

11. A system comprising a processor that is communicatively coupled to memory, The aforementioned processor, Identifying the user interface window of a software product, Creating a first completion graph for the user interface window, wherein the first completion graph includes two nodes representing each user interface control of the user interface window. Capturing multiple screenshots of the user interface window while using the software product, The method involves calculating a completion percentage for each of the plurality of screenshots, wherein the completion percentage for each of the plurality of screenshots is calculated by inputting the first completion graph into a three-layer graph convolutional network, and the completion percentage calculated for each of the plurality of screenshots is based on embeddings representing the current completion graph, a second completion graph having each no-input node connected to the window node, and a third completion graph having each input node belonging to all user interface controls connected to the window node. Identifying a subset of the multiple screenshots to be included in the software product documentation based on the completion percentage calculated for each of the multiple screenshots. A system configured to perform the following actions.

12. The system according to claim 11, wherein the completion graph is an undirected weighted graph created by identifying and grouping the user interface elements of the user interface window.

13. A system comprising a processor communicatively coupled to memory, The aforementioned processor, Identifying the user interface window of a software product, To create a completion graph for the aforementioned user interface window, Capturing multiple screenshots of the user interface window while using the software product, Calculate the completion percentage for each of the aforementioned multiple screenshots, Identifying a subset of the multiple screenshots to be included in the software product documentation based on the aforementioned completion percentage. It is configured to do the following: The completion graph is an undirected weighted graph created by identifying and grouping the user interface elements of the user interface window in the system.

14. The system according to claim 11, wherein calculating the completion percentage for each of the plurality of screenshots comprises analyzing the completion graph corresponding to each of the plurality of screenshots.

15. The system according to claim 14, wherein analyzing the completion graph comprises inputting the completion graph into a three-layer graph convolutional network, and the completion percentage is calculated via window_node_embedding_1, window_node_embedding_2, and window_node_embedding_3 of the corresponding completion graph.

16. A system comprising a processor communicatively coupled to memory, The aforementioned processor, Identifying the user interface window of a software product, To create a completion graph for the aforementioned user interface window, Capturing multiple screenshots of the user interface window while using the software product, Calculate the completion percentage for each of the aforementioned multiple screenshots, Identifying a subset of the multiple screenshots to be included in the software product documentation based on the aforementioned completion percentage. It is configured to do the following: Calculating the completion percentage for each of the aforementioned multiple screenshots involves analyzing the completion graph corresponding to each of the aforementioned multiple screenshots. The system comprises analyzing the completion graph by inputting the completion graph into a three-layer graph convolutional network, wherein the completion percentage is calculated via window_node_embedding_1, window_node_embedding_2, and window_node_embedding_3 of the corresponding completion graph.

17. The system according to claim 11, wherein capturing the plurality of screenshots of the user interface window during the use of the software product comprises periodically capturing screenshots while the user interface window is open.

18. The system according to claim 11, wherein capturing the plurality of screenshots of the user interface window during the use of the software product comprises capturing a screenshot each time a user interface event occurs with respect to the user interface window.

19. The aforementioned processor, Obtain documentation for older versions of the aforementioned software product, and identify screenshots included within the documentation. Each of the identified screenshots is classified into a user interface window category, Determining the user interface window category for the aforementioned user interface window, For each of the identified screenshots belonging to the same user interface window category as the aforementioned user interface window, calculate the completion percentage. The system according to any one of claims 11 to 18, further configured to perform the following:

20. A computer program for automatically capturing user interface screenshots for use in software product documentation, wherein the processor... Procedure for identifying the user interface window of a software product, A procedure for creating a first completion graph for the user interface window, wherein the first completion graph includes two nodes representing each user interface control of the user interface window; A procedure for capturing multiple screenshots of the user interface window while using the software product, A procedure for calculating a completion percentage for each of the plurality of screenshots, wherein the completion percentage for each of the plurality of screenshots is calculated by inputting the first completion graph into a three-layer graph convolutional network, and the completion percentage calculated for each of the plurality of screenshots is based on an embedding representing the current completion graph, a second completion graph having each no-input node connected to the window node, and a third completion graph having each input node belonging to all user interface controls connected to the window node, A procedure for identifying a subset of the aforementioned screenshots to be included in the software product documentation based on the completion percentage calculated for the aforementioned screenshots, and A computer program that executes something.

21. The computer program according to claim 20, wherein the completion graph is an undirected weighted graph created by identifying and grouping the user interface elements of the user interface window.

22. A computer program for automatically capturing user interface screenshots for use in software product documentation, comprising a processor, Procedure for identifying the user interface window of a software product, The procedure for creating a completion graph for the aforementioned user interface window, A procedure for capturing multiple screenshots of the user interface window while using the software product, A procedure for calculating the completion percentage for each of the aforementioned multiple screenshots, A procedure for identifying a subset of the multiple screenshots to be included in the software product documentation based on the completion percentage, and Make it run, The aforementioned completion graph is an undirected weighted graph created by identifying and grouping the user interface elements of the user interface window. Computer program.

23. The computer program according to claim 20, wherein the procedure for calculating the completion percentage for each of the plurality of screenshots comprises the procedure for analyzing the completion graph corresponding to each of the plurality of screenshots.

24. The computer program according to claim 20, wherein the procedure for capturing the plurality of screenshots of the user interface window during the use of the software product comprises the procedure for periodically capturing screenshots while the user interface window is open.

25. The aforementioned processor, A procedure for obtaining documentation for older versions of the aforementioned software product and identifying screenshots included within the documentation, A procedure for classifying each of the identified screenshots into a user interface window category, A procedure for determining the user interface window category for the aforementioned user interface window, A procedure for calculating the completion percentage for each of the identified screenshots belonging to the same user interface window category as the aforementioned user interface window, and A computer program according to any one of claims 20 to 24, which further performs the following:

Images