Processing method, device and storage medium of application program
By acquiring screen recordings from the application's testing phase, extracting keyframes, and constructing a functional knowledge graph, the problem of the application lacking user manuals was solved, enabling efficient functional Q&A and automated testing support.
Patent Information
- Application Number
- CN202411885591.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2026-06-23
AI Technical Summary
The lack of effective solutions for building application functional knowledge graphs in existing technologies has resulted in a lack of support for application scenarios such as functional question answering and automated testing.
By acquiring screen recordings of the application during the testing phase, keyframes are extracted to identify the interface, components, and their relationships, a knowledge graph of the application's functions is constructed, and the access paths of target interfaces and components are determined.
It enables the rapid and convenient construction of application function knowledge graphs, reducing costs, improving efficiency, and supporting application scenarios such as functional Q&A and automated testing.
Smart Images

Figure CN122264048A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer and network communication technology, and in particular to a method, apparatus and storage medium for processing an application. Background Technology
[0002] Knowledge graphs are essentially semantic networks that reveal the relationships between entities. If there is a relationship between two nodes, they will be connected by an undirected edge. These nodes are called entities, and the edge between them is called a relationship.
[0003] Traditional software usually comes with a user manual, but terminal applications (Apps) generally do not. Therefore, an application function knowledge graph can be built for the application as a user manual. However, there is no publicly available scheme for building application function knowledge graphs in the existing technology. Summary of the Invention
[0004] This disclosure provides an application processing method, device, and storage medium to quickly and conveniently construct an application function knowledge graph.
[0005] In a first aspect, embodiments of this disclosure provide a method for processing an application, comprising:
[0006] Acquire screen recordings of the application during its testing phase and extract keyframes from the screen recordings;
[0007] The keyframes are used to determine the interfaces, components, and the relationships between components and interfaces in the application.
[0008] Based on the interfaces, components, and relationships between components and interfaces in the application, construct a functional knowledge graph of the application.
[0009] The opening path of at least one of the target interface and target component is determined based on the application function knowledge graph, wherein the opening path is the path from the initial interface or the initial target component to at least one of the target interface and target component.
[0010] Secondly, embodiments of this disclosure provide an application processing device, including:
[0011] The extraction unit is used to acquire screen recording videos during the application's testing phase and extract keyframes from the screen recording videos.
[0012] The determining unit is used to determine the interface, components, and the relationship between components and the interface in the application based on the keyframe;
[0013] The construction unit is used to construct a functional knowledge graph of the application based on the interface, components, and the relationship between components and interface in the application.
[0014] An application unit is configured to determine, based on the application function knowledge graph, an opening path for at least one of a target interface and a target component, wherein the opening path is a path from an initial interface or an initial target component to at least one of the target interface and the target component.
[0015] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;
[0016] The memory stores computer-executed instructions;
[0017] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the processing method of the application described in the first aspect and various possible designs of the first aspect.
[0018] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the application processing method described in the first aspect and various possible designs of the first aspect.
[0019] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the processing method of the application described in the first aspect and various possible designs of the first aspect.
[0020] The application processing method, device, and storage medium provided in this disclosure involve acquiring screen recording videos of the application during its testing phase and extracting keyframes from the recording videos; determining interfaces, components, and relationships between components and interfaces within the application based on the keyframes; constructing an application functional knowledge graph based on the interfaces, components, and relationships between components and interfaces; and determining an access path for at least one of a target interface and a target component based on the application functional knowledge graph. This access path is the path from an initial interface or an initial target component to at least one of the target interface and target component. This disclosure allows for the rapid and convenient construction of an application functional knowledge graph based on screen recording videos of the application during its testing phase. It is low-cost, highly efficient, and can support application scenarios such as functional Q&A or automated testing. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 A schematic diagram illustrating a scenario of an application processing method provided in an embodiment of this disclosure;
[0023] Figure 2 A schematic flowchart of an application processing method provided in an embodiment of this disclosure;
[0024] Figure 3 A schematic flowchart of an application processing method provided in another embodiment of this disclosure;
[0025] Figure 4 A structural block diagram of an application processing device provided in an embodiment of this disclosure;
[0026] Figure 5 A schematic diagram of the hardware structure of an application processing device provided in an embodiment of this disclosure. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0028] Knowledge graphs are essentially semantic networks that reveal the relationships between entities. If there is a relationship between two nodes, they will be connected by an undirected edge. These nodes are called entities, and the edge between them is called a relationship.
[0029] Traditional software typically includes a user manual, while terminal applications (Apps) generally do not. Therefore, this disclosure proposes constructing an application function knowledge graph to represent the relationships between various functions within the application, serving as a user manual for question-and-answer services or automation services. However, no publicly available scheme for constructing an application function knowledge graph exists in the prior art.
[0030] To address the aforementioned technical problems, this disclosure provides an application processing method. This method involves acquiring screen recordings of the application during its testing phase and extracting keyframes from the recordings. Based on the keyframes, the method determines the interfaces, components, and relationships between components and interfaces within the application. Then, based on these keyframes, it constructs a functional knowledge graph for the application. Finally, it determines an access path for at least one of a target interface and a target component, where the access path is the path from an initial interface or initial target component to at least one of the target interface and target component. This disclosure allows for the rapid and convenient construction of an application functional knowledge graph based on screen recordings of the application during its testing phase. It is cost-effective and efficient, and can support application scenarios such as functional Q&A or automated testing.
[0031] The application scenarios of the processing method of the application in this disclosure are as follows: Figure 1 As shown, when users (such as developers or operations personnel) need to build an application functional knowledge graph, they can record screen recordings of the application's operation during the testing phase. For example, they can use each function of the application one by one while recording the screen. Furthermore, keyframes are extracted from the screen recordings. Based on the keyframes, the interfaces, components, and relationships between components and interfaces in the application are determined. Specifically, this may include the inclusion relationship between any interface and the component operated on within that interface, and the jump relationship between the component operated on within any interface and the next adjacent interface of that interface. Based on the interfaces, components, and relationships between components and interfaces in the application, an application functional knowledge graph is constructed.
[0032] After constructing the application function knowledge graph, it can be applied in various ways. For example, the application function knowledge graph can be used as an application user manual, or it can be used for function Q&A, that is, to query the application's functions and provide the opening path. It can also be used for automated testing of the application, or other applications.
[0033] The processing method of the application disclosed herein will be described in detail below with reference to specific embodiments.
[0034] refer to Figure 2 , Figure 2 This is a schematic flowchart illustrating an application processing method according to an embodiment of the present disclosure. The method of this embodiment can be applied to any electronic device such as a terminal device or server. The processing method of this application includes:
[0035] S201. Obtain the screen recording video during the application's operation in the testing phase, and extract key frames from the screen recording video.
[0036] In this embodiment, screen recordings of the application during its testing phase can be obtained. Optionally, during the testing phase, a screen recording program can be started on an electronic device with the application installed, and the application can continue to be used, including using at least one component in at least one interface of the application. While using any component, the application can jump from one interface to another, or it may not jump at all. Optionally, screen recordings of the application during its testing phase can also be obtained through any other means, such as being transmitted from another device, or downloaded or read from any storage device, etc. There are no limitations here. The screen recordings can be obtained by developers or operations personnel using or testing various functions of the application to build an application functional knowledge graph, and may not involve user data.
[0037] Furthermore, keyframes can be extracted from the screen recording video. Keyframes are usually frames in which key actions of characters or moving objects are performed. In this embodiment, keyframes can specifically be video frames in the screen recording video that are changing, including but not limited to changes in the interface of the application (such as interface jumps, interface swipes, etc.), user operations on components in the interface (such as operation point markers that can be displayed in the screen recording video), changes in components, etc.
[0038] S202. Determine the interface, components, and the relationship between components and interface in the application based on the keyframes.
[0039] In this embodiment, keyframes can be identified to determine the interfaces opened, the components operated on, and the pages navigated to after operating on the components in the screen recording video. This allows for the determination of interfaces, components, and the relationships between components and interfaces within the application. Identifying interfaces, components, and the relationships between components and interfaces from keyframes can be done in any feasible manner, such as using a pre-trained machine learning model. Specifically, the relationships between components and interfaces can include the inclusion relationship between any interface and the operated component within that interface, and the navigation relationship between the operated component within any interface and the next adjacent interface.
[0040] S203. Construct a functional knowledge graph of the application based on the interfaces, components, and the relationships between components and interfaces in the application.
[0041] In this embodiment, after determining the interfaces, components, and relationships between components and interfaces in the application, interfaces and components can be treated as entities in the application's functional knowledge graph. Edges connecting interface entities and component entities are constructed based on the relationships between components and interfaces. Specifically, based on the inclusion relationship between any interface and the manipulated component within that interface, an edge pointing from that interface to the manipulated component within that interface can be constructed. Similarly, based on the jump relationship between the manipulated component within any interface and the next adjacent interface of that interface, an edge pointing from the manipulated component to the next adjacent interface of that interface can be constructed. For example, if interface 1 includes component 1, and component 1 can be manipulated to jump to interface 2, the following can be constructed: (interface 1) → [component inclusion] → (component 1) → [operation jump] → (interface 2). By connecting all interface entities and component entities based on the relationships between components and interfaces, a complete application functional knowledge graph can be obtained.
[0042] S204. Determine the opening path of at least one of the target interface and target component based on the application function knowledge graph, wherein the opening path is the path from the initial interface or the initial target component to at least one of the target interface and target component.
[0043] In this embodiment, after constructing the application function knowledge graph, the application function knowledge graph can also be applied. The opening path of at least one of the target interface and target component can be determined based on the application function knowledge graph. That is, the path from the initial interface or the initial target component to at least one of the target interface and target component. For example, from the initial interface A, clicking component a jumps to interface B, clicking component b jumps to interface C, clicking component c jumps to interface D, and so on.
[0044] For example, an application function knowledge graph can serve as an application user manual. Functional Q&A can be performed based on the application function knowledge graph, i.e., querying application functions and providing activation paths. Automated testing of the application can also be performed based on the application function knowledge graph. In specific implementation, instructions related to the application function knowledge graph can be received, and then related operations can be performed based on the application function knowledge graph according to those instructions. The target interface and target component are the interfaces and components to be queried or tested. The initial interface and initial component can be the root interface and root component, or it can be the currently used interface and component, or it can be specified by the user, or determined through other means.
[0045] In one optional embodiment, a query instruction for the interface to be queried and / or the component to be queried can be received, and the opening path of the interface to be queried and / or the component to be queried can be queried according to the application function knowledge graph.
[0046] In this embodiment, if a user needs to query how to open at least one of the target interface and target component, a query command can be triggered for the interface and / or component. This allows the system to retrieve at least one of the target interface entity and target component entity from the application's functional knowledge graph. A path from the initial interface or initial target component to the target interface entity or target component entity is then provided as the opening path. The user can then operate step-by-step from the initial interface or initial target component based on this opening path until reaching the target interface or target component. For example, a user can input a query request for component d included in interface D. By querying the application's functional knowledge graph, an opening path can be provided: Interface A, click component a, jump to interface B, click component b, jump to interface C, click component c, jump to interface D.
[0047] Optionally, the query command may also provide the starting point of the opening path of at least one of the target interface and target components, i.e. the initial interface or the initial target component, and then the path from the initial interface or the initial target component to at least one of the target interface and target components can be queried based on the application function knowledge graph.
[0048] In another alternative embodiment, a test instruction for at least one of the target interface and target component to be tested can be received, the opening path of at least one of the target interface and target component can be queried according to the application function knowledge graph, and an automatic test script for the interface to be tested and / or the component to be tested can be generated according to the opening path of at least one of the target interface and target component, and the automatic test script can be executed.
[0049] In this embodiment, when it is necessary to test at least one of the target interface and target component to be tested, the opening path of at least one of the target interface and target component can be queried according to the application function knowledge graph. The method of obtaining the opening path is as shown in the above embodiment. Further, an automatic test script for at least one of the target interface and target component can be generated using a preset programming language based on the opening path of at least one of the target interface and target component. The automatic test script can be a test script for any test scenario such as functional testing, performance testing, stress testing, etc., of the target interface and target component. The automatic test script can be generated in any feasible way. Optionally, a test script generation model can be pre-trained. The test script generation model can generate an automatic test script for the interface and / or component based on the input opening path of the interface and / or component using a preset programming language. Of course, other methods can also be used, which will not be elaborated here.
[0050] The application processing method provided in this embodiment acquires screen recording videos of the application during its testing phase and extracts keyframes from the recordings. Based on the keyframes, it determines the interfaces, components, and relationships between components and interfaces within the application. Based on these interfaces, components, and relationships, it constructs a functional knowledge graph for the application. Finally, it determines an access path for at least one of the target interfaces and target components based on the application's functional knowledge graph. This access path is the path from the initial interface or initial target component to at least one of the target interfaces and target components. This embodiment can quickly and conveniently construct an application functional knowledge graph based on screen recording videos of the application during its testing phase. It is low-cost, highly efficient, and can support application scenarios such as functional Q&A or automated testing.
[0051] Based on any of the above embodiments, in S202, the interface, components, and the relationship between components and the interface in the application are determined according to the keyframe, such as... Figure 3 As shown, it may specifically include:
[0052] S301. Cluster adjacent keyframes to obtain multiple keyframe sets, and determine an interface based on each keyframe set.
[0053] In this embodiment, when no interface transition occurs, adjacent keyframes belong to the same interface. Therefore, adjacent keyframes can be clustered, aggregating adjacent keyframes belonging to the same interface into a keyframe set, with each keyframe set corresponding to one interface. In this embodiment, clustering adjacent keyframes is used to identify transitions between interfaces. For example, if a user transitions from interface 1 to interface 2 and then back to interface 1, clustering adjacent keyframes will yield three keyframe sets: the keyframe set corresponding to the initial interface 1, the keyframe set corresponding to interface 2, and the keyframe set corresponding to interface 1 after the transition from interface 2.
[0054] Optionally, each keyframe can be vectorized, and cosine similarity can be used to cluster the vectors of adjacent keyframes. Of course, other methods can also be used to cluster adjacent keyframes.
[0055] Optionally, a clustering model can be pre-trained to vectorize keyframes and cluster adjacent keyframes based on vector similarity. The clustering model can be any machine learning model, such as a neural network model.
[0056] It should be noted that when scrolling within the same interface (e.g., scrolling up and down, scrolling left and right), there are still overlapping parts in adjacent keyframes. In the above clustering process, these keyframes can still be aggregated into a keyframe set.
[0057] In addition, when determining an interface based on each set of keyframes, the content of the keyframes in the set of keyframes can also be identified to determine the content, function or purpose of the interface, so that it can be used in subsequent scenarios such as functional question answering.
[0058] S302. Identify the manipulated component in the corresponding interface based on each keyframe set.
[0059] In this embodiment, since adjacent keyframe sets represent interface transitions, and interface transitions are usually caused by the user performing certain operations on the previous interface, especially operations on a certain manipulated component in the previous interface, resulting in a transition from the previous interface to the next interface, the manipulated component in the corresponding interface can be identified for each keyframe set.
[0060] In this embodiment, during screen recording, the user's operation position is typically also recorded. This is displayed in the screen recording video with specific markers, such as indicating the user's click location. Based on this, the user's operation position within the corresponding interface can be identified from the keyframe set, thereby identifying the manipulated component on the interface. Of course, for operations other than clicks, corresponding methods can also be used to identify the manipulated component. For example, for voice operations, speech recognition can be used to determine the manipulated component.
[0061] In practice, we can first identify all the components included in the corresponding interface and the user's operation position in the corresponding interface based on any set of keyframes. Then, based on all the components included in the interface and the operation position, we can determine the component at the operation position, thereby determining the component being operated on in the interface.
[0062] Optionally, identifying all components included in the corresponding interface based on any set of keyframes can be achieved using any method. For example, a recognition model can be pre-trained to identify all components included in an interface screenshot. Then, any set of keyframes can be input into the recognition model to identify all components included in the set of keyframes, thus determining them as all components included in the corresponding interface. The recognition model can be any machine learning model, such as a neural network model.
[0063] S303, Create the containment relationship between the interface and the manipulated components in the interface, and the jump relationship between the manipulated components in the interface and the next adjacent interface.
[0064] In this embodiment, after determining the adjacent interfaces and the manipulated components in each interface, the inclusion relationship between any interface and the manipulated components in that interface, as well as the jump relationship between the manipulated components in that interface and the next adjacent interface, can be created.
[0065] If the previous interface includes multiple manipulated components, the inclusion relationship between the interface and the manipulated components in the interface, as well as the jump relationship between the last manipulated component in the interface and the next adjacent interface, can be constructed.
[0066] In this interface, the other manipulated components, except for the last one, did not cause the interface to jump. For example, input box controls and keyboard controls did not cause the interface to jump during the input process. These manipulated components can be regarded as jumping back to the interface. Therefore, when constructing the application function knowledge graph, if it is determined that the manipulated component in the current interface does not jump from the current interface to another interface after being manipulated, then an edge pointing from the manipulated component to the current interface is constructed in the application function knowledge graph. For example, after component 2 in interface 1 is manipulated, no interface jump occurs and it is still in interface 1. The following can be constructed: (interface 1) → [component contains] → (component 2) → [operation jump] → (interface 1).
[0067] Furthermore, an application function knowledge graph can be constructed based on the inclusion relationship between any interface and the manipulated component within that interface, as well as the jump relationship between the manipulated component within any interface and the next adjacent interface of that interface. This involves constructing edges from interface entities to manipulated component entities based on the inclusion relationship between any interface and the manipulated component within that interface, and constructing edges from manipulated component entities to interface entities based on the jump relationship between the manipulated component within any interface and the next adjacent interface of that interface, ultimately resulting in a complete application function knowledge graph.
[0068] The processing method of the application corresponding to the above embodiment, Figure 4 This is a structural block diagram of a processing device for an application provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 4 The processing device 400 for the application includes: an extraction unit 401, a determination unit 402, a construction unit 403, and an application unit 404.
[0069] The extraction unit 401 is used to acquire screen recording videos of the application during the test phase and extract key frames from the screen recording videos.
[0070] The determining unit 402 is used to determine the interface, components, and the relationship between components and the interface in the application based on the key frame;
[0071] The construction unit 403 is used to construct a functional knowledge graph of the application based on the interface, components, and the relationship between components and interface in the application.
[0072] Application unit 404 is used to determine, based on the application function knowledge graph, an opening path for at least one of the target interface and target components, wherein the opening path is a path from the initial interface or the initial target component to at least one of the target interface and target components.
[0073] In one or more embodiments of this disclosure, the determining unit 402, when determining the interface, components, and the relationship between components and the interface in the application based on the keyframe, is configured to:
[0074] Cluster adjacent keyframes to obtain multiple keyframe sets, and determine an interface based on each keyframe set.
[0075] Identify the manipulated component in the corresponding interface based on each set of keyframes;
[0076] Create the containment relationship between the interface and the components being manipulated within the interface, as well as the navigation relationship between the components being manipulated within the interface and the next adjacent interface.
[0077] In one or more embodiments of this disclosure, when the determining unit 402 identifies the manipulated component in the corresponding interface based on each keyframe set, it is configured to:
[0078] Identify all components included in the corresponding interface and their operation positions in the corresponding interface based on the keyframe set;
[0079] Based on all the components and the operation location, the component being operated on is determined among all the components.
[0080] In one or more embodiments of this disclosure, when the determining unit 402 identifies all components included in the corresponding interface based on any set of keyframes, it is configured to:
[0081] Based on the keyframe set, a preset recognition model is invoked to identify all components included in the keyframe set and determine them as all components included in the corresponding interface.
[0082] In one or more embodiments of this disclosure, when the determining unit 402 clusters adjacent keyframes to obtain multiple keyframe sets, it is used to:
[0083] Based on all keyframes, a preset clustering model is invoked to vectorize each keyframe, and adjacent keyframes are clustered according to vector similarity to obtain multiple keyframe sets.
[0084] In one or more embodiments of this disclosure, when the construction unit 403 constructs the application functional knowledge graph based on the interfaces, components, and relationships between components and interfaces in the application, it is used to:
[0085] The interface and components are respectively treated as entities in the application function knowledge graph. The edges connecting the components and the interface are determined according to the relationship between the components and the interface to obtain the application function knowledge graph.
[0086] In one or more embodiments of this disclosure, the building unit 403 is further configured to:
[0087] If it is determined that after the manipulated component in the current interface is manipulated, the user does not jump to another interface from the current interface, then an edge is constructed in the application function knowledge graph from the manipulated component to the current interface.
[0088] In one or more embodiments of this disclosure, when the application unit 404 determines the opening path of at least one of the target interface and target component based on the application function knowledge graph, it is configured to:
[0089] Receive a query instruction for at least one of the target interface and target components to be queried, and query the opening path of at least one of the target interface and target components based on the application function knowledge graph.
[0090] In one or more embodiments of this disclosure, when the application unit 404 determines the opening path of at least one of the target interface and target component based on the application function knowledge graph, it is configured to:
[0091] Receive test instructions for at least one of the target interface and target components to be tested, and query the opening path of at least one of the target interface and target components according to the application function knowledge graph;
[0092] Accordingly, after determining the opening path of at least one of the target interface and target component based on the application function knowledge graph, the application unit 404 is further configured to:
[0093] An automatic test script is generated for at least one of the target interface and the target component based on the opening path of the target interface and the target component, and the automatic test script is executed.
[0094] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.
[0095] To implement the above embodiments, this disclosure also provides an electronic device.
[0096] refer to Figure 5 The diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The electronic device 500 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0097] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0098] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
[0099] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.
[0100] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0101] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0102] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.
[0103] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0105] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".
[0106] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0107] In a first aspect, according to one or more embodiments of this disclosure, a method for processing an application is provided, comprising:
[0108] Acquire screen recordings of the application during its testing phase and extract keyframes from the screen recordings;
[0109] The keyframes are used to determine the interfaces, components, and the relationships between components and interfaces in the application.
[0110] Based on the interfaces, components, and relationships between components and interfaces in the application, construct a functional knowledge graph of the application.
[0111] The opening path of at least one of the target interface and target component is determined based on the application function knowledge graph, wherein the opening path is the path from the initial interface or the initial target component to at least one of the target interface and target component.
[0112] According to one or more embodiments of this disclosure, determining the interface, components, and the relationship between components and the interface in the application based on the keyframe includes:
[0113] Cluster adjacent keyframes to obtain multiple keyframe sets, and determine an interface based on each keyframe set.
[0114] Identify the manipulated component in the corresponding interface based on each set of keyframes;
[0115] Create the containment relationship between the interface and the components being manipulated within the interface, as well as the navigation relationship between the components being manipulated within the interface and the next adjacent interface.
[0116] According to one or more embodiments of this disclosure, identifying the manipulated component in the corresponding interface based on each keyframe set includes:
[0117] Identify all components included in the corresponding interface and their operation positions in the corresponding interface based on the keyframe set;
[0118] Based on all the components and the operation location, the component being operated on is determined among all the components.
[0119] According to one or more embodiments of this disclosure, identifying all components included in the corresponding interface based on a set of keyframes includes:
[0120] Based on the keyframe set, a preset recognition model is invoked to identify all components included in the keyframe set and determine them as all components included in the corresponding interface.
[0121] According to one or more embodiments of this disclosure, the clustering of adjacent keyframes to obtain multiple keyframe sets includes:
[0122] Based on all keyframes, a preset clustering model is invoked to vectorize each keyframe, and adjacent keyframes are clustered according to vector similarity to obtain multiple keyframe sets.
[0123] According to one or more embodiments of this disclosure, constructing a functional knowledge graph of the application based on the interfaces, components, and relationships between components and interfaces in the application includes:
[0124] The interface and components are respectively treated as entities in the application function knowledge graph. The edges connecting the components and the interface are determined according to the relationship between the components and the interface to obtain the application function knowledge graph.
[0125] According to one or more embodiments of this disclosure, the method further includes:
[0126] If it is determined that after the manipulated component in the current interface is manipulated, the user does not jump to another interface from the current interface, then an edge is constructed in the application function knowledge graph from the manipulated component to the current interface.
[0127] According to one or more embodiments of this disclosure, determining the access path of at least one of the target interface and target component based on the application functional knowledge graph includes:
[0128] Receive a query instruction for at least one of the target interface and target components to be queried, and query the opening path of at least one of the target interface and target components based on the application function knowledge graph.
[0129] According to one or more embodiments of this disclosure, determining the access path of at least one of the target interface and target component based on the application functional knowledge graph includes:
[0130] Receive test instructions for at least one of the target interface and target components to be tested, and query the opening path of at least one of the target interface and target components according to the application function knowledge graph;
[0131] Accordingly, after determining the opening path of at least one of the target interface and target component based on the application function knowledge graph, the method further includes:
[0132] An automatic test script is generated for at least one of the target interface and the target component based on the opening path of the target interface and the target component, and the automatic test script is executed.
[0133] Secondly, according to one or more embodiments of this disclosure, an application processing device is provided, comprising:
[0134] The extraction unit is used to acquire screen recording videos during the application's testing phase and extract keyframes from the screen recording videos.
[0135] The determining unit is used to determine the interface, components, and the relationship between components and the interface in the application based on the keyframe;
[0136] The construction unit is used to construct a functional knowledge graph of the application based on the interface, components, and the relationship between components and interface in the application.
[0137] An application unit is configured to determine, based on the application function knowledge graph, an opening path for at least one of a target interface and a target component, wherein the opening path is a path from an initial interface or an initial target component to at least one of the target interface and the target component.
[0138] According to one or more embodiments of this disclosure, when determining the interface, components, and the relationship between components and the interface in the application based on the keyframe, the determining unit is configured to:
[0139] Cluster adjacent keyframes to obtain multiple keyframe sets, and determine an interface based on each keyframe set.
[0140] Identify the manipulated component in the corresponding interface based on each set of keyframes;
[0141] Create the containment relationship between the interface and the components being manipulated within the interface, as well as the navigation relationship between the components being manipulated within the interface and the next adjacent interface.
[0142] According to one or more embodiments of this disclosure, when the determining unit identifies the manipulated component in the corresponding interface based on each set of keyframes, it is configured to:
[0143] Identify all components included in the corresponding interface and their operation positions in the corresponding interface based on the keyframe set;
[0144] Based on all the components and the operation location, the component being operated on is determined among all the components.
[0145] According to one or more embodiments of this disclosure, when the determining unit identifies all components included in the corresponding interface based on the keyframe set, it is configured to:
[0146] Based on the keyframe set, a preset recognition model is invoked to identify all components included in the keyframe set and determine them as all components included in the corresponding interface.
[0147] According to one or more embodiments of this disclosure, when the determining unit clusters adjacent keyframes to obtain multiple keyframe sets, it is used to:
[0148] Based on all keyframes, a preset clustering model is invoked to vectorize each keyframe, and adjacent keyframes are clustered according to vector similarity to obtain multiple keyframe sets.
[0149] According to one or more embodiments of this disclosure, when the construction unit constructs the application functional knowledge graph based on the interfaces, components, and relationships between components and interfaces in the application, it is used to:
[0150] The interface and components are respectively treated as entities in the application function knowledge graph. The edges connecting the components and the interface are determined according to the relationship between the components and the interface to obtain the application function knowledge graph.
[0151] According to one or more embodiments of this disclosure, the building unit is further configured to:
[0152] If it is determined that after the manipulated component in the current interface is manipulated, the user does not jump to another interface from the current interface, then an edge is constructed in the application function knowledge graph from the manipulated component to the current interface.
[0153] According to one or more embodiments of this disclosure, when the application unit determines the opening path of at least one of the target interface and target component based on the application function knowledge graph, it is configured to:
[0154] Receive a query instruction for at least one of the target interface and target components to be queried, and query the opening path of at least one of the target interface and target components based on the application function knowledge graph.
[0155] According to one or more embodiments of this disclosure, when the application unit determines the opening path of at least one of the target interface and target component based on the application function knowledge graph, it is configured to:
[0156] Receive test instructions for at least one of the target interface and target components to be tested, and query the opening path of at least one of the target interface and target components according to the application function knowledge graph;
[0157] Accordingly, after determining the opening path of at least one of the target interface and target component based on the application function knowledge graph, the application unit is further configured to:
[0158] An automatic test script is generated for at least one of the target interface and the target component based on the opening path of the target interface and the target component, and the automatic test script is executed.
[0159] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;
[0160] The memory stores computer-executed instructions;
[0161] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the processing method of the application described in the first aspect and various possible designs of the first aspect.
[0162] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the processing method of the application described in the first aspect and various possible designs of the first aspect is implemented.
[0163] Fifthly, according to one or more embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the processing method of the application described in the first aspect and various possible designs of the first aspect.
[0164] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.
[0165] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.
[0166] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.
Claims
1. A method for processing an application, characterized in that, include: Acquire screen recordings of the application during its testing phase and extract keyframes from the screen recordings; The keyframes are used to determine the interfaces, components, and the relationships between components and interfaces in the application. Based on the interfaces, components, and relationships between components and interfaces in the application, construct a functional knowledge graph of the application. The opening path of at least one of the target interface and target component is determined based on the application function knowledge graph, wherein the opening path is the path from the initial interface or the initial target component to at least one of the target interface and target component.
2. The method according to claim 1, characterized in that, Determining the interface, components, and relationships between components and the interface in the application based on the keyframes includes: Cluster adjacent keyframes to obtain multiple keyframe sets, and determine an interface based on each keyframe set. Identify the manipulated component in the corresponding interface based on each set of keyframes; Create the containment relationship between the interface and the components being manipulated within the interface, as well as the navigation relationship between the components being manipulated within the interface and the next adjacent interface.
3. The method according to claim 2, characterized in that, The step of identifying the manipulated component in the corresponding interface based on each keyframe set includes: Identify all components included in the corresponding interface and their operation positions in the corresponding interface based on the keyframe set; Based on all the components and the operation location, the component being operated on is determined among all the components.
4. The method according to claim 3, characterized in that, The step of identifying all components included in the corresponding interface based on the keyframe set includes: Based on the keyframe set, a preset recognition model is invoked to identify all components included in the keyframe set and determine them as all components included in the corresponding interface.
5. The method according to claim 2, characterized in that, The clustering of adjacent keyframes yields multiple keyframe sets, including: Based on all keyframes, a preset clustering model is invoked to vectorize each keyframe, and adjacent keyframes are clustered according to vector similarity to obtain multiple keyframe sets.
6. The method according to any one of claims 1-5, characterized in that, The step of constructing a functional knowledge graph of the application based on the interfaces, components, and the relationships between components and interfaces in the application includes: The interface and components are respectively treated as entities in the application function knowledge graph. The edges connecting the components and the interface are determined according to the relationship between the components and the interface to obtain the application function knowledge graph.
7. The method according to claim 6, characterized in that, The method further includes: If it is determined that after the manipulated component in the current interface is manipulated, the user does not jump to another interface from the current interface, then an edge is constructed in the application function knowledge graph from the manipulated component to the current interface.
8. The method according to claim 1, characterized in that, Determining the access path of at least one of the target interface and target component based on the application function knowledge graph includes: Receive a query instruction for at least one of the target interface and target components to be queried, and query the opening path of at least one of the target interface and target components based on the application function knowledge graph.
9. The method according to claim 1, characterized in that, Determining the access path of at least one of the target interface and target component based on the application function knowledge graph includes: Receive test instructions for at least one of the target interface and target components to be tested, and query the opening path of at least one of the target interface and target components according to the application function knowledge graph; Accordingly, after determining the opening path of at least one of the target interface and target component based on the application function knowledge graph, the method further includes: An automatic test script is generated for at least one of the target interface and the target component based on the opening path of the target interface and the target component, and the automatic test script is executed.
10. A processing device for an application, characterized in that, include: The extraction unit is used to acquire screen recording videos during the application's testing phase and extract keyframes from the screen recording videos. The determining unit is used to determine the interface, components, and the relationship between components and the interface in the application based on the keyframe; The construction unit is used to construct a functional knowledge graph of the application based on the interface, components, and the relationship between components and interface in the application. An application unit is configured to determine, based on the application function knowledge graph, an opening path for at least one of a target interface and a target component, wherein the opening path is a path from an initial interface or an initial target component to at least one of the target interface and the target component.
11. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-9.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-9.
13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.
