Data processing method and device, equipment and storage medium
By generating an interface transformation diagram containing scene nodes and action nodes, and using semantic information recognition model for semantic annotation, the problem of poor readability of interface transformation diagrams in the prior art is solved, and a clearer interface action information display is achieved.
Patent Information
- Application Number
- CN202411748110.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-21
AI Technical Summary
The interface conversion diagram generated by existing test tools only contains multiple scene nodes, and only edges are used for connection between each scene node, which makes it impossible for users to understand the action information between interface scenes and is poor in readability.
By obtaining page interaction data, an interface transformation diagram containing multiple scene nodes and action nodes is generated, scene key information and action key information are extracted based on preset rules, and scene semantic information recognition model is used to obtain scene semantic information and action semantic information for semantic annotation.
It improves the readability of the interface conversion diagram, enables users to better understand the logic and process of interface conversion, and enhances their understanding of interface action information.
Smart Images
Figure CN119669042A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium. Background Art
[0002] GUI (Graphical User Interface) is a way for users to interact with applications. At present, in the process of GUI testing, a test tool can be used to generate an interface transition diagram of the application, and the interface transition diagram can be used to assist in analyzing the dynamic interaction between the interface states of the application.
[0003] However, the interface transition graph generated by the existing testing tools only contains multiple scene nodes, and each scene node is only connected by edges. Users cannot understand the action information between the interface scenes through the interface transition graph. Obviously, the readability of the interface transition graph generated by the existing testing tools is poor. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a data processing method, apparatus, device and storage medium.
[0005] In a first aspect, an embodiment of the present disclosure provides a data processing method, the method comprising:
[0006] Obtaining page interaction data collected for the object to be tested, and an interface transition graph generated based on the page interaction data; wherein the page interaction data includes multiple session data, the first session data in the multiple session data includes page data and interaction operation data, the interface transition graph includes multiple scene nodes and an action node for connecting two scene nodes, and the first scene node and the first action node in the interface transition graph are generated based on the first session data;
[0007] Extracting scene key information from page data in the first session data based on a preset scene information extraction rule, and extracting action key information from interactive operation data in the first session data based on a preset action information extraction rule;
[0008] Based on the scene key information and the action key information, the scene semantic information of the first scene node in the interface transition graph and the action semantic information of the first action node are obtained; wherein the scene semantic information is used to semantically annotate the first scene node in the interface transition graph, and the action semantic information is used to semantically annotate the first action node in the interface transition graph.
[0009] In an optional implementation, the acquiring, based on the scene key information and the action key information, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph includes:
[0010] The scene key information and the action key information are input into a semantic information recognition model, and after being processed by the semantic information recognition model, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition diagram are output.
[0011] In an optional implementation, the step of inputting the scene key information and the action key information into a semantic information recognition model, and before outputting the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph after being processed by the semantic information recognition model, further includes:
[0012] Acquire scene semantic information of a second scene node and action semantic information of a second action node in the interface transition graph; wherein the second scene node is a previous scene node adjacent to the first scene node in the interface transition graph, and the second action node is a previous action node adjacent to the first action node in the interface transition graph;
[0013] Correspondingly, the step of inputting the scene key information and the action key information into a semantic information recognition model, and outputting the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph after being processed by the semantic information recognition model, includes:
[0014] The scene semantic information of the second scene node, the action semantic information of the second action node, the scene key information and the action key information are input into a semantic information recognition model. After being processed by the semantic information recognition model, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition diagram are output.
[0015] In an optional implementation manner, the page data in the first session data includes workflow attribute information and page code data of a starting page and workflow attribute information and page code data of an end page; and extracting scene key information from the page data in the first session data based on a preset scene information extraction rule includes:
[0016] Extracting user interaction UI component related information from the page code data of the starting page and the ending page in the first session data;
[0017] Extracting page workflow attribute information from the workflow attribute information of the starting page and the end page in the first session data;
[0018] The UI component related information and the page workflow attribute information are determined as the scene key information corresponding to the first session data.
[0019] In an optional implementation manner, before determining the UI component related information and the page workflow attribute information as the scenario key information corresponding to the first session data, the method further includes:
[0020] Extracting the reused component information from the page code data of the starting page and the ending page in the first session data;
[0021] Correspondingly, determining the UI component related information and the page workflow attribute information as the scenario key information corresponding to the first session data includes:
[0022] The UI component related information, the page workflow attribute information and the reused component information are determined as the scene key information corresponding to the first session data.
[0023] In an optional implementation manner, the interactive operation data in the first session data includes interactive action description information of the starting page, and extracting action key information from the interactive operation data in the first session data based on a preset action information extraction rule includes:
[0024] At least one of information of interactive controls, information of subcomponents of interactive controls, and information of parent components of interactive controls is extracted from the interactive action description information of the starting point page in the first session data as action key information corresponding to the first session data.
[0025] In an optional implementation manner, the first session data further includes a screenshot of a starting page and a screenshot of an end page, and before acquiring the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph based on the scene key information and the action key information, the method further includes:
[0026] By performing image recognition on the starting page screenshot and the ending page screenshot, scene key information and action key information corresponding to the first session data are obtained.
[0027] In an optional implementation, after acquiring the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph based on the scene key information and the action key information, the method further includes:
[0028] The scene semantic information of the first scene node is stored in the node information of the first scene node in the graph database of the interface transition graph, and the action semantic information of the first action node is stored in the node information of the first action node in the graph database.
[0029] In a second aspect, the present disclosure provides a data processing device, the device comprising:
[0030] A first acquisition module is used to acquire page interaction data collected for the object to be tested, and an interface transition graph generated based on the page interaction data; wherein the page interaction data includes multiple session data, the first session data in the multiple session data includes page data and interaction operation data, the interface transition graph includes multiple scene nodes and an action node for connecting two scene nodes, and the first scene node and the first action node in the interface transition graph are generated based on the first session data;
[0031] an extraction module, configured to extract scene key information from page data in the first session data based on a preset scene information extraction rule, and to extract action key information from interactive operation data in the first session data based on a preset action information extraction rule;
[0032] A second acquisition module is used to obtain the scene semantic information of the first scene node in the interface transition graph and the action semantic information of the first action node based on the scene key information and the action key information; wherein the scene semantic information is used to semantically annotate the first scene node in the interface transition graph, and the action semantic information is used to semantically annotate the first action node in the interface transition graph.
[0033] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a data processing method as provided in an embodiment of the present disclosure.
[0034] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the data processing method provided by the embodiment of the present disclosure.
[0035] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0036] In the data processing method provided by the embodiment of the present disclosure, firstly, the page interaction data collected for the object to be tested and the interface transition graph generated based on the interface interaction data are obtained; wherein the interface transition graph includes a plurality of scene nodes and an action node for connecting two scene nodes, the page interaction data includes a plurality of session data, and the first session data in the plurality of session data includes page data and interaction operation data; then, based on the preset scene information extraction rule, the scene key information is extracted from the page data in the first session data, and based on the preset action information extraction rule, the action key information is extracted from the interaction operation data in the first session data; then, based on the scene key information and the action key information, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph are obtained, so as to semantically annotate the first scene node and the first action node using the scene semantic information and the action semantic information.
[0037] It can be seen that the embodiment of the present disclosure can obtain the corresponding scene semantic information and action semantic information based on the key information after extracting the scene key information and action key information from the page interaction data, and use the above semantic information to semantically annotate each node in the interface transition diagram, thereby obtaining an interface transition diagram with semantic annotations, thereby improving the readability of the interface transition diagram. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0039] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;
[0040] Figure 2 A schematic diagram of an interface conversion diagram provided by an embodiment of the present disclosure;
[0041] Figure 3 A flowchart for generating semantic information provided by an embodiment of the present disclosure;
[0042] Figure 4 A schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure;
[0043] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0044] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0045] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0046] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0047] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0048] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0049] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0050] Graphical User Interface (GUI) testing is a type of application testing, which is used to test whether the application's user interface meets the design requirements. Currently, during the GUI testing process, the test tool can be used to generate an application's interface transition diagram, which can be used to assist in analyzing the dynamic interactions between the application's interface states.
[0051] However, the interface transition graph generated by the existing testing tools only contains multiple scene nodes, and each scene node is only connected by edges. Users cannot understand the action information between the interface scenes through the interface transition graph. Obviously, the readability of the interface transition graph generated by the existing testing tools is low.
[0052] Therefore, an embodiment of the present disclosure provides a data processing method, which first obtains page interaction data collected for an object to be tested, and an interface transition graph generated based on the interface interaction data; wherein the interface transition graph includes multiple scene nodes and action nodes for connecting two scene nodes, and the page interaction data includes multiple session data, and the first session data in the multiple session data includes page data and interaction operation data; then based on a preset scene information extraction rule, scene key information is extracted from the page data in the first session data, and based on a preset action information extraction rule, action key information is extracted from the interaction operation data in the first session data; then based on the scene key information and the action key information, scene semantic information of the first scene node and action semantic information of the first action node in the interface transition graph are obtained, so as to semantically annotate the first scene node and the first action node using the scene semantic information and the action semantic information.
[0053] It can be seen that the embodiment of the present disclosure can obtain the corresponding scene semantic information and action semantic information based on the key information after extracting the scene key information and action key information from the page interaction data, and use the above semantic information to semantically annotate each node in the interface transition diagram, thereby obtaining an interface transition diagram with semantic annotations, thereby improving the readability of the interface transition diagram.
[0054] Based on this, an embodiment of the present disclosure provides a data processing method, which is introduced below in conjunction with a specific embodiment.
[0055] Figure 1 The present invention provides a flowchart of a data processing method provided in an embodiment of the present invention. The method can be executed by a data processing device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:
[0056] S101: Acquire page interaction data collected for an object to be tested, and an interface transition diagram generated based on the page interaction data.
[0057] Among them, the page interaction data includes multiple session data, the first session data among the multiple session data includes page data and interactive operation data, the interface transition graph includes multiple scene nodes and action nodes for connecting two scene nodes, and the first scene node and the first action node in the interface transition graph are generated based on the first session data.
[0058] The data processing method provided in the embodiments of the present disclosure can be applied to a client. Specifically, the client can be deployed on a terminal such as a smart phone, a tablet computer, or a desktop computer.
[0059] In the embodiment of the present disclosure, the object to be tested may include an application program, and may also include one or more functional modules on the application program.
[0060] In the disclosed embodiment, the page interaction data includes multiple session data, and the page interaction data can be used to generate an interface transition diagram of the object to be tested. The session data refers to a combination of data generated by multiple consecutive pages and interactive operations with a time sequence relationship. For example, "When displaying the mall homepage, click the search control on the mall homepage to enter the search page" can be a session data.
[0061] The first session data may be any session data of the plurality of session data, and the first session data includes page data and interactive operation data. The page data is also called the original scene information, and the page data may include data such as page identification, page content, and page layout. The page content includes element content such as text, pictures, videos, and components, and the page layout is used to describe information such as the display position of each element content on the page.
[0062] Interaction data, also known as raw action information, refers to information related to the operations performed by users or test tools on the interface. These operations can trigger the interface to switch from one state to another, or switch from one interface to another. For example, in a shopping scenario, when a user clicks the "Add to Cart" control, the interaction data will record the relevant attribute information of the control, such as the text label "Add to Cart" and the location information of the control on the current page.
[0063] In the disclosed embodiment, the interface transition diagram is a graphical representation method for describing the interaction process between a user or a test tool and an application. The interface transition diagram may include multiple scene nodes and action nodes for connecting two scene nodes. Among them, the scene node can be used to represent scene pages with different functions in the application, such as the mall homepage and the search page, etc.; the action node is used to connect two scene nodes, and can be used to represent the interactive operations required to switch from one scene page to another scene page. For example, when entering the personal user homepage by clicking the user avatar on the mall homepage, the action of clicking the user avatar can be regarded as an action node.
[0064] The interface transition graph includes a first scene node and a first action node. Specifically, the first scene node and the first action node may be generated based on the first session data.
[0065] In an optional implementation, the page interaction data of the object to be tested can be obtained by inserting a stub. Specifically, insert a stub in the source code of the object to be tested to collect page information and interactive operation information as page interaction data, wherein the page information may include at least one of a page screenshot and page control attribute information, and the page control attribute information includes the size, position, color, font, etc. of the control; the page interactive operation information may include at least one of a page screenshot marked with an interactive position, page control information, and page control status information. The page screenshot marked with an interactive position can represent the area where the page interactive operation occurs, and the page control information may include the name, ID, and other information of the control. The page control status information is used to represent the state change of the control during the interaction process.
[0066] S102: extracting scene key information from page data in the first session data based on preset scene information extraction rules, and extracting action key information from interactive operation data in the first session data based on preset action information extraction rules.
[0067] In the embodiment of the present disclosure, the preset scene information extraction rule is used to extract information related to the page scene from the page data of the session data, that is, scene key information, such as component information contained in the page that can be used to characterize the page function, etc. After the page interaction data of the object to be tested is obtained, the scene key information is extracted from the page data of the first session data based on the preset scene information extraction rule.
[0068] The preset action information extraction rule is used to extract information related to user interaction, i.e., action key information, from the interactive operation data of the session data. For example, when a user clicks on a like control on a video playback page to achieve a like effect, the text information of the like control and other information that can be used to characterize the function of the control can be determined as action key information. After obtaining the page interaction data of the object to be tested, the action key information is extracted from the interactive operation data of the first session data based on the preset action information extraction rule.
[0069] S103: Based on the scene key information and the action key information, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph are obtained.
[0070] The scene semantic information is used to perform semantic annotation for the first scene node in the interface transition graph, and the action semantic information is used to perform semantic annotation for the first action node in the interface transition graph.
[0071] In the disclosed embodiment, after extracting the scene key information and the action key information, the scene semantic information of the first scene node in the interface transition diagram is obtained based on the scene key information, for example, by analyzing the data related to the scene page in the page interaction data, the corresponding scene name and detailed description information are determined for the first scene node as the scene semantic information of the first scene node, so as to mark the first scene node. And, the action semantic information of the first action node in the interface transition diagram is obtained based on the action key information, for example, according to the interactive operation of the user on the scene page, the corresponding action name and action description information are determined for the first action node as the action semantic information corresponding to the first action node, so as to mark the first action node.
[0072] In the disclosed embodiment, by utilizing scene key information and action key information, more detailed and accurate semantic annotations can be performed on scene nodes and action nodes in the interface transition diagram, thereby improving the readability of the interface transition diagram and helping users better understand the logic and process of interface transitions.
[0073] In an optional implementation, after extracting scene key information and action key information from page interaction data, the scene key information and action key information can also be input into a semantic information recognition model. After being processed by the semantic information recognition model, the scene semantic information of the first scene node and the action semantic information of the first action node in the input interface conversion graph are input.
[0074] In the disclosed embodiment, the semantic information recognition model is used to identify and parse the input scene key information and action key information to convert them into more readable and understandable semantic expressions. For example, based on the scene semantic information of the first scene node, the interface name, detailed description and other information of the scene page represented by the first scene node can be clearly understood. For another example, based on the action semantic information of the first action node, the functional relationship between interfaces and the user's operation intention can be intuitively understood.
[0075] like Figure 2 , which is a schematic diagram of an interface transition diagram provided by an embodiment of the present disclosure. The interface transition diagram includes multiple scene nodes and action nodes connecting two scene nodes, such as scene node 201 and scene semantic information corresponding to the scene node 201, and action node 202 and action semantic information corresponding to the action node 202, i.e., "click on the blogger image".
[0076] In the data method provided by the embodiment of the present disclosure, firstly, the page interaction data collected for the object to be tested and the interface transition graph generated based on the interface interaction data are obtained; wherein, the interface transition graph includes a plurality of scene nodes and an action node for connecting two scene nodes, the page interaction data includes a plurality of session data, and the first session data in the plurality of session data includes page data and interaction operation data; then, based on the preset scene information extraction rule, the scene key information is extracted from the page data in the first session data, and based on the preset action information extraction rule, the action key information is extracted from the interaction operation data in the first session data; then, based on the scene key information and the action key information, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph are obtained, so as to semantically annotate the first scene node and the first action node using the scene semantic information and the action semantic information.
[0077] It can be seen that the embodiment of the present disclosure can obtain the corresponding scene semantic information and action semantic information based on the key information after extracting the scene key information and action key information from the page interaction data, and use the above semantic information to semantically annotate each node in the interface transition diagram, thereby obtaining an interface transition diagram with semantic annotations, thereby improving the readability of the interface transition diagram.
[0078] In an optional implementation, when using a semantic recognition model to generate scene semantic information and action semantic information, in order to further improve the semantic recognition accuracy of the semantic recognition model, it is also possible to input the scene semantic information of the previous scene node adjacent to the first scene node and the action semantic information of the previous action node adjacent to the first action node while inputting the scene key information of the first scene node and the action key information of the first action node.
[0079] In the disclosed embodiment, the scene semantic information of the second scene node and the action semantic information of the second action node in the interface transition graph are first obtained; then the scene semantic information of the second scene node, the action semantic information of the second action node, the scene key information of the first scene node and the action key information of the first action node are input into the semantic information recognition model; after being processed by the semantic recognition model, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph are output.
[0080] The second scene node is a previous scene node adjacent to the first scene node in the interface transition graph, and the second action node is a previous action node adjacent to the first action node in the interface transition graph.
[0081] It can be seen that when the embodiments of the present disclosure use a semantic information recognition model to generate scene semantic information of a scene node and action semantic information of an action node, the scene semantic information of the previous scene node adjacent to the scene node can be simultaneously input into the semantic information recognition model to improve the semantic recognition accuracy of the scene node by the semantic recognition model, and the action semantic information of the previous action node adjacent to the action node can be simultaneously input into the semantic information recognition model to improve the semantic recognition accuracy of the action node by the semantic recognition model.
[0082] In an optional implementation, after obtaining the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph, the scene semantic information of the first scene node can also be stored in the node information of the first scene node in the graph database of the interface transition graph, and the action semantic information of the first action node can be stored in the node information of the first action node in the graph database.
[0083] like Figure 3 The figure shows a flow chart of generating semantic information provided by an embodiment of the present disclosure.
[0084] First, use the instrumentation tool to instrument the source code of the object to be tested. Specifically, traverse the source code and packaged files of the object to be tested, identify the event listeners in the packaged files and the event processing functions associated with the event listeners, and insert the functional code for obtaining page information into the event processing function to obtain the code file after instrumentation.
[0085] The test tool is used to run the instrumented code file to collect page interaction data of the object to be tested, and an interface transition graph is generated based on the page interaction data. In addition, the collected page interaction data of the object to be tested can also be stored in a database, the page interaction data includes the first session data, and the first scene node and the first action node in the interface transition graph are generated based on the first session data.
[0086] The first session data includes a starting page and an ending page. Based on a preset scene information extraction rule, scene key information is extracted from the page data of the starting page and the ending page. Based on a preset action information extraction rule, action key information is extracted from the interaction data of the first session data.
[0087] The scene key information of the first scene node, the action key information of the first action node, the scene semantic information of the previous scene node adjacent to the first scene node, and the action semantic information of the previous action node adjacent to the first action node are input into the semantic information recognition model. After being processed by the semantic recognition model, the scene semantic information of the first scene node in the interface transition diagram and the action semantic information of the first action node are output.
[0088] Finally, the scene semantic information of the first scene node and the action semantic information of the first action node are stored in the graph semantic database.
[0089] In an optional implementation, the page data in the first session data may include workflow attribute information of the starting page, page code data, and workflow attribute information of the end page. Page code data.
[0090] The workflow attribute information may include attributes such as process control, data transfer, and state management, through which functions such as lifecycle management, user interface navigation, data transfer and communication, authority management, event processing, and workflow logic processing can be realized. The page code data may include page layout information and page element description information written in code, etc. In practical applications, the page code data of the starting page and the end page may be described using a page XML (Extensible Markup Language) tree. For example, the nodes in the page XML tree may be used to represent page elements on the starting page or the end page.
[0091] In the embodiment of the present disclosure, the scene key information can be extracted from the page data of the first session data in the following manner:
[0092] Extract user interaction UI component related information from the page code data of the starting page and the end page in the first session data; extract page workflow attribute information from the workflow attribute information of the starting page and the end page of the first session data; determine the UI component related information and the page workflow attribute information as the scene key information corresponding to the first session data.
[0093] In the disclosed embodiment, since the page workflow attribute information can reflect the scenarios of the starting page and the end page, for example, assuming that the workflow attribute information of the starting page carries a search field, it can be determined that the starting page is a search scenario page, and assuming that the workflow attribute information of the end page carries a live field, it can be determined that the starting page is a live scenario page.
[0094] In actual applications, the page code data of the starting page and the end page may include page layout information and information related to user interaction UI components. Since the page layout information cannot reflect the personalized information of pages in different scenarios, the layout information can be eliminated and only the information related to user interaction UI components can be retained as the scene key information.
[0095] In the page XML tree, multi-layer node nesting is a way to describe the page structure, where each node can contain other nodes to form a tree structure. There may be non-functional nodes and invisible nodes in the multi-layer nested nodes. Obviously, it is impossible to extract scene key information from these nodes. Therefore, the embodiment of the present disclosure can also merge non-functional nodes and invisible nodes upwards, and concentrate the description information in the parent node, so as to facilitate the subsequent extraction of scene key information from the parent node.
[0096] In an optional implementation, the page code data also includes a component with a state description, such as a switch component. The switch component generally includes two basic states: on (true) and off (false). For example, a like control on a video playback page can be switched from a non-liked state to a liked state by triggering an operation on the like control.
[0097] The disclosed embodiment extracts the state description information of the switch components in the page code data to enrich the scene key information input into the semantic information recognition model, thereby enhancing the focus of the semantic information recognition model and further improving the readability and practicality of the scene semantic information output based on the semantic information recognition model.
[0098] In an optional implementation, the reused component information and user interaction UI component related information can be extracted from the page code data of the starting page and the end page in the first session data, and the page workflow attribute information can be extracted from the workflow attribute information of the starting page and the end page in the first session data; and then the UI component related information, the page workflow attribute information and the reused component information can be determined as the scene key information corresponding to the first session data.
[0099] Among them, the reused component refers to the component that can be reused in different pages or different windows of the same page, for example, the video component can be reused in the search page and the video playback page, and in the personal introduction page. The reused component information can include key information such as UI component reuse, dynamic UI construction, and multi-window support.
[0100] It can be seen that the embodiment of the present disclosure extracts the reused component information in the page code data so that the scene key information input into the semantic information recognition model can more accurately reflect the functions of the starting page and the end page, thereby improving the recognition accuracy of the semantic information recognition model.
[0101] In an optional implementation, the interactive operation data in the first session data may include interactive action description information of the starting page, and the interactive action description information may be used to record interactive actions between the user and the starting page or between the test tool and the starting page. For example, when a collection control on the starting page is clicked, the text information "Collect" of the collection control may be recorded as the interactive action description information.
[0102] In an embodiment of the present disclosure, after obtaining the page interaction data of the object to be tested, it is also possible to extract at least one of the information of the interactive control, the sub-component information of the interactive control, and the parent component information of the interactive control from the interactive action description information of the starting point page in the first session data as the action key information corresponding to the first session data.
[0103] Among them, the types of interactive controls may include buttons, text boxes, drop-down lists, etc., and the information of the interactive controls may be, for example, text information contained in the code corresponding to the interactive control, etc. The text information can usually directly indicate the function or meaning of the interactive control. If it is determined that the interactive action description information of the starting page includes text information, the text information is extracted as the key information of the interactive control.
[0104] In addition, for icon type controls in interactive controls, since the icon control itself does not contain text information, it is impossible to directly extract key information. For example, when adding an image resource to a control, a unique identifier will be assigned to the image resource. Therefore, the resource identifier of the icon control can be extracted as the key information of the icon control.
[0105] In the design of page controls, operable interactive components may be surrounded by multiple layers of controls, which means that the outer wrapped controls may not contain useful key information, such as functional information, and the truly operable interactive components may be hidden in their subcomponents.
[0106] Therefore, in order to accurately extract key information from the interactive control, the embodiment of the present disclosure can also extract subcomponent information of the interactive control from the interactive action description information of the starting page as the action key information corresponding to the first session data. In actual applications, the position attribute in the control can be used to determine the position information of the interactive control and the subcomponent in the interactive control to more accurately extract the information of the subcomponent of the interactive control.
[0107] In addition, in order to make the extracted subcomponent information more accurate, the embodiment of the present disclosure can also judge the area of the interactive control and the area of the corresponding subcomponent. Specifically, when it is determined that the interactive control contains only one subcomponent and the area of the subcomponent occupies a percentage of the interactive control area that reaches a preset threshold (e.g., 75%), the key information of the subcomponent is extracted as the action key information corresponding to the first session data. For subcomponents whose percentage does not reach the preset threshold, it may not be possible to accurately characterize the function or meaning of the corresponding interactive control, so the key information is not extracted.
[0108] In practical applications, if neither the interactive control nor its subcomponents contain key information, it may be considered to extract the key information from the parent component of the interactive control.
[0109] In practical applications, in order to make the extracted parent component information more accurate, the embodiment of the present disclosure can also judge the area of the interactive control and the area of the corresponding parent component. Specifically, when it is determined that the parent component corresponding to the interactive control only contains the interactive control, and the percentage of the area of the interactive control occupied by the parent component does not exceed a preset threshold (for example, 80%), the key information of the parent component is extracted as the action key information corresponding to the first session data.
[0110] In an optional implementation, after extracting the interactive control information, the interactive control's subcomponent information or the interactive control's parent component information, the interactive control information, subcomponent information and parent component information can also be stored as information in HTML (HyperText Markup Language) format to facilitate understanding and analysis by the semantic information recognition model.
[0111] In an optional implementation, the first session data also includes a screenshot of the starting page and a screenshot of the end page. Since the screenshot of the starting page and the screenshot of the end page may contain key information that can characterize the page scene, the scene key information and action key information corresponding to the first session data can be obtained by performing image recognition on the screenshot of the starting page and the screenshot of the end page.
[0112] In order to implement the above embodiments, the present disclosure also proposes a data processing device. Figure 4The structure diagram of a data processing device provided by an embodiment of the present disclosure is shown in FIG. 1 , which can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 4 As shown, the device comprises:
[0113] A first acquisition module 401 is used to acquire page interaction data collected for the object to be tested, and an interface transition graph generated based on the page interaction data; wherein the page interaction data includes multiple session data, the first session data in the multiple session data includes page data and interaction operation data, the interface transition graph includes multiple scene nodes and an action node for connecting two scene nodes, and the first scene node and the first action node in the interface transition graph are generated based on the first session data;
[0114] An extraction module 402, configured to extract scene key information from page data in the first session data based on a preset scene information extraction rule, and to extract action key information from interactive operation data in the first session data based on a preset action information extraction rule;
[0115] The second acquisition module 403 is used to obtain the scene semantic information of the first scene node in the interface transition graph and the action semantic information of the first action node based on the scene key information and the action key information; wherein the scene semantic information is used to semantically annotate the first scene node in the interface transition graph, and the action semantic information is used to semantically annotate the first action node in the interface transition graph.
[0116] In an optional implementation manner, the second acquisition module includes:
[0117] An output submodule is used to input the scene key information and the action key information into a semantic information recognition model, and after being processed by the semantic information recognition model, output the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition diagram.
[0118] In an optional implementation manner, the second acquisition module further includes:
[0119] An acquisition submodule, used to acquire scene semantic information of a second scene node and action semantic information of a second action node in the interface transition graph; wherein the second scene node is a previous scene node adjacent to the first scene node in the interface transition graph, and the second action node is a previous action node adjacent to the first action node in the interface transition graph;
[0120] Accordingly, the output submodule is specifically used for:
[0121] The scene semantic information of the second scene node, the action semantic information of the second action node, the scene key information and the action key information are input into a semantic information recognition model. After being processed by the semantic information recognition model, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition diagram are output.
[0122] In an optional implementation manner, the page data in the first session data includes workflow attribute information and page code data of the starting page and workflow attribute information and page code data of the ending page; the extraction module includes:
[0123] A first extraction submodule, configured to extract information related to user interaction UI components from page code data of a starting page and an end page in the first session data;
[0124] A second extraction submodule is used to extract page workflow attribute information from the workflow attribute information of the starting page and the end page in the first session data;
[0125] A determination submodule is used to determine the UI component related information and the page workflow attribute information as the scene key information corresponding to the first session data.
[0126] In an optional implementation, the extraction module further includes:
[0127] A second extraction submodule is used to extract the reuse component information from the page code data of the starting page and the ending page in the first session data;
[0128] Accordingly, the determination submodule is specifically used for:
[0129] The UI component related information, the page workflow attribute information and the reused component information are determined as the scene key information corresponding to the first session data.
[0130] In an optional implementation manner, the interactive operation data in the first session data includes interactive action description information of the starting point page, and the extraction module includes:
[0131] The third extraction submodule is used to extract at least one of the information of the interactive control, the subcomponent information of the interactive control and the parent component information of the interactive control from the interactive action description information of the starting page in the first session data as the action key information corresponding to the first session data.
[0132] In an optional implementation manner, the first session data also includes a screenshot of the starting page and a screenshot of the ending page, and the device further includes:
[0133] The third acquisition module is used to acquire the scene key information and the action key information corresponding to the first session data by performing image recognition on the screenshot of the starting page and the screenshot of the ending page.
[0134] In an optional implementation, the device further includes:
[0135] A storage module is used to store the scene semantic information of the first scene node in the node information of the first scene node in the graph database of the interface transition graph, and to store the action semantic information of the first action node in the node information of the first action node in the graph database.
[0136] In the data processing device provided by the embodiment of the present disclosure, firstly, the page interaction data collected for the object to be tested and the interface transition graph generated based on the interface interaction data are obtained; wherein the interface transition graph includes multiple scene nodes and action nodes for connecting two scene nodes, the page interaction data includes multiple session data, and the first session data in the multiple session data includes page data and interaction operation data; then, based on the preset scene information extraction rule, the scene key information is extracted from the page data in the first session data, and based on the preset action information extraction rule, the action key information is extracted from the interaction operation data in the first session data; then, based on the scene key information and the action key information, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph are obtained, so as to semantically annotate the first scene node and the first action node using the scene semantic information and the action semantic information.
[0137] It can be seen that the embodiment of the present disclosure can obtain the corresponding scene semantic information and action semantic information based on the key information after extracting the scene key information and action key information from the page interaction data, and use the above semantic information to semantically annotate each node in the interface transition diagram, thereby obtaining an interface transition diagram with semantic annotations, thereby improving the readability of the interface transition diagram.
[0138] In order to implement the above embodiments, the present disclosure further proposes a computer program product, including a computer program / instruction, which implements the data processing method in the above embodiments when executed by a processor.
[0139] In addition, the present disclosure also provides a data processing device, see Figure 5 As shown, it may include:
[0140] Processor 501, memory 502, input device 503 and output device 504. The number of processors 501 in the data processing device can be one or more. Figure 5In some embodiments of the present disclosure, the processor 501, the memory 502, the input device 503 and the output device 504 may be connected via a bus or other means, wherein: Figure 5 The example of connecting through bus is taken in the following.
[0141] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing of the data processing device by running the software programs and modules stored in the memory 502. The memory 502 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory 502 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The input device 503 can be used to receive input digital or character information, and generate signal input related to user settings and function control of the data processing device.
[0142] Specifically in this embodiment, the processor 501 will load the executable files corresponding to the processes of one or more applications into the memory 502 according to the following instructions, and the processor 501 will run the applications stored in the memory 502, thereby realizing various functions of the above-mentioned data processing device.
[0143] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0144] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing method, characterized in that: include: Obtaining page interaction data collected for the object to be tested, and an interface transition graph generated based on the page interaction data; wherein the page interaction data includes multiple session data, the first session data in the multiple session data includes page data and interaction operation data, the interface transition graph includes multiple scene nodes and an action node for connecting two scene nodes, and the first scene node and the first action node in the interface transition graph are generated based on the first session data; Extracting scene key information from page data in the first session data based on a preset scene information extraction rule, and extracting action key information from interactive operation data in the first session data based on a preset action information extraction rule; Based on the scene key information and the action key information, the scene semantic information of the first scene node in the interface transition graph and the action semantic information of the first action node are obtained; wherein the scene semantic information is used to semantically annotate the first scene node in the interface transition graph, and the action semantic information is used to semantically annotate the first action node in the interface transition graph.
2. The method according to claim 1, characterized in that The acquiring, based on the scene key information and the action key information, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph includes: The scene key information and the action key information are input into a semantic information recognition model, and after being processed by the semantic information recognition model, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition diagram are output.
3. The method according to claim 2, characterized in that The method further includes: inputting the scene key information and the action key information into a semantic information recognition model, and outputting the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph after being processed by the semantic information recognition model. Acquire scene semantic information of a second scene node and action semantic information of a second action node in the interface transition graph; wherein the second scene node is a previous scene node adjacent to the first scene node in the interface transition graph, and the second action node is a previous action node adjacent to the first action node in the interface transition graph; Correspondingly, the step of inputting the scene key information and the action key information into a semantic information recognition model, and outputting the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph after being processed by the semantic information recognition model, includes: The scene semantic information of the second scene node, the action semantic information of the second action node, the scene key information and the action key information are input into a semantic information recognition model. After being processed by the semantic information recognition model, the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition diagram are output.
4. The method according to claim 1, characterized in that: The page data in the first session data includes workflow attribute information and page code data of the starting page and workflow attribute information and page code data of the ending page; The extracting scene key information from the page data in the first session data based on the preset scene information extraction rule includes: Extracting user interaction UI component related information from the page code data of the starting page and the ending page in the first session data; Extracting page workflow attribute information from the workflow attribute information of the starting page and the end page in the first session data; The UI component related information and the page workflow attribute information are determined as the scene key information corresponding to the first session data.
5. The method according to claim 4, characterized in that Before determining the UI component related information and the page workflow attribute information as the scenario key information corresponding to the first session data, the method further includes: Extracting the reused component information from the page code data of the starting page and the ending page in the first session data; Correspondingly, determining the UI component related information and the page workflow attribute information as the scenario key information corresponding to the first session data includes: The UI component related information, the page workflow attribute information and the reused component information are determined as the scene key information corresponding to the first session data.
6. The method according to claim 1, characterized in that The interactive operation data in the first session data includes interactive action description information of the starting page, and extracting action key information from the interactive operation data in the first session data based on a preset action information extraction rule includes: At least one of information of interactive controls, information of subcomponents of interactive controls, and information of parent components of interactive controls is extracted from the interactive action description information of the starting point page in the first session data as action key information corresponding to the first session data.
7. The method according to claim 1, characterized in that The first session data also includes a screenshot of the starting page and a screenshot of the ending page. Before acquiring the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph based on the scene key information and the action key information, the method further includes: By performing image recognition on the starting page screenshot and the ending page screenshot, scene key information and action key information corresponding to the first session data are obtained.
8. The method according to claim 1, characterized in that After acquiring the scene semantic information of the first scene node and the action semantic information of the first action node in the interface transition graph based on the scene key information and the action key information, the method further includes: The scene semantic information of the first scene node is stored in the node information of the first scene node in the graph database of the interface transition graph, and the action semantic information of the first action node is stored in the node information of the first action node in the graph database.
9. A data processing device, characterized in that: The device comprises: A first acquisition module is used to acquire page interaction data collected for the object to be tested, and an interface transition graph generated based on the page interaction data; wherein the page interaction data includes multiple session data, the first session data in the multiple session data includes page data and interaction operation data, the interface transition graph includes multiple scene nodes and an action node for connecting two scene nodes, and the first scene node and the first action node in the interface transition graph are generated based on the first session data; an extraction module, configured to extract scene key information from page data in the first session data based on a preset scene information extraction rule, and to extract action key information from interactive operation data in the first session data based on a preset action information extraction rule; A second acquisition module is used to obtain the scene semantic information of the first scene node in the interface transition graph and the action semantic information of the first action node based on the scene key information and the action key information; wherein the scene semantic information is used to semantically annotate the first scene node in the interface transition graph, and the action semantic information is used to semantically annotate the first action node in the interface transition graph.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the method according to any one of claims 1 to 8.
11. A data processing device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.
12. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.