Data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a data processing method and device, computer equipment, a storage medium and a computer program product, which can be applied to application testing, game testing and other scenes related to intelligent traffic. The method comprises the following steps: acquiring test case data recorded by an operation terminal aiming at a plurality of continuously triggered interface operation events, and interface images respectively acquired when each interface operation event is triggered; according to an operation sequence between operation nodes represented by each interface operation event, determining a sequence relationship of operation metadata describing each interface operation event in the test case data; determining a triggered interface element in the interface image matched with the interface operation event as a key interface element; the operation metadata and the key interface elements serve as node data and are mapped to the case knowledge graph according to the sequence relation, and graph structured test case data are obtained. By adopting the method, the complexity of managing the test case data can be reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a data processing method, apparatus, computer device, storage medium, and computer program product. Background Art

[0002] With the development of computer technology, more and more software is used on operating terminals. In the process of software development, maintenance, and application, testing technology plays an important role. Test cases are the key data for testing. For the data management of test cases, traditional technologies generally store them in the form of text, such as using excel, xmind, or tabular documentation.

[0003] However, in the current test case data management method, there is a phenomenon of huge data redundancy in the use case data, which easily leads to the problem of difficult maintenance in test case management. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can simplify the management of test case data.

[0005] In a first aspect, the present application provides a data processing method. The method includes:

[0006] Obtaining test case data recorded for a plurality of continuously triggered interface operation events on an operating terminal, and interface images respectively collected when each of the interface operation events is triggered;

[0007] Determining the sequence relationship of the operation metadata describing each of the interface operation events in the test case data according to the operation sequence between the operation nodes represented by each of the interface operation events;

[0008] Determining the interface elements triggered in the interface image matching the interface operation event as the key interface elements matching the operation metadata of the interface operation event;

[0009] Mapping the operation metadata and the matching key interface elements as node data to a use case knowledge graph according to the sequence relationship to obtain graph-structured test case data.

[0010] In a second aspect, the present application further provides a data processing apparatus. The apparatus includes:

[0011] A data acquisition module, configured to obtain test case data recorded for a plurality of continuously triggered interface operation events on an operating terminal, and interface images respectively collected when each of the interface operation events is triggered;

[0012] A sequence relationship determination module, configured to determine the sequence relationship of the operation metadata describing each of the interface operation events in the test case data according to the operation sequence between the operation nodes represented by each of the interface operation events;

[0013] An image processing module, configured to determine the triggered interface element in the interface image matching the interface operation event as the key interface element matching the operation metadata of the interface operation event;

[0014] A node mapping module, configured to map the operation metadata and the matching key interface elements as node data to a use case knowledge graph according to the sequence relationship, so as to obtain graph-structured test case data.

[0015] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0016] Obtain test case data recorded for a plurality of continuously triggered interface operation events on an operation terminal, and interface images respectively collected when each of the interface operation events is triggered;

[0017] Determine the sequence relationship of the operation metadata describing each of the interface operation events in the test case data according to the operation sequence between the operation nodes represented by each of the interface operation events;

[0018] Determine the triggered interface element in the interface image matching the interface operation event as the key interface element matching the operation metadata of the interface operation event;

[0019] Map the operation metadata and the matching key interface elements as node data to a use case knowledge graph according to the sequence relationship, so as to obtain graph-structured test case data.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0021] Obtain test case data recorded for a plurality of continuously triggered interface operation events on an operation terminal, and interface images respectively collected when each of the interface operation events is triggered;

[0022] Determine the sequence relationship of the operation metadata describing each of the interface operation events in the test case data according to the operation sequence between the operation nodes represented by each of the interface operation events;

[0023] Determine the interface element triggered in the interface image that matches the interface operation event as the key interface element that matches the operation metadata of the interface operation event;

[0024] Use the operation metadata and the matching key interface element as node data, and map them to the use case knowledge graph according to the sequence relationship to obtain graph-structured test case data.

[0025] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0026] Obtain test case data recorded for multiple continuously triggered interface operation events on an operation terminal, and interface images respectively collected when each of the interface operation events is triggered;

[0027] Determine the sequence relationship of the operation metadata describing each interface operation event in the test case data according to the operation sequence between the operation nodes represented by each interface operation event;

[0028] Determine the interface element triggered in the interface image that matches the interface operation event as the key interface element that matches the operation metadata of the interface operation event;

[0029] Use the operation metadata and the matching key interface element as node data, and map them to the use case knowledge graph according to the sequence relationship to obtain graph-structured test case data.

[0030] The above data processing method, device, computer device, storage medium, and computer program product record test case data on an operation terminal, obtain the operation metadata describing each interface operation event and the interface images respectively collected when each interface operation event is triggered, determine the key interface element that matches the operation metadata of the interface operation event from the interface image, use the operation metadata and the matching key interface element as node data, and map them to the use case knowledge graph according to the sequence relationship, so as to construct a use case data knowledge graph based on the forms of the key interface element and the operation metadata. By recording test case data on the operation terminal, it is possible to accurately record data during normal operation on the operation terminal. At the same time, parsing the use case data during the operation process into corresponding knowledge graph structure data realizes the automatic parsing and mapping of test case data into graph-structured data, providing great convenience and data intuitiveness for test case data management, thereby effectively reducing the complexity of test case data management. Description of the Drawings

[0031] Figure 1 It is an application environment diagram of the data processing method in an embodiment;

[0032] Figure 2 It is a schematic flowchart of the data processing method in an embodiment;

[0033] Figure 3 It is a schematic diagram of operation metadata in the data processing method in an embodiment;

[0034] Figure 4 It is a schematic diagram of data change in the data processing method in an embodiment;

[0035] Figure 5 It is a schematic flowchart of the startup process of the target process with shell permission in an embodiment;

[0036] Figure 6 It is a schematic diagram of partial content of the test data stored as a json object in an embodiment;

[0037] Figure 7 It is a schematic diagram of the use case knowledge graph of the game scenario in an embodiment;

[0038] Figure 8 It is a schematic diagram of the overall architecture of the use case knowledge graph in an embodiment;

[0039] Figure 9 It is a schematic diagram of the display interface of the use case knowledge graph example diagram in an embodiment;

[0040] Figure 10 It is a structural block diagram of the data processing device in an embodiment;

[0041] Figure 11 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in multiple fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart healthcare, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0044] Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling machines to have the functions of perception, reasoning, and decision-making.

[0045] Artificial intelligence technology is an interdisciplinary subject with a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, and mechatronics. Among them, the pre-trained model, also known as the large model or the foundation model, can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0046] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes for tasks such as object recognition and measurement in machine vision, and further performing graphic processing to make the computer-processed images more suitable for human eyes to observe or be transmitted to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems that can obtain information from images or multi-dimensional data. The large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the visual field such as swin-transformer, ViT, V-MOE, and MAE can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology usually includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0047] System-level program: A program in the operating terminal that has the function of starting a native Java program in the shell. Based on this function, a process with system-level shell execution permissions can be started. In this application, the system-level program app_process will be used as an example for illustration. app_process is a native executable program provided by the Android system based on Linux and is located in the / system / bin / directory of the Android system. The most primitive zygote process is also started by this executable file. The role of app_process is that based on this executable program, a native Java program can be started in the shell. Therefore, based on this function, a process with system-level shell execution permissions can be started.

[0048] Event acquisition tool: A tool in the operating terminal that provides information about real-time dumps of input devices and kernel input events. In this application, the event acquisition tool getevent will be used as an example for illustration. getevent is a tool in the Android system and is essentially an executable file of the Linux system. Getevent can provide information about real-time dumps of input devices and kernel input events in the Android system. It can be simply understood that through this tool, the real-time information flow in the Android event system can be obtained, including screen input events (such as click, swipe, long press, home, back, etc. operations), key input events (such as power key, volume control key, etc.), and real-time information of some kernel input events. The results will be displayed in the terminal as a string stream of data in a specific format for the tool user to view.

[0049] UI (i.e., user interaction interface) recording and playback: Refers to two functions, UI operation recording and UI operation playback. UI operation recording means that through some technical solutions, all operation data of the user on the UI operation of the terminal device (some may include application data and environmental data of the UI scene) is persisted in a special storage method. UI operation playback is to inject the previously persisted UI operation data into the corresponding UI interface through some framework capabilities, that is, to reproduce the previously operated steps through event injection or data recovery, so as to achieve the purpose of recording and UI automation, greatly improving the efficiency of UI automation and reducing the cost of UI automation.

[0050] Use case knowledge graph: It is a structured semantic knowledge base, and the form of presenting data generally describes concepts in the physical world and their interrelationships in symbolic form. The basic unit of a use case knowledge graph is the triple of "entity-relationship-entity", as well as the entity and its related attribute value pairs. Entities are interconnected through relationships to form a networked knowledge structure. Based on the form of abstracting the core steps of test case data, it presents test case data and the relationships between test case data in a graphical form, enabling users to understand and identify changes in test case data more intuitively and systematically.

[0051] Interface operation event: The operation events in the embodiments of the present application can, but are not limited to, various types of user operations implemented through the user interface. For example, they can, but are not limited to, include screen input events (such as click, slide, long press, home, back, etc.), key input events (such as power key, volume control key, etc.), and some kernel input events, etc.

[0052] Test case data: The test case data in the embodiments of the present application can, but are not limited to, correspond to the functions of the product or user interface to be tested in UI automation testing. For example, it can, but is not limited to, directly use the operation process of one or more functions of the product or user interface as a test case; it can, but is not limited to, use the operation processes of functions such as account login, account registration, purchasing goods, publishing or editing information streams (such as including audio, video, or picture and text, etc.), and viewing or editing or publishing media streams (such as including audio or video, etc.) as a test case.

[0053] The data processing method provided by the embodiments of the present application can be applied to, for example Figure 1 the application environment shown. Among them, the operation terminal 102 communicates with the control terminal 104 through the network. The data storage system can store the data that the control terminal 104 needs to process. The control terminal can be a server, and the data storage system can be integrated on the server, or placed in the cloud or on other servers. Among them, the operation terminal 102 can, but is not limited to, be various notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The control terminal 104 can be implemented by various desktop computers, notebook computers, etc.

[0054] The design concept of the present application will be described below.

[0055] The data management of test cases is generally stored in a completely plain text format. Generally speaking, the vast majority of industry engineering R & D teams store test cases in ways such as using Excel, XMind, or tabular documentation. Among them, using the XMind document as the form of test case management and storage has its specific advantages, such as clear structure, visualization, high flexibility, convenience for collaboration, and good agility, etc. For simplicity, the form of directly using a table to store test case data can also be adopted. However, no matter which of the above-mentioned test case storage forms is used, there are problems of high maintenance cost and high maintenance difficulty. Moreover, for these two forms, the same problem will be encountered, that is, there is a huge data redundancy in test case data. Test cases with highly similar scenarios will generate a large amount of redundant data in the above two storage methods, and many test case steps and descriptions are basically exactly the same. Especially when there are the same precondition step test cases, this problem becomes even more prominent. In addition, when the business project undergoes version iteration and modification, once it involves the reconstruction or modification of the key test case step functions, a small engineering change and optimization may lead to a corresponding large number of adaptability modifications to the test case set. When the scale of the test case data set is large, this disadvantage will be extremely obvious.

[0056] In view of this, a data processing method is provided, which fully reuses the existing test data of automated testing and explores a brand-new test case knowledge graph from the perspective of visualization in combination with the knowledge graph. This new type of test case data management strategy can make good use of the self-driving nature of interface operation automation and the uniqueness of scenario interface elements, and analyze and automatically classify various test case node data from the above two perspectives. When the interface operation processes the corresponding interface element, its operation type will also be serialized and parsed into the operation metadata corresponding to the interface element, forming a graph node with a unique identification feature in the test case knowledge graph.

[0057] Specifically, this application records test case data on the operation terminal, obtains the operation metadata describing each interface operation event and the interface image when each interface operation event is triggered, determines the key interface element that matches the operation metadata of the interface operation event from the interface image, and uses the operation metadata and the matching key interface element as node data, and maps them to the test case knowledge graph according to the sequence relationship, so as to construct the test case data knowledge graph according to the form of the key interface element and the operation metadata. By means of recording test case data, it can achieve accurate data recording during the normal operation of the operation terminal. At the same time, the test case data during the operation process is parsed into the corresponding knowledge graph structure data, so as to realize the automatic parsing and mapping of test case data into graph-structured data, which provides great convenience and data intuitiveness for test case data management, and thus effectively reduces the complexity of test case data management.

[0058] In some specific embodiments, the control terminal 104 obtains the test case data recorded for a plurality of consecutively triggered interface operation events at the operation terminal 102, and the interface images respectively collected when each interface operation event is triggered; the control terminal 104 determines the sequence relationship of the operation metadata describing each interface operation event in the test case data according to the operation sequence between the operation nodes represented by each interface operation event, determines the interface elements triggered in the interface image matching the interface operation event as the key interface elements matching the operation metadata of the interface operation event, takes the operation metadata and the matching key interface elements as node data, and maps them to the use case knowledge graph according to the sequence relationship to obtain the graph-structured test case data.

[0059] In one embodiment, as Figure 2 shown, a data processing method is provided. Taking the control terminal in Figure 1 as an example, the method includes the following steps:

[0060] Step 202, obtain the test case data recorded for a plurality of consecutively triggered interface operation events at the operation terminal, and the interface images respectively collected when each interface operation event is triggered.

[0061] Herein, the operation terminal refers to a terminal installed with a product or program to be tested, and the user can operate through the operation page of the product. Opposite to the operation terminal is the control terminal. Through the control terminal, the control of the recording process in the operation terminal can be realized, the test case data recorded and collected in the operation terminal can be obtained, and through the analysis and processing of the test case data, the graph structuring of the test case data can be realized to obtain the use case knowledge graph.

[0062] The interface operation event refers to an event triggered by an operation when the interface of the test object is displayed on the operation terminal. The interface operation event can include, but is not limited to, screen input events and key input events. The screen input events include screen trigger operations such as click, swipe, long press, home, back, etc., and the key input events include key trigger operations such as triggering the power key, volume control key, etc. Taking the test object as a game program or a part of the functions in the game program as an example, the interface operation event is the game operation event triggered by the game user in the game interface. Operation controls for realizing human-computer interaction are configured in the game interface, and the game user can control the currently displayed game interface or update the content currently displayed in the game interface through the operation of the operation controls to realize a continuous game processing process. Since in the game scenario, the page changes of the game are continuous, and with the change of one operation step, the relevance between this operation step and other operation steps will change synchronously. For the game scenario, there is a strong continuity feature between various operation events.

[0063] Multiple consecutively triggered interface operation events refer to multiple interface operation events that are triggered sequentially in chronological order and have a logical relationship. Multiple consecutively triggered interface operation events can correspond to multiple operation interfaces with an associated relationship. Interface operation events include game operation events. During the process of a game user participating in a game, it is a continuous multi-interaction operation process. By responding to the operation triggered by the game user on the first game interface, the operation terminal will, according to the operation instruction corresponding to the operation, jump to the second game interface indicated by the operation instruction, realizing the jump between different game interfaces.

[0064] Among them, the associated relationship between operation interfaces can be a link relationship, a nested relationship, etc. For example, taking a game scenario as an example, multiple consecutively triggered interface operation events include clicking on the main page entrance -> clicking on battle to enter the secondary page -> clicking on quick match -> clicking on the matching mode. Multiple consecutively triggered interface operation events can also include clicking on the main page entrance -> clicking on battle to enter the secondary page -> clicking on the lane of the turret -> clicking on opening a room -> clicking on the matching mode.

[0065] Different interface operation events may correspond to different operation interfaces, or some of the interface operation events may correspond to the same operation interface. Each interface operation event must have an operation interface corresponding to it. The image collected for this operation interface is the interface image when this interface operation event is triggered. Taking the example of multiple consecutively triggered interface operation events including clicking on the main page entrance -> clicking on battle to enter the secondary page -> clicking on quick match -> clicking on the matching mode, the main page entrance is displayed on the game home page. Therefore, the interface image when clicking on the main page entrance to trigger this interface operation event is the game home page. Battle is displayed on the game main page. Therefore, the interface image when clicking on battle to enter the secondary page to trigger this interface operation event is the game main page. Similarly, the interface image when clicking on quick match is the secondary page of the game.

[0066] Specifically, the control terminal controls the operation terminal to record and collect test case data by sending relevant instructions for starting recording to the operation terminal. When the operation terminal is performing normal operations, the control terminal can obtain the test case data recorded for multiple consecutively triggered interface operation events through the operation terminal. For each interface operation event, the operation terminal will automatically collect the interface image corresponding to the interface operation event, thereby realizing the acquisition of data.

[0067] Step 204, determine the sequence relationship of the operation metadata describing each interface operation event in the test case data according to the operation sequence between the operation nodes represented by each interface operation event.

[0068] Among them, since a test case data includes multiple continuously triggered interface operation events, each interface operation event can be regarded as a node in a series of operations, and each operation node is arranged in the order of triggering of the operations. Based on the node position corresponding to the operation node, the sequence of occurrence of the page operation events in the test case data can be determined.

[0069] As Figure 3 shown, operation metadata refers to the various attribute data corresponding to the interface operation events and is used to describe the interface operation events. The operation metadata may include multiple pieces of description data for the interface operation events, such as at least a part of information such as page information, process information, location information, size information, and description information corresponding to the interface operation events. The operation metadata of different interface operation events is different to reflect the unique attributes of the interface operation events. In a game scenario, human-computer interaction has characteristics such as frequent interaction operations and high requirements for operation recognition accuracy. For example, for the same game page, the response results to be displayed for game operations triggered at different positions may be different, and for different processes of the game, the response results to be displayed for game operations triggered at the same position may also be different. Therefore, for each interface operation event, it is necessary to define the attributes of the interface operation event through the operation metadata, so as to effectively distinguish each interface operation event.

[0070] The operation metadata is used to uniquely represent one of the multiple continuously triggered interface operation events. Based on the operation sequence between the operation nodes represented by each interface operation event respectively, the sequence relationship between each operation metadata can be determined, and the playback of the test case data can be realized through the sequence relationship between each operation metadata.

[0071] Specifically, the test case data obtained by the control terminal can be the data after serialization processing on the operation terminal, or the control terminal can perform serialization processing by itself after receiving the test case data. Among them, performing serialization processing on the operation terminal can directly perform serialization based on the recording order, which can simplify the processing process of serialization processing and improve the efficiency of serialization processing. Performing serialization processing on the control terminal can reduce the data processing volume of the operation terminal, reduce the data processing pressure of the operation terminal, and avoid abnormal situations such as freezing.

[0072] Furthermore, the serialized test case data can intuitively reflect the operation sequence between the operation nodes represented by each interface operation event respectively. Based on the serialized test case data, the control terminal can directly determine the sorting result of the operation metadata as the sequence relationship between each operation metadata.

[0073] Step 206: Determine the triggered interface element in the interface image that matches the interface operation event as the key interface element that matches the operation metadata of the interface operation event.

[0074] Among them, the interface element is an element in the operation interface. The essence of the operation interface is a UI page. The UI elements in the UI page all have their unique characteristics, such as the corresponding coordinates, the graphic elements of the corresponding UI area, the hierarchical structure of multiple graphic elements, etc. In a specific embodiment, the triggered interface element can be an image element directly displayed in the operation interface, or other elements associated with the operation interface. For example, when the operation interface is triggered by a key on the operation terminal, the key can be used as

[0075] When the interface operation event, that is, the UI operation, processes the corresponding UI element, the UI element is the triggered interface element in the interface image. This interface element can be determined as the key interface element that matches the operation metadata of the interface operation event. Among them, the operation type of this interface element can also be parsed as the operation metadata corresponding to this key interface element.

[0076] Specifically, after the operation terminal obtains the interface image and operation metadata, it needs to further analyze and process the interface image to determine the key interface element that matches the operation metadata based on the interface image, that is, the interface element through which the user triggers the interface operation event.

[0077] In a specific embodiment, the operation terminal can send an image element segmentation instruction to an image processing service with image processing capabilities, so that the image processing service performs image element segmentation processing on the interface image to obtain each image element in the interface image. Among them, the image processing service can be deployed with a SAM model, which can realize image element segmentation of the interface image. The SAM model is a general model for processing image segmentation tasks, which can process a variety of different types of tasks and determine the content to be segmented in the image by identifying various input prompts. The segmented image elements can determine the position of each image element in the image. Based on the trigger coordinates recorded in the operation metadata of the interface operation event, the triggered interface element in the interface image, that is, the key interface element that matches the operation metadata, can be determined.

[0078] Step 208: Map the operation metadata and the matching key interface elements as node data to the use case knowledge graph according to the sequence relationship to obtain the graph-structured test case data.

[0079] Among them, node data is the data corresponding to the nodes in the use case knowledge graph. The use case knowledge graph is a network knowledge structure composed of "entity-relationship-entity" triples as basic components. Based on the form of abstracting the core steps of test case data, the test case data and the relationship between the test case data can be displayed in a new graphical form, which more intuitively and systematically reflects the association and changes between the test case data.

[0080] Operation metadata is the specific description data of the operation, and key interface elements are the graphical expressions of the operation, which can intuitively and comprehensively record interface operation events. By taking operation metadata and matching key interface elements as node data and mapping them to the use case knowledge graph according to the sequence relationship, it is possible to achieve comprehensive recording of test case data in the use case knowledge graph. Figure 4 As shown, the operation metadata and matching key interface elements of each operation node in the test case data can be used as node data, mapped to the use case knowledge graph according to the sequence relationship, and the graph-structured test case data can be obtained. By recording each test case data through the use case knowledge graph, the test case data can be split according to the operation node, and then the data at the operation node level can be collected in the use case knowledge graph to obtain the graph-structured test case data, which provides great convenience and data intuitiveness for test case data management, thereby effectively reducing the complexity of test case data management.

[0081] The above data processing method records the test case data at the operation terminal, obtains the operation metadata describing each interface operation event and the interface image collected when each interface operation event is triggered, determines the key interface elements that match the operation metadata of the interface operation event from the interface image, and uses the operation metadata and the matching key interface elements as node data, and maps them to the use case knowledge graph according to the sequence relationship, so as to construct the use case data knowledge graph according to the form of the key interface elements and the operation metadata. By recording the test case data at the operation terminal, it is possible to achieve accurate recording of the data when the operation terminal is in normal operation. At the same time, the use case data during the operation process is parsed into the corresponding knowledge graph structure data, and the test case data is automatically parsed and mapped into graph structured data, which provides great convenience and data intuitiveness for the test case data management, thereby effectively reducing the complexity of the test case data management.

[0082] In some embodiments, the recording of test case data and image acquisition of the operation terminal are started by the control of the control terminal. It is understood that in other embodiments, the recording of test case data and image acquisition can also be realized by the user at the operation terminal.

[0083] Specifically, taking the recording and image acquisition of test case data of the control terminal to control the operation terminal as an example, the test case data recorded for multiple interface operation events triggered continuously at the operation terminal and the interface images collected respectively when each interface operation event is triggered are obtained, including:

[0084] Based on the system-level program of the operation terminal, the target process for the test object in the operation terminal is started, and the target process has shell permissions;

[0085] Based on the shell permissions of the target process, call the event acquisition tool of the operation terminal to obtain the interface operation events of the test object in the operation terminal;

[0086] Record test case data according to interface operation events, and capture interface images when interface operation events are triggered.

[0087] The system-level program refers to a program in the operation terminal that has the function of starting a Java native program in the shell. Based on the system-level program, the operation terminal can start a process with system-level shell execution authority.

[0088] In one of the embodiments, the operating terminal is a terminal of the Android system, and the system-level program is an Android system app_process program. Specifically, app_process is a system-level executable program pre-embedded in the Android system, located in the / system / bin / directory of the Android file system, wherein the original application process zygote in the operating terminal is also constructed by the executable program app_process. Among them, although the ordinary user-level App in the operating terminal is obtained by the zygote process through fork, various permissions are restricted in the initialization logic related to fork. Therefore, in the Android system, the application is started by conventional means such as clicking the application icon, which are all processes with restrictive permissions. Therefore, it is necessary to start the target process for the test object in the operating terminal based on the system-level program of the operating terminal.

[0089] Shell permissions are specific functions of system processes. The essence of shell is a command line interpreter that can translate user commands to the system core and translate the processing results of the system core to the user. The operation terminal can use the shell permissions of the target process to call the event acquisition tool of the operation terminal to obtain the interface operation events of the test object in the operation terminal.

[0090] Specifically, the operating terminal can start an application based on a system-level program. It should be noted that when the operating terminal starts a process through app_process, the actual processing logic is different from the normal App startup logic. The process started through app_process is a native java process of the Linux system. Further, since app_process will default to inherit from the init process with a pid of 1 on the operating terminal, the processes started through app_process will all have the shell permission capabilities corresponding to the init process. The shell permission capabilities are the core dependencies of the event acquisition tool used in the test case data recording function.

[0091] Among them, the target process includes the process corresponding to the test object. Taking the test object as game A as an example, the target process is the process corresponding to game A. Taking the test object as shopping application B as an example, the target process is the process corresponding to shopping application B. By means of the system-level program of the operating terminal, the target process for the test object in the operating terminal is started. Compared with directly starting the target process of the test object, it can make the target process have shell permissions, and then, based on the shell permissions, the event acquisition tool of the operating terminal can be called to obtain the interface operation events for the test object in the operating terminal.

[0092] In this embodiment, by starting the target process for the test object in the operating terminal based on the system-level program of the operating terminal, the started target process can have shell permissions, and then the operating terminal can utilize the shell permissions possessed by the target process to call the event acquisition tool of the operating terminal to obtain the interface operation events for the test object in the operating terminal, realizing the recording of test case data and the acquisition of page images, and then realizing the functions of automatically recording test case data and automatically collecting page images on the operating terminal, which can effectively simplify the process of obtaining test case data and page images and effectively improve the data acquisition efficiency.

[0093] In one of the embodiments, starting the target process for the test object in the operating terminal based on the system-level program of the operating terminal includes:

[0094] Sending a driving instruction of the system-level program to the operating terminal through the control tool of the control terminal, and the control tool is implemented based on a cross-platform desktop application framework;

[0095] In response to the driving instruction, controlling the startup of the target process for the test object through the system-level program of the operating terminal.

[0096] Among them, the driving instruction of the system-level program is an instruction sent by the control terminal to drive the system-level application of the operating terminal to start a process with shell permissions on the operating terminal. In response to the driving instruction of the system-level program, the operating terminal can start a target process for the test object based on this system-level program.

[0097] As Figure 5 shown, taking the system-level program app_process as an example, although the ordinary App process on the operating terminal cannot utilize the executable program app_process, it can drive the app_process on the operating terminal through the driving instruction of the control terminal. Among them, the driving instruction of the control terminal can specifically be an adb instruction. By sending an adb driving instruction to the operating terminal, the control terminal can drive the app_process on the operating terminal to start the process of the operating terminal Linux, so that the started process has system-level execution permissions.

[0098] Among them, the control tool in the control terminal can be a tool implemented based on a cross-platform desktop application framework. It can use the cross-platform desktop application framework as the underlying basic framework, and then implement the interface of the recording and playback tool through the react technology. And in the node.js backend capabilities provided by the cross-platform desktop application framework, through the adb command, it can drive the core system-level program and event acquisition tool of the operating terminal.

[0099] Specifically, the control terminal uses the control tool implemented based on the cross-platform desktop application framework to send a driving instruction of the system-level program to the operating terminal. The operating terminal can respond to the driving instruction to drive the system-level program of the operating terminal, and then control the start of a target process for the test object through the system-level program, so that the target process has system-level shell permissions.

[0100] In this embodiment, through the cross-platform desktop application framework provided by the control tool of the control terminal, a driving instruction of the system-level program for the operating terminal can be constructed. Then, by sending this driving instruction to the operating terminal, it can control the start of a target process for the test object through the system-level program, realizing the convenient and effective start of a target program with system-level shell permissions, and ensuring that the operating terminal can have the functions of automatically recording test case data and automatically collecting page images.

[0101] In one of the embodiments, the data processing method further includes:

[0102] Sending a component initialization instruction to the operating terminal through the control tool of the control terminal; based on the component initialization instruction, performing component initialization processing on the operating terminal to obtain an initialized set of components.

[0103] Further, record test case data according to the interface operation event, and collect the interface image when the interface operation event is triggered, including:

[0104] Based on the recording engine component in the component set, record test case data according to the interface operation event; based on the image acquisition component in the component set, collect the interface image when the interface operation event is triggered.

[0105] Among them, the component initialization instruction refers to an instruction for initializing the components used to obtain data on the operation terminal. When the operation terminal is installed with each component for data acquisition, the component set containing each component can be directly initialized. When the operation terminal is not installed with each component for data acquisition, the components can be installed on the operation terminal based on this component initialization instruction to obtain an initialized component set.

[0106] The initialized component set includes various components for data acquisition, such as the recording engine component for recording test case data, and for another example, the image acquisition component for collecting interface images. Each component in the component set cooperates with each other to implement the recording of test case data for interface operation events and the image acquisition of the interface image when the interface operation event is triggered.

[0107] In one embodiment, the control tool of the control terminal will, based on the relevant initialization logic of node.js, control the automatic installation of a tool apk on the operation terminal. This apk is the component set of the operation terminal, and the component set can include a recording engine component, a playback engine component, an event acquisition component, and an image acquisition component.

[0108] Specifically, each component module in the component set is a Java backend execution program. Each component module in the component set can be started based on the system-level program and continuously run in the Android system in the form of a background process. Among them, the recording engine component will, after receiving the relevant instructions from the control tool, collect all event data during the recording process through the event acquisition tool of the operation terminal; the playback engine component will parse the previously recorded and saved data, and then implement the injection of event playback through various shell instructions. The event acquisition component is a special part. This component is encapsulated and implemented based on the event acquisition tool of the operation terminal. Essentially, it is an independent shell subprocess that continuously reads relevant event data through the poll function in the background. The event data will be redirected to a specific buffer area. By reading the event data in the buffer area, and then the entire process will be sent back to the corresponding process of the control tool through the socket thread. The corresponding process of the control tool will then parse the collected data.

[0109] In this embodiment, the control terminal sends a component initialization instruction to the operation terminal through a control tool; performs component initialization processing on the operation terminal to obtain an initialized set of components, so that the operation terminal can use the set of components to record test case data for interface operation events and collect interface images when interface operation events are triggered, ensuring the accuracy of the acquired data. By dividing the work according to components, the efficiency and comprehensiveness of data acquisition can be effectively improved.

[0110] In some of these embodiments, performing component initialization processing on the operation terminal to obtain an initialized set of components includes:

[0111] In the case where the component indicated by the initialization instruction is not installed on the operation terminal, determine the database storing the component installation package; install the component installation package downloaded from the database to the operation terminal to obtain an initialized set of components.

[0112] Among them, the database storing the component installation package can be obtained based on the database identifier carried in the initialization instruction. The database storing the component installation package can be a database that has a pre-established association relationship with the operation terminal or the control terminal. The operation terminal can directly determine the database storing the component installation package based on this association relationship, then download the component installation package from the database and install it to the operation terminal, so that the operation terminal obtains an initialized set of components, and further enables the operation terminal to use the set of components to record test case data for interface operation events and collect interface images when interface operation events are triggered, ensuring the accuracy of the acquired data. By dividing the work according to components, the efficiency and comprehensiveness of data acquisition can be effectively improved.

[0113] In one of the embodiments, the data processing method further includes:

[0114] Determine the operation sequence between the operation nodes represented by each interface operation event according to the respective triggering times of each interface operation event; perform serialization processing on the operation metadata describing each interface operation event in the test case data based on the operation sequence to obtain a serialized result of the operation metadata;

[0115] Determine the sequence relationship of the operation metadata describing each interface operation event in the test case data according to the operation sequence between the operation nodes represented by each interface operation event, including:

[0116] Determine the sequence relationship between the operation metadata based on the serialized result of the operation metadata.

[0117] Among them, the serialization of operation metadata can be implemented on the operation terminal or the control terminal, and can be specifically determined based on whether a component for serialization processing is deployed on the operation terminal. Correspondingly, the test case data obtained by the control terminal can be the data after serialization processing on the operation terminal. Among them, the serialization processing on the operation terminal can be directly serialized based on the recording order, which can simplify the processing process of serialization and improve the efficiency of serialization processing. The test case data obtained by the control terminal can also be the result obtained by the control terminal through self-serialization processing after receiving the test case data. By performing serialization processing on the control terminal, the data processing volume of the operation terminal can be reduced, the data processing pressure of the operation terminal can be reduced, and abnormal situations such as freezing can be avoided.

[0118] Furthermore, the serialized test case data can intuitively reflect the operation order between the operation nodes represented by each interface operation event. Based on the serialized test case data, the control terminal can directly determine the sorting result of the operation metadata as the sequence relationship between the operation metadata.

[0119] In this embodiment, the operation terminal determines the operation order between the operation nodes represented by each interface operation event according to the trigger time of each interface operation event, and then based on the operation order, serializes the operation metadata describing each interface operation event in the test case data to obtain the serialization result of the operation metadata, and then feeds back the serialization result of the operation metadata to the control terminal through the control tool, so that the control terminal can determine the sequence relationship between the operation metadata based on the serialization result of the operation metadata, which is convenient for mapping processing of the operation metadata and key interface elements in the use case knowledge graph, and improves the accuracy of the node data in the use case knowledge graph.

[0120] In one of the embodiments, serializing the operation metadata describing each interface operation event in the test case data based on the operation order to obtain the serialization result of the operation metadata includes:

[0121] Respectively obtain the operation metadata describing each interface operation event from the test case data;

[0122] Based on the operation order, serialize each operation metadata into a json object respectively to obtain the serialization result of the operation metadata.

[0123] Among them, each piece of data in the test case data corresponds to an operation node identifier representing an operation node. Based on the operation node identifier, the interface operation event described by the operation metadata can be determined. Therefore, the operation terminal or the control terminal can obtain a set of operation metadata corresponding to each interface operation event respectively based on the operation node identifiers corresponding to the data in the test case data. For each set of operation metadata, the operation terminal or the control terminal serializes each set of operation metadata into a json object respectively based on the operation sequence between the operation nodes represented by each interface operation event, and obtains the serialization result of the operation metadata.

[0124] By adding the operation metadata to the case knowledge graph as a json object, the structured data in json format represents a unique test case data. Among them, the jsonarray data related to the caseData field in the json object is mainly strongly related to the interface operation event.

[0125] In Figure 6 For the example json shown, due to space limitations, only a json data of the click type is shown. Taking this interface operation data of the click type as an example, after recording the test case data corresponding to the interface operation event, the json data will be serialized into the corresponding local data by the control tool framework, and then the corresponding rendering and mapping analysis processing will be carried out. In Figure 6 For the data shown, each piece of description data in the click_action data is used as the node data in the case knowledge graph. Specifically, the interface elements where the UI operations in the business system are located are unique. For example, the description data, location, size, page where it is located, process where it is located, etc. of the interface elements can all be used as the identifiers of their unique attributes. By making full use of this unique data feature, all the data collected during the UI recording process can be mapped and automatically analyzed for attributes.

[0126] Furthermore, the operation terminal or the control terminal can also serialize the UI data of the real-time operation on the Android device into local data with case characteristics, and at the same time save the serialized data to the cloud in a persistent form. The operation data is essentially a json form of each UI operation, and multiple UI operation data form a serializable test case data.

[0127] In this embodiment, the operation terminal or the control terminal serializes each group of operation metadata into a JSON object respectively based on the operation sequence between the operation nodes represented by each interface operation event, and obtains the serialization result of the operation metadata, which facilitates the storage of the operation metadata. At the same time, by using the JSON object to record the operation metadata, it is convenient to play back the test case data later, improving the integrity and reusability of the data record.

[0128] In one embodiment, the determination method of the triggered interface element in the interface image can be based on the trigger coordinates for element positioning. First, determine the position of the interface element, and then cut out the interface element from the interface image as the key interface element that matches the operation metadata. Through this processing method, the requirement for the processing ability of interface element segmentation can be reduced, and it has a wider applicability.

[0129] In another embodiment, the determination method of the triggered interface element in the interface image can be implemented based on interface element cutting. Specifically, the triggered interface element in the interface image that matches the interface operation event is determined as the key interface element that matches the operation metadata of the interface operation event, including:

[0130] Perform interface element cutting processing on the interface image that matches the interface operation event to obtain multiple candidate interface elements belonging to the image interface;

[0131] From the candidate interface elements, screen out the selected interface elements that match the trigger parameter information in the interface operation event;

[0132] Determine the selected interface element as the key interface element that matches the operation metadata of the interface operation event.

[0133] Among them, the interface element cutting processing refers to the process of cutting each element in the interface image into individual interface elements. The principle of performing interface element cutting processing on the interface image is to identify each interface element included in the interface image, and then cut the identified interface elements to obtain multiple candidate interface elements belonging to the image interface.

[0134] The trigger parameter information refers to the trigger parameters recorded for the interface operation event, including the trigger coordinates on the operation terminal and the coordinate information of the interface image on the operation terminal, such as the boundary position where the interface image is displayed on the operation terminal. Based on the coordinate information of the interface image on the operation terminal, the display range of each candidate interface element can be determined. By matching the trigger coordinates with the display range of each candidate interface element, the selected interface element that matches the trigger parameter information in the interface operation event can be obtained, and this selected interface element is the key interface element that matches the operation metadata of the interface operation event.

[0135] Furthermore, the interface element cutting process can be achieved by deploying the SAM image segmentation model. The SAM image segmentation model is an image cutting service built based on segment-anything of facebook. Through this powerful open-source image recognition service, all corresponding interface elements in the interface image can be well cut out. These interface elements are also one of the important components of the node data in the atlas database.

[0136] In this embodiment, the operation terminal or the control terminal can obtain multiple candidate interface elements of the image interface through the interface element cutting process, and determine the key interface elements that match the operation metadata of the interface operation event through the trigger parameter information of the interface operation event, which can further improve the accuracy of the segmented key interface elements and effectively improve the recognition efficiency of the key interface elements.

[0137] In one of the embodiments, the data processing method further includes the processing of multiple node data mapped to the same node in the use case knowledge graph. Specifically, the processing of multiple node data mapped to the same node in the use case knowledge graph includes:

[0138] Obtain multiple node data mapped to the same node from the atlas database storing the node data of each node in the use case knowledge graph;

[0139] Determine the target node data that meets the screening conditions from the node data of the same node;

[0140] Use the target node data as the updated node data of the node to obtain an updated use case knowledge graph.

[0141] Among them, the atlas database is a database used to store the node data mapped to each node in the use case knowledge graph. From the atlas database, multiple node data mapped to the same node can be obtained. As Figure 7 shown, the operation terminal recorded four test case data and wrote these four test case data into the atlas database; the operation nodes of these four test case data are respectively the following contents: The first one: Click the main page entrance -> Click on Battle to enter the secondary page -> Click on the King's Canyon mode; The second one: Click the main page entrance -> Click on Battle to enter the secondary page -> Click on Quick Match -> Click on the Matching mode; The third one: Click the main page entrance -> Click on Battle to enter the secondary page -> Click on the Changping Siege Warfare -> Click on the Matching mode; The fourth one: Click the main page entrance -> Click on Battle to enter the secondary page -> Mo's Mechanism Road -> Click on Create Room -> Click on the Matching mode.

[0142] As Figure 7As shown, all operation nodes in the above four use cases are parsed and mapped to node data in the graph database. Since the attributes of each node data, in addition to the id attribute that uniquely identifies it, are more related to the description data, position, size, page where it is located, process where it is located, etc. of the interface elements, these attributes combine to form its unique data characteristics. Therefore, precise aggregation and deduplication can be achieved. After the overall recording is completed, in addition to the original recorded use case data, a corresponding composite graph structure will also be automatically generated in the graph data structure, such as Figure 7 As shown, all use case steps will be mapped to the nodes in the graph structure and correspond one by one.

[0143] For these four test case data, for the node of "click on Battle to enter the secondary page", four mapped node data can be obtained. From these four node data, the target node data that meets the screening conditions can be determined as the target node data of the node of "click on Battle to enter the secondary page". Specifically, when the content of these four node data is exactly the same, one of them can be randomly selected as the node data of the node of "click on Battle to enter the secondary page". When the content of these four node data is not exactly the same, the different content can be analyzed and processed to screen out one node data that meets the pre-set screening conditions as the target node data of the node of "click on Battle to enter the secondary page".

[0144] In the case where node data already exists for the node of "click on Battle to enter the secondary page", the target node data can be used as the updated node data of the node to obtain an updated use case knowledge graph. In the case where node data already exists for the node of "click on Battle to enter the secondary page", the target node data can be directly used as the initial node data of the node, and the initial node data can be updated according to the node data newly mapped to this node later.

[0145] In this embodiment, by determining the target node data that meets the screening conditions from the node data of the same node and updating the data of the nodes in the use case knowledge graph based on the target node data, the timely update of the use case knowledge graph can be achieved. Through the above method, the test case data is serialized into a corresponding network graph structure, so that all the originally discrete and independent use case data in the same business system can be automatically identified and displayed accordingly, which can reduce a large amount of redundant use case data. For test case data with multiple highly similar repeated nodes, they can be globally displayed more clearly with a strongly related relationship. When some core key nodes change, it will be automatically synchronized to all graph nodes, thereby automatically updating the use case dataset, and can effectively handle various test scenarios such as version updates of the test object.

[0146] There are various ways to determine the target node data from the node data of the same node. For example, screening rules can be set for screening, or screening can be performed through data statistical results, etc. Specifically, it can be set according to actual needs.

[0147] In some embodiments, to determine the target node data that meets the screening conditions from the node data of the same node, it includes: obtaining the attribute description data from the node data with the same node identifier; the node identifier is used to represent the node in the use case knowledge graph; performing data aggregation processing on each attribute description data to obtain the target attribute description data that matches the node identifier; and determining the target node data of the node represented by the node identifier based on the target attribute description data.

[0148] Among them, the node identifier is an identifier used to represent the node in the use case knowledge graph, and the node data mapped to the same node will be marked with the same node identifier. Through the node identifier, the node data with the same node identifier can be obtained from the graph database, that is, the node data mapped to the same node. Based on the node identifier, the node data mapped to the same node can be obtained quickly and accurately, improving the data acquisition efficiency.

[0149] The attribute description data is operation metadata used to describe the operation-related attributes of the interface operation event. The data nature of the attribute description data can be variable data or fixed data. For each attribute description data, at least one of the methods of abnormal data elimination and data aggregation can be used for processing according to the data nature of each attribute description data, and the processing result is used as the target attribute description data that matches the node identifier. When the node data only contains attribute description data, the target attribute description data can be used as the target node data of the node represented by the node identifier. When the node data contains other specific data in addition to the attribute description data, the target attribute description data and the specific data can be used together as the target node data of the node represented by the node identifier.

[0150] In this embodiment, by performing data aggregation processing on each attribute description data, effective screening of the data can be achieved, obtaining the target attribute description data that matches the node identifier, and improving the accuracy of the target node data of the node represented by the node identifier.

[0151] In one embodiment, the operation metadata and the matching key interface elements are used as node data and mapped to the use case knowledge graph according to the sequence relationship to obtain the graph-structured test case data, including:

[0152] Taking the operation metadata and the matching key interface elements as node data and passing them transparently to the background service so that the background service can parse the node data to obtain the node parsing data;

[0153] Map the node parsing data to the use case knowledge graph according to the sequence relationship to obtain the graph-structured test case data.

[0154] Among them, the background service refers to a service with the function of parsing node data. Pass-through refers to a data transmission method that transmits the transmitted content from the source address to the destination address without making any changes to the business data content. By combining the pass-through with the data parsing function of the background service, the processing process of data parsing can be transferred to the background service for processing, the decomposition of the execution subject of data processing can be realized, the balance of resources required for data processing can be improved, and at the same time, the data processing ability of the background service can be more reasonably utilized to perform a deeper analysis of the node data, further improving the accuracy of the data mapped in the use case knowledge graph.

[0155] Further, taking the Django service as an example of the background service, the collected test case data and interface images are stored by the operating terminal in the form of operation metadata of images and json data, and the core data is passed through to the Django background. In the Django background, relevant apis based on the knowledge graph database capabilities are implemented, and the data generated in the process is parsed to obtain the node parsing data mapped to the use case knowledge graph.

[0156] This application also provides an application scenario that applies the above data processing method. Specifically, the application of the data processing method in this application scenario is as follows:

[0157] This application mainly provides a test case data based on real-time data stream and interface image serialization, and constructs a use case knowledge graph by parsing data in the form of pictures + operation metadata. Through the UI recording method, it is possible to accurately record UI data when performing normal operations completely on the mobile terminal, that is, the mobile terminal has the ability of instant UI recording. At the same time, during the recording process, the use case data in the process is automatically parsed into the corresponding knowledge graph structure data, so as to realize the automatic parsing and mapping of all use case data in the business project into graph-structured data. The above implementation provides great convenience and data intuitiveness for the test case data management of game projects.

[0158] Specifically, based on the capabilities of app_process and getevent on Android, and combined with image recognition technology, the UI data of real-time operations on Android devices can be serialized into local data with use case characteristics, and at the same time, it is saved to the cloud in a persistent form. The UI data is essentially a json form of each UI operation, and multiple UI operation data form a serializable test case data.

[0159] Based on the interface images collected in real time when the operation is triggered and the corresponding UI operation metadata, the real-time data analysis in these processes is abstracted into corresponding knowledge graph nodes. Each use case knowledge graph node essentially corresponds to a describable UI operation. The data of the use case knowledge graph node is comprehensively generated based on the current process page information, the event metadata description of the UI operation, and the key image elements where the UI operation is located. Relatively speaking, for a business system, the data of each node in the use case knowledge graph is basically unique and mutually exclusive. This gives the use case knowledge graph the ability to structure data, and all operations and corresponding elements in the UI operation process can be structured into a mesh data knowledge structure.

[0160] Based on the UI data collected by interface trigger events, it can be serialized into a corresponding mesh graph structure in the order of the trigger events, so that all previously discrete and independent use case data in the same business system can be automatically identified and displayed accordingly, which can reduce a large amount of redundant use case data and allow multiple use cases with highly similar repeated paths to be displayed more clearly globally with a strongly correlated relationship. When some of the core key nodes change, they will be automatically synchronized to all test cases through the node correlation, thereby automatically updating the use case data set.

[0161] By combining with UI recording and playback, the use case node data in all scenarios in the same business system can be automatically obtained when recording the use case. After the use case data is synchronized to the graph database and related service analysis, the node merger and use case data deduplication will be automatically performed, which provides a new idea for the project's use case data management. At the same time, during playback, it is possible to analyze whether the use case data nodes have changed based on the latest business system scenario at the time of playback. For example, if the UI elements have changed, the knowledge graph will prompt to use the latest UI elements to re-update all relevant use case data in the graph dataset. This can achieve the effect of automatically updating the use case data and reduce the cost of manual intervention.

[0162] The technical solution of the present application fully reuses the existing data of UI automation testing and explores a new use case knowledge graph from a graphical perspective in combination with the knowledge graph. This new use case data management strategy can make good use of the self-driving nature of UI automation and the uniqueness of scene UI elements, and analyzes and automatically classifies various use case node data from the above two perspectives.

[0163] In the technical solution of this application, all UI elements in the UI pages have their unique characteristics, such as corresponding coordinates, graphic elements in the corresponding UI area, the hierarchical structure of graphic elements, etc. When a UI operation processes the corresponding UI element, the operation type will also be serialized and parsed into the operation metadata of the corresponding UI element. All the above data is combined into a node with a unique identifier feature in the use case knowledge graph, which is another abstract expression of the use case step data.

[0164] The technical solution of this application further realizes in-depth data analysis based on the UI recording and playback technology. It combines the combination of app_process and getevent, and based on the image recognition ability of the opencv open source library, a general recording and playback solution for the terminal real machine environment is realized. Taking the operation terminal as the mobile terminal and the control terminal as the PC terminal as an example, a set of backend recording and playback engine sets are started on the mobile terminal through app_process. The getevent is used to monitor all operation data of the Android system in a specific time period in real time. After being parsed by algorithms and processed logically, these operation metadata are automatically recorded. At the same time, all interface image data will also be recorded in real time during the process. The opencv open source library is used to automatically identify the UI image area covered by the operation, and the general UI recording and playback technology is realized in a way that is completely comparable to the regular operations of the real machine.

[0165] In this process, all the data collected by all UI operations under the framework of this application, such as UI element images, UI operation metadata, UI operation types, etc., will be parsed and mapped to the data sources with unique identifier features in the knowledge graph database. In the data set stored in the knowledge graph, all data will correspond to the use case data stored in the recording. Each use case step data will be serialized into the corresponding structured data, and these structured data will be further mapped to the node data in the knowledge graph. When the recorded data is increasing, there will be multiple use case step data that actually map to the same node data in the knowledge graph, which is the aggregation of use case data. Through this aggregation, all test case data in the business system can be graphically structured well, and the impact of the change of the core step use case data on the global data can be intuitively analyzed.

[0166] Specifically, app_process is a system-level executable program pre-embedded in the Android system, located in the / system / bin / directory of the Android file system. The original application process zygote is also constructed by this executable program. Although ordinary user-level Apps are forked from the zygote process, various permissions are generally restricted in the initialization logic related to forking. Therefore, when starting an application through conventional means such as clicking on the application icon in the Android system, it is generally a process with restricted permissions. However, although the executable program of app_process cannot be utilized by ordinary App processes, it can be driven by the adb execution program on the PC side. So, as long as the adb execution program on the PC side sends a driving instruction to drive app_process, and then starts a Linux process through app_process, the started process can have system-level execution permissions.

[0167] When starting a process through app_process, the actual logic is different from the normal App startup logic. The process started in this way belongs to the native java process of the Linux system. Since app_process inherits from the init process with a pid of 1 by default, the process started in this form naturally has the shell permission capabilities corresponding to the init process. And the shell permission capabilities are the core capabilities that getevent in the recording function depends on.

[0168] When a process with shell permission capabilities is started through app_process, system-level shell commands can be executed through the process component of java in this process. Another core capability is to perform data monitoring and collection of operation events based on getevent. getevent is an Android-built executable program. By executing this program, kernel event data serialized in a specific format in the Android event system can be obtained in real time.

[0169] As Figure 8 shown, the main core part of the use case knowledge graph technology is divided into six modules, namely: PC-side tools, mobile component sets, SAM image recognition module, general Django background service, cos (Cloud Object Storage) data background, and the graph database KonisGraph. Among them, the module related to the knowledge graph is mainly the KonisGraph database. The other modules are basically used to generate relevant use case metadata for mapping analysis. The following will introduce these core modules one by one.

[0170] The PC - side tool module is the entry point for using the entire solution. Essentially, it is a tool implemented based on Electron. Its principle is based on a special compilation method of binary packaging, which can encapsulate the UI logic implemented based on front - end technologies such as HTML, CSS, and JS into an executable file for mainstream PC platforms. In the PC - side tool, an underlying basic framework is provided based on Electron, and then the UI of the recording and playback tool is implemented through the technology of React. And in the Node.js backend capabilities provided by Electron, the core app_process and getevent capabilities are started through adb commands.

[0171] The mobile - side component set module centrally encapsulates the implementation of the main mobile - side capabilities, mainly including a recording processing engine, a playback processing engine, an event collection module, an image recognition module, and a global monitoring module, etc. Essentially, these component modules are all Java backend execution programs, which are controlled and started by the PC - side based on app_process and continuously run in the Android system in the form of background processes. The recording processing engine will, after receiving relevant instructions from the PC - side tool, collect all event data during the recording process through getevent; the playback processing engine will parse the previously recorded and saved data, and then implement the injection of event playback through various shell commands. The image recognition module is implemented based on minicap, and its logic is controlled and scheduled by the engine object of the mobile - side component. When starting recording or playback, it will perform real - time screen image data collection based on minicap, and then upload relevant data or perform comparison and parsing through the cos cloud storage server; the event collection module is a special part. This module is encapsulated and implemented based on getevent. Essentially, it is also an independent shell subprocess that continuously polls and reads relevant event data in the background. These event data will be redirected to a specific buffer area. By reading the event data in the buffer area, and then through the socket thread, the entire process will be sent back to the tool process, and the tool process will then parse the collected data.

[0172] The data obtained by the above - mentioned mobile - side component set will be stored in the form of use - case metadata of image and JSON data, and the core data will be transmitted to the Django backend. In the Django backend, relevant APIs based on the knowledge graph database capabilities are implemented to parse the data generated during the process, and the data obtained by the mobile - side component set is one of the important components of the node data in the graph database.

[0173] The cos data backend is a module responsible for caching the use case data and image data generated during the process, and can reuse the cloud storage capabilities of the cos object storage in the cloud server. Specifically, the cos data backend is divided according to the access frequency and disaster tolerance level, and provides various storage types for the data to be cached of the test object. Among them, the cached data of the test object includes use case data and image data. Each storage type of the cos data backend has different characteristics, such as object access frequency, data persistence, data availability, and access latency, etc. You can choose which storage type to upload the data to be cached to the cos data backend according to your own scenario. The storage types provided by the cos data backend specifically include standard storage, low-frequency storage, intelligent tiered storage, intelligent tiered storage, standard storage, low-frequency storage, archive storage, deep archive storage, etc.

[0174] Specifically, standard storage is applicable to business scenarios such as real-time access to a large number of hot files and frequent data interactions. For example, when the test object is a hot video, social picture, mobile application, game program, static website, etc., the use case data and image data can be cached through the standard storage method of the cos data backend.

[0175] In the specific application process, the cos data backend also provides various storage types for the use case data and image data of different test objects, which can reflect the storage level and activity degree of different test objects in the cos data backend. In actual use, users can still modify the storage type of the use case data and image data of the test object according to actual needs, or sink the use case data and image data of the test object to a storage type with a lower activity degree to achieve the adaptive storage of the use case data and image data of different test objects.

[0176] The SAM image recognition module is an image cutting service built based on facebook's segment-anything. Through this powerful open-source image recognition service, all the corresponding UI element data in the UI operation can be well cut out, and these data are also one of the important components of the node data in the graph database.

[0177] Such as Figure 9As shown, the data and display of the use case knowledge graph will show the corresponding data in the area shown in the example figure. The nodes in the step shown in the figure are the node data in the graph dataset, which correspond to a step in the use case data, that is, an operation step. In the technical implementation solution of recording and playback, through getevent, minicap, and related recording engines, all the operation process data in the UI operation can be serialized into a use case data in a json structure. A json-formatted structured data represents a unique recorded use case data. Among them, the jsonarray data related to the caseData field is mainly strongly related to the UI operation. Taking this click-type UI operation data as an example, when the corresponding UI operation is recorded, this type of json data will be Figure 8 serialized into the corresponding local data by the framework sequence, and then the data is transparently transmitted to the KonisGraph graph database through the Django service for corresponding rendering and mapping analysis processing. All the description data in this click_action data will be used as the graph node attributes for the node data.

[0178] Generally speaking, the UI elements where the UI operations in the business system are located are unique. For example, the description data, location, size, page where it is located, process where it is located, etc. of the UI element can all be used as the identifiers of its unique attributes. This application makes full use of this unique data feature and maps and automatically analyzes all the data collected during the UI recording.

[0179] For each step data in each use case json, the framework will automatically parse out the core attribute data. For example, the page information, process information, location information, size information, description information, picture information, etc. listed in the figure. These data will be uniquely mapped to a unique node in the knowledge graph database and stored in the database as the attributes of the node. In this way, all the use case data in all business systems during the UI recording can be automatically parsed into unique nodes in the graph structure, and the lines between nodes depend on the relationships in the use case step. Generally speaking, sequential steps will form a series of node lines in the graph.

[0180] In summary, after using the tool provided by this solution, the Android mobile device can be directly operated through the UI entry on the PC side to record and playback the UI operation data, and then the UI elements corresponding to the use cases in the process will be automatically parsed and mapped into the node data in the knowledge graph structure and rendered. The overall process is as introduced before. According to the content introduced before, the basic working principle of the technical solution of this application is as follows:

[0181] This technical solution relies on the app_process capability to start the recording capability. Therefore, when using the capabilities of this technical solution, it is necessary to start the corresponding java service class in the control tool through the adb command on the PC side. This javaservice class will serve as the main entry point and start a socket local service in the form of a native java process. The ability to collect event system data by getevent also depends on this native java process and is encapsulated and started through the shell capability in the native java process.

[0182] After the PC-side tool is installed on the PC side, the PC-side tool will automatically install a tool apk on the mobile device in the relevant initialization logic of node.js. This apk is the collection of mobile components introduced above, which mainly includes the cos background service sdk capability, recording processing engine, playback processing engine, event collection module, image recognition module, and global monitoring module, etc.

[0183] After the tool is started on the PC side, various initializations will be automatically performed on the mobile device. The initialization process mainly includes downloading the mobile tool apk from the cos side, installing the apk, setting the adb forward port, starting the foreground service, and starting the app_process driver engine service, etc. After the services required by the tool are started, a foreground service associated with the cos side will be resident in the Android system. The Android sdk components of the cloud server cos side, and at the same time, the opencv framework on which image recognition depends is also integrated in the process where this foreground service is located. Therefore, in addition to starting the process services related to app_process each time, it is also necessary to start the foreground service.

[0184] When starting recording from the tool UI operation, the PC server will forward the request to the mobile service through adbforward in the socket communication mode. All the data generated by the mobile service during the entire recording process will be serialized into a json object, which mainly includes the use case name, use case creation time, an array of operation metadata, etc. During the recording process, the mobile side will continuously take screenshots based on minicap, parse and save the image data at the time point corresponding to each UI operation. All the metadata related to the images, such as which operation metadata it belongs to, the operation time, etc., will also be saved in the use case json object data. These two types of data will be transmitted to the background cloud storage through the cos interface together, and then the Django background-related knowledge graph rendering API will be triggered to parse and map the step data of these use cases into the node data in the graph structure.

[0185] Taking the game scenario as an example, when the control tool function starts recording scenario use cases, a total of four test case data are recorded in this embodiment; the steps of these four test cases are assumed to be the following content:

[0186] i. Click the main page entry -> Click on Battle to enter the secondary page -> Click on the King's Canyon mode;

[0187] ii. Click the main page entry -> Click on Battle to enter the secondary page -> Click on Quick Match -> Click on the Matching mode;

[0188] iii. Click the main page entry -> Click on Battle to enter the secondary page -> Click on the Changping Siege -> Click on the Matching mode;

[0189] iv. Click the main page entry -> Click on Battle to enter the secondary page -> Mo's Mechanism Road -> Click on Create Room -> Click on the Matching mode;

[0190] After successfully recording the above four test case data through the tool, the framework will automatically parse and map all the step data in the above four use cases into node data in the KonisGraph graph database. Since the attributes of each node data, in addition to its uniquely identified id attribute, are more related to the description data, location, size, page where it is located, process where it is located, etc. of the UI elements, these attribute combinations form its unique data characteristics. Therefore, accurate aggregation and deduplication can be achieved. After the overall recording, in addition to the original recorded use case data, a corresponding composite graph structure will be automatically generated in the graph data structure, and all the use case steps will be mapped to the nodes in the graph structure and correspond one by one.

[0191] Due to the characteristics of frequent human-computer interaction and close association between consecutive operations in the game scenario, in the development or testing stage of a game program or a certain function in a game program, by adopting the above data processing method to manage the test cases in the game scenario, the management efficiency of game test case data can be effectively improved. During the process of recording game test cases, all game use case data units will be automatically parsed and aggregated into the corresponding game test case data graph structure, effectively reducing the maintenance cost of test cases in the game scenario with complex operations and close operation associations.

[0192] When it is necessary to use game test case data to conduct functional tests on the game, based on the graph-structured game test case data, test case data can be quickly retrieved for testing. And by adjusting some parameters in the graph-structured game test case data, new test cases can be generated conveniently and efficiently to meet the requirements for test cases during the test process, effectively improving the test efficiency.

[0193] Using the solution of the present application, on any Android mobile device, after starting the tool apk provided by the present application through the adb command on the PC machine, the generalized UI recording and playback ability can be started on the corresponding Android mobile device. At the same time, during the process of recording test cases, all test case data units will be automatically parsed and aggregated into corresponding test case data graph structures. Therefore, this solution can support all UI automations on Android devices, make full use of the data during the process, and further structure it. For some scenarios that have long required a large amount of cost to maintain the test case set of the business system, this solution can combine the technologies and principles introduced above to solve the pain points of such test data management.

[0194] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps in other steps.

[0195] Based on the same inventive concept, the embodiments of the present application also provide a test case data processing device for implementing the data processing method involved above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the test case data processing device provided below can refer to the limitations on the data processing method in the above text, and will not be repeated here.

[0196] In one embodiment, as Figure 10 shown, a test case data processing device 1000 is provided, including: a data acquisition module 1002, a sequence relationship determination module 1004, an image processing module 1006, and a node mapping module 1008, where:

[0197] The data acquisition module 1002 is configured to acquire test case data recorded for a plurality of continuously triggered interface operation events on the operation terminal, and interface images respectively acquired when each of the interface operation events is triggered.

[0198] A sequence relationship determination module 1004 is configured to determine the sequence relationship of the operation metadata describing each of the interface operation events in the test case data according to the operation sequence between the operation nodes represented by each of the interface operation events.

[0199] An image processing module 1006 is configured to determine the key interface elements that match the operation metadata of the interface operation event from the triggered interface elements in the interface image that matches the interface operation event.

[0200] A node mapping module 1008 is configured to map the operation metadata and the matching key interface elements as node data to a use case knowledge graph according to the sequence relationship, so as to obtain graph-structured test case data.

[0201] In one embodiment, the data acquisition module 1002 is specifically configured to start a target process for a test object in the operation terminal based on a system-level program of the operation terminal, where the target process has shell permissions; call an event acquisition tool of the operation terminal based on the shell permissions of the target process to obtain interface operation events for the test object in the operation terminal; record test case data according to the interface operation events, and collect the interface images when the interface operation events are triggered.

[0202] In one embodiment, the data acquisition module 1002 is specifically configured to send a driving instruction of a system-level program to the operation terminal through a control tool of a control terminal, where the control tool is implemented based on a cross-platform desktop application framework; in response to the driving instruction, control the start of a target process for a test object through the system-level program of the operation terminal.

[0203] In one embodiment, the test case data processing device further includes an initialization module, configured to send a component initialization instruction to the operation terminal through a control tool of a control terminal; perform component initialization processing on the operation terminal based on the component initialization instruction to obtain an initialized component set; the data acquisition module 1002 is further configured to record test case data according to the interface operation events based on the recording engine component in the component set; collect the interface images when the interface operation events are triggered based on the image acquisition component in the component set.

[0204] In one embodiment, the initialization module is specifically configured to determine a database storing a component installation package in the case where the component indicated by the initialization instruction is not installed on the operation terminal; install the component installation package downloaded from the database to the operation terminal to obtain an initialized component set.

[0205] In one embodiment, the test case data processing device further includes a serialization processing module, configured to determine the operation sequence between the operation nodes represented by each of the interface operation events according to the respective trigger times of each of the interface operation events; based on the operation sequence, perform serialization processing on the operation metadata describing each of the interface operation events in the test case data to obtain a serialization result of the operation metadata; the sequence relationship determination module 1004 is specifically configured to determine the sequence relationship between the operation metadata based on the serialization result of the operation metadata.

[0206] In one embodiment, the serialization processing module is configured to respectively obtain the operation metadata describing each of the interface operation events from the test case data; based on the operation sequence, serialize each of the operation metadata into a json object to obtain a serialization result of the operation metadata.

[0207] In one embodiment, the image processing module 1006 is specifically configured to perform interface element cutting processing on the interface image matching the interface operation event to obtain a plurality of candidate interface elements belonging to the image interface; screen out the selected interface elements matching the trigger parameter information in the interface operation event from the candidate interface elements; determine the selected interface elements as the key interface elements matching the operation metadata of the interface operation event.

[0208] In one embodiment, the test case data processing device further includes a use case knowledge graph update module, configured to obtain a plurality of node data mapped to the same node from the graph database storing the node data of each node in the use case knowledge graph; determine target node data meeting the screening conditions from each of the node data of the same node; use the target node data as the updated node data of the node to obtain an updated use case knowledge graph.

[0209] In one embodiment, the use case knowledge graph update module is specifically configured to obtain attribute description data from the node data with the same node identifier; the node identifier is used to represent a node in the use case knowledge graph; perform data aggregation processing on each of the attribute description data to obtain target attribute description data matching the node identifier; based on the target attribute description data, determine the target node data of the node represented by the node identifier.

[0210] In one embodiment, the node mapping module 1008 is specifically configured to transparently transmit the operation metadata and the matched key interface elements as node data to the background service, so that the background service parses the node data to obtain node parsing data; and maps the node parsing data to the use case knowledge graph according to the sequence relationship to obtain graph-structured test case data.

[0211] The above test case data processing device records test case data on the operation terminal, obtains operation metadata describing each interface operation event and interface images respectively collected when each interface operation event is triggered, determines key interface elements matching the operation metadata of the interface operation event from the interface images, takes the operation metadata and the matched key interface elements as node data, and maps them to the use case knowledge graph according to the sequence relationship, so as to construct a use case data knowledge graph according to the forms of the key interface elements and the operation metadata. By recording test case data on the operation terminal, it is possible to accurately record data during normal operation on the operation terminal. At the same time, the use case data in the operation process is parsed into corresponding knowledge graph structure data, realizing the automatic parsing and mapping of test case data into graph-structured data, providing great convenience and data intuitiveness for test case data management, and thus effectively reducing the complexity of test case data management.

[0212] Each module in the above test case data processing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in hardware form or be independent of it, or be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0213] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 11As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. The computer program, when executed by the processor, implements a data processing method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0214] Those skilled in the art can understand that Figure 11 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0215] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0216] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0217] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0218] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0219] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0220] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0221] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A data processing method, characterized in that, the method includes: obtaining test case data recorded for a plurality of continuously triggered interface operation events on an operation terminal, and interface images respectively collected when each of the interface operation events is triggered; determining the sequence relationship of operation metadata describing each of the interface operation events in the test case data according to the operation sequence between operation nodes represented by each of the interface operation events; determining the interface elements triggered in the interface images matching the interface operation events as key interface elements matching the operation metadata of the interface operation events; taking the operation metadata and the matching key interface elements as node data, and mapping them to a use case knowledge graph according to the sequence relationship to obtain graph-structured test case data.

2. The method according to claim 1, characterized in that, the obtaining test case data recorded for a plurality of continuously triggered interface operation events on an operation terminal, and interface images respectively collected when each of the interface operation events is triggered, includes: starting a target process for a test object in the operation terminal based on a system-level program of the operation terminal, and the target process has shell permissions; invoking an event acquisition tool of the operation terminal based on the shell permissions of the target process to obtain interface operation events for the test object in the operation terminal; recording test case data according to the interface operation events, and collecting interface images when the interface operation events are triggered.

3. The method according to claim 2, characterized in that, the starting a target process for a test object in the operation terminal based on a system-level program of the operation terminal includes: sending a driving instruction of the system-level program to the operation terminal through a control tool of a control terminal, and the control tool is implemented based on a cross-platform desktop application framework; responding to the driving instruction, and controlling to start a target process for a test object through the system-level program of the operation terminal.

4. The method according to claim 3, characterized in that, the method further includes: sending a component initialization instruction to the operation terminal through a control tool of a control terminal; performing component initialization processing on the operation terminal based on the component initialization instruction to obtain an initialized component set; the recording test case data according to the interface operation events, and collecting interface images when the interface operation events are triggered, includes: recording test case data according to the interface operation events based on a recording engine component in the component set; collecting interface images when the interface operation events are triggered based on an image acquisition component in the component set.

5. The method according to claim 4, characterized in that, the performing component initialization processing on the operation terminal to obtain an initialized component set includes: when a component indicated by the initialization instruction is not installed on the operation terminal, determining a database storing a component installation package; installing the component installation package downloaded from the database to the operation terminal to obtain an initialized component set.

6. The method according to claim 1, wherein, the method further includes: determining the operation sequence between the operation nodes represented by each of the interface operation events according to the respective trigger times of each of the interface operation events; serializing the operation metadata describing each of the interface operation events in the test case data based on the operation sequence to obtain a serialization result of the operation metadata; The determining the sequence relationship of the operation metadata describing each of the interface operation events in the test case data according to the operation sequence between the operation nodes represented by each of the interface operation events includes: determining the sequence relationship between the operation metadata based on the serialization result of the operation metadata.

7. The method according to claim 6, wherein, the serializing the operation metadata describing each of the interface operation events in the test case data based on the operation sequence to obtain a serialization result of the operation metadata includes: respectively obtaining the operation metadata describing each of the interface operation events from the test case data; serializing each of the operation metadata into a json object based on the operation sequence to obtain a serialization result of the operation metadata.

8. The method according to claim 1, wherein, the determining the key interface element that matches the operation metadata of the interface operation event from the interface elements triggered in the interface image that matches the interface operation event includes: performing interface element cutting processing on the interface image that matches the interface operation event to obtain a plurality of candidate interface elements belonging to the image interface; screening out the selected interface elements that match the trigger parameter information in the interface operation event from the candidate interface elements; determining the selected interface element as the key interface element that matches the operation metadata of the interface operation event.

9. The method according to any one of claims 1 to 8, wherein, the method further includes: obtaining a plurality of node data mapped to the same node from the graph database storing the node data of each node in the use case knowledge graph; determining target node data that meets the screening conditions from the node data of the same node; using the target node data as the updated node data of the node to obtain an updated use case knowledge graph.

10. The method according to claim 9, wherein, the determining target node data that meets the screening conditions from the node data of the same node includes: obtaining attribute description data from the node data with the same node identifier; the node identifier is used to represent the node in the use case knowledge graph; performing data aggregation processing on each of the attribute description data to obtain target attribute description data that matches the node identifier; determining the target node data of the node represented by the node identifier based on the target attribute description data.

11. The method according to any one of claims 1 to 8, wherein, Mapping the operation metadata and the matched key interface elements as node data to a use case knowledge graph according to the sequence relationship to obtain graph-structured test case data, including: Taking the operation metadata and the matched key interface elements as node data and transparently transmitting them to a background service, so that the background service parses the node data to obtain node parsing data; Mapping the node parsing data to a use case knowledge graph according to the sequence relationship to obtain graph-structured test case data.

12. A test case data processing device, characterized in that the device includes: a data acquisition module, configured to acquire test case data recorded for a plurality of continuously triggered interface operation events on an operation terminal, and interface images respectively acquired when each of the interface operation events is triggered; a sequence relationship determination module, configured to determine the sequence relationship of operation metadata describing each of the interface operation events in the test case data according to the operation sequence between operation nodes represented by each of the interface operation events; an image processing module, configured to determine the triggered interface elements in the interface image matching the interface operation event as key interface elements matching the operation metadata of the interface operation event; a node mapping module, configured to map the operation metadata and the matched key interface elements as node data to a use case knowledge graph according to the sequence relationship to obtain graph-structured test case data.

13. A computer device, including a memory and a processor, where the memory stores a computer program, characterized in that when the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images