Methods and apparatuses for testing, clusters, media and products

Abstract

Embodiments of the present disclosure relate to a method and apparatus, a cluster, a medium and a product for testing. The method comprises determining a script similarity between a target test script and an associated test script based on a comparison between the target test script and the associated test script, the target test script and the associated test script indicating at least one of a training phase and an inference phase of an AI-based performance test. The method further comprises identifying test process features indicative of behavior characteristics and file characteristics on a cluster comprising a plurality of computing devices during a running of the target test script on the cluster, and determining a process similarity between the plurality of computing devices for the test process features. The method further comprises providing a test analysis of the cluster with respect to the target test script based on the script similarity and the process similarity. In this way, the entire test process can be comprehensively monitored at the training phase, and the test process is prevented from being tampered with in multiple dimensions, thereby improving the reliability of the test.

Description

Technical Field

[0001] The embodiments of this application generally relate to the field of computers, and more specifically, to methods and apparatus for testing, computing device clusters, computer-readable storage media, and computer program products. Background Technology

[0002] An intelligent computing cluster is a high-performance computing cluster designed and optimized for Artificial Intelligence (AI) computing tasks. It typically consists of a large number of computing nodes coupled together to work collaboratively, providing powerful computing capabilities to meet the computational demands of complex application scenarios. Such AI clusters can be used to process and analyze massive amounts of data, accelerate the training and inference processes of machine learning models, support the application of abstract algorithm models, and meet other high-performance computing needs. With the continuous development of AI technology and the sustained growth of intelligent computing demands, it is necessary to focus on the performance testing process for AI clusters. Summary of the Invention

[0003] This application provides a testing solution. This solution offers a secure testing strategy for AI clusters, enabling comprehensive monitoring of the entire performance testing process during the training phase and preventing tampering of the testing process from multiple dimensions, thereby avoiding test cheating and improving test reliability.

[0004] In a first aspect, a method for testing is provided. The method includes determining a script similarity between a target test script and at least one associated test script, based on a comparison between the target test script and at least one associated test script, wherein the target test script and the at least one associated test script at least indicate at least one of a training phase and an inference phase in an AI-based performance test. The method also includes identifying test process features indicating behavioral and file characteristics on a cluster during the execution of the target test script on a cluster comprising multiple computing devices, and determining process similarity among the multiple computing devices based on the test process features. The method further includes providing a test analysis of the cluster regarding the target test script based on the script similarity and process similarity. In this manner, the entire AI performance testing process can be controlled, and various forms of cheating and tampering can be prevented across multiple dimensions, including tasks and devices, thereby improving the security and reliability of the testing process.

[0005] In some embodiments of the first aspect, the target test script and each of the at least one test script corresponds to the same test benchmark and at least indicates the test model, test data, and test stopping conditions used for testing; and the target test script differs from the at least one test script in at least one of the following: AI framework, acceleration library, or programming language. The test benchmark indicates the test objectives, test elements, test requirements, etc., of all these test scripts, while the individual test scripts may differ in their specific implementations.

[0006] In some embodiments of the first aspect, determining script similarity includes: determining a first script similarity between the target test script and at least one test script by comparing the target abstract syntax tree corresponding to the target test script with the abstract syntax tree corresponding to at least one test script; determining a second script similarity between the target test script and at least one test script by comparing the target nesting relationship of the target test script with the nesting relationship of at least one test script using the call stack; and determining script similarity based on the first script similarity and the second script similarity. Thus, through static and dynamic code analysis of the test scripts, it is possible to ensure comprehensive and accurate coverage of the script tracking sequence in the test code itself corresponding to the test task and its nesting relationships, thereby achieving precise tracking of the testing process at the task level.

[0007] In some embodiments of the first aspect, the test process characteristics include behavioral characteristics. Identifying the test process characteristics includes: monitoring the behavioral characteristics of multiple computing devices, whereby the behavioral characteristics include at least one of the following: file upload, file modification, file deletion, login behavior, process creation, network connection establishment, command behavior, or privilege escalation; and extracting the timestamp, device name, source Internet Protocol (IP) address, and destination (IP) address for each behavioral characteristic. In this way, it is possible to comprehensively understand the activities of computing devices in the cluster during the testing process and identify key information for each behavior, thereby achieving precise tracking of machine characteristics and changes during the testing process at the device level.

[0008] In some embodiments of the first aspect, the method further includes: determining a sub-performance similarity of the respective computing devices for each behavioral feature by comparing performance metrics of each computing device responding to each behavioral feature; and determining the performance similarity of the respective computing devices relative to other computing devices based on the sub-performance similarity, wherein the performance metrics include at least one of the following: central processing unit (CPU) utilization, memory usage, request response time, or error rate. This allows for comparison of the performance responses of computing devices in the cluster to various behaviors during the testing process, effectively preventing cheating such as partial hardware device or configuration modifications during testing, thereby ensuring the fairness and authenticity of the test.

[0009] In some embodiments of the first aspect, the test process characteristics further include file characteristics. Identifying the test process characteristics includes monitoring the file characteristics of each of the multiple computing devices. The file characteristics include at least one of the following: an operation log indicating a sequence of file operations, or file attributes. This allows for a comprehensive understanding of the file status associated with the computing devices in the cluster during the test, thereby enabling precise tracking of file characteristics and changes during the test at the device level.

[0010] In some embodiments of the first aspect, the method further includes determining the file similarity of a given computing device relative to other computing devices by comparing file features of each of the plurality of computing devices. Thus, file anomalies can be captured through file comparisons between the devices to indicate anomalies in the respective devices.

[0011] In some embodiments of the first aspect, determining process similarity includes: determining process similarity based on performance similarity and file similarity. Thus, based on performance similarity characterizing the performance responses of multiple devices to various behaviors, and file similarity characterizing the file states of multiple devices during the testing process, the process similarity between the multiple computing devices during the testing process can be accurately determined in multiple dimensions.

[0012] In some embodiments of the first aspect, the method further includes: establishing a first feature matrix of the time series and a second feature matrix of the machine sequence based on unsupervised clustering analysis for script similarity; establishing a third feature matrix of the time series based on unsupervised clustering analysis for process similarity; and identifying test anomalies based on the first, second, and third feature matrices. Thus, by establishing matrix clustering models of time series and machine sequences, it is possible to indicate the similarities and differences between tasks and devices, and based on this, a feature space can be searched to provide accurate test analysis.

[0013] In some embodiments of the first aspect, identifying test anomalies includes: identifying elements in a first feature matrix, a second feature matrix, and a third feature matrix whose values ​​do not meet a predetermined threshold, wherein the identified elements indicate abnormal call operations or abnormal machine behavior. Thus, if the value of an element corresponding to a task or device in the feature matrix deviates significantly from the threshold, an anomaly can be determined at that task or device, thereby facilitating the detection of cheating or tampering.

[0014] In a second aspect, an apparatus for testing is provided. The apparatus includes a script similarity determination module configured to determine script similarity between a target test script and at least one associated test script based on a comparison between the target test script and at least one associated test script, wherein the target test script and the at least one associated test script at least indicate at least one of a training phase and an inference phase in an AI-based performance test. The apparatus also includes a process similarity determination module configured to identify test process features indicating behavioral and file characteristics on a cluster during the execution of the target test script on a cluster comprising multiple computing devices, and to determine process similarity among the multiple computing devices for these test process features. The apparatus further includes a test analysis module configured to provide test analysis of the cluster regarding the target test script based on script similarity and process similarity. In this manner, the entire AI performance testing process can be controlled, and various forms of cheating and tampering can be prevented across multiple dimensions, including tasks and machines, thereby improving the security and reliability of the testing process.

[0015] In some embodiments of the second aspect, the target test script and each of the at least one test script corresponds to the same test benchmark and at least indicates the test model, test data, and test stopping conditions used for testing; and the target test script differs from the at least one test script in at least one of the following: AI framework, acceleration library, or programming language. The test benchmark indicates the test objectives, test elements, test requirements, etc., of all these test scripts, while the specific implementations of the individual test scripts may differ.

[0016] In some embodiments of the second aspect, the script similarity determination module is further configured to: determine a first script similarity between the target test script and at least one test script by comparing the target abstract syntax tree corresponding to the target test script with the abstract syntax tree corresponding to at least one test script; determine a second script similarity between the target test script and at least one test script by comparing the target nesting relationship of the target test script with the nesting relationship of at least one test script using the call stack; and determine script similarity based on the first script similarity and the second script similarity. Thus, through static and dynamic code analysis of the test scripts, it is possible to ensure comprehensive and accurate coverage of the script tracking sequence in the test code itself corresponding to the test task and its nesting relationships, thereby achieving precise tracking of the testing process at the task level.

[0017] In some embodiments of the second aspect, where the test process characteristics include behavioral characteristics, the device further includes a behavioral characteristic identification module configured to: monitor the behavioral characteristics of multiple computing devices, the behavioral characteristics including at least one of the following: file upload, file modification, file deletion, login behavior, process creation, network connection establishment, command command behavior, or privilege escalation; and extract the timestamp, device name, source IP address, and destination IP address of each behavioral characteristic. In this way, it is possible to comprehensively understand the activities of computing devices in the cluster during the test process and identify key information for each behavior, thereby achieving precise tracking of machine characteristics and changes during the test process at the device level.

[0018] In some embodiments of the second aspect, the apparatus further includes: a sub-performance similarity determination module configured to determine the sub-performance similarity of a corresponding computing device for each behavioral feature by comparing performance metrics of each computing device among a plurality of computing devices in response to each behavioral feature; and a performance similarity determination module configured to determine the performance similarity of a corresponding computing device relative to other computing devices based on the sub-performance similarity. The performance metrics include at least one of the following: CPU utilization, memory usage, request response time, or error rate. This allows for comparison of the performance responses of computing devices in the cluster to various behaviors during the testing process, effectively preventing cheating such as partial hardware device or configuration modifications during testing, thereby ensuring the fairness and authenticity of the test.

[0019] In some embodiments of the second aspect, the test process characteristics further include file characteristics. The device also includes a file characteristic identification module configured to monitor the file characteristics of each of a plurality of computing devices. The file characteristics include at least one of the following: an operation log indicating a sequence of file operations, or file attributes. This allows for a comprehensive understanding of the file status associated with computing devices in the cluster during the test, thereby enabling precise tracking of file characteristics and changes during the test at the device level.

[0020] In some embodiments of the second aspect, the apparatus further includes a file similarity determination module configured to determine the file similarity of a given computing device relative to the other computing devices by comparing file features of each of the plurality of computing devices. Thus, file anomalies can be captured through file comparisons between the devices to indicate anomalies in the respective devices.

[0021] In some embodiments of the second aspect, the process similarity determination module is further configured to determine process similarity based on performance similarity and file similarity. Thus, based on performance similarity, which characterizes the performance responses of multiple devices to various behaviors, and file similarity, which characterizes the file states of multiple devices during the testing process, the process similarity between the multiple computing devices during the testing process can be accurately determined in multiple dimensions.

[0022] In some embodiments of the second aspect, the apparatus further includes: a first feature matrix building module configured to build a first feature matrix of the time series and a second feature matrix of the machine sequence based on unsupervised clustering analysis for script similarity; a second feature matrix building module configured to build a third feature matrix of the time series based on unsupervised clustering analysis for process similarity; and an anomaly identification module configured to identify test anomalies based on the first, second, and third feature matrices. Thus, by building matrix clustering models of the time series and machine sequences, the similarities and differences between tasks and devices can be indicated, and based on this, the feature space can be searched to provide accurate test analysis.

[0023] In some embodiments of the second aspect, the anomaly identification module is further configured to: identify elements in the first, second, and third feature matrices whose values ​​do not meet a predetermined threshold, wherein the identified elements indicate abnormal call operations or abnormal machine behavior. Thus, if the value of an element corresponding to a task or device in the feature matrix deviates significantly from the threshold, an anomaly can be determined at that task or device, thereby facilitating the detection of cheating or tampering.

[0024] According to a third aspect of this disclosure, a computing device is provided. The computing device includes a processor and a memory storing instructions that, when executed by the processor, cause the computing device to perform the method described according to a first aspect of this disclosure.

[0025] According to a fourth aspect of this disclosure, a computing device cluster is provided. The computing device cluster includes at least one computing device, each computing device including a processor and memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method described in the first aspect of this disclosure. In some embodiments, the computing device cluster includes a single computing device. In other embodiments, the computing device cluster includes multiple computing devices. In some embodiments, the computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0026] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program instructions that, when executed by a computing device or cluster of computing devices, cause the computing device or cluster of computing devices to perform the method described according to a first aspect of this disclosure.

[0027] According to a sixth aspect of this disclosure, a computer program product is provided. The computer program product includes instructions that, when executed by a computing device or cluster of computing devices, cause the computing device or cluster of computing devices to perform the method described in accordance with a first aspect of this disclosure. In some embodiments, the computer program product may include one or more software installation packages that can be downloaded or copied and executed on the computing device or cluster of computing devices when the method provided in the first aspect or its possible variations is required.

[0028] Please note that the Summary of the Invention is provided to introduce a series of concepts in a simplified form, which will be further described below in the Detailed Description. The Summary of the Invention is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0029] The above and other objects, features, and advantages of this disclosure will become clearer through a more detailed description of the embodiments thereof in conjunction with the accompanying drawings, in which:

[0030] Figure 1 This is a schematic diagram illustrating an exemplary environment in which methods and / or processes according to embodiments of the present disclosure may be implemented;

[0031] Figure 2 This is a schematic illustration of a flowchart of a method for testing according to an embodiment of the present disclosure;

[0032] Figure 3 The illustration shows a process for determining script similarity using static analysis according to an embodiment of the present disclosure;

[0033] Figure 4 The illustration shows a process for determining script similarity using dynamic analysis according to an embodiment of the present disclosure;

[0034] Figure 5 The illustration shows a process for determining the behavioral characteristics of a computing device according to an embodiment of the present disclosure;

[0035] Figure 6 The illustration shows a process for determining performance similarity according to an embodiment of the present disclosure;

[0036] Figure 7 The illustration shows a diagram of a document similarity determination process according to an embodiment of the present disclosure;

[0037] Figure 8 The illustration shows an example of a feature space for characterizing similarity differences according to an embodiment of the present disclosure;

[0038] Figure 9 A schematic block diagram of a testing apparatus according to an embodiment of the present disclosure is shown schematically;

[0039] Figure 10 A schematic block diagram of an example device is shown, which can be used to implement exemplary implementations of this disclosure.

[0040] Figure 11 A schematic block diagram of an example device cluster that can be used to implement exemplary implementations of this disclosure is shown; and

[0041] Figure 12 A schematic block diagram of another example device is shown, which can be used to implement exemplary implementations of this disclosure.

[0042] In all the accompanying drawings, the same or similar reference numerals usually indicate the same or similar elements. Detailed Implementation

[0043] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0044] In the description of embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0045] As mentioned above, intelligent computing clusters are high-performance computing clusters specifically designed and optimized for AI-related computing tasks. They can effectively meet the computing needs of various AI scenarios and are therefore called AI clusters. Generally, such clusters consist of a large number of computing nodes that are coupled together and work collaboratively to provide powerful computing capabilities, thus meeting the high computing power requirements of AI tasks. AI clusters have the ability to process and analyze massive amounts of data, accelerate the training and inference process of machine learning models, support the deployment of abstract algorithm models, and meet other high-performance computing needs.

[0046] Given the continuous advancement of AI technology and the ever-increasing demand for intelligent computing, the process of performance testing for AI clusters is particularly important. Excellent performance results will foster user acceptance of the cluster product. Cluster performance testing is a comprehensive and systematic undertaking, involving numerous factors and complex processes comprised of multiple sub-processes. Conventional solutions often involve manual review, but this is frequently costly and yields unsatisfactory results. Especially in AI scenarios, the testing difficulty increases significantly, and problems become more prominent. For example, the increased difficulty of large-scale model computation tasks may be due to the complexity of the computing environment, the richness of the software stack, and the diverse communication behaviors between processes and user groups.

[0047] To provide truly reliable test results, it is necessary to ensure that the task flow or timing sequence is consistent across the systems under test during the testing process, and it is expected that there will be no skipped steps. The test benchmark provides requirements for writing test code and the test flow, but may not require the test scripts to use a uniform hardware or software implementation. The test benchmark specifies the task flow of the test tasks, but the specific implementation of each test script corresponding to the test benchmark on the corresponding system may have the possibility of skipped steps. This may be due to one or more of the following: inherent characteristics of the programming language of the test scripts, device differences between AI servers, acceleration libraries, AI frameworks, etc. Under such degraded conditions, although each test may eventually reach the test endpoint, because some steps were not actually executed, a fair and realistic test performance evaluation cannot be formed.

[0048] Performance testing of AI clusters faces numerous challenges. For example, ensuring a secure testing environment is difficult, and the system under test may be uncontrollable. The construction and maintenance of a secure testing environment directly impacts whether the system under test can maintain a stable and controllable state. The AI ​​clusters to be tested can form testing environments with hundreds or even thousands of GPUs. "Hundreds" and "thousands" refer to large-scale computing cluster architectures composed of hundreds or thousands of graphics processing units (GPUs) or other computing units. Such complex testing environments consist of numerous hardware components and intricate software configurations, implying a degree of concealment. Modifications to local hardware devices or configurations within the AI ​​cluster during testing may go undetected. Conventional solutions often struggle to capture these hardware or configuration changes in real-time and comprehensively. Unauthorized hardware adjustments or configuration changes during testing will affect the accurate assessment of cluster performance. For instance, introducing higher-performance devices not specified in the testing benchmark can lead to "unexpected" performance test results for the AI ​​cluster.

[0049] As mentioned above, conventional solutions typically involve manual review, which is costly and ineffective. Reviewers not only need deep expertise but also must invest significant time and effort in meticulously examining scripts, logs, and other documents. Even so, the review results remain unsatisfactory, frequently resulting in omissions or misjudgments. Furthermore, conventional solutions fail to ensure strong consistency. For example, conflicts may arise between the development frameworks, acceleration libraries, and training scripts in the test system configuration and AI chips or Domain Specific Architecture (DSA) chips in terms of hardware-software coordination.

[0050] Current testing solutions for AI clusters, such as controlling requests on the load generation side (e.g., recording points when the load generator sends and receives requests), achieve relatively fair performance testing, which to some extent ensures the consistency of the point sequence. However, load generation only applies to the inference phase, while there are no related outputs during the training phase, making it impossible to achieve tamper-proof testing throughout the entire AI cluster testing process. Furthermore, while monitoring and verification at the beginning and end of the test process can contribute to the consistency of the point sequence, such solutions lack effective control over intermediate steps, thus failing to prevent cheating such as skipping steps. Similarly, current solutions also suffer from the shortcomings of manual review. For example, dispatching test verification experts for on-site or remote access review is insufficient to comprehensively monitor all behaviors and states on the cluster system. Due to the numerous components and complex interactions between computing nodes in AI clusters, relying on peer review based on manual review to check complete execution process logs and numerous test scripts with implementation differences is extremely inefficient. This results in the sampling validity of the current solutions failing to represent the effectiveness of test execution.

[0051] To address at least some of the aforementioned problems and other potential issues, a testing scheme according to embodiments of this disclosure is proposed. The method includes determining a script similarity between a target test script and at least one associated test script, based on a comparison between the target test script and at least one associated test script, wherein the target test script and the at least one associated test script at least indicate at least one of a training phase and an inference phase in an AI-based performance test. The method also includes identifying test process features indicating behavioral and file characteristics on a cluster during the execution of the target test script on a cluster comprising multiple computing devices, and determining process similarity among the multiple computing devices based on the test process features. The method further includes providing a test analysis of the cluster regarding the target test script based on the script similarity and process similarity. In this manner, the entire AI performance testing process can be controlled, and various forms of cheating and tampering can be prevented across multiple dimensions, including tasks and devices, thereby improving the security and reliability of the testing process.

[0052] The following is for reference. Figures 1 to 12 The present disclosure is provided to illustrate its basic principles and several exemplary implementations. It should be understood that these exemplary embodiments are given only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the disclosure in any way.

[0053] Figure 1 This is a schematic diagram illustrating an exemplary environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 1As shown, environment 100 includes multiple test scripts (e.g., Figure 1 The test scripts 111, 112, and 113, cluster 120, and test report 130 are illustrated schematically. Figure 1 As shown, cluster 120 can serve as the AI ​​cluster to be tested, schematically including computing devices 121, 122, and 123. These computing devices can be deployed in a distributed manner in the cloud for dynamic access or allocation of resources. The individual computing devices in such a cluster system may not be geographically located, but are coupled together through cloud computing or distributed technologies. It should be understood that... Figure 1 The description of the implementation environment is merely exemplary and not limiting, and appropriate environment configurations can be selected according to actual usage needs.

[0054] Each of the multiple test scripts runs separately on cluster 120 to evaluate its performance. The test scripts consist of a series of instructions that guide the test process. They facilitate the automation of the test process and the maintenance and reuse of test steps. The test scripts specify the test tasks and the test requirements for those tasks. Running the test scripts simulates operations and interactions to verify whether cluster 120 meets expectations in terms of accuracy, completeness, and stability for the test tasks. Test scripts are typically written by test engineers using specific programming languages ​​(such as C++, Python, etc.). The test scripts instruct a series of markers, which are actions that set markers at key nodes, operations, and steps during the test script's execution. Detailed information about the behavior and status during the test can be recorded at these markers for subsequent analysis and troubleshooting.

[0055] According to embodiments of this disclosure, in an AI scenario, the test script also involves training and inference of the AI ​​model used to perform the relevant test task (e.g., image classification task), selection and use of the dataset, and construction of the test platform (e.g., the AI ​​server equipment involved, basic software such as the operating system (OS), AI framework, acceleration library, etc.). For example, assuming the test task is an image classification task, the corresponding AI model can be a classification model for image classification (such as a support vector machine, neural network model, etc.). Embodiments of this disclosure also support performance testing of language models (Large Language Model (LLM)) on AI clusters.

[0056] Performance testing of a cluster system typically involves the following steps: 1) Selecting a test script; 2) Preparing the test environment by installing the required software and dependencies (such as deep learning frameworks, operating systems, etc.); 3) Configuring the evaluation tools, such as obtaining the testing tools and associated configuration files; 4) Preparing the dataset, such as acquiring and preparing the corresponding data according to the test requirements, and ensuring that the data preprocessing, format, and logs conform to the specifications; 5) Running the test task; 6) Submitting the test results, ensuring that the test termination conditions are met, and submitting according to the standard format and process; 7) Analyzing and comparing the results, such as comparing the performance metrics (such as latency, throughput, energy efficiency, etc.) if the task accuracy meets the standards.

[0057] It is important to note that in step 1), the test benchmark specifies the task flow and model type, as well as the corresponding dataset and stopping conditions. The test script corresponding to the test benchmark is implemented on the system under test (i.e., the hardware and software platform). Different test scripts corresponding to the same test benchmark may involve different AI server devices, basic software, AI frameworks, acceleration libraries, etc., which may result in predefined steps being omitted or skipped. Furthermore, in step 5), the test script is run on the AI ​​cluster under test, and during the run, data acquisition and transmission operations are performed according to the points marked in the test script. Inconsistent point sequences may distort the test results.

[0058] According to embodiments of this disclosure, cluster 120 may refer to a cloud-based computing and service model that couples computing resources together via a network. Cluster 120 is not merely a simple collection of resources, but a highly integrated, intelligently scheduled, and flexibly scalable environment. In this environment, key elements such as resources and applications are tightly coupled, allowing users to dynamically acquire, configure, and use these resources as needed without having to own or manage them locally.

[0059] Within the Cluster 120 framework, high-performance computing systems such as supercomputers and large-scale computing clusters are not only deployed efficiently, but also achieve a high level of flexibility and scalability. These high-performance computing resources, which were originally limited by high deployment costs and complex operation and maintenance management, are now seamlessly integrated into the cloud through cloud computing technology and elastic scheduling technology, forming a powerful and flexible computing ecosystem.

[0060] exist Figure 1In the example, computing devices 121, 122, and 123 included in cluster 120 can be coupled to each other via a network. Taking a server cluster as an example, the servers can interconnect to communicate, for example, to achieve data synchronization or sharing between servers. In this configuration scenario, the servers can process tasks in parallel through distributed computing. This parallel computing capability enables the cloud computing platform to handle more complex and massive tasks. Furthermore, each server in the cluster can have redundancy and fault tolerance mechanisms to ensure reliable task execution. When a node fails, the system can automatically migrate the job to other normally functioning nodes, ensuring task continuity and availability.

[0061] According to the performance testing of AI clusters in embodiments of this disclosure, in order to perform AI tasks, the AI ​​cluster to be tested can form a test environment with hundreds or even thousands of GPUs. "Hundreds of GPUs" and "thousands of GPUs" refer to a large-scale computing cluster architecture composed of hundreds or thousands of graphics processing units (GPUs) or other computing units. Examples of such "computing units" include, but are not limited to, central processing units (CPUs), tensor processing units (TPUs), field-programmable gate arrays (FPGAs), etc. In embodiments of this disclosure, the computing power corresponding to a "computing device" can be regarded as unit computing power, which can be regarded as the computing power provided by a single server, or the computing power provided by a single processing unit within a server, etc.

[0062] In the example configuration of cluster 120 as a server cluster, examples of computing devices or resource nodes may include a central server, an edge server, or a local server in a local data center. Besides the servers mentioned above, other devices with computing capabilities can serve as resource nodes in cluster 120, such as supercomputers, personal computers, laptops, in-vehicle computing devices, mobile devices (such as smartphones, tablets, etc.), wearable electronic devices, multimedia devices, personal digital assistants (PDAs), smart home devices, consumer electronics, or combinations thereof. It should be understood that the resource nodes described herein are merely exemplary and not limiting; for example, other different types of resource nodes may exist as technology evolves.

[0063] Test report 130 can be based on, for example, the performance (such as latency, throughput, energy efficiency, etc.) and status attributes of each test script (e.g., each of test scripts 111, 112, and 113) running on cluster 120. By comprehensively analyzing these performance indicators and status attributes, the test results can not only be used to determine whether a test is normal, suspicious, or invalid, but also further support the deductive analysis of the evidence chain, providing data support and theoretical basis for problem localization, system optimization, and improvement.

[0064] It should be understood that, for the purpose of ease of explanation and illustration, in Figure 1 The diagram only shows one cluster system (i.e., cluster 120) and three computing devices (i.e., computing devices 121, 122, and 123). However, this is merely an example; other cluster systems and computing devices with different configurations and numbers can exist, and they can be customized and arranged according to specific use cases and needs, as will be discussed below. The above, combined with... Figure 1 The following describes an example environment in which methods and / or processes according to embodiments of this disclosure may be implemented, in conjunction with... Figure 2 To describe a testing method 200 according to embodiments of the present disclosure.

[0065] Figure 2 This is a schematic illustration of a flowchart of a testing method 200 according to an embodiment of the present disclosure. (Based on...) Figure 2 The described method 200 provides a security testing strategy for AI clusters, which enables comprehensive monitoring during the training phase, closely tracks every step of the testing process, and prevents cheating and tampering during the testing process in multiple dimensions between and within the tested systems, thereby improving the authenticity and reliability of the test.

[0066] At point 210, based on a comparison between the target test script and at least one associated test script, a script similarity is determined between the target test script and at least one associated test script, wherein the target test script and at least one associated test script indicate at least one of the training and inference phases of an AI-based performance test. The test script specifies the test flow for the test task. Here, multiple associated test scripts for the test task have the same or similar test flows, wherein the target test script may be a test script subsequently prepared to be run on the AI ​​cluster. In the performance test for an AI cluster according to embodiments of this disclosure, the test script may cover the training phase of training the AI ​​model performing the test task, specifying key inputs, outputs, operation steps, etc., in the training phase. The test script may cover the inference phase of using the trained AI model for inference (such as classification, prediction, etc.) to verify the accuracy, stability, reliability, etc., of the inference results. According to embodiments of this disclosure, the test script covers both the training and inference phases of the model used to execute the test task; that is, it utilizes the AI ​​model to execute the entire process or complete flow of the test task. This is used to comprehensively monitor and evaluate the consistency and completeness of the test task's process sequence throughout the entire testing process, and to promptly identify and resolve potential problems. By comparing multiple related test scripts, task consistency can be guaranteed across the systems under test. If the test scripts differ significantly, it is necessary to promptly investigate whether there are inconsistencies in the testing process, missing steps, etc.

[0067] According to embodiments of this disclosure, the target test script and each of the at least one test script can correspond to the same test benchmark and at least indicate the test model, test data, and test stopping conditions used for testing. In other words, in performance testing of an AI cluster according to embodiments of this disclosure, the target test script and its associated test scripts can have the same test task flow, the same test model executing the AI ​​task, the same dataset (including training data, test data, validation data, etc.), and the same test stopping conditions (such as the model accuracy reaching a predetermined accuracy threshold). Although the target test script and its associated test scripts are generally consistent in test direction, test objectives, etc., they may differ in specific implementations due to, for example, hardware or software differences or conflicts. For example, the target test script and at least one test script may differ in at least one of the following: AI framework, acceleration library, or programming language, etc. By way of example and not limitation, multiple associated test scripts employ different deep learning frameworks (such as TensorFlow, PyTorch, etc.), different acceleration libraries (such as Deep Speed, Apex), and different programming languages ​​(such as C++, Python, etc.).

[0068] At point 220, during the execution of the target test script on a cluster comprising multiple computing devices, test process features indicating behavioral and file characteristics on the cluster are identified, and process similarity among the multiple computing devices is determined based on these test process features. Beyond the task level, performance testing of AI clusters according to embodiments of this disclosure also provides, for example, monitoring of the test process at the machine level using process group calls to prevent unauthorized hardware or configuration changes from distorting test results. Within the system under test, multiple computing devices collaborate to complete test tasks. For the same task flow, the states, attributes, etc., of multiple computing devices in the cluster should be identical or similar. Based on comparisons among multiple computing devices in the cluster, device consistency can be guaranteed within the system under test. In cases where there are significant differences between computing devices and other computing devices, it is necessary to promptly investigate whether there is hardware or configuration not specified by the test benchmark.

[0069] At point 230, based on script similarity and process similarity, test analysis of the cluster regarding the target test script is provided. After identifying trends or abrupt changes that deviate from normal behavior, the relationships between different data can be analyzed, for example, using algorithms such as graph search, to find key factors that may lead to anomalies. Logical connections between events can be established, for example, using causal graphs and dependency graphs. Based on this analysis, possible causes can be inferred, for example, using rule engines, decision trees, and deep learning. According to embodiments of this disclosure, script similarity can indicate task consistency between systems under test, and process similarity can indicate device consistency within systems under test. The combination of these two can prevent cheating and tampering during the testing process from multiple dimensions. This test analysis process can be unsupervised, thereby promoting automation of the testing process to eliminate human interference.

[0070] According to the testing method 200 of the embodiments of this disclosure, a security testing strategy for AI clusters is provided. Between systems under test, comparative analysis of multiple associated test scripts enables comprehensive monitoring of the entire testing process during the training phase, closely tracking the complete flow of the testing task. Furthermore, within the system under test, comparative analysis of multiple computing devices in the cluster can detect device anomalies, for example, in terms of behavior and file characteristics. Thus, cheating and tampering during the testing process can be effectively detected and prevented at multiple dimensions, both between and within the systems under test, aiming to fundamentally improve the authenticity of the test and ensure accurate and reliable test results.

[0071] According to embodiments of this disclosure, consistency of task flows can be ensured between systems under test by comparing multiple associated test scripts. For example, the more missing steps in a test script's test task flow, the lower its similarity to other test scripts. To ensure consistency of task sequences, the script similarity of test scripts needs to be explicitly defined.

[0072] Figure 3 The illustration shows a script similarity determination process 300 using static analysis according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, a first script similarity (i.e., static script similarity) between the target test script and at least one test script can be determined by comparing a target abstract syntax tree corresponding to a target test script and an abstract syntax tree corresponding to at least one test script. Figure 3 As shown, an abstract syntax tree 312 can be extracted from the target test script 310, and a corresponding abstract syntax tree 322 can be extracted for one or more associated test scripts 320. It should be understood that, for ease of understanding and illustration, only one abstract syntax tree is shown at abstract syntax tree 322.

[0073] An Abstract Syntax Tree (AST) is a tree-like structure used to represent program source code. It describes the syntactic structure of the source code in an abstract way, where each node represents a construct (e.g., expression, statement, declaration) in the source code. It's called "abstract" because such a syntax tree doesn't represent every detail of the actual syntax of the source code, but rather serves as an intermediate representation during the compilation or parsing phase, using higher-level representations such as "if-then". ASTs are typically obtained using programming language-specific tools or libraries. Code analysis based on ASTs possesses static characteristics. When comparing the target AST of a test script with those of other test scripts, the syntactic structure of the code can be compared, such as the order of statements and variable declarations. Thus, even if multiple related test scripts differ in programming language, because they share the same or similar test flows, the task sequence can still be accurately identified using ASTs, effectively capturing task exceptions in the test scripts and preventing the skipping of critical steps.

[0074] Figure 4 The illustration shows a diagram of a script similarity determination process 400 using dynamic analysis according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, a second script similarity (i.e., dynamic script similarity) between the target test script and at least one test script can be determined by comparing the target nesting relationship of the target test script with the nesting relationship of the at least one test script using the call stack. Figure 4 As shown, the target nesting relationship 412 of the target test script 310 and the corresponding nesting relationship 422 of one or more associated test scripts 320 can be obtained through the call stack.

[0075] Call stacks can be configured to manage function calls during code execution. They are a first-in, first-out (FIFO) data structure used to store function call information. By analyzing call stacks, the sequence of function calls during code execution can be traced, revealing the program's execution flow and dependencies between functions. The dynamic nature of call stack-based code analysis means it is closely related to the real-time state and execution process of the code. Test scripts may contain complex function calls and nested structures. Dynamic analysis using call stacks allows for a more accurate assessment of the similarity between test scripts, further ensuring the consistency and integrity of the task sequence.

[0076] According to embodiments of this disclosure, script similarity can be determined based on a first script similarity and a second script similarity. In some embodiments, different weights can be set for static script similarity and dynamic script similarity. For the same test task, the static features of each test script should be substantially consistent, and the dynamic features should be highly similar. In the script static analysis and script dynamic analysis of embodiments of this disclosure, anomalies in the task sequence can be easily detected based on the singular values ​​of static script similarity and dynamic script similarity. Among the systems under test, by comprehensively considering the static script similarity and dynamic script similarity of multiple related test scripts, it is possible to ensure that the test task is complete and consistent without missing or skipping key steps in the test code itself and its internal nesting relationships, thereby achieving detailed tracking and precise control of the test process at the task level.

[0077] Figure 5 The illustration shows a process 500 for determining the behavioral characteristics of a computing device according to an embodiment of this disclosure. Above, the discovery of anomalies at the task dimension was described through static and dynamic analysis of the test scripts. Below, strategies to avoid cheating and tampering in the machine-level testing process will be discussed. As mentioned above, AI clusters are large-scale computing cluster architectures composed of numerous hardware components and complex software configurations. Performance testing of AI clusters covers complex testing environments. During testing, modifications to local hardware devices or configurations within the cluster may go undetected, leading to distorted test results.

[0078] According to embodiments of this disclosure, at point 510, the behavioral characteristics of multiple computing devices can be monitored. These behavioral characteristics include at least one of the following: file upload, file modification, file deletion, login behavior, process creation, network connection establishment, command behavior, or privilege escalation. When the target test script runs on the AI ​​cluster, multiple computing devices within the system under test collaborate to complete the test task. For the same task flow, the states, attributes, etc., of the multiple computing devices in the cluster should be the same or similar. For example, during a file upload operation, the request response times of the multiple computing devices in the cluster should be the same or similar. If the request response time of a computing device is too fast or too slow relative to other computing devices, then the hardware or configuration of that computing device is likely to have been tampered with. In some embodiments, the behavioral characteristics of multiple computing devices are detected through process group calls. It should be understood that the above behavioral characteristics are merely exemplary, and other different host behaviors of computing devices can be detected according to specific test requirements.

[0079] To gain a more comprehensive and accurate understanding of each behavioral feature, at point 520 in the monitoring of behavioral features from multiple computing devices, the timestamp, device name, source IP address, and destination IP address of each behavioral feature can be extracted for subsequent analysis. The timestamp precisely records the exact moment the behavior occurred, which can be used to analyze the sequence and temporal relationships of the behaviors. The device name clearly identifies which computing devices or devices the behavior occurred on, facilitating device-level research. Furthermore, the source and destination IP addresses need to be extracted. These IP addresses clearly show the flow of the behavior at the network layer and the interacting objects, thus facilitating better identification of behavioral features. It should be understood that the information items extracted or represented for the behavioral features described above are merely exemplary and not restrictive. Other different information items can be extracted for each behavioral feature to meet the needs of actual testing and analysis.

[0080] Figure 6 The illustration shows a performance similarity determination process 600 according to an embodiment of the present disclosure. For each monitored behavioral feature, the performance response of each computing device in the AI ​​cluster to the corresponding behavioral feature can be determined. In some embodiments, monitoring of performance metrics can be performed by configuring thread groups. Such performance metrics may include, but are not limited to, CPU utilization, memory usage, request response time, or error rate. It should be understood that other different performance metrics of the computing devices can be detected according to specific testing requirements.

[0081] According to embodiments of this disclosure, at point 610, the sub-performance similarity of the respective computing devices for each behavioral feature is determined by comparing the performance metrics of each computing device in response to each behavioral feature. By way of example, for login behavior, the CPU utilization, memory usage, request response time, or error rate of each computing device in the AI ​​cluster at or during login can be determined. Accordingly, by comparing these performance metrics of each computing device in response to login behavior (i.e., comparison between devices), the sub-performance similarity of each computing device for login behavior can be determined. The process for determining the sub-performance similarity of other behavioral features is similar and will not be described further here. In some embodiments, considering that different performance metrics have different rates of change during each behavioral feature, weights can be customized for the selected performance metrics during comparison.

[0082] At point 620, the performance similarity of the corresponding computing device relative to other computing devices is determined based on sub-performance similarity. By comprehensively considering the performance of each computing device in the cluster system for various behavioral characteristics (characterized by the sub-performance metric determined at point 610), the performance similarity between computing devices can be finally determined. This allows for a comparison of the performance feedback of computing devices within the cluster system for various behaviors during testing, effectively preventing cheating caused by local hardware changes or configuration adjustments during testing, thus ensuring the fairness and authenticity of the test.

[0083] Figure 7 The illustration shows a file similarity determination process 700 according to an embodiment of the present disclosure. In addition to detecting from the perspective of the performance of each device, it is also possible to determine whether there are device or machine-level anomalies by examining the file characteristics of each device. To achieve machine-level anomaly detection from a file perspective, the key lies in monitoring the file characteristics of each computing device in the AI ​​cluster.

[0084] According to embodiments of this disclosure, at point 710, file characteristics of each of the multiple computing devices can be monitored. These file characteristics include at least one of the following: operation logs indicating a file operation sequence, or file attributes. When the target test script runs on the AI ​​cluster, multiple computing devices within the system under test collaborate to complete the test task. Therefore, the log files of each computing device, or files corresponding to the test task, have the same or similar recorded content and operation characteristics. For example, such operation characteristics may include, but are not limited to, log retention period, file refresh frequency, file attribute conversion, intermediate file generation, etc. By knowing the similarity in the file characteristics of the computing devices, a comprehensive understanding of the changing trends of operations, attributes, etc., of each computing device in the cluster system at the file level can be achieved. Based on this, at the device level, accurate tracking of the characteristics and dynamic changes of files during the testing process can be realized.

[0085] At point 720, the file similarity of a given computing device relative to the others is determined by comparing the file characteristics of each of the multiple computing devices. If the file characteristics of a computing device in the cluster system deviate significantly from those of other computing devices, it can be considered that the computing device is highly likely to be abnormal. In this way, abnormal changes at the file level during the operation of computing devices can be detected acutely, allowing for rapid and effective indication of anomalies on the corresponding computing devices.

[0086] According to embodiments of this disclosure, process similarity can be determined based on performance similarity and file similarity. Within the system under test, by comparing and analyzing multiple computing devices in a cluster system, device anomalies can be keenly observed, for example, in dimensions such as behavior and file characteristics. Based on performance similarity, which characterizes the performance responses of multiple devices to various behaviors, and file similarity, which characterizes the file states of multiple devices during the testing process, the process similarity between the multiple computing devices during the testing process can be accurately determined in multiple dimensions.

[0087] Figure 8 The illustration shows an example 800 of a feature space for characterizing similarity differences according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, for script similarity, a first feature matrix of the time series and a second feature matrix of the machine sequence can be established using unsupervised clustering analysis; for process similarity, a third feature matrix of the time series can be established using unsupervised clustering analysis. Thus, test anomalies can be identified based on the established first, second, and third feature matrices. In some embodiments, such unsupervised clustering analysis can be performed using an unsupervised algorithm (such as the density-based spatial clustering algorithm DBSCAN). Based on the clustering results, the feature space can be searched to provide test result judgments, confidence assessments, etc.

[0088] According to embodiments of this disclosure, elements whose values ​​do not meet a predetermined threshold are identified in the first, second, and third feature matrices, wherein the identified elements indicate abnormal call operations or abnormal machine behavior. Here, the predetermined threshold may be the average similarity difference between multiple associated test scripts or multiple computing devices, and failure to meet the predetermined threshold may, for example, indicate that the absolute value of the value differs too much from the average value.

[0089] like Figure 8 As shown, by way of example and not limitation, the first feature matrix (1) in Example 800 can be a feature matrix that indicates the similarity of task flow between test scripts over time. The rows of the matrix can represent the similarity differences between multiple associated test scripts at a given time, and the columns can represent the progression of time. Through normalization, larger values ​​in the feature matrix indicate greater distance from the cluster center and more significant differences, while smaller values ​​indicate closer proximity to the cluster center and less significant differences. For example, in the first feature matrix (1), the second row indicates that at time 2, the third test script among multiple associated test scripts shows significant differences, with a higher risk of skipping steps, missing steps, etc.

[0090] By way of example and not limitation, the second feature matrix (2) in Example 800 may be a feature matrix indicating the similarity of task flows of test scripts across a machine (or, host) sequence. In the case of multi-machine collaborative operation, multiple computing devices in a cluster system will cooperate with each other for the same task flow. The rows of the matrix may represent the similarity differences between multiple associated test scripts on that computing device, and the columns of the matrix may represent the machine sequence. For example, in the second feature matrix (2), the third row indicates that at the third computing device, there is a significant difference among the multiple associated test scripts, and the risk of task sequence variation is high.

[0091] For example, the third feature matrix (3) in Example 800 could be a feature matrix that indicates the similarity of performance and / or file characteristics of various computing devices over a time series for the test process. The rows of the matrix could represent the process similarity differences between multiple computing devices at a given time, and the columns could represent the time series. For instance, in the third feature matrix (3), the fifth row indicates that at time 5, the second computing device among the multiple computing devices in the cluster system has a high probability of device tampering.

[0092] According to embodiments of this disclosure, similarities and differences between tasks and devices can be indicated by establishing matrix clustering models of time series and machine sequences. Based on this, a feature space can be searched to provide accurate test analysis. In the feature matrix, if the value of an element corresponding to a task or device deviates significantly from a threshold, an anomaly can be determined at that task or device, thereby facilitating the detection of cheating or tampering.

[0093] Figure 9 An apparatus 900 for testing according to some embodiments of the present disclosure is shown. Each module in the apparatus 900 can be configured to perform corresponding steps in a method for testing according to embodiments of the present disclosure. Figure 9As shown, the testing apparatus 900 includes a script similarity determination module 910, a process similarity determination module 920, and a test analysis module 930. According to embodiments of this disclosure, the apparatus 900 includes a script similarity determination module 910 configured to determine script similarity between a target test script and at least one associated test script based on a comparison between the target test script and at least one associated test script, wherein the target test script and the at least one associated test script at least indicate at least one of a training phase and an inference phase in an AI-based performance test. The apparatus 900 also includes a process similarity determination module 920 configured to identify test process features indicating behavioral and file characteristics on a cluster during the execution of the target test script on a cluster comprising multiple computing devices, and to determine process similarity among the multiple computing devices for the test process features. The apparatus 900 also includes a test analysis module 910 configured to provide test analysis of the cluster regarding the target test script based on script similarity and process similarity. This approach allows for control over the entire AI performance testing process and prevents various forms of cheating and tampering across multiple dimensions, including tasks and machines, thereby improving the security and reliability of the testing process.

[0094] In some embodiments, the target test script and each of the at least one test scripts corresponds to the same test benchmark and at least indicates the test model, test data, and test stopping conditions used for testing; and the target test script differs from at least one of the following: AI framework, acceleration library, or programming language. The test benchmark indicates the test objectives, test elements, test requirements, etc., of all these test scripts, while the specific implementations of the individual test scripts may differ.

[0095] In some embodiments, the script similarity determination module 910 is further configured to: determine a first script similarity between the target test script and at least one test script by comparing the target abstract syntax tree corresponding to the target test script with the abstract syntax tree corresponding to at least one test script; determine a second script similarity between the target test script and at least one test script by comparing the target nesting relationship of the target test script with the nesting relationship of at least one test script using the call stack; and determine script similarity based on the first script similarity and the second script similarity. Thus, through static and dynamic code analysis of the test scripts, it is possible to ensure comprehensive and accurate coverage of the script tracking sequence in the test code itself corresponding to the test task and its nesting relationships, thereby achieving precise tracking of the testing process at the task level.

[0096] In some embodiments, where the test process characteristics include behavioral characteristics, the apparatus 900 further includes a behavioral characteristic identification module configured to: monitor the behavioral characteristics of multiple computing devices, the behavioral characteristics including at least one of the following: file upload, file modification, file deletion, login behavior, process creation, network connection establishment, command behavior, or privilege escalation; and extract the timestamp, device name, source Internet Protocol (IP) address, and destination (IP) address for each behavioral characteristic. In this way, it is possible to comprehensively understand the activities of computing devices in the cluster during the test process and identify key information for each behavior, thereby achieving precise tracking of machine characteristics and changes during the test process at the device level.

[0097] In some embodiments, the apparatus 900 further includes: a sub-performance similarity determination module configured to determine the sub-performance similarity of a corresponding computing device for each behavioral feature by comparing performance metrics of each computing device among a plurality of computing devices in response to each behavioral feature; and a performance similarity determination module configured to determine the performance similarity of a corresponding computing device relative to other computing devices based on the sub-performance similarity. The performance metrics include at least one of the following: central processing unit (CPU) utilization, memory usage, request response time, or error rate. This allows for comparison of the performance responses of computing devices in the cluster to various behaviors during the testing process, effectively preventing cheating such as partial hardware device or configuration modifications during testing, thereby ensuring the fairness and authenticity of the test.

[0098] In some embodiments, the test process characteristics further include file characteristics, and the apparatus 900 further includes a file characteristic identification module configured to monitor the file characteristics of each of a plurality of computing devices. The file characteristics include at least one of the following: an operation log indicating a sequence of file operations, or file attributes. This allows for a comprehensive understanding of the file status associated with computing devices in the cluster during the test, thereby enabling precise tracking of file characteristics and changes during the test at the device level.

[0099] In some embodiments, the apparatus 900 further includes a file similarity determination module configured to determine the file similarity of a given computing device relative to other computing devices by comparing file features of each of the plurality of computing devices. This allows file anomalies to be captured through file comparisons between the devices to indicate anomalies in the respective devices.

[0100] In some embodiments, the process similarity determination module 920 is further configured to determine process similarity based on performance similarity and file similarity. Thus, based on performance similarity, which characterizes the performance responses of multiple devices to various behaviors, and file similarity, which characterizes the file states of multiple devices during the testing process, the process similarity between the multiple computing devices during the testing process can be accurately determined in multiple dimensions.

[0101] In some embodiments, the apparatus 900 further includes: a first feature matrix building module configured to build a first feature matrix of the time series and a second feature matrix of the machine sequence based on unsupervised clustering analysis for script similarity; a second feature matrix building module configured to build a third feature matrix of the time series based on unsupervised clustering analysis for process similarity; and an anomaly identification module configured to identify test anomalies based on the first, second, and third feature matrices. Thus, by building matrix clustering models of the time series and machine sequences, the similarities and differences between tasks and devices can be indicated, and based on this, the feature space can be searched to provide accurate test analysis.

[0102] In some embodiments, the anomaly identification module is further configured to: identify elements in the first, second, and third feature matrices whose values ​​do not meet a predetermined threshold, wherein the identified elements indicate abnormal call operations or abnormal machine behavior. Thus, if the value of an element corresponding to a task or device in the feature matrix deviates significantly from the threshold, an anomaly can be determined at that task or device, thereby facilitating the detection of cheating or tampering.

[0103] As an example of a software functional unit, the script similarity determination module 910 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the script similarity determination module 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0104] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0105] As an example of a hardware functional unit, the script similarity determination module 910 may include at least one computing device, such as a server. Alternatively, the script similarity determination module 910 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0106] The script similarity determination module 910 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the script similarity determination module 910 includes multiple computing devices that can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the script similarity determination module 910 includes multiple computing devices that can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0107] It should be noted that, in other embodiments, the script similarity determination module 910 can be used to execute any step in the testing method according to embodiments of the present disclosure, the process similarity determination module 920 can be used to execute any step in the testing method according to embodiments of the present disclosure, and the test analysis module 930 can be used to execute any step in the testing method according to embodiments of the present disclosure. Furthermore, the aforementioned behavior feature identification module, sub-performance similarity determination module, performance similarity determination module, file feature identification module, file similarity determination module, first feature matrix establishment module, second feature matrix establishment module, and anomaly identification module can also be used to execute any step in the testing method according to embodiments of the present disclosure.

[0108] The steps implemented by the script similarity determination module 910, process similarity determination module 920, test analysis module 930, behavior feature identification module, sub-performance similarity determination module, performance similarity determination module, file feature identification module, file similarity determination module, first feature matrix establishment module, second feature matrix establishment module, and anomaly identification module can be specified as needed. The script similarity determination module 910, process similarity determination module 920, test analysis module 930, behavior feature identification module, sub-performance similarity determination module, performance similarity determination module, file feature identification module, file similarity determination module, first feature matrix establishment module, second feature matrix establishment module, and anomaly identification module respectively implement different steps in the testing method according to the embodiments of this disclosure to realize all the functions of the testing apparatus 900.

[0109] This disclosure also provides a computing device 1000. Figure 10 A schematic block diagram of an example computing device 1000, which can be used to implement exemplary implementations of the present disclosure, is illustrated. The computing device 1000 can be used to perform a method for testing according to embodiments of the present disclosure. Figure 10 As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that this disclosure does not limit the number of processors and memories in the computing device 1000.

[0110] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1002 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1002 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0111] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0112] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0113] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the aforementioned script similarity determination module 910, process similarity determination module 920, test analysis module 930, etc., thereby realizing the testing method according to the embodiments of this disclosure. That is, the memory 1006 stores instructions for executing the testing method according to the embodiments of this disclosure.

[0114] Alternatively, the memory 1006 may store executable code, which the processor 1004 executes to implement the functions of the aforementioned testing apparatus 1100, thereby implementing the testing method according to an embodiment of the present disclosure. That is, the memory 1006 stores instructions for executing the testing method according to an embodiment of the present disclosure.

[0115] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0116] This disclosure also provides a computing device cluster. Figure 11 A schematic block diagram of an example computing device cluster 1100 that can be used to implement exemplary implementations of the present disclosure is shown. The computing device cluster 1100 enables the methods provided in this disclosure for testing to be executed in a distributed manner by multiple servers within the cluster. The computing device cluster 1100 includes at least one computing device. This computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0117] like Figure 11 As shown, the computing device cluster 1100 includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster 1100 may store the same instructions for performing a testing method according to embodiments of this disclosure.

[0118] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster 1100 may also store partial instructions for executing the testing method according to embodiments of the present disclosure. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the testing method according to embodiments of the present disclosure.

[0119] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster 1100 can store different instructions, which are used to execute certain functions of the testing device 1100. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules such as the script similarity determination module 910, the process similarity determination module 920, and the test analysis module 930.

[0120] In some possible implementations, one or more computing devices in the computing device cluster 1100 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 One possible implementation is shown. For example... Figure 12As shown, the two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1006 in computing device 1000A stores instructions for executing the functions of the script similarity determination module 910 and the process similarity determination module 920. Simultaneously, the memory 1006 in computing device 1000B stores instructions for executing the functions of the test analysis module 930.

[0121] Figure 12 The connection method between the computing device cluster 1100 shown may be that, considering that the testing method provided in this disclosure may be for distributed storage devices, the functions implemented by the test analysis module 930 are to be executed by the computing device 1000B.

[0122] It should be understood that, Figure 12 The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0123] This disclosure also provides another computing device cluster 1100. The connection relationships between the computing devices in this computing device cluster 1100 can be similarly referred to... Figure 12 The computing device cluster 1100 is connected in a manner that differs from the others. The memory 1006 of one or more computing devices 1000 within the computing device cluster 1100 may store the same instructions for executing the testing method according to embodiments of this disclosure.

[0124] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster 1100 may also store partial instructions for executing the testing method according to embodiments of the present disclosure. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the testing method according to embodiments of the present disclosure.

[0125] This disclosure also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform a testing method according to embodiments of this disclosure.

[0126] This disclosure also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a testing method according to embodiments of this disclosure.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for testing, comprising: Based on a comparison between the target test script and at least one associated test script, a script similarity between the target test script and the at least one associated test script is determined, wherein the target test script and the at least one associated test script indicate at least one of the training and inference phases of an AI-based performance test. During the execution of the target test script on a cluster comprising multiple computing devices, test process features that indicate behavioral and file characteristics on the cluster are identified, and process similarity among the multiple computing devices is determined based on the test process features. as well as Based on the script similarity and the process similarity, the cluster provides test analysis of the target test script.

2. The method according to claim 1, wherein, The target test script and each of the at least one test scripts correspond to the same test benchmark and at least indicate the test model, test data, and test stopping conditions used for testing. The target test script differs from the at least one test script in at least one of the following: AI framework, acceleration library, or programming language.

3. The method according to claim 1, wherein determining the script similarity comprises: A first script similarity between the target test script and the at least one test script is determined by comparing the target abstract syntax tree corresponding to the target test script and the abstract syntax tree corresponding to the at least one test script. The second script similarity between the target test script and the at least one test script is determined by comparing the target nesting relationship of the target test script with the nesting relationship of the at least one test script using the call stack. as well as The script similarity is determined based on the first script similarity and the second script similarity.

4. The method according to claim 1, wherein the test process features include behavioral features. The characteristics that identify the testing process include: Monitoring the behavioral characteristics of the plurality of computing devices, wherein the behavioral characteristics include at least one of the following: file upload, file modification, file deletion, login behavior, process creation, network connection establishment, command execution, or privilege escalation; and Extract the timestamp, device name, source IP address, and destination IP address for each behavioral feature.

5. The method according to claim 4, further comprising: The sub-performance similarity of the respective computing devices for each behavioral feature is determined by comparing the performance metrics of each computing device in response to each behavioral feature. as well as Based on the sub-performance similarity, the performance similarity of the corresponding computing device relative to other computing devices is determined. The performance metrics mentioned therein include at least one of the following: CPU utilization, memory usage, request response time, or error rate.

6. The method according to claim 5, wherein the test process features further include file features, and identifying the test process features includes: The file characteristics of each of the plurality of computing devices are monitored, the file characteristics including at least one of the following: operation logs indicating file operation sequences, or file attributes.

7. The method according to claim 6, further comprising: The file similarity of a given computing device relative to the other computing devices is determined by comparing the file features of each of the plurality of computing devices.

8. The method of claim 7, wherein determining the process similarity comprises: The process similarity is determined based on the performance similarity and the file similarity.

9. The method according to claim 1, further comprising: For the script similarity, a first feature matrix of the time series and a second feature matrix of the machine sequence are established by unsupervised clustering analysis; For the process similarity, a third feature matrix of the time series is established by unsupervised clustering analysis; as well as Based on the first feature matrix, the second feature matrix, and the third feature matrix, test anomalies are identified.

10. The method of claim 9, wherein identifying the test anomaly comprises: In the first feature matrix, the second feature matrix, and the third feature matrix, elements whose values ​​do not meet a predetermined threshold are identified. The element identified indicates an abnormal call operation or abnormal machine behavior.

11. An apparatus for testing, comprising: A script similarity determination module is configured to determine the script similarity between a target test script and at least one associated test script based on a comparison between the target test script and at least one associated test script, wherein the target test script and the at least one associated test script indicate at least one of the training and inference phases of an AI-based performance test. A process similarity determination module is configured to identify test process features that indicate behavioral and file characteristics on the cluster during the execution of the target test script on a cluster comprising multiple computing devices, and to determine process similarity between the multiple computing devices for the test process features; as well as The test analysis module is configured to provide test analysis of the cluster regarding the target test script based on the script similarity and the process similarity.

12. A computing device cluster comprising a plurality of computing devices, each computing device including a processor and a memory, wherein the processor of the plurality of computing devices is configured to execute instructions stored in the memory of the plurality of computing devices to cause the computing device cluster to perform the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 10.

Images