A method for discovering software performance defects based on cross-architecture comparison

Abstract

This invention provides a method for discovering software performance defects based on cross-architecture comparison, relating to the field of computer software technology. The method includes: when the test set of the software under test is run on a cross-CPU architecture, collecting corresponding performance data and using the ratio of these performance data as the performance data points to be detected; then calling an autoencoder to detect abnormal test cases; when the abnormal test cases are run on a cross-CPU architecture, collecting corresponding multi-dimensional performance data and performing cross-CPU architecture function matching to obtain successfully matched functions; when the ratio of the number of clock cycles of the successfully matched functions on the cross-CPU architecture is greater than a benchmark threshold, the successfully matched function is determined to be the function causing the performance defect in the software under test. This application solves the problems of relying on manually set indicators to judge performance anomalies when performing software performance testing on a single architecture, which cannot meet cross-architecture requirements and makes it difficult to identify performance bottlenecks during architecture adaptation.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and in particular to a method for discovering software performance defects based on cross-architecture comparison. Background Technology

[0002] Software performance defect discovery refers to identifying defects or bottlenecks affecting system performance during the development, testing, and execution of a software system. These performance defects typically manifest as excessively long response times, excessively high resource utilization, and insufficient throughput, which may impact user experience, system stability, or efficient resource utilization. Discovering performance defects is not only about identifying whether software functions correctly, but also a crucial step in ensuring that software can operate efficiently under real-world loads and various scenarios.

[0003] Software performance testing is an essential part of the software performance defect discovery process. It verifies whether the software meets the user's expected performance requirements, identifies potential performance bottlenecks, analyzes performance issues, and provides performance optimization solutions. In actual testing, different testing methods can be adopted based on specific performance requirements and application domains to verify the software's performance and application requirements. In practical applications, software needs to be deployed and run on different CPU architecture platforms (x86, ARM64, RISC-V, etc.), and the microarchitectural characteristics of each architecture lead to differences in application software performance. To improve software performance on a specific architecture, performance defect discovery becomes crucial in the software optimization process. However, due to differences between architectures, traditional performance testing methods cannot meet the requirements under cross-architecture conditions. The execution performance of the same software on different architectures is affected by hardware characteristics such as instruction sets, memory, I / O, cache levels and sizes, making direct cross-architecture performance comparisons and anomaly identification complex.

[0004] Currently, the industry typically uses performance benchmarking tools to conduct performance tests on a single architecture to evaluate application software performance. However, these testing methods cannot accurately reflect cross-architecture performance differences and struggle to effectively identify performance bottlenecks during architecture adaptation. Furthermore, existing methods often rely on manually set thresholds or metrics to determine performance anomalies, which is subjective and limited, resulting in low accuracy. Summary of the Invention

[0005] This invention provides a software performance defect discovery method based on cross-architecture comparison, which solves the problems of existing performance testing methods that only perform performance testing on a single architecture, rely on manually setting thresholds or indicators to judge performance anomalies, cannot meet the requirements under cross-architecture conditions, and are difficult to effectively identify performance bottlenecks in the architecture adaptation process.

[0006] This invention provides a method for discovering software performance defects based on cross-architecture comparison, the method comprising the following steps:

[0007] When the test set of the software to be tested is run on the first CPU architecture and the second CPU architecture respectively, the first performance data and the second performance data are collected respectively.

[0008] The ratio of the first performance data to the second performance data is used as the performance data point to be detected, and the autoencoder is called to perform anomaly detection on the performance data point to be detected to obtain abnormal test cases. The autoencoder is adaptively optimized by the particle swarm optimization algorithm.

[0009] When the abnormal test cases are run for the second time on the first CPU architecture and the second CPU architecture respectively, the corresponding multi-dimensional performance data are collected respectively.

[0010] Based on the multidimensional performance data, cross-architecture function matching is performed on the first CPU architecture and the second CPU architecture to obtain the successfully matched function;

[0011] Determine the first clock cycle number of the successful matching function when running on the first CPU architecture and the second clock cycle number of the successful matching function when running on the second CPU architecture;

[0012] When the ratio of the first clock cycle number to the second clock cycle number is greater than a reference threshold, the successful matching function is determined to be a function that causes performance defects in the software under test.

[0013] In some embodiments, the step of performing cross-architecture function matching based on the multidimensional performance data in the first CPU architecture and the second CPU architecture to obtain a successfully matched function includes:

[0014] The objective function in the multidimensional performance data collected from the first CPU architecture is used as the function index;

[0015] Based on the function index, each function to be matched is traversed in the multidimensional performance data collected by the second CPU architecture;

[0016] During the traversal, the function to be matched is determined to be a successfully matched function when the function index and the function to be matched satisfy the following conditions:

[0017] The function to be matched and the function index come from the same base class and command pattern;

[0018] The similarity between the function name of the function index and the function name of the function to be matched reaches a preset threshold.

[0019] In some embodiments, the process of calculating the similarity between the function name of the function index and the function name of the function to be matched includes:

[0020] Obtain the first function name string of the function index and the second function name string of the function to be matched;

[0021] Vectorize the first function name string and the second function name string to obtain the corresponding first string vector and second string vector;

[0022] Determine the vector similarity between the first string vector and the second string vector, and use the vector similarity as the similarity between the function name of the function index and the function name of the function to be matched.

[0023] In some embodiments, the benchmark threshold is determined according to the SPEC CPU benchmark program, and the process of determining the benchmark threshold includes:

[0024] In the first CPU architecture and the second CPU architecture, the SPEC CPU benchmark test program is executed respectively, and the corresponding first performance test results and second performance test results are collected during the execution process.

[0025] The ratio of the first performance test result to the second performance test result is used as the benchmark threshold.

[0026] In some embodiments, after the corresponding multidimensional performance data are collected respectively, the method further includes:

[0027] Based on the multidimensional performance data, cross-architecture function matching is performed on the first CPU architecture and the second CPU architecture to obtain unmatched functions. The unmatched functions are target functions whose multidimensional performance data corresponding to the first CPU architecture is used as function index and whose target functions are not matched in the multidimensional performance data corresponding to the second CPU architecture.

[0028] The resource usage data of the unsuccessfully matched function in the first CPU architecture is determined. When the resource usage data reaches a set threshold, the unsuccessfully matched function is determined to be a function that causes performance defects in the software under test.

[0029] In some embodiments, the method further includes:

[0030] The function that causes performance defects in the software under test is optimized to obtain the optimized function;

[0031] When the optimized function is run on the first CPU architecture and the second CPU architecture respectively, the corresponding running performance data is collected respectively.

[0032] The performance optimization result of the optimized function is determined based on the aforementioned operational performance data;

[0033] When the performance optimization result indicates that the optimized function is not a function that causes performance defects in the software under test, the function with performance defects in the software under test is updated according to the optimized function.

[0034] This invention also provides a software performance defect discovery device based on cross-architecture comparison, the device comprising the following modules:

[0035] The data acquisition module is used to collect first performance data and second performance data respectively when the test set of the software to be tested is run on the first CPU architecture and the second CPU architecture respectively.

[0036] An anomaly detection module is used to take the ratio of the first performance data and the second performance data as the performance data point to be detected, and call the autoencoder to perform anomaly detection on the performance data point to be detected to obtain anomaly test cases. The autoencoder is adaptively optimized by the particle swarm optimization algorithm.

[0037] The data acquisition module is also used to collect corresponding multi-dimensional performance data when the abnormal test cases are run for the second time on the first CPU architecture and the second CPU architecture respectively.

[0038] The function matching module is used to perform cross-architecture function matching based on the multidimensional performance data in the first CPU architecture and the second CPU architecture to obtain successfully matched functions.

[0039] The defect localization module is used to determine the first clock cycle number of the successful matching function when running on the first CPU architecture and the second clock cycle number of the successful matching function when running on the second CPU architecture.

[0040] When the ratio of the first clock cycle number to the second clock cycle number is greater than a reference threshold, the successful matching function is determined to be a function that causes performance defects in the software under test.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the software performance defect discovery method based on cross-architecture comparison as described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the software performance defect discovery method based on cross-architecture comparison as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the software performance defect discovery method based on cross-architecture comparison as described above.

[0044] This invention provides a software performance defect discovery method based on cross-architecture comparison. It automatically collects multi-dimensional performance data by comparing the performance of the software under test across different CPU architectures within the same software. Based on this multi-dimensional performance data, it identifies abnormal test cases and further determines functions with performance defects. Through cross-architecture testing and verification, it not only accurately locates the performance bottlenecks of the software under test but also deeply explores and analyzes the root causes of performance differences, providing data support and decision-making basis for subsequent software optimization. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the software performance defect discovery method based on cross-architecture comparison provided by the present invention.

[0047] Figure 2 This is an application framework diagram of the software performance defect discovery method based on cross-architecture comparison provided by the present invention.

[0048] Figure 3 This is a schematic diagram illustrating the selection of the anomaly detection algorithm provided by the present invention.

[0049] Figure 4 This is a schematic diagram illustrating the principle of the function matching process provided by the present invention.

[0050] Figure 5 This is a schematic diagram of the software performance defect discovery device based on cross-architecture comparison provided by the present invention.

[0051] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] The software performance defect discovery method based on cross-architecture comparison provided by this invention can be applied in software testing scenarios, specifically deployed in the server or terminal of a software testing system or platform, to assist the software testing system or platform in testing the software under test, thereby accurately discovering software performance defects and feeding them back to the software testing system or platform.

[0054] The following description, in conjunction with the accompanying drawings, illustrates the software performance defect discovery method based on cross-architecture comparison according to the present invention. Figure 1 This is a flowchart illustrating the software performance defect discovery method based on cross-architecture comparison provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps 101 to 106.

[0055] Step 101: When the test set of the software to be tested is run on the first CPU architecture and the second CPU architecture respectively, the first performance data and the second performance data are collected respectively.

[0056] To support anomaly detection and defect localization, raw data related to the performance of the software under test needs to be collected during the testing process. To achieve cross-architecture compatibility, this embodiment of the invention designs a first CPU (Central Processing Unit) architecture and a second CPU architecture, which differ in performance and hardware parameters. The software under test is first deployed on both the first and second CPU architectures. Then, a test set for the software under test is obtained. This test set includes test cases for testing the performance of the software under test. Test cases can involve the software under test performing application functions or running algorithmic programs, such as loop structures. When the software under test runs on both the first and second CPU architectures, its performance is tested by executing these test sets.

[0057] Therefore, when the test set of the software under test is run on the first CPU architecture and the second CPU architecture respectively, first performance data and second performance data are collected respectively. To efficiently capture data and ensure its accuracy and reliability so that subsequent analysis can reflect the true system state, this embodiment of the invention uses the perf performance analysis tool to collect performance data during the test set execution of the software under test. The perf performance analysis tool can delve into various levels of the operating system and accurately capture various types of performance data. Its functions not only cover monitoring of overall system performance but also support function-level data collection. This provides a more reliable basis for subsequent anomaly detection and defect localization. The perf performance analysis tool runs based on hardware performance monitoring counters, utilizing the CPU's built-in performance monitoring hardware to collect counts of various hardware events such as cache hits, branch predictions, instruction execution, and cycle counts. Because it operates directly at the hardware level, this collection method has extremely low performance overhead, can obtain key system performance indicators in real time, and has almost no interference with the operation of the monitored program, thereby maximizing the preservation of the system's true performance.

[0058] Here, when the test suite of the software under test is run for the first time on the first CPU architecture, the corresponding first performance data is collected using the perf performance analysis tool. Similarly, when the test suite of the software under test is run for the first time on the second CPU architecture, the corresponding second performance data is collected using the perf performance analysis tool. Generally speaking, execution time is a key indicator for evaluating the performance of the software under test in executing test cases, and potential anomalies can be quickly identified based on the execution time of each test case. Therefore, the performance data collected by the perf performance analysis tool mainly includes the execution time of the test cases.

[0059] Step 102: Take the ratio of the first performance data and the second performance data as the performance data point to be tested, and call the autoencoder to perform anomaly detection on the performance data point to be tested to obtain abnormal test cases.

[0060] To compare the performance differences of the software under test on different CPU architectures, this embodiment of the invention, after collecting performance data on different CPU architectures, uses the ratio of the first performance data and the second performance data as the performance data point to be tested. For example, the ratio of the execution time of the corresponding test cases on the first CPU architecture and the second CPU architecture is used as the performance data point to be tested. This ratio can measure the performance differences in executing test cases on different CPU architectures.

[0061] Next, an anomaly detection algorithm is selected to detect anomalies in the performance data points to be tested, resulting in anomaly test cases. In some embodiments, such as... Figure 3As shown, there are various anomaly detection algorithms, including those based on ensemble learning, generative adversarial networks, and autoencoders. These algorithms can all be optimized in real-time, performing anomaly detection on performance data and yielding corresponding results. By comparing the silhouette coefficients of the detection results, the optimal algorithm can be selected as the anomaly detection algorithm.

[0062] In this embodiment of the invention, anomaly detection requires eliminating interference from hardware differences to identify data deviating from the normal range. Autoencoders, however, can capture and learn complex patterns in data, detecting minute anomalies in high-dimensional data. By compressing data to a low-dimensional space and reconstructing it back to the original space, autoencoders identify anomalous data points with large reconstruction errors. Because of their excellent performance and stable results on test datasets, this embodiment of the invention selects an autoencoder-based method as the anomaly detection algorithm. By calling the autoencoder to perform anomaly detection on the performance data points to be detected, anomaly test cases are obtained.

[0063] Autoencoders can accurately identify anomalous data points in performance data points, thereby determining abnormal test cases. Their identification effectiveness depends on the anomaly detection algorithm, i.e., the performance of the autoencoder itself. To optimize the anomaly detection effect, the anomaly detection algorithm needs adaptive optimization. In this embodiment of the invention, the anomaly detection algorithm is adaptively optimized using the particle swarm optimization algorithm; that is, the autoencoder is adaptively optimized through particle swarm optimization. The particle swarm optimization algorithm can automatically adjust the parameters of the autoencoder to ensure optimal model performance. Based on this, more accurate and efficient anomaly detection can be achieved, ensuring reliable detection of abnormal performance data.

[0064] Step 103: When the abnormal test cases are run for the second time on the first CPU architecture and the second CPU architecture respectively, the corresponding multi-dimensional performance data are collected respectively.

[0065] After identifying the abnormal test cases in step 102, it is necessary to determine the performance defects in the software under test based on these test cases, i.e., defect localization. This is done by re-executing the abnormal test cases in the software under test. Before execution, the compilation options and debugging environment are modified for the abnormal test cases. When loading the debugging environment, it is necessary to load the test case symbol table, which is a list of function names called by the software under test during the execution of these test cases.

[0066] After loading the debugging environment, the software under test can be run on both the first and second CPU architectures, and abnormal test cases can be executed. When the abnormal test cases are run for the second time on the first and second CPU architectures respectively, corresponding multidimensional performance data can be collected. The multidimensional performance data here refers to the functions called by the software under test when executing the abnormal test cases, that is, function-level multidimensional performance data. The collection method is also to use the perf performance analysis tool. The function-level multidimensional performance data specifically includes: number of instructions, number of cycles, number of cache misses, number of cache references, number of branch predictions, number of branch prediction misses, number of L1 data cache loads, number of L1 data cache load misses, number of L1 data cache stores, number of L1 data cache store misses, as well as function execution time and function call stack.

[0067] After collecting multidimensional performance data, the target function with performance defects is identified based on this data. Since loading the debugging environment requires loading a symbol table containing test cases, analyzing this multidimensional performance data allows us to determine the performance status of the target function in the software under test, which executes abnormal test cases. Based on the test case symbol table, we can then identify the key functions within the target function that cause performance bottlenecks.

[0068] Step 104: Based on multidimensional performance data, perform cross-architecture function matching on the first CPU architecture and the second CPU architecture to obtain successfully matched functions.

[0069] To further determine whether performance differences exist among functions and to accurately identify performance-defective functions, this embodiment of the invention performs function localization after collecting multi-dimensional performance data. Here, cross-architecture function matching is performed based on the multi-dimensional performance data across the first and second CPU architectures to obtain successfully matched functions.

[0070] Specifically, in the first CPU architecture, the functions that execute abnormal test cases are first determined based on the collected multidimensional performance data. Then, based on these functions, the corresponding target functions are determined from the multidimensional performance data of the first CPU architecture. The target functions are then used as indexes to match the corresponding functions to be matched from the multidimensional performance data of the second CPU architecture, thereby achieving cross-architecture function matching. When the function is successfully matched, the function to be matched is the successfully matched function.

[0071] During the matching process, it is necessary to ensure that the two functions involved in the matching (i.e., the function index corresponding to the first CPU architecture and the function to be matched corresponding to the second CPU architecture) come from the same object base class and command pattern. The matching principle is that the similarity of the function name strings of the two functions must reach a preset threshold.

[0072] Step 105: Determine the first clock cycle number of the successful matching function when running on the first CPU architecture and the second clock cycle number of the successful matching function when running on the second CPU architecture.

[0073] Once the matching is complete and the successfully matched function is identified, the corresponding performance metrics can be tested on both the first and second CPU architectures. Here, the number of clock cycles consumed by the successfully matched function on each CPU architecture is used as the performance metric. First, the successfully matched function is executed on both the first and second CPU architectures. During execution, the first clock cycle count of the successfully matched function on the first CPU architecture and the second clock cycle count of the successfully matched function on the second CPU architecture are determined and used as the performance metrics for the successfully matched function.

[0074] Step 106: When the ratio of the first clock cycle number to the second clock cycle number is greater than the reference threshold, the successfully matched function is determined to be the function that causes the software under test to have performance defects.

[0075] Furthermore, to eliminate normal performance deviations caused by architectural differences, this embodiment of the invention calls some CPU benchmark programs to run on two CPU architectures respectively, and determines the corresponding performance standard values ​​as benchmark thresholds. When the ratio of the first clock cycle count to the second clock cycle count is greater than the benchmark threshold, the successfully matched function is determined to be the function that causes performance defects in the software under test.

[0076] If the ratio of the first clock cycle count to the second clock cycle count is greater than the baseline threshold, it indicates that the performance problem caused by the successful matching function is not due to architectural differences, but rather a performance bottleneck inherent in the successful matching function itself. This allows us to identify the successful matching function as the function causing the performance defect in the software under test, thus pinpointing the performance defect function. This provides support and data for the subsequent maintenance of the software under test, and facilitates optimization feedback for the software under test.

[0077] like Figure 2 As shown, Figure 2 This is an application framework diagram of the software performance defect discovery method based on cross-architecture comparison provided by this invention. The following is combined with... Figure 2 This describes the application process of the software performance defect discovery method based on cross-architecture comparison provided by this invention. For example... Figure 2 As shown, the application process is divided into two phases. Phase 1 consists of the data acquisition module 1 and the anomaly detection module, which mainly uses cross-architecture comparison based on the selected test set to quickly identify test cases with performance defects. Phase 2 consists of the data acquisition module 2 and the defect localization module, which mainly uses the initially screened abnormal test cases to locate the root cause from the full call chain function and perf multidimensional performance data, and optimize the feedback.

[0078] First, in data acquisition module 1, the software ecosystem, i.e., the software to be tested, is selected, including cloud-native, edge computing, artificial intelligence (AI), and big data software. Test sets are selected based on the software package dependencies. Of course, specific test sets are selected for each type of software, including cloud-native test sets, edge computing test sets, AI test sets, and big data test sets. Next, the software ecosystem runs the corresponding test sets on architectures A and B respectively, and performance data such as execution time and CPU time are collected using the perf performance analysis tool. Architectures A and B are two different CPU architectures, for example, SOPHON SG2042 and Kunpeng 920-4826 respectively. The software environment running the software ecosystem must be consistent, for example, the compiler is gcc12.3.1, the kernel is Linux 6.6, the operating system is OpenEuler 24.03, and the C runtime library is glibc2.38, etc.

[0079] In the anomaly detection module, anomaly detection is performed based on the performance data collected in data acquisition module 1. The performance data includes the execution time of the test cases in architecture A and architecture B. The ratio of these two execution times is used as the anomaly data point to be detected and input into the anomaly detection algorithm (e.g., an autoencoder) to detect abnormal test cases. Thus, phase 1 ends, and phase 2 begins.

[0080] In Phase 2, the data acquisition module 2 first loads the debugging environment based on the abnormal test cases selected in Phase 1, loads symbol table information, and configures perf multidimensional performance sampling. Then, the abnormal test cases are executed on both architecture A and architecture B, collecting multidimensional performance data. This includes function-level multidimensional performance data from the anomaly test in architecture A and architecture B. In the defect localization module, defect localization is performed based on the multidimensional performance data collected by the data acquisition module 2. This multidimensional performance data includes time, instruction count, cycle count, cache, etc., and also involves full call chain function performance analysis of the test cases, including application, framework, base library, and kernel. The multidimensional performance data identifies functions with performance defects, and function matching is performed in architectures A and B to determine successfully matched functions. Then, the clock cycle count of the successfully matched functions in architectures A and B is tested, and combined with CPU benchmark differences, the problematic functions—that is, functions with performance defects in the software ecosystem—are identified. For problematic functions, the description of the performance anomaly and the function location can be provided to the testers. The testers can then perform manual analysis and optimization, and the optimization will be fed back into the software ecosystem. This concludes Phase 2 of the process, and the entire application process has been introduced.

[0081] This invention, through cross-architecture testing, automatically collects multi-dimensional performance data by comparing the performance of the software under test across different CPU architectures within the same software. Based on this multi-dimensional performance data, it identifies abnormal test cases and further determines the objective function containing performance defects. This cross-architecture testing verification not only accurately locates the performance bottlenecks of the software under test but also deeply explores and analyzes the root causes of performance differences, providing data support and decision-making basis for subsequent software optimization. Furthermore, this invention eliminates the need to align compilation options in the software system, simplifying the operation process, lowering the technical implementation threshold, and improving applicability.

[0082] In some embodiments, the process of locating performance defects mainly focuses on function-level matching, that is, cross-architecture function matching is performed based on the target function on the first CPU architecture and the second CPU architecture to obtain the successfully matched function.

[0083] Specifically, the target functions in the multi-dimensional performance data corresponding to the first CPU architecture are used as function indices. These target functions are the functions that the software under test needs to call when executing abnormal test cases, and they can be identified one by one through the multi-dimensional performance data. Then, each function to be matched is traversed from the multi-dimensional performance data corresponding to the second CPU architecture based on the function index. Based on the multi-dimensional performance data corresponding to the second CPU architecture, the functions that the software under test needs to call when executing abnormal test cases can also be determined, and these are used as the functions to be matched.

[0084] After traversing to a function to be matched, a match is performed with the function index. Here, during the matching process, the function to be matched is determined to be a successfully matched function when the function index and the function to be matched meet the following conditions.

[0085] The first condition must be met: the function to be matched and the function index must come from the same base class (object base class) and command pattern (Command command pattern). This is to ensure that the call stacks of the two functions being matched are the same, which is a prerequisite for the matching to be satisfied.

[0086] Next, the second condition must be met: the similarity between the function name at the function index and the function name of the function to be matched must reach a preset threshold. In other words, the similarity between the strings represented by the Symbols of the two functions must reach the preset threshold. Of course, similarity is calculated by comparing strings; it can be calculated using Jaccard distance, cosine similarity, or by mapping to binary strings, multi-dimensional vectors, etc., before calculating the similarity.

[0087] The preset threshold is a similarity threshold, for example, set to 95%, which is used to measure the degree of matching of functions. The similarity threshold can be dynamically adjusted according to the actual situation to ensure the accuracy of matching.

[0088] Only when the function index corresponding to the first CPU architecture and the function to be matched corresponding to the second CPU architecture meet the above two conditions can it be said that the two functions are successfully matched.

[0089] In some embodiments, the objective function in the multidimensional performance data collected by the first CPU architecture is multiple, and therefore the second CPU architecture also has multiple functions to be matched. Thus, this is a many-to-many matching process, which can be found in [reference needed]. Figure 4As shown. Initially, all functions of architecture A are traversed, pointing to the first function of architecture A, which is the first target function in the multi-dimensional performance data corresponding to the first CPU architecture, as the function index. Then, it is determined whether all functions of architecture A have been traversed. If not, starting from the first function of architecture A, all functions of architecture B are traversed, pointing to the first function of architecture B, which is the first matching function in the multi-dimensional performance data corresponding to the second CPU architecture. Next, it is determined whether all functions of architecture B have been traversed. If not, the first functions of architecture A and architecture B are matched to determine if the functions belong to the same Object (base class) and Command (command pattern). If they do, the similarity of the function symbols is further determined to see if it reaches a preset threshold. If so, the match is successful, the matching information is stored, and the process continues to determine whether all functions of architecture B have been traversed, pointing to the next function of architecture B, which is the second matching function in the multi-dimensional performance data corresponding to the second CPU architecture.

[0090] When checking if all functions in architecture B have been traversed, if so, it means the first function in architecture A has been matched. The process then moves to the next function in architecture A, which is the second target function in the multi-dimensional performance data corresponding to the first CPU architecture, and continues to use it as a function index to match each function in architecture B. This process continues until all functions in architecture A have been traversed, ending the matching process.

[0091] This invention proposes a method of matching two functions across architectures by matching function name strings. This simplifies the cross-architecture matching process and can quickly locate function performance issues even with large amounts of data and a large number of functions, without any omissions, thereby improving the efficiency of function performance mining.

[0092] In some embodiments, when performing cross-architecture function matching on the first CPU architecture and the second CPU architecture, it is necessary to calculate the similarity between the function name of the function index and the function name of the function to be matched. The calculation process of the similarity between the function name of the function index and the function name of the function to be matched is described below.

[0093] First, obtain the first function name string of the function index and the second function name string of the function to be matched.

[0094] Here, we directly extract the function name from the function index and the function name of the function to be matched. The function name is generally composed of at least one word or a string of English characters. Here, we directly extract the function name strings of both to calculate the similarity.

[0095] In the similarity calculation process, the first function name string and the second function name string are vectorized to obtain corresponding first string vectors and second string vectors. Vectorization maps strings to vector dimensions. One method for vectorization is to use a text encoder to encode characters, obtaining the corresponding encoded vectors.

[0096] Next, the vector similarity between the first string vector and the second string vector is determined. Methods for calculating vector similarity include edit distance, Euclidean distance, and cosine similarity. Finally, the vector similarity is used as the similarity between the function name of the function index and the function name of the function to be matched. This similarity is then used to determine if the similarity exceeds a preset threshold, thus confirming whether the function index and the function to be matched have successfully matched.

[0097] In this embodiment of the invention, the degree of matching between two functions is measured by calculating the similarity of their function name strings. The similarity calculation, performed using string vectorization, ensures accuracy even with large amounts of data and a large number of functions. Furthermore, using string vectorization to calculate similarity and determine matching simplifies the matching process.

[0098] To eliminate normal performance deviations caused by architectural differences, it is necessary to run some CPU benchmark programs on both CPU architectures and determine the corresponding performance standard values ​​as benchmark thresholds. Then, it is determined whether the ratio of the first clock cycle count to the second clock cycle count is greater than the benchmark threshold to determine whether the successfully matched function is the function that causes the performance defects in the software under test.

[0099] This CPU benchmark program determines whether the performance differences between functions are within a normal range. In some implementations, the benchmark threshold is determined based on the SPEC CPU benchmark program, which is a CPU benchmark program from Standard Performance Evaluation Corporation. It is used to evaluate the differences in computing power between different architectures, has high operating system independence, and hardly relies on the kernel interface, thus effectively avoiding the influence of the kernel on performance differences. Through SPEC CPU testing, the computing power between different architectures can be compared, accurately reflecting the performance differences brought about by the hardware platform. The process of determining the benchmark threshold using the SPEC CPU benchmark program is described below.

[0100] First, the SPEC CPU benchmark program is executed in both the first and second CPU architectures, and the corresponding first and second performance test results are collected during execution. The ratio of the first performance test result to the second performance test result is then used as the benchmark threshold. The first and second performance test results here are similar to the multi-dimensional performance data mentioned above, and will not be elaborated further.

[0101] In this embodiment of the invention, the SPEC CPU benchmark test program is used to determine the benchmark threshold, which is used to determine whether there is a performance anomaly in the successfully matched function, ensuring that the performance difference between functions is within the normal range, eliminating normal performance deviations caused by architectural differences, and more accurately identifying and locating actual performance anomalies.

[0102] In some embodiments, after the abnormal test cases are run a second time on the first CPU architecture and the second CPU architecture respectively, and the corresponding multidimensional performance data are collected, cross-architecture function matching is performed based on the objective function on the first CPU architecture and the second CPU architecture to obtain the unsuccessfully matched functions. These unsuccessfully matched functions are the objective functions whose function index is the multidimensional performance data corresponding to the first CPU architecture, and which do not find a matching function in the multidimensional performance data corresponding to the second CPU architecture.

[0103] When cross-architecture function matching is performed based on multidimensional performance data on the first and second CPU architectures, two different function matching results will be generated. When a match is successful, that is, when the function index and the function to be matched satisfy the two conditions mentioned above, the function to be matched is taken as the successfully matched function.

[0104] However, there are also cases where matching fails, meaning the function index and the function to be matched do not meet the two conditions mentioned above. For example, the similarity between the function name of the function index and the function name of the function to be matched is less than or equal to the similarity threshold. Therefore, it is possible that a target function is used as a function index in the multidimensional performance data collected in the first CPU architecture, but fails to match the function to be matched in the second CPU architecture. In this case, the target function is considered an unmatched function.

[0105] These unsuccessfully matched functions cannot be directly compared across architectures due to differences in architecture optimization or function characteristics. However, we can still determine the resource usage data of the unsuccessfully matched functions in the first CPU architecture. The resource usage data includes the proportion of CPU resources used and the execution time of the unsuccessfully matched functions. We can also determine whether these are potential abnormal data points through the resource usage data.

[0106] The criterion for judgment can be a preset threshold based on actual conditions. When resource usage data reaches the set threshold, the unsuccessfully matched function is determined to be the function causing performance defects in the software under test. For example, when the proportion of CPU resource usage reaches a preset proportion threshold, or the execution time exceeds a preset time threshold, it indicates that the unsuccessfully matched function is the function causing performance defects in the software under test.

[0107] Of course, in practice, this standard for judgment may not be effective. If there is a deviation, further determination is needed, such as through manual analysis. Other testing environments and performance testing methods should be used to further test this unsuccessfully matched function.

[0108] In this embodiment of the invention, when a function in multidimensional performance data fails to achieve cross-architecture matching, the unsuccessfully matched function is further judged by the resource usage data running on a single architecture, and combined with manual assisted analysis, thereby eliminating the impact of architecture optimization or function characteristic differences on performance anomalies when cross-architecture matching fails.

[0109] In some embodiments, after identifying the functions causing performance defects in the software under test through cross-architecture matching, the functions causing performance defects can be optimized to obtain optimized functions. Performance optimization can involve rewriting the function computation logic, optimizing code length, optimizing loop iteration structures, etc. For the optimized functions in the software under test, cross-architecture comparison matching can continue, that is, steps 101 to 103 above can be executed again using the test set of the software under test.

[0110] When the optimized function is run on both the first and second CPU architectures, corresponding performance data is collected. This data can be collected using the perf performance analysis tool. The performance data can be the multi-dimensional performance data mentioned above, which will not be elaborated upon here.

[0111] Next, the performance optimization result of the optimized function is determined based on the runtime performance data. This process involves cross-architecture function matching based on the runtime performance data in both the first and second CPU architectures to obtain successfully matched functions. Then, the clock cycles required for the successfully matched function to run on the first CPU architecture and the clock cycles required for the successfully matched function to run on the second CPU architecture are determined, and the ratio of these two clock cycle counts is used as the performance optimization result of the optimized function. The cross-architecture function matching process can be referred to in step 104 above, and the process of determining the clock cycle count can be referred to in step 105 above; these details are not elaborated here.

[0112] Next, we determine whether the performance optimization result of the optimized function is greater than the benchmark threshold determined by the SPEC CPU benchmark program. If it is less than the benchmark threshold, it means that the performance optimization result indicates that the optimized function is not a function that causes performance defects in the software under test. If it is greater than the benchmark threshold, it means that the performance optimization result indicates that the optimized function is still a function that causes performance defects in the software under test.

[0113] When the performance optimization result indicates that the optimized function is no longer the function causing performance defects in the software under test, the function with performance defects in the software under test is updated based on the optimized function. Here, when the optimized function is no longer the function causing performance defects in the software under test, it means that the effectiveness of the performance optimization has been verified. Therefore, by updating the function with performance defects in the software under test using the optimized function, the performance defects in the software under test are overcome.

[0114] If the performance optimization result indicates that the optimized function is the function that causes performance defects in the software under test, it means that the performance optimization is poor. If the optimized function still causes performance defects in the software under test, then performance optimization can continue to be performed. For example, the function calculation logic, code length optimization, and loop iteration structure optimization can be performed. The corresponding runtime performance data can continue to be collected and the test can continue until the performance optimization result indicates that the optimized function is no longer the function that causes performance defects in the software under test.

[0115] This invention, after identifying functions causing performance defects in the software under test, optimizes those functions and then tests the performance optimization results to ensure the effectiveness of the optimization. The optimized functions are then used to update the functions causing performance defects in the software under test. If the performance optimization results indicate poor performance, iterative optimization continues, thus realizing a closed-loop optimization mechanism in the software under test. This mechanism automates performance testing and tuning of the software under test, thereby promoting its performance improvement and widespread application within the software ecosystem.

[0116] The software performance defect discovery device based on cross-architecture comparison provided by the present invention will be described below. The software performance defect discovery device based on cross-architecture comparison described below and the software performance defect discovery method based on cross-architecture comparison described above can be referred to in correspondence.

[0117] See Figure 5 , Figure 5 This is a schematic diagram of the software performance defect discovery device based on cross-architecture comparison provided by the present invention, as shown below. Figure 5As shown, the software performance defect discovery device based on cross-architecture comparison includes: a data acquisition module 501, an anomaly detection module 502, a function matching module 503, and a defect localization module 504. Specifically, the data acquisition module 501 is used to collect first performance data and second performance data respectively when the test set of the software to be tested is run on the first CPU architecture and the second CPU architecture respectively; the anomaly detection module 502 is used to take the ratio of the first performance data and the second performance data as the performance data point to be detected, and call the autoencoder to perform anomaly detection on the performance data point to be detected to obtain abnormal test cases, wherein the autoencoder is adaptively optimized by the particle swarm optimization algorithm; the data acquisition module 501 is also used to, when the abnormal test cases are run on the first CPU architecture and the second CPU architecture respectively, divide... The system collects corresponding multidimensional performance data and determines the target function with performance defects based on the multidimensional performance data; the function matching module 503 is used to perform cross-architecture function matching on the first CPU architecture and the second CPU architecture based on the target function to obtain a successfully matched function; the defect location module 504 is used to determine the first clock cycle number of the successfully matched function when running on the first CPU architecture and the second clock cycle number of the successfully matched function when running on the second CPU architecture; when the ratio of the first clock cycle number to the second clock cycle number is greater than a benchmark threshold, the successfully matched function is determined to be the function that causes the software under test to have performance defects.

[0118] It should be noted that the beneficial effects of the software performance defect discovery device based on cross-architecture comparison mentioned above correspond to each other, so the beneficial effects of the software performance defect discovery device based on cross-architecture comparison will not be elaborated here.

[0119] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communications bus 640. The processor 610 can call logic instructions in the memory 630 to execute a software performance defect discovery method based on cross-architecture comparison. This method includes: when the test set of the software under test is run for the first time on a first CPU architecture and a second CPU architecture respectively, collecting first performance data and second performance data respectively; using the ratio of the first performance data to the second performance data as the performance data point to be detected, and calling an autoencoder to perform anomaly detection on the performance data point to be detected, obtaining abnormal test cases, wherein the autoencoder is adaptively optimized using a particle swarm optimization algorithm; when the abnormal test cases are run for the second time on the first CPU architecture and the second CPU architecture respectively, collecting corresponding multi-dimensional performance data respectively; performing cross-architecture function matching based on the multi-dimensional performance data on the first CPU architecture and the second CPU architecture to obtain a successfully matched function; determining the first clock cycle number of the successfully matched function when running on the first CPU architecture and the second clock cycle number of the successfully matched function when running on the second CPU architecture; when the ratio of the first clock cycle number to the second clock cycle number is greater than a benchmark threshold, determining the successfully matched function as a function causing a performance defect in the software under test.

[0120] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the software performance defect discovery method based on cross-architecture comparison provided by the above methods. This method includes: when the test set of the software to be tested is run on a first CPU architecture and a second CPU architecture respectively, collecting first performance data and second performance data respectively; using the ratio of the first performance data to the second performance data as the performance data point to be tested, and calling an autoencoder to perform anomaly detection on the performance data point to be tested to obtain abnormal test cases, wherein... The autoencoder is adaptively optimized using a particle swarm optimization algorithm. When the abnormal test cases are run for the second time on the first CPU architecture and the second CPU architecture respectively, corresponding multi-dimensional performance data are collected. Based on the multi-dimensional performance data, cross-architecture function matching is performed on the first CPU architecture and the second CPU architecture to obtain a successfully matched function. The first clock cycle number of the successfully matched function when running on the first CPU architecture and the second clock cycle number of the successfully matched function when running on the second CPU architecture are determined. When the ratio of the first clock cycle number to the second clock cycle number is greater than a benchmark threshold, the successfully matched function is determined to be a function that causes performance defects in the software under test.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the software performance defect discovery method based on cross-architecture comparison provided by the above methods. This method includes: when a test set of the software to be tested is run for the first time on a first CPU architecture and a second CPU architecture, first performance data and second performance data are collected respectively; the ratio of the first performance data to the second performance data is used as a performance data point to be detected, and an autoencoder is invoked to perform anomaly detection on the performance data point to be detected to obtain abnormal test cases, wherein the autoencoder is adaptively optimized using a particle swarm optimization algorithm; when the abnormal test cases are run for the second time on the first CPU architecture and the second CPU architecture, corresponding multidimensional performance data are collected respectively; cross-architecture function matching is performed based on the multidimensional performance data on the first CPU architecture and the second CPU architecture to obtain a successfully matched function; the first clock cycle number of the successfully matched function during the first CPU architecture run and the second clock cycle number of the successfully matched function during the second CPU architecture run are determined; when the ratio of the first clock cycle number to the second clock cycle number is greater than a benchmark threshold, the successfully matched function is determined to be a function causing performance defects in the software to be tested.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for discovering software performance defects based on cross-architecture comparison, characterized in that, The method includes: When the test set of the software to be tested is run on the first CPU architecture and the second CPU architecture respectively, the first performance data and the second performance data are collected respectively. The ratio of the first performance data to the second performance data is used as the performance data point to be detected, and the autoencoder is called to perform anomaly detection on the performance data point to be detected to obtain abnormal test cases. The autoencoder is adaptively optimized by the particle swarm optimization algorithm. When the abnormal test cases are run for the second time on the first CPU architecture and the second CPU architecture respectively, the corresponding multi-dimensional performance data are collected respectively. Based on the multidimensional performance data, cross-architecture function matching is performed on the first CPU architecture and the second CPU architecture to obtain the successfully matched function; Determine the first clock cycle number of the successful matching function when running on the first CPU architecture and the second clock cycle number of the successful matching function when running on the second CPU architecture; When the ratio of the first clock cycle number to the second clock cycle number is greater than a reference threshold, the successful matching function is determined to be a function that causes performance defects in the software under test. The process of performing cross-architecture function matching based on the multidimensional performance data in the first and second CPU architectures to obtain successfully matched functions includes: The objective function in the multidimensional performance data collected from the first CPU architecture is used as the function index; Based on the function index, each function to be matched is traversed in the multidimensional performance data collected by the second CPU architecture; During the traversal, the function to be matched is determined to be a successfully matched function when the function index and the function to be matched satisfy the following conditions: The function to be matched and the function index come from the same base class and command pattern; The similarity between the function name of the function index and the function name of the function to be matched reaches a preset threshold.

2. The software performance defect discovery method based on cross-architecture comparison according to claim 1, characterized in that, The process of calculating the similarity between the function name of the function index and the function name of the function to be matched includes: Obtain the first function name string of the function index and the second function name string of the function to be matched; Vectorize the first function name string and the second function name string to obtain the corresponding first string vector and second string vector; Determine the vector similarity between the first string vector and the second string vector, and use the vector similarity as the similarity between the function name of the function index and the function name of the function to be matched.

3. The software performance defect discovery method based on cross-architecture comparison according to claim 1, characterized in that, The benchmark threshold is determined according to the SPEC CPU benchmark test program. The process of determining the benchmark threshold includes: In the first CPU architecture and the second CPU architecture, the SPEC CPU benchmark test program is executed respectively, and the corresponding first performance test results and second performance test results are collected during the execution process. The ratio of the first performance test result to the second performance test result is used as the benchmark threshold.

4. The software performance defect discovery method based on cross-architecture comparison according to claim 1, characterized in that, After the corresponding multidimensional performance data are collected, the method further includes: Based on the multidimensional performance data, cross-architecture function matching is performed on the first CPU architecture and the second CPU architecture to obtain unmatched functions. The unmatched functions are target functions whose multidimensional performance data collected on the first CPU architecture is used as function index and whose target functions are not matched in the multidimensional performance data collected on the second CPU architecture. The resource usage data of the unsuccessfully matched function in the first CPU architecture is determined. When the resource usage data reaches a set threshold, the unsuccessfully matched function is determined to be a function that causes performance defects in the software under test.

5. The software performance defect discovery method based on cross-architecture comparison according to claim 1, characterized in that, The method further includes: The function that causes performance defects in the software under test is optimized to obtain the optimized function; When the optimized function is run on the first CPU architecture and the second CPU architecture respectively, the corresponding running performance data is collected respectively. The performance optimization result of the optimized function is determined based on the aforementioned operational performance data; When the performance optimization result indicates that the optimized function is not a function that causes performance defects in the software under test, the function with performance defects in the software under test is updated according to the optimized function.

6. A software performance defect discovery device based on cross-architecture comparison, characterized in that, The device includes: The data acquisition module is used to collect first performance data and second performance data respectively when the test set of the software to be tested is run on the first CPU architecture and the second CPU architecture respectively. An anomaly detection module is used to take the ratio of the first performance data and the second performance data as the performance data point to be detected, and call the autoencoder to perform anomaly detection on the performance data point to be detected to obtain anomaly test cases. The autoencoder is adaptively optimized by the particle swarm optimization algorithm. The data acquisition module is also used to collect corresponding multi-dimensional performance data when the abnormal test cases are run for the second time on the first CPU architecture and the second CPU architecture respectively. The function matching module is used to perform cross-architecture function matching based on the multidimensional performance data in the first CPU architecture and the second CPU architecture to obtain successfully matched functions. The defect localization module is used to determine the first clock cycle number of the successful matching function when running on the first CPU architecture and the second clock cycle number of the successful matching function when running on the second CPU architecture. When the ratio of the first clock cycle number to the second clock cycle number is greater than a reference threshold, the successful matching function is determined to be a function that causes performance defects in the software under test. The process of performing cross-architecture function matching based on the multidimensional performance data in the first and second CPU architectures to obtain successfully matched functions includes: The objective function in the multidimensional performance data collected from the first CPU architecture is used as the function index; Based on the function index, each function to be matched is traversed in the multidimensional performance data collected by the second CPU architecture; During the traversal, the function to be matched is determined to be a successfully matched function when the function index and the function to be matched satisfy the following conditions: The function to be matched and the function index come from the same base class and command pattern; The similarity between the function name of the function index and the function name of the function to be matched reaches a preset threshold.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the software performance defect discovery method based on cross-architecture comparison as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the software performance defect discovery method based on cross-architecture comparison as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the software performance defect discovery method based on cross-architecture comparison as described in any one of claims 1 to 5.

Images