Methods, apparatus and storage media for performance testing of algorithm libraries for RISC-V devices

Abstract

This invention provides a method, apparatus, and storage medium for performance testing of algorithm libraries on RISC-V devices, relating to the field of software testing technology. The method includes: selecting multiple RISC-V devices as test nodes from a RISC-V platform; constructing an algorithm library based on a compiler to obtain multiple target compilation products of the algorithm library; running the same target compilation product on different test nodes for each target compilation product of the algorithm library to obtain test data corresponding to each test node; and using the test data corresponding to different test nodes to characterize the performance differences of different RISC-V devices. The algorithm library performance testing method, apparatus, and storage medium provided by this invention cover multiple RISC-V architectures, can automatically and comprehensively acquire performance data of the algorithm library on different devices, and improves the reliability of algorithm library performance testing.

Description

Technical Field

[0001] This invention relates to the field of software testing technology, and in particular to a method, apparatus and storage medium for testing the performance of algorithm libraries for RISC-V devices. Background Technology

[0002] Current high-performance algorithm libraries typically include performance test suites (test sets consisting of multiple unit test cases). Their compiled outputs (performance testing tools) can be run on devices to obtain performance data, which is used to evaluate the optimization effect during the development of the algorithm library. For a code modification aimed at optimizing performance, developers are usually required to use performance testing tools to compare the performance differences between the original solution before the modification and the optimized solution after the modification. By statistically analyzing and comparing the performance of the two solutions on key performance indicators, the effectiveness of the performance optimization can be ensured.

[0003] However, current algorithm library performance testing methods rely on manual testing, which is costly in terms of equipment and manpower, and is inefficient and lacks comprehensive analysis, resulting in poor reliability of performance testing. Summary of the Invention

[0004] This invention provides a method, apparatus, and storage medium for testing the performance of algorithm libraries for RISC-V devices, in order to solve the technical problem of poor reliability in the performance testing of algorithm libraries in the prior art.

[0005] In a first aspect, the present invention provides a method for testing the performance of algorithm libraries for RISC-V devices, comprising:

[0006] Multiple RISC-V devices were selected as test nodes from the fifth-generation RISC-V platform.

[0007] An algorithm library is built based on a compiler, resulting in multiple target compilation products of the algorithm library;

[0008] For each target compilation product of the algorithm library, the same target compilation product is run on different test nodes to obtain test data corresponding to each test node; the test data corresponding to different test nodes are used to characterize the performance differences of different RISC-V devices.

[0009] In some embodiments, the step of building an algorithm library based on a compiler to obtain multiple target compilation products of the algorithm library includes:

[0010] Obtain different versions of the compiler;

[0011] The same code version of the algorithm library is built based on different versions of compilers, resulting in multiple target compilation products in the algorithm library.

[0012] In some embodiments, the step of building an algorithm library based on a compiler to obtain multiple target compilation products of the algorithm library includes:

[0013] Using the same version of the compiler and different compilation options to build the same code version of the algorithm library, multiple target compilation products are obtained.

[0014] In some embodiments, the method further includes:

[0015] By combining different versions of compilers, different compilation options, and different test nodes, multiple target test environments can be obtained;

[0016] For each target test environment, obtain test data corresponding to different versions of the algorithm library code within the same target test environment.

[0017] In some embodiments, the method further includes:

[0018] If a new code submission is detected in the algorithm library, the test data corresponding to the new code in the target test environment is obtained.

[0019] In some embodiments, the method further includes:

[0020] By comparing the test data corresponding to the new code with the test data corresponding to the previous version of the code, a comparison result is obtained to determine whether there is a performance loss.

[0021] If a performance penalty occurs, the comparison results will be communicated.

[0022] In some embodiments, the method further includes:

[0023] Based on a preset period, test data corresponding to the current version of the code in the target test environment is periodically acquired;

[0024] Performance change data is generated based on test data for the current version of the code and test data for historical versions of the code.

[0025] Secondly, the present invention provides a performance testing device for algorithm libraries for RISC-V devices, comprising:

[0026] The master node is used to display the user interface of the algorithm library performance testing tools for RISC-V devices, and is used for triggering, generating and scheduling tasks, as well as saving test data.

[0027] Compilation nodes are used to build an algorithm library based on the compiler, and obtain multiple target compilation products of the algorithm library;

[0028] Test nodes are used to run the same compiled product of the algorithm library on different test nodes to obtain test data corresponding to each test node; the test nodes are multiple RISC-V devices selected from the RISC-V platform; the test data corresponding to different test nodes are used to characterize the performance differences of different RISC-V devices.

[0029] Resource nodes are used to store the test datasets required by the test nodes during the testing process.

[0030] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the algorithm library performance testing method for RISC-V devices as described above.

[0031] Fourthly, a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the algorithm library performance testing method for RISC-V devices as described above.

[0032] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the algorithm library performance testing method for RISC-V devices as described above.

[0033] The present invention provides a method, apparatus, and storage medium for performance testing of algorithm libraries for RISC-V devices. By constructing a RISC-V device test cluster and using a compiler to generate multi-target compilation products of the algorithm library, the same compilation product is executed on different test nodes in the RISC-V device test cluster and running data is collected. This allows for quantitative analysis of the impact of differences between different RISC-V devices on algorithm efficiency, thereby achieving fully automated testing. It can obtain performance data of the algorithm library on different devices while covering multiple RISC-V architectures, improving the reliability and evaluation efficiency of algorithm library performance testing. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the performance testing method for algorithm libraries for RISC-V devices provided by this invention.

[0036] Figure 2 This is a schematic diagram of the structure of the algorithm library performance testing device for RISC-V devices provided by the present invention.

[0037] Figure 3 This is a schematic diagram of the initialization process of the test node provided by the present invention.

[0038] Figure 4 This is a schematic diagram of the initialization process of the compilation node and resource node provided by the present invention.

[0039] Figure 5 This is a schematic diagram of the initialization process of the master node provided by the present invention.

[0040] Figure 6 This is a flowchart illustrating a specific example of the performance testing method for algorithm libraries for RISC-V devices provided by this invention.

[0041] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0042] Algorithm libraries typically need to be compatible with various hardware devices. However, RISC-V, as an emerging hardware platform, has not yet formed a mature hardware platform solution. (Compared to traditional devices such as ARM / x86 architecture hosts and servers, RISC-V is an open-source instruction set architecture (ISA) based on the Reduced Instruction Set Computer (RISC) principle, while ARM is an Advanced RISC Machine. RISC-V devices mostly exist in the form of development boards, and there are instruction set differences between various development boards, making the acquisition, configuration, and use of these boards quite difficult.) Performance may vary across different devices, requiring comprehensive testing covering various devices. Developers may lack the necessary testing equipment, and equipping each developer with all the necessary devices would be extremely costly.

[0043] Comparing the performance differences between the original solution before modification and the optimized solution after modification requires developers to compile and run two versions (or more, depending on the complexity of the optimization solution) separately. Furthermore, performance test suites for large algorithm libraries typically include thousands of unit tests, resulting in long runtimes. Additionally, different compilers (such as GCC and LLVM), different compiler versions (such as GCC 13.3 and GCC 14.2), and different compiler configuration options (such as whether to enable automatic vectorization) can all affect the performance of the algorithm library. These differences all require separate compilation and execution during performance testing. Using current testing techniques is not only laborious and time-consuming, resulting in low testing efficiency, but also leads to incomplete and inaccurate manual analysis and comparison.

[0044] Other modifications to the algorithm library (non-RISC-V platform optimizations, such as adding / modifying features, fixing vulnerabilities, or performance optimizations for other platforms) may unintentionally affect the performance of RISC-V devices. It is necessary to perform the aforementioned performance tests (on different options, multiple versions, and multiple devices) for each modification to promptly identify unexpected performance impacts (i.e., to conduct performance monitoring). Current solutions that rely on developers to perform performance testing cannot automatically monitor the performance impact of code repository modifications, resulting in poor reliability.

[0045] To address the aforementioned technical issues, this invention proposes a performance testing method for algorithm libraries on RISC-V devices. The method involves selecting multiple RISC-V devices as test nodes from the RISC-V platform; constructing an algorithm library based on a compiler to obtain multiple target compilation artifacts for the algorithm library; running the same target compilation artifact on different test nodes for each target compilation artifact to obtain test data corresponding to each test node; and using the test data from different test nodes to characterize the performance differences between different RISC-V devices. This method achieves automated performance testing, covers multiple RISC-V architectures, and comprehensively obtains performance data of the algorithm library on different devices, thereby improving the reliability of algorithm library performance testing.

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] Figure 1 This is a flowchart illustrating the performance testing method for algorithm libraries targeting RISC-V devices provided by this invention, as shown below. Figure 1As shown, this invention provides a method for performance testing of algorithm libraries for RISC-V devices. The method includes:

[0048] Step 101: Select multiple RISC-V devices from the RISC-V platform as test nodes.

[0049] Specifically, taking into account the modularity of the instruction set of the fifth-generation Reduced Instruction Set Computer-V (RISC-V) platform, representative devices that may affect the performance of the algorithm library are categorized and selected as hardware for running tests, thus obtaining multiple test nodes.

[0050] Test nodes are categorized based on differences in RISC-V instruction extensions supported by the devices and resources such as clock speed and memory. Each test node must contain at least one type of device, and each type of device must have at least one device instance. The more types of test nodes there are, the more hardware features the tool can cover, the more device instances can be used for testing, and the more test tasks can be executed concurrently.

[0051] Step 102: Build an algorithm library based on the compiler to obtain multiple target compilation products of the algorithm library.

[0052] Specifically, an algorithm library is built based on a compiler, resulting in different compilation artifacts, i.e., multiple target compilation artifacts. These compilation artifacts are performance test programs—programs driven by testing tools that run and time given performance test cases using test data.

[0053] Step 103: For each target compilation product of the algorithm library, run the same target compilation product on different test nodes to obtain test data corresponding to each test node; the test data corresponding to different test nodes are used to characterize the performance differences of different RISC-V devices.

[0054] Specifically, for each target compilation product of the algorithm library, it is run on different test nodes to obtain test data corresponding to each test node. The test data corresponding to different test nodes is used to characterize the performance differences of different RISC-V devices.

[0055] The algorithm library performance testing method for RISC-V devices provided in this application constructs a RISC-V device test cluster and uses a compiler to generate multi-target compilation products of the algorithm library. Then, the same compilation product is executed on different test nodes in the RISC-V device test cluster and running data is collected to quantitatively analyze the impact of differences between different RISC-V devices on algorithm efficiency. This achieves fully automated testing and can obtain performance data of the algorithm library on different devices while covering multiple RISC-V architectures, thus improving the reliability and evaluation efficiency of algorithm library performance testing.

[0056] In some embodiments, the step of building an algorithm library based on a compiler to obtain multiple target compilation products of the algorithm library includes:

[0057] Obtain different versions of the compiler;

[0058] The same code version of the algorithm library is built based on different versions of compilers, resulting in multiple target compilation products in the algorithm library.

[0059] Specifically, taking advantage of the rapid iteration speed of RISC-V compiler backend development, the release status of mainstream compilers is tracked, new versions of compilers are automatically built, and historical versions of compilers are maintained to form a set of compilers containing different versions. This set is used to evaluate the performance impact of different compilers.

[0060] The same code version of the algorithm library was built using different versions of compilers, resulting in different target compilation products. The test data can be used to analyze the impact of compiler differences on performance.

[0061] The algorithm library performance testing method for RISC-V devices provided in this application takes into account the rapid updates and iterations of the RISC-V compiler backend, which may affect the execution performance of the program on the RISC-V device. By automatically acquiring and building new versions of the compiler, it is helpful to locate the performance changes caused by compiler version differences (such as the implementation of new optimization algorithms / adaptation to more RISC-V features, etc.) during performance analysis, making the algorithm library performance analysis more comprehensive and accurate.

[0062] In some embodiments, the step of building an algorithm library based on a compiler to obtain multiple target compilation products of the algorithm library includes:

[0063] Using the same version of the compiler and different compilation options to build the same code version of the algorithm library, multiple target compilation products are obtained.

[0064] Specifically, we set a set of compilation options or project configuration options that might change in real-world application scenarios for the algorithm library. Using the same compiler version and different compilation options, we build the same code version of the algorithm library, obtaining different target compilation artifacts. The test data can then be used to analyze the impact of compilation options on performance.

[0065] Different optimization options are helpful in pinpointing the source of performance changes during performance analysis. For example, compilers have automatic vectorization capabilities. When compiler optimizations are enabled (compile options O2 / O3 or project configuration option Release), the compiler will attempt to automatically generate vector instructions, but the optimization effect is somewhat unpredictable. When algorithm library developers manually implement vector optimizations for RISC-V vector extension devices using vectorization optimization methods, the source of performance changes is difficult to determine (e.g., whether it stems from manual optimization, compiler automatic vectorization, or a combination of both).

[0066] The algorithm library performance testing method for RISC-V devices provided in this application embodiment can disable automatic vectorization by setting additional compilation options (such as fno-vectorize) without affecting other optimizations, thereby helping developers better pinpoint the causes of performance changes.

[0067] In some embodiments, the method further includes:

[0068] By combining different versions of compilers, different compilation options, and different test nodes, multiple target test environments can be obtained;

[0069] For each target test environment, obtain test data corresponding to different versions of the algorithm library code within the same target test environment.

[0070] Specifically, by combining different versions of compilers, different compilation options, and different test nodes, multiple target test environments can be obtained. Each target test environment is associated with a version of the compiler, a compilation option, and a test node. For a specific version of the algorithm library source code, a set of performance test data can be obtained in multiple target test environments. This set of test data can characterize the performance of the algorithm library in the current code version.

[0071] In this embodiment of the application, developers are allowed to specify multiple different code versions of the algorithm library, and multiple sets of test data will be automatically obtained in the target test environment (each code version of the algorithm library corresponds to a set of test data).

[0072] The algorithm library performance testing method for RISC-V devices provided in this application embodiment can automatically obtain multiple sets of test data for multiple code versions, thereby comparing and analyzing the performance differences of source code modifications in different environments, helping developers to more easily and comprehensively optimize the performance of algorithm libraries on RISC-V devices.

[0073] In some embodiments, the method further includes:

[0074] If a new code submission is detected in the algorithm library, the test data corresponding to the new code in the target test environment is obtained.

[0075] Specifically, the source code changes of the algorithm library are monitored in real time. When a new code submission is detected, a test process of the new code in the target test environment will be automatically triggered to obtain the test data corresponding to the new code.

[0076] When monitoring changes to the source code of the algorithm library code repository, a Webhook can be set in the code repository so that when changes occur, the code repository actively sends a command to the platform to trigger a task, thereby automatically triggering the pipeline script.

[0077] The algorithm library performance testing method for RISC-V devices provided in this application embodiment triggers a testing process based on the new code when new code is submitted, thereby automating performance testing and enabling real-time acquisition of the latest algorithm library performance results.

[0078] In some embodiments, the method further includes:

[0079] By comparing the test data corresponding to the new code with the test data corresponding to the previous version of the code, a comparison result is obtained to determine whether there is a performance loss.

[0080] If a performance penalty occurs, the comparison results will be communicated.

[0081] Specifically, after obtaining the test data corresponding to the new code, the test data corresponding to the new code is compared with the test data corresponding to the previous version of the code to obtain the comparison results of whether there is a performance change or loss. If a performance loss occurs, a notification is sent to inform the developers of the comparison results, which can be done via email or other means.

[0082] For example, using the test data corresponding to the previous version of the code as the baseline data and the test data corresponding to the new code as the comparison data, calculate the speedup ratio corresponding to the test data corresponding to the previous version of the code and the test data corresponding to the new code. That is, compare the baseline time with the time. If the speedup ratio is less than a certain preset value (such as 0.95), it indicates that the performance has degraded.

[0083] The algorithm library performance testing method for RISC-V devices provided in this application compares the test data corresponding to the new code with the test data corresponding to the previous version of the code to obtain the comparison results of whether there is a performance change or loss. This helps developers better locate the performance changes caused by code changes (such as the implementation of optimized algorithms) and improve the quality of the code repository.

[0084] In some embodiments, the method further includes:

[0085] Based on a preset period, test data corresponding to the current version of the code in the target test environment is periodically acquired;

[0086] Performance change data is generated based on test data for the current version of the code and test data for historical versions of the code.

[0087] Specifically, based on a preset period, such as daily or weekly, the testing process for the current version of the code is periodically triggered to obtain the test data corresponding to the current version of the code in the target test environment. Combined with the test data corresponding to the retained historical versions of the code, traceable performance change data is generated to monitor the performance changes of the algorithm library on the RISC-V device.

[0088] The algorithm library performance testing method for RISC-V devices provided in this application embodiment can set periodic triggering of performance tests and save the test data for each test, so that developers can improve the quality of the code repository by monitoring the performance changes of the algorithm repository on the RISC-V device.

[0089] The following specific examples further illustrate the performance testing methods for the algorithm libraries for RISC-V devices provided in the above embodiments:

[0090] This application provides an algorithm library performance analysis and monitoring tool based on an algorithm library performance testing method for RISC-V devices. The tool involves both hardware and software components.

[0091] Figure 2 This is a schematic diagram of the structure of the algorithm library performance testing device for RISC-V devices provided by the present invention, as shown below. Figure 2 As shown, the performance testing device for the algorithm library on RISC-V devices is the hardware part of the aforementioned tool. Specifically, the hardware part consists of different device nodes, which can be divided into four types according to the different tasks undertaken by the device nodes: master node 201, compilation node 202, test node 203, and resource node 204.

[0092] The tool platform runs on the master node 201 and is responsible for functions such as user interface presentation, task triggering, subtask (compilation, testing) generation and scheduling, and result collection and saving.

[0093] Compilation node 202 needs to deploy different versions of various compilers to undertake the compilation of algorithm libraries for multiple test environments, and the generated compilation artifacts are used for performance testing.

[0094] Test node 203 actually runs the compiled artifacts (i.e., the performance test program, a program driven by the testing tool that runs and times the test data for a given performance test case. It is designed and implemented by each algorithm library itself) and generates performance test data results.

[0095] Resource node 204 stores shared resources such as test datasets required by nodes during the testing process, which facilitates unified maintenance and management of the required data and saves storage resources (hard disk space) for each test node.

[0096] Each node should have an independent IP address. The master node and resource nodes should be able to establish bidirectional network communication with any other node in the form of IP address. Communication between other nodes is not required.

[0097] In some embodiments, there is typically one and only one master node, which can run on a device with any instruction set architecture.

[0098] There can be one or more compilation nodes. Due to the high computational resource requirements of the compilation process, they typically run on high-performance servers based on x86 or ARM architectures. The more compilation nodes there are, the more concurrent compilation tasks can be performed, and the better the performance of the compilation nodes, the shorter the time required for each individual task.

[0099] Test nodes are RISC-V architecture devices, categorized based on differences in supported RISC-V instruction extensions and resources such as clock speed and memory. Each test node must contain at least one type of device, and each type must have at least one device instance. The more types of test nodes there are, the more hardware features the tool can cover, the more device instances can be used for testing, and the more test tasks can be executed concurrently.

[0100] There is usually only one resource node. It can exist independently or be the master node or any compilation node.

[0101] In this embodiment, the test node may include four types of RISC-V devices, focusing on RISC-V Vector (RVV) instruction set extensions: Lichee Pi 4A (a development board equipped with TH1520, supporting RVV 0.7 extended instruction set, VLEN=128bit); CanMV-k230 (a development board equipped with k230 processor, supporting RVV 1.0 instruction set extension, VLEN=128bit); and BPI-F3 (equipped with K1 processor, supporting RVV 1.0 instruction set extension, VLEN=256bit).

[0102] The software consists of a tool platform on the master node, agent programs and compilers on the compilation nodes, agent programs and datasets on the resource nodes, and agent programs on the test nodes. After initialization, it is available for developers (users). The initialization process mainly includes three steps: using an open-source continuous integration software (such as Jenkins) build tool platform, establishing network connections between the master node and other nodes using its device connection and management functions, and describing tasks using its pipeline task scripting capabilities.

[0103] Figure 3 This is a schematic diagram of the initialization process of the test node provided by the present invention, as shown below. Figure 3 As shown, the initialization steps for the test node are as follows:

[0104] Step 301: Select suitable equipment as test nodes according to the algorithm library requirements.

[0105] Algorithm libraries can be optimized for performance based on the different hardware characteristics of the RISC-V platform. The devices used as test nodes should possess the hardware characteristics employed by the algorithm library to better analyze the impact of hardware differences on performance. Currently, many algorithm libraries employ vectorization techniques to focus on optimizing their performance on RISC-V vector processors (processors with RISC-V vector extensions). This invention evaluates the performance optimization effect of RISC-V vector extensions by selecting different RISC-V vector processors as test devices. Specifically, RISC-V vector extensions have two widely adopted versions, v0.7 and v1.0, with different vector register lengths (VLEN) within the same version, all of which can affect performance. Therefore, this invention uses the following development board to support the analysis of the impact of different factors on performance:

[0106] (1) Lichee Pi 4A (RVV v0.7, VLEN=128bit);

[0107] (2) CanMV-k230 (RVV v1.0, VLEN=128bit);

[0108] (3) BPI-F3 (RVV v1.0, VLEN=256bit);

[0109] Development boards of versions (1) and (2) have the same vector register length but use different versions of the instruction set: comparing their performance data can help analyze the impact of instruction set version differences on performance. Development boards of versions (2) and (3) have the same instruction set version but different vector register lengths: comparing their performance data can help analyze the impact of vector register length differences on performance.

[0110] Step 302: Install the agent program and its required runtime environment.

[0111] Install the agent program corresponding to the selected integration testing platform (provided by the platform, usually installed using a package manager / script) so that it can handle tasks issued by the master node.

[0112] Step 303: Configure the network to enable it to communicate with the master node and resource nodes.

[0113] Obtain the IP address so that the master node can communicate with it (to send tasks and upload results).

[0114] Step 304: Mount the resource node and set environment variables.

[0115] Mount the test dataset from the resource node to a file directory on the test node device, and set that directory as an environment variable (e.g., ...). Environment variables will be used for writing test scripts in the master node, thus allowing test tasks to access test data when running on different test devices.

[0116] Figure 4 This is a schematic diagram of the initialization process of the compilation node and resource node provided by the present invention, as shown below. Figure 4 As shown, the initialization steps for compilation nodes and resource nodes are as follows:

[0117] Step 401: Install the agent program and its required runtime environment (same as step 302).

[0118] Step 402: Configure the network to enable it to communicate with the master node (same as step 303).

[0119] Step 403: Build a specific version of the compiler / load specific resources according to requirements.

[0120] For the compilation node, different versions of different compilers are needed to compile different outputs for algorithm library performance testing. The different compilers supported by the platform are specified during initialization. For example, this invention supports the mainstream GCC and Clang compilers; one or more supported compiler versions can be specified during initialization (e.g., GCC 14.2.0 and Clang 18.1.8, Clang 19.1.0, with at least one version of each compiler being initialized and built). Subsequently, for each compiler (GCC and Clang), the master node will periodically trigger the task of "checking and building new compiler versions" (corresponding to step 504), thereby building new compilers as their versions are released. The build node should maintain a "list of built compilers" (e.g., saved as a text file in the compiler's build directory), which contains all built compilers and their versions.

[0121] For the resource server, the test dataset it provides to the test nodes needs to be downloaded during initialization. Additionally, since the dataset may be updated as the algorithm library iterates, the master node will periodically trigger the "update test dataset" task (corresponding to step 503).

[0122] Step 404: Set environment variables.

[0123] Set the directory containing each compiler in the compilation node as an environment variable (e.g., This allows compilation tasks to find the compiler on different compilation node devices when running.

[0124] Figure 5 This is a schematic diagram of the master node initialization process provided by the present invention, as follows: Figure 5 As shown, the initialization steps for the master node are as follows:

[0125] Step 501: Install the integration test platform and its required operating environment.

[0126] Select an integration testing platform that meets the functional requirements (this invention selects the open-source software Jenkins), and install the integration testing platform and its required runtime environment on the master node.

[0127] Step 502: Add build node, test node, and resource node.

[0128] Add the initialized nodes to the test platform and add tags to each node: Distinguish by node type, adding tags such as "master" (master node), "builder" (builder node), "runner" (test node), and "feeder" (resource node); distinguish by node architecture, adding tags such as "RISC-V" and "x86". For RISC-V architecture devices, additional tags describing their vector extension characteristics should be added, such as "RVV1.0" / "RVV0.7", "VLEN128" / "VLEN256", etc.

[0129] Step 503: Write and add pipeline scripts for updating the test dataset.

[0130] This script's function is to pull data from a specified Git repository containing test data. Since the test dataset only exists on resource nodes, the script should be configured to run on nodes with the tag "feeder". When executed, the script will navigate to the directory containing the test dataset and perform a pull operation (git pull), thus updating the dataset. Add the script to the platform and set a daily trigger (e.g., trigger at midnight every day) to achieve automatic daily dataset updates.

[0131] Step 504: Write and add pipeline scripts for checking the build of the new version of the compiler.

[0132] This script's function is to determine if a new version has been released for an existing compiler source code repository based on its commit tag. If so, it switches to the new version of the source code and builds the corresponding new compiler. This script should be set to run on nodes with the tag "builder". When the script executes, it will navigate to the path containing the compiler source code repository, check for the existence of a new commit tag, and if found, build the new compiler and add it to the list of built compilers. The check for a new version is achieved by checking if the latest commit tag of the source code repository appears in the list of built compilers. Add the script to the platform and set a daily task trigger (e.g., trigger at 00:00 every day) to automatically check for and build new compiler versions daily.

[0133] Step 505: Write and add pipeline scripts for compiling the performance test program.

[0134] This script's function is to build a performance test program for a specific version of an algorithm library, given a version and configuration parameters, using a specified compiler and compilation options. The task first checks out the algorithm library to the given version, then compiles it into a performance test program using the specified compiler, compilation options, and algorithm library configuration parameters. This task should be set to run on nodes with the tag "builder". The compilation process should be performed in a specific directory to prevent interference between different builds. The naming convention for the compilation directory is build-<algorithm library name>-<algorithm library code version ID>-<compiler name and version>-<algorithm library configuration parameter identifier>, for example, the build-opencv-4.x-llvm-19.1.1-normal directory contains the compilation artifacts of the OpenCV algorithm library version 4.x built using the llvm 19.1.1 compiler with normal configuration. The compilation process should save log files (e.g., dumping the command-line output during compilation to a file) for review in case of compilation failures. After compilation, the resulting performance test program should be temporarily stored for use in subsequent tasks.

[0135] Step 506: Write and add pipeline scripts for running the performance test program.

[0136] This script executes all given performance test programs on specific devices to obtain performance data. The task first iterates through all performance test programs to be executed. Then, for each type of device, it creates parallel subtasks that should run on nodes labeled "runner". Specifically, to differentiate between device types and execute test programs separately, more labels should be used, and a logical AND operation should be used to ensure that specific subtasks run on specific devices: for example, the labels "runner&&RVV1.0&&VLEN128" identify the CanMV-k230 device (only devices of this type have all three labels). Finally, the test results are temporarily stored. The test results are saved in XML format. To distinguish the test results of different tasks on different devices, a naming convention should be applied to each test output: result-<test program name>-<test device node name>.xml.

[0137] Step 507: Write and add pipeline scripts to trigger tasks, collect test results, and process them.

[0138] The function of this script is to trigger one or more compile-run tasks as needed when certain conditions are met, and to collect and process the test results. These specific conditions include:

[0139] ① Code change trigger: When code changes occur in the algorithm library code repository (this is achieved by setting up Webhooks in the code repository; when new code is submitted to the repository, a POST request will be sent to the platform, thereby triggering this pipeline script).

[0140] ② Periodic trigger: Triggered at specific times (weekly / daily periodic trigger).

[0141] ③ Manual triggering: triggered by manual operation (developers manually using the platform's interactive interface).

[0142] When the trigger condition is ①, the script triggers all compilers and all versions of them to compile and run the current version of the algorithm library (i.e., the modified source code) on all types of devices, and compares the performance data of the current version with the version before the change. If a performance loss occurs, a notification is sent (via email, etc.).

[0143] When the trigger condition is ②, the script triggers the compilation-run task of all compilers and all versions of them on all types of devices for the current default version of the algorithm library at the current time point, and saves the test results for future reference, thus allowing developers to backtrack on performance changes by viewing the results of previous tests.

[0144] When the trigger condition is ③, developers are allowed to select specific parameters such as compiler / compiler version / compilation options / performance test program to run / device for running the test program, and specify one or more source code versions from code repositories. If the developer specifies one version of the source code, a compile-run task is triggered using the given parameters, and its results are collected. If the developer specifies multiple versions of the source code, a compile-run task is triggered for each version of the source code using the given parameters, and the performance differences between different versions are compared after all test results are collected.

[0145] Figure 6 This is a flowchart illustrating a specific example of the performance testing method for algorithm libraries aimed at RISC-V devices provided by this invention. Figure 6 As shown, the trigger type is first determined. If it is triggered by code changes, the two code repository versions before and after the change are regarded as the code version to be compiled. If it is triggered periodically, the default branch of the code repository at the current moment is regarded as the code version to be compiled. If it is triggered manually, one or more code repository versions specified by the user are regarded as the code version to be compiled.

[0146] Then, a code version to be compiled is detected, and different compilers and compilation options are combined to form a set of compilation configurations. If an unused compilation configuration exists, it is selected to trigger a compilation task on the compilation node using the current configuration. The compilation task is executed, and the compilation artifacts (i.e., performance test programs) are temporarily stored. Simultaneously, it continues to check if any unused compilation configurations exist. If no unused compilation configurations exist and an undeployed device type exists, an undeployed device type is selected. It then checks if any unexecuted performance test programs exist. If so, the test programs are started on devices matching the current device type, and the test tasks are executed. The test results (i.e., test data) are temporarily stored. Simultaneously, it continues to check if any unexecuted performance test programs exist. If no unexecuted performance test programs exist, it continues to check if any undeployed device types exist. If no unused compilation configurations exist, no undeployed device types exist, and a code version to be compiled exists, the testing process for that code version is re-executed. If there are no unused compilation configurations, no undeployed device types, and no code versions to be compiled, collect test results. If there is only one code version, save its performance data as the result and end the process. If there is more than one code version, select the test results from the earlier code repository as the baseline and select another version's test results as comparison data. Calculate the speedup ratio by dividing the baseline time by the total time. If the speedup ratio is greater than 1.05, it is considered a performance improvement; if it is less than 0.95, it is considered a performance degradation; if the speedup ratio is between 0.95 and 1.05, it is considered no performance difference. If there are no uncalculated speedup ratios and no uncompared test result versions, save this performance comparison data as the result and end the process. If there are no uncalculated speedup ratios but there are still uncompared test result versions, continue comparing the test results of other versions.

[0147] The algorithm library performance testing device for RISC-V devices provided by the present invention can implement all the method steps implemented in the above-described algorithm library performance testing method embodiment for RISC-V devices, and can achieve the same technical effect. Here, the beneficial effects of the parts that are the same as those in the method embodiment will not be described in detail.

[0148] It should be noted that the division of units / modules in the above embodiments of the present invention is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0149] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include: a processor 701, a communications interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communications interface 702, and the memory 703 communicate with each other via the communication bus 704. The processor 701 can call logical instructions in the memory 703 to execute a performance testing method for an algorithm library targeting RISC-V devices. This method includes:

[0150] Multiple RISC-V devices were selected as test nodes from the fifth-generation RISC-V platform.

[0151] An algorithm library is built based on a compiler, resulting in multiple target compilation products of the algorithm library;

[0152] For each target compilation product of the algorithm library, the same target compilation product is run on different test nodes to obtain test data corresponding to each test node; the test data corresponding to different test nodes are used to characterize the performance differences of different RISC-V devices.

[0153] Specifically, the processor 701 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor can also adopt a multi-core architecture.

[0154] When the logical instructions in memory 703 can be implemented as software functional units and sold or used as independent products, they can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] In some embodiments, a computer program product is also provided, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the algorithm library performance testing method for RISC-V devices provided in the above method embodiments, the method including:

[0156] Multiple RISC-V devices were selected as test nodes from the fifth-generation RISC-V platform.

[0157] An algorithm library is built based on a compiler, resulting in multiple target compilation products of the algorithm library;

[0158] For each target compilation product of the algorithm library, the same target compilation product is run on different test nodes to obtain test data corresponding to each test node; the test data corresponding to different test nodes are used to characterize the performance differences of different RISC-V devices.

[0159] Specifically, the computer program product provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0160] In some embodiments, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program, the computer program being used to cause a computer to execute the algorithm library performance testing method for RISC-V devices provided in the above method embodiments, the method comprising:

[0161] Multiple RISC-V devices were selected as test nodes from the fifth-generation RISC-V platform.

[0162] An algorithm library is built based on a compiler, resulting in multiple target compilation products of the algorithm library;

[0163] For each target compilation product of the algorithm library, the same target compilation product is run on different test nodes to obtain test data corresponding to each test node; the test data corresponding to different test nodes are used to characterize the performance differences of different RISC-V devices.

[0164] Specifically, the computer-readable storage medium provided by the present invention can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0165] It should be noted that the computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical storage (e.g., CD, DVD, BD, HVD), and semiconductor storage (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0166] In this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0167] In this invention, "determining B based on A" means that factor A must be considered when determining B. It is not limited to "B can be determined based solely on A," but should also include: "determining B based on A and C," "determining B based on A, C, and E," "determining C based on A, and further determining B based on C," etc. Additionally, it can include using A as a condition for determining B, for example, "when A satisfies the first condition, B is determined using the first method"; or "when A satisfies the second condition, B is determined," etc.; or "when A satisfies the third condition, B is determined based on the first parameter," etc. Of course, it can also be a condition where A is a factor in determining B, for example, "when A satisfies the first condition, C is determined using the first method, and B is further determined based on C," etc.

[0168] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0169] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These processors can execute instructions that can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0172] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for testing the performance of an algorithm library for a RISC-V device, characterized in that, The method comprises the following steps: selecting multiple RISC-V devices as test nodes from a fifth-generation reduced instruction set RISC-V platform; constructing an algorithm library based on a compiler to obtain multiple target compilation products of the algorithm library; for each target compilation product of the algorithm library, running the same target compilation product on different test nodes to obtain test data corresponding to each test node; the test data corresponding to different test nodes is used to represent the performance difference of different RISC-V devices; obtaining a code version to be compiled; based on the code version to be compiled, combining different compilers and compilation options to form a group of compilation configurations; if there is an unused compilation configuration, selecting an unused compilation configuration to trigger a compilation task using the current compilation configuration on a compilation node, executing the compilation task, and temporarily storing the compilation product; at the same time, it is continuously judged whether there is an unused compilation configuration, if there is no unused compilation configuration and there is an undeployed device type, an undeployed device type is selected, and it is judged whether there is an unexecuted performance test program; if there is an unexecuted performance test program, starting the test program on a device conforming to the current device type and executing a test task, and temporarily storing the test result; at the same time, it is continuously judged whether there is an unexecuted performance test program; if there is no unexecuted performance test program, it is continuously judged whether there is an undeployed device type; if there is no unused compilation configuration and no undeployed device type, and there is a code version to be compiled, the test process of the code version to be compiled is re-executed; if there is no unused compilation configuration and no undeployed device type, and there is no code version to be compiled, the test results are collected; if there is only one code version, the performance data of this version is saved as the result, and the process ends; if there is not only one code version, the test result corresponding to the earlier version of the code repository is selected as the benchmark, and the test result of another version is selected as the comparison data, the speedup ratio is calculated by dividing the benchmark time by the test result time of another version, if it is judged that there is no uncalculated speedup ratio and no uncomparable test result version, the obtained performance comparison data is saved as the result, and the process ends; if it is judged that there is no uncalculated speedup ratio but there is an uncomparable test result version, other version test results are continuously compared.

2. The RISC-V device oriented algorithm library performance test method of claim 1, wherein, The method further comprises the following steps: obtaining different versions of compilers; constructing an algorithm library of the same code version based on different versions of compilers to obtain multiple target compilation products in the algorithm library.

3. The RISC-V device oriented algorithm library performance test method of claim 1, wherein, The method further comprises the following steps: using the same version of the compiler and different compilation options to construct an algorithm library of the same code version to obtain multiple target compilation products.

4. The RISC-V device oriented algorithm library performance test method of claim 1, wherein, The method further comprises the following steps: combining different versions of compilers, different compilation options and different test nodes to obtain multiple target test environments; for each target test environment, obtaining test data corresponding to different versions of codes of the algorithm library under the same target test environment.

5. The RISC-V device oriented algorithm library performance test method of claim 4, wherein, The method further comprises the following steps: In a case where it is monitored that new code is submitted to the algorithm library, test data corresponding to the new code under the target test environment is acquired.

6. The RISC-V device oriented algorithm library performance test method of claim 5, wherein, The method further includes: By comparing the test data corresponding to the new code with test data corresponding to a previous version of code, a comparison result of whether performance loss is generated is obtained; In a case where performance loss is generated, the comparison result is notified.

7. The RISC-V device oriented algorithm library performance test method of claim 4, wherein, The method further includes: Based on a preset period, test data corresponding to a current version of code under the target test environment is periodically acquired; Based on the test data corresponding to the current version of code and test data corresponding to historical versions of code, performance variation data is generated.

8. A RISC-V device oriented algorithm library performance testing apparatus, characterized in that, The method includes: A master node is configured to display a user interface of an algorithm library performance test tool for RISC-V devices, trigger, generate and schedule tasks, and save test data; A compilation node is configured to build an algorithm library based on a compiler to obtain a plurality of target compilation products of the algorithm library; A test node is configured to run the same target compilation product on different test nodes for each target compilation product of the algorithm library to obtain test data corresponding to each test node; the test node is a plurality of RISC-V devices selected from a RISC-V platform; and the test data corresponding to different test nodes is used to represent performance differences of different RISC-V devices; A resource node is configured to store a test data set required by the test node in a test process; The device is specifically configured to: Acquire a code version to be compiled; Based on the code version to be compiled, different compiler-compile options are combined to form a group of compile configurations; If there is an unused compile configuration, an unused compile configuration is selected to trigger a compilation task using the current compile configuration on the compilation node, execute the compilation task, and temporarily store the compilation product; at the same time, it is continuously determined whether there is an unused compile configuration, if there is no unused compile configuration and there is an undeployed device type, an undeployed device type is selected, and it is determined whether there is an unexecuted performance test program; If there is an unexecuted performance test program, the test program is started on a device conforming to the current device type, and a test task is executed, and the test result is temporarily stored; at the same time, it is continuously determined whether there is an unexecuted performance test program; If there is no unexecuted performance test program, it is continuously determined whether there is an undeployed device type; If there is no unused compile configuration and no undeployed device type, and there is a code version to be compiled, the test process for the code version to be compiled is re-executed; If there is no unused compile configuration and no undeployed device type, and there is no code version to be compiled, test results are collected; If there is only one code version, the performance data of this version is saved as a result, and the process ends; If there is only one code version, the performance data of this version is saved as a result, and the process ends; If there is not only one code version, the test result corresponding to the code repository of the earlier version is selected as the benchmark, and the test result of another version is selected as the comparison data, the speedup ratio is calculated by dividing the benchmark time by the test result time of another version, if it is judged that there is no uncalculated speedup ratio and no uncomparable test result version, the performance comparison data obtained is saved as the result, and the process ends; If it is judged that there is no uncalculated speedup ratio but there is an uncomparable test result version, the test results of other versions are continuously compared.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the algorithm library performance test method for RISC-V devices as claimed in any one of claims 1 to 7 when executing the computer program.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the algorithm library performance test method for RISC-V devices as claimed in any one of claims 1 to 7.

Images