Method and system for testing compatibility and performance of multiple frameworks

Abstract

The invention provides a method and a system for testing compatibility and performance of multiple frameworks. The method comprises the following steps: creating a virtual test environment according to environment information of different autonomous controllable platforms based on a virtualization technology; each virtual test environment simulates the combination of different environment information; obtaining a to-be-tested operator based on the operation information of the different autonomous controllable platforms, and generating a test case based on the characteristic information of the operator and the environment information of the different autonomous controllable platforms; distributing each test case to a corresponding virtual test environment for execution based on the test environment configuration information, and obtaining a test execution result and performance index data in the execution process of each test case; and determining compatibility evaluation results and performance analysis results of the operator under different autonomous controllable platforms based on the test execution result and the performance index data. According to the embodiment of the invention, the efficiency, accuracy and universality of operator compatibility and performance testing are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a method and system for testing the compatibility and performance of multiple frameworks. Background Art

[0002] In the current era of rapid development of information technology, with the continuous advancement of independent and controllable technologies, more and more software and algorithms need to run on different independent and controllable platforms. Among them, as the basic computing unit in software and algorithms, the compatibility and performance of operators on different platforms are crucial.

[0003] Currently, for the compatibility and performance testing of operators, mainly manual testing methods are used. That is, testers need to manually install and configure the test environment on different hardware devices and operating systems, and run the operators one by one for testing. First, the manual testing process is cumbersome and time-consuming, requiring a large amount of human and time costs. Testers need to frequently switch between different hardware and operating systems, performing repetitive installation, configuration, and testing operations, resulting in extremely low efficiency. Second, the accuracy and reliability of manual testing are difficult to guarantee. Due to human factors in the testing process, such as operation errors and missed tests, the test results may be inaccurate and cannot truly reflect the compatibility and performance of operators on different platforms. Moreover, most of the existing methods can only test for a single type of platform or environment and cannot achieve cross-platform compatibility evaluation. Therefore, the existing methods have low efficiency, poor accuracy, and poor generality for the compatibility and performance testing of operators. Summary of the Invention

[0004] The present invention provides a method and system for testing the compatibility and performance of multiple frameworks to improve the efficiency, accuracy, and generality of operator compatibility and performance testing.

[0005] In a first aspect, the present invention provides a method for testing the compatibility and performance of multiple frameworks, including:

[0006] Creating a virtual test environment based on virtualization technology according to the environmental information of different independent and controllable platforms; each virtual test environment simulates a combination of different environmental information;

[0007] Obtaining the operators to be tested based on the running information of different independent and controllable platforms, and generating test cases based on the characteristic information of the operators and the environmental information of different independent and controllable platforms; the test cases specify input parameters, expected output results, and test environment configuration information;

[0008] Distributing each test case to the corresponding virtual test environment for execution based on the test environment configuration information, and obtaining the test execution results and performance index data during the execution of each test case;

[0009] Determine the compatibility evaluation results and performance analysis results of the operator under different domestically controllable platforms based on the test execution results and performance metric data.

[0010] In a second aspect, the present invention further provides a test system for the compatibility and performance of multiple frameworks, which is applied to the test method for the compatibility and performance of multiple frameworks as described in the first aspect; the test system for the compatibility and performance of multiple frameworks includes:

[0011] A virtual test environment module, configured to create a virtual test environment based on virtualization technology according to the environmental information of different domestically controllable platforms; each virtual test environment simulates a combination of different environmental information.

[0012] A test case generation module, configured to obtain the operators to be tested based on the running information of different domestically controllable platforms, and generate test cases based on the characteristic information of the operators and the environmental information of different domestically controllable platforms; the input parameters, expected output results, and test environment configuration information are specified in the test cases.

[0013] A test case execution module, configured to distribute each test case to the corresponding virtual test environment for execution based on the test environment configuration information, and obtain the test execution results and performance metric data during the execution of each test case.

[0014] A test evaluation module, configured to determine the compatibility evaluation results and performance analysis results of the operator under different domestically controllable platforms based on the test execution results and performance metric data.

[0015] In a third aspect, the present invention further provides an electronic device, including: a memory, configured to store a computer software program; a processor, configured to read and execute the computer software program, thereby implementing the test method for the compatibility and performance of multiple frameworks as described in any one of the above.

[0016] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, in which a computer software program is stored, and when the computer software program is executed by a processor, the test method for the compatibility and performance of multiple frameworks as described in any one of the above is implemented.

[0017] In a fifth aspect, the present invention further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the test method for the compatibility and performance of multiple frameworks as described in any one of the above is implemented.

[0018] The test method for the compatibility and performance of multiple frameworks provided by the embodiments of the present invention distributes test cases to each virtual test environment through an automated script, avoiding the cumbersome installation, configuration, and test operations in the manual test process, improving the test efficiency. At the same time, by using virtualization technology to create multiple virtual test environments, test cases can be executed in parallel in different environments simultaneously, shortening the test time. On the other hand, by generating standardized test cases, clearly specifying the input parameters and expected output results of the operator, the influence of human factors on the test results is reduced, improving the test accuracy. At the same time, through comprehensive analysis based on the test execution results and performance index data, the compatibility of the operator can be evaluated more accurately. Further, by using virtualization technology to simulate virtual test environments with different hardware and operating systems, cross-platform compatibility evaluation of the operator is achieved, improving the universality of the test. Therefore, the embodiments of the present invention improve the efficiency, accuracy, and universality of the operator compatibility and performance tests. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic flowchart of the test method for the compatibility and performance of multiple frameworks provided by the embodiments of the present invention;

[0020] Figure 2 is a schematic structural diagram of the test system for the compatibility and performance of multiple frameworks provided by the embodiments of the present invention;

[0021] Figure 3 is an embodiment diagram of the electronic device provided by the embodiments of the present invention;

[0022] Figure 4 is an embodiment diagram of the computer-readable storage medium provided by the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0024] In the description of the present invention, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present invention, "a plurality" means two or more, unless otherwise specifically defined.

[0025] In the description of the present invention, the term "for example" is used to mean "serving as an example, illustration, or explanation". Any embodiment described as "for example" in the present invention is not necessarily construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to implement and use the present invention. In the following description, details are set forth for purposes of explanation. It should be understood that those of ordinary skill in the art can recognize that the present invention can be implemented without the use of these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0026] Optionally, refer to Figure 1 , Figure 1 which is a schematic flowchart of a method for testing the compatibility and performance of multiple frameworks provided by the present invention. In the embodiments of the present invention, the execution subject of the method for testing the compatibility and performance of multiple frameworks is a test system. Therefore, the method for testing the compatibility and performance of multiple frameworks includes:

[0027] Step 10: Create a virtual test environment based on virtualization technology according to the environmental information of different domestically controllable platforms.

[0028] Optionally, the test system collects the environmental information of different domestically controllable platforms, including hardware models (such as the CPU model used by a certain platform is XX-FT-2000 / 4, and the memory is 8GB DDR4 of a certain brand, etc.), operating system versions (such as YY-Desktop Operating System V7.0), and software framework versions (such as TensorFlow 2.3.0). Optionally, the test system creates a virtual test environment by using virtualization technology with the collected environmental information, specifically as described in Steps 101 to 104, where each virtual test environment simulates a combination of different environmental information. In one embodiment, for the domestically controllable platform A, the hardware of platform A is XX-FT-2000 / 4-CPU, 16GB memory, the operating system is YY-Desktop Operating System V7.0, and the software framework is PyTorch 1.7.1; the hardware of platform B is ZZ-3A4000-CPU, 8GB memory, the operating system is Deepin Operating System V20, and the software framework is MindSpore 1.5.0. The test system uses KVM virtualization technology to create a virtual test environment according to the environmental information of platform A, configures the corresponding hardware simulation parameters therein (such as simulating the performance of FT-2000 / 4-CPU, etc.), installs YY-Desktop Operating System V7.0, and deploys PyTorch 1.7.1. Similarly, a virtual test environment of platform A can be created.

[0029] Step 20: Obtain the operators to be tested based on the operation information of different domestically controllable platforms, and generate test cases based on the characteristic information of the operators and the environmental information of different domestically controllable platforms.

[0030] Further, the test system obtains the operators to be tested according to the operation information of different domestically controllable platforms (for example, platform A is mainly used for image recognition tasks, and platform B is mainly used for natural language processing tasks), as specifically described in steps 201 to 204.

[0031] Further, the test system generates test cases by combining the characteristic information of the operators (such as characteristics of matrix operations, convolution operations, and logical operations, etc.) and the environmental information of different domestically controllable platforms, as specifically described in steps 205 to 208. Among them, the input parameters (such as when performing matrix operations, the dimensions and element values of the input matrix, etc.), the expected output results (the matrix operation results obtained through theoretical calculations), and the test environment configuration information (that is, the configuration information of a certain virtual test environment) are clearly specified in each test case.

[0032] Continuing with the above embodiment, taking platform A (used for image recognition, using the PyTorch framework) as an example, a convolution operation operator is obtained. Since the common image size in image recognition is 224*224, the input parameters are set as an image matrix with 3 channels and a size of 224*224, and the convolution kernel size is 3*3. Through theoretical calculations, the expected output result is a feature map matrix after convolution operations. The test environment configuration information points to the virtual test environment that simulates the environment of platform A created in step 10. For platform B (used for natural language processing, using the MindSpore framework), a logical operation operator is obtained. The input parameters are set as two boolean value vectors, and the expected output result is a boolean value vector after logical operations. The test environment configuration information corresponds to the virtual test environment that simulates the environment of platform B.

[0033] Step 30: Distribute each test case to the corresponding virtual test environment for execution based on the test environment configuration information, and obtain the test execution results and performance metric data during the execution of each test case.

[0034] Optionally, the test execution result in the embodiment of the present invention represents whether the actual test output result matches the expected output result. The performance index data includes the execution time of the test case and the resource occupancy. Therefore, the test system distributes each test case to the corresponding virtual test environment for execution according to the test environment configuration information in the test case. During the execution of each test case, the test system collects the test execution result (judging whether the actual test output result matches the expected output result, passing if it matches and failing if it does not match) and the performance index data. The performance index data includes the execution time of the test case (the time interval from the start to the end of the test case execution) and the resource occupancy (such as CPU usage rate, memory occupancy, etc.).

[0035] Continuing with the convolution operation test cases previously generated for Platform A, the test system distributes them to the virtual test environment that simulates the environment of Platform A for execution. During the execution, the start time is recorded, and the end time is recorded when the convolution operation is completed. The difference between the two is the execution time. At the same time, the resource occupancy such as CPU usage rate and memory occupancy is obtained through the monitoring tool in the virtual environment. The actual output feature map matrix is compared with the expected output result to determine whether the test execution result passes. Similarly, the logical operation test cases of Platform B are distributed to the corresponding virtual test environment for execution, and the execution time, resource occupancy are recorded and the test execution result is judged.

[0036] Step 40: Determine the compatibility evaluation result and performance analysis result of the operator under different domestically controllable platforms based on the test execution result and the performance index data.

[0037] Furthermore, the test system determines the compatibility evaluation result of the operator under different domestically controllable platforms according to the test execution result. If all the test cases of a certain operator on a certain platform are executed successfully, it means that the operator is compatible with the platform; if some or all of the test cases fail, it means that there are compatibility problems. Furthermore, the test system makes a judgment on the performance analysis result according to the performance index data (execution time and resource occupancy). For example, compare the execution times of the same operator on different platforms, and the platform with a shorter execution time has better performance; compare the resource occupancy, and the platform with less resource occupancy has better performance, as specifically described in Steps 401 to 403.

[0038] In one embodiment, for the convolution operation operator of Platform A, if all test cases are executed successfully, then the convolution operation operator is compatible with Platform A. In terms of performance analysis, the average execution time of Platform A for executing this convolution operation test case is 0.5 seconds, the average CPU utilization rate is 60%, and the average memory occupancy is 500 MB. While for Platform C (another domestically controllable platform), the average execution time for executing the same convolution operation test case is 0.8 seconds, the average CPU utilization rate is 70%, and the average memory occupancy is 600 MB. Thus, in terms of this convolution operation operator, the performance of Platform A is superior to that of Platform C. For the logical operation operator of Platform B, if some test cases fail, it indicates that there are compatibility issues between the logical operation operator and Platform B. Meanwhile, based on the performance metric data and comparison with other platforms, its performance is analyzed.

[0039] In the embodiment of the present invention, test cases are distributed to each virtual test environment for execution through automated scripts, which avoids the cumbersome installation, configuration, and testing operations in the manual testing process, improves the testing efficiency. By using virtualization technology to create multiple virtual test environments, test cases can be executed in parallel in different environments simultaneously, shortening the testing time. On the other hand, by generating standardized test cases, clearly specifying the input parameters and expected output results of the operator, the influence of human factors on the test results is reduced, improving the accuracy of the test. Through comprehensive analysis based on the test execution results and performance metric data, the compatibility of the operator can be evaluated more accurately. Further, by using virtualization technology to simulate virtual test environments with different hardware and operating systems, cross-platform compatibility evaluation of the operator is achieved, improving the generality of the test.

[0040] In one embodiment, the descriptions of steps 101 to 104 are as follows:

[0041] Step 101: Based on the compatibility, performance adaptability, and function matching degree of the hardware model, operating system version, and software framework version of the domestically controllable platform in different application scenarios, potential correlation relationships between the hardware model, operating system version, and software framework version are explored, and a correlation relationship model is constructed.

[0042] Optionally, the test system collects a large amount of data on the hardware model, operating system version, and software framework version of domestically controllable platforms in different application scenarios. These data cover compatibility performance (such as whether it can run normally, whether there are error reports, etc.), performance adaptability (such as running speed, resource utilization efficiency), and function matching degree (whether the functions of the software framework can be fully implemented on the hardware and operating system).

[0043] Furthermore, through in-depth analysis of these data, the test system discovers potential correlation relationships among the hardware model, operating system version, and software framework version. For example, it is found that when a specific hardware model is paired with a specific operating system version, the operating efficiency of certain software frameworks will be significantly improved, and a correlation relationship model among them is constructed, which can describe the mutual influence and dependency relationships among the three.

[0044] In one embodiment, there are three self-controlled platforms. Platform X uses the hardware model AA-KX-U6780A-CPU, the operating system BB-18.04, and the software framework Scikit-learn 0.23.2, and is applied to the data mining scenario; Platform Y uses the hardware model MM-SW26010-CPU, the operating system NN-V10, and the software framework TensorFlow 2.4.1, and is applied to the deep learning scenario; Platform Z uses the hardware model XX-FT-1500A-CPU, the operating system Deepin 20, and the software framework PyTorch 1.7.1, and is applied to the image recognition scenario. The test system collects that on Platform X, when Scikit-learn 0.23.2 is running, the execution speed of the data mining task under BB-18.04 will be significantly improved with the full utilization of the core number of AA-KX-U6780A-CPU, reflecting the performance adaptation correlation among the hardware, operating system, and software framework, and a correlation relationship model is constructed, such as showing the influence relationship of different hardware models on the performance and functions of each software framework under different operating systems in the form of a chart.

[0045] Step 102: Plan the initial virtual environment architecture according to the correlation relationship model, combined with the platform characteristics and test requirements of the self-controlled platform, to determine the organization form and interaction method of virtual hardware, operating system, and software framework in the virtual environment.

[0046] Furthermore, the test system plans the initial virtual environment architecture based on the correlation relationship model, combined with the platform characteristics of the self-controlled platform (such as the computing power characteristics of the hardware, the stability of the operating system, etc.) and test requirements (for example, whether it is to test the functional integrity or performance of a specific software framework in different environments), and the obtained initial virtual environment architecture includes the allocation architecture of virtual hardware resources, such as determining how to allocate the virtual CPU core number, memory size, etc. corresponding to different hardware models; planning the installation and deployment architecture of the operating system, for example, which installation method can better adapt to the virtual hardware and software framework, and determining the running architecture of the software framework, such as which virtual resources the software framework depends on for running and in what way it interacts with the virtual hardware and operating system.

[0047] In one embodiment, taking the test requirements in the deep learning scenario as an example, the association relationship model shows that when the MM-SW26010-CPU is paired with the NN-V10 operating system, TensorFlow 2.4.1 can perform well. According to the platform characteristics, the MM-SW26010-CPU is good at large-scale parallel computing. Therefore, in the virtual hardware resource allocation architecture, a relatively large number of virtual cores, such as 32 virtual cores (determined according to actual test requirements and hardware simulation capabilities), are allocated to the virtual MM-SW26010-CPU, and 16GB of virtual memory is allocated. In terms of the operating system installation and deployment architecture, the NN-V10 is installed using an automated script to ensure good driver adaptation to the virtual hardware during the installation process. For the operating architecture of the software framework, when TensorFlow 2.4.1 runs, it is set to preferentially call the parallel computing resources of the virtual MM-SW26010-CPU, and efficient interaction with the virtual hardware and the operating system is achieved through specific environment variable configurations.

[0048] Step 103: Plan the virtual resource allocation information based on the adaptability requirements of virtual resources for combinations of different hardware models, operating system versions, and software framework versions.

[0049] Optionally, combinations of different hardware models, operating system versions, and software framework versions have different adaptability requirements for virtual resources. Further, the test system plans the virtual resource allocation information according to the adaptability requirements of virtual resources for combinations of different hardware models, operating system versions, and software framework versions. For example, complex deep learning software frameworks require a large amount of memory and high-performance CPU computing resources during operation, while simple data processing software frameworks may have relatively low memory requirements.

[0050] Therefore, the test system determines the specific allocation of virtual resources in each virtual test environment, such as information on the number of virtual CPU cores, memory size, disk space, etc., based on the functional complexity of the software framework, the performance characteristics of the hardware, and the resource management capabilities of the operating system.

[0051] In one embodiment, for a virtual test environment used to run the complex image processing software framework OpenCV 4.5.2, considering that it has high requirements for the CPU computing power during complex operations such as image convolution and also requires a large amount of memory to cache a large number of image data. If the virtual hardware simulates XX-FT-2000 / 4-CPU, the test system plans to allocate 8 virtual cores to it to meet the computing requirements, and at the same time allocates 8GB of virtual memory for data caching. For disk space, considering that a large amount of test image data may need to be stored, 50GB of virtual disk space is allocated. If it is running a simple text processing software framework, such as NLTK (a basic framework for natural language processing), the requirements for CPU and memory are relatively low, and 2 virtual cores and 2GB of virtual memory, and 10GB of virtual disk space may be allocated.

[0052] Step 104, create a virtual test environment based on the initial virtual environment architecture and virtual resource allocation information.

[0053] Furthermore, the test system creates a virtual test environment according to the initial virtual environment architecture and virtual resource allocation information, that is, uses virtualization technology to create multiple virtual test environment instances on a physical server according to the organizational form and interaction method of the virtual hardware, operating system, and software framework determined in the initial virtual environment architecture. During the creation process, corresponding virtual CPU cores, memory, disk space and other resources are allocated to each virtual test environment according to the virtual resource allocation information, and the installation and deployment of the operating system and the configuration of the software framework are completed to meet the previously planned architecture requirements, creating multiple virtual test environments for different application scenarios, different hardware and software combinations, as specifically described in Steps 1041 to 1044.

[0054] The embodiment of the present invention can create a highly customized virtual test environment that meets the requirements of different application scenarios and can truly simulate the environment of an independent and controllable platform. The virtual test environment not only considers the potential correlation between the hardware, operating system, and software framework, but also makes a reasonable architecture plan and resource allocation according to the platform characteristics and test requirements, so that the virtual test environment can provide a more accurate and targeted test environment for subsequent operator tests, greatly improving the accuracy and effectiveness of the test results, helping to more deeply analyze the compatibility of the operator under different independent and controllable platforms, and improving the generality of the test.

[0055] In one embodiment, the descriptions of Steps 1041 to 1044 are as follows:

[0056] Step 1041, based on the virtual resource allocation information, the correlation relationship model, and the storage structure characteristics of the independent and controllable platform, plan the optimal deployment path of the operating system and software framework in the initial virtual environment architecture.

[0057] Optionally, the test system deeply analyzes the virtual resource allocation information to clarify the details of virtual resources corresponding to various hardware models, such as the number of virtual CPU cores and memory capacity. At the same time, referring to the association relationship model, it understands the adaptation status among different hardware, operating systems, and software frameworks. And combining with the storage structure characteristics of the self-controlled platform, for example, some platforms use solid-state drives (SSDs) with fast read and write speeds, while some use hard disk drives (HDDs) with relatively slower read and write speeds. Considering these factors comprehensively, the test system plans the optimal deployment path of the operating system and software framework in the initial virtual environment architecture. For example, for a software framework that frequently reads and writes data, if the platform uses an SSD storage structure, it can plan to set its data storage directory near the high-speed read and write area of the SSD to improve data access efficiency; if the operating system has special requirements for memory management, according to the virtual memory allocation situation, it plans the loading order and memory resident location of its kernel modules.

[0058] In an embodiment, there is a self-controlled platform with a hardware model of XX-FT-2000 / 4, virtual resource allocation of 8 virtual CPU cores and 16GB of virtual memory, and a storage structure of SSD. The software framework is Spark for big data processing, which needs to frequently read and write a large amount of data. The association relationship model shows that Spark can be well adapted when running YY-operating system V7.0 on this hardware. The test system plans to store the core files of YY-operating system V7.0 in the virtual storage location corresponding to the cache area of the SSD to speed up the system startup speed. For the Spark software framework, set its data storage directory in the virtual location corresponding to the continuous storage space of the SSD to reduce fragmented data read and write. At the same time, according to its multi-threaded computing characteristics, plan to let Spark preferentially use the virtual memory area close to the physical memory controller.

[0059] Step 1042: Update the initial virtual environment architecture based on the optimal deployment path of the operating system and software framework in the initial virtual environment architecture to obtain the target virtual environment architecture.

[0060] Furthermore, the test system updates the initial virtual environment architecture according to the optimal deployment path of the operating system and software framework planned in step 1041. This includes adjusting the interaction mode between virtual hardware resources and the operating system and software framework, re-planning the data storage and access paths of the virtual storage structure and the operating system and software framework, and updating the running configuration parameters of the software framework to meet the requirements of the optimal deployment path. Through these updates, the target virtual environment architecture is obtained, which can make more full use of virtual resources and improve the running efficiency of the operating system and software framework in the virtual environment.

[0061] Continuing with the above embodiment, taking the example of the XX-FT-2000 / 4 platform, in the initial virtual environment architecture, the deployment of the operating system and software framework is relatively conventional. According to the optimal deployment path planned in step 1041, the test system updates the virtual storage architecture. In the virtual hard disk partition, a dedicated high-speed virtual disk partition is allocated for the YY-operating system V7.0 to store system core files and frequently accessed cache data. For the Spark software framework, its configuration file is modified to specify the new data storage path to the previously planned SSD high-speed continuous virtual storage area. At the same time, the scheduling policy of the virtual CPU cores with the operating system and software framework is adjusted, so that Spark preferentially uses the virtual CPU cores with better performance when performing big data processing tasks, thereby completing the update of the initial virtual environment architecture and obtaining the target virtual environment architecture.

[0062] Step 1043: Based on the requirements of network communication for combinations of different hardware models, operating system versions, and software framework versions, construct a virtual environment network communication model.

[0063] Furthermore, different combinations of hardware models, operating system versions, and software framework versions have different requirements for network communication. The test system analyzes these requirements. For example, some software frameworks for distributed computing have high requirements for network bandwidth and latency; while some simple local data processing software frameworks have low requirements for network communication. The test system constructs a virtual environment network communication model according to these different requirements. This virtual environment network communication model defines the configuration of network devices in the virtual environment, such as the number of virtual network cards, bandwidth limits, and network topology structures. For example, whether a star topology or a bus topology is more suitable for a specific software framework and hardware combination, and at the same time determines the selection of network communication protocols, such as parameter optimization of the TCP / IP protocol in different scenarios.

[0064] In one embodiment, based on the autonomous and controllable platform of ZZ-3A5000-CPU, running Deepin operating system V20, the software framework is Horovod for distributed deep learning training. Horovod has high requirements for network communication, requiring low latency and high bandwidth. The test system constructs a virtual environment network communication model, configures multiple high-performance virtual network cards for this virtual environment, and sets a relatively high bandwidth limit for each virtual network card, such as 10 Gbps. The star network topology structure is adopted to reduce network transmission latency, and the central node connects each computing node (virtual server). In terms of network communication protocol, parameters such as the buffer size and timeout retransmission of the TCP / IP protocol are optimized to adapt to the large amount of data transmission requirements of Horovod during the distributed training process.

[0065] Step 1044: Based on the target virtual environment architecture and the virtual environment network communication model, perform fusion packaging to create a virtual test environment.

[0066] Furthermore, the test system integrates and packages the target virtual environment architecture and the virtual environment network communication model. During the integration process, it ensures that the hardware, operating system, software framework, and network communication in the virtual environment can work together. For example, seamless integration of virtual network devices into the target virtual environment architecture enables the operating system to correctly identify and manage virtual network cards, and the software framework can perform efficient data transmission through the optimized network communication model. Through this integration and packaging, a complete virtual test environment is created, which not only has reasonable resource allocation and architecture deployment but also optimized network communication capabilities, and can meet the diverse needs of operator testing under different domestically controllable platforms.

[0067] Continuing with the example of the ZZ-3A5000 platform paired with the Deepin operating system V20 and the Horovod software framework, the test system integrates the configured optimized hardware, operating system, and software framework in the target virtual environment architecture with the constructed virtual environment network communication model. During the creation of the virtual environment, multiple configured high-performance virtual network cards are added to the virtual environment, and the Deepin operating system V20 automatically identifies these virtual network cards and performs initialization configuration according to the settings of the network communication model. When the Horovod software framework starts, it can use the high-bandwidth virtual network cards to quickly transmit distributed training data through the optimized network communication model, completing the creation of the virtual test environment.

[0068] The virtual test environment finally created in the embodiment of the present invention provides an almost real and optimized operating environment for operator testing, can more accurately reflect the actual operating conditions of operators under different domestically controllable platforms, improve the reliability and effectiveness of test results, and provide more powerful support for subsequent compatibility evaluation and performance analysis.

[0069] In one embodiment, the descriptions of steps 201 to 204 are as follows:

[0070] Step 201: Identify and analyze the operation information of the domestically controllable platform to obtain various platform operation themes of the domestically controllable platform, and classify the operators in the operator library according to the theme classification system to obtain operator categories.

[0071] Optionally, the test system collects the operation information of the domestically controllable platform, which may cover various aspects such as the types of services carried by the platform, the characteristics of the processed data, and the running task processes. Through the identification and analysis of this operation information, various platform operation themes are summarized, such as "image recognition and processing theme", "data mining and analysis theme", "natural language processing theme", etc. Further, the test system classifies the operators in the operator library according to a pre-established theme classification system, which may be constructed based on factors such as the functional characteristics and application fields of the operators, and obtains different operator categories, such as "image processing operator category", "data calculation operator category", "language logic operator category", etc.

[0072] In one embodiment, the operation information of the domestically controllable platform shows that the platform is mainly used for image recognition tasks in the security monitoring field, including the recognition of people and vehicles in the video stream, etc. The test system determines a main operation theme of the platform as the "security image recognition theme" by analyzing this operation information. In the operator library, operators such as edge detection operators and target box positioning operators are classified into the "image processing operator category" according to the theme classification system. Another example is that if the platform operation information indicates that it often performs statistical analysis tasks on financial data, the test system determines its operation theme as the "financial data statistical analysis theme", and operators such as mean calculation operators and variance calculation operators are classified into the "data calculation operator category".

[0073] Step 202: Based on the shared features and dependency metrics between the operation theme and the operator category, establish an association mapping relationship between each type of platform operation theme and its corresponding operator category.

[0074] Further, the test system deeply studies the shared features and dependency metrics between the operation theme and the operator category. The shared features may include similarities in data types, processing flows, etc., and the dependency metrics are reflected in aspects such as the frequency and importance of a certain operation theme's use of a specific operator category.

[0075] Further, the test system establishes an association mapping relationship between each type of platform operation theme and its corresponding operator category according to the shared features and dependency metrics between the operation theme and the operator category. For example, in the "image recognition and processing theme", links such as image preprocessing and feature extraction highly rely on image processing operators, so a close association mapping relationship is established between the "image processing operator category" and the "image recognition and processing theme".

[0076] In one embodiment, for the "security image recognition theme", its data is mainly image data, and the processing process includes links such as image acquisition, preprocessing, feature extraction, and target recognition. And for the operators in the "image processing operator category", such as the grayscale operator for image preprocessing and the SIFT feature extraction operator for feature extraction, they have obvious shared features and a high degree of dependence on this running theme. By sorting out these relationships, the test system establishes an associated mapping between the "security image recognition theme" and the "image processing operator category", which is recorded in tabular form as: "security image recognition theme" - "image processing operator category". Similarly, for the "financial data statistical analysis theme", this theme mainly processes financial numerical data and performs statistical calculations on the data, which is closely related to various statistical calculation operators in the "data calculation operator category", thus establishing a corresponding relationship.

[0077] Step 203, according to the theme popularity of each type of platform running theme, combined with the calculation complexity and application frequency of each operator in the operator category corresponding to each type of platform running theme, determine the priority score of each operator in the operator category under each type of platform running theme.

[0078] Furthermore, the test system determines the theme popularity of each type of platform running theme, and the theme popularity can be measured by factors such as the task execution frequency and resource investment of the platform under this theme. At the same time, analyze the calculation complexity of each operator in the operator category corresponding to each type of platform running theme, and the calculation complexity can be judged based on the time complexity and space complexity of the algorithm. In addition, consider the application frequency of each operator, that is, the number of times this operator is used during the actual operation of the platform. Combining the theme popularity, operator calculation complexity, and application frequency, determine the priority score of each operator in the operator category under each type of platform running theme. Generally speaking, an operator with a high theme popularity, high calculation complexity, and high application frequency has a relatively high priority score.

[0079] In one embodiment, under the "security image recognition theme", there are an edge detection operator and a target box localization operator in the "image processing operator category". The theme popularity of this theme is relatively high because the security monitoring task is ongoing continuously. The edge detection operator has a medium calculation complexity, but a very high application frequency, and is almost used in the processing of each frame of image; the target box localization operator has a relatively high calculation complexity, but a relatively low application frequency. According to the preset rules, such as the theme popularity accounting for a certain proportion, the calculation complexity and application frequency each accounting for a certain proportion, the test system determines that the priority score of the edge detection operator is relatively high, and the priority score of the target box localization operator is relatively low. For example, it is set that a high theme popularity corresponds to a higher score, a medium calculation complexity corresponds to a certain score, and a very high application frequency corresponds to a high score. The comprehensive priority score of the edge detection operator is 80 points (out of 100); the priority score of the target box localization operator is 60 points because of its high calculation complexity but low application frequency.

[0080] Step 204: Traverse the priority scores of each operator, and determine the operators with priority scores greater than or equal to the preset score as the operators to be tested.

[0081] Further, the test system traverses the priority scores of each operator, and determines the operators with priority scores greater than or equal to the preset score as the operators to be tested. The preset score can be set in advance according to factors such as test resources and test objectives. Through this screening process, key operators that are more important to the platform operation theme and may affect the platform performance or functions can be selected from numerous operators for focused testing.

[0082] In one embodiment, the preset score is 70 points. In the "image processing operator category" under the "security image recognition theme", the priority score of the edge detection operator is 80 points, which is greater than the preset score of 70 points, and it is determined as the operator to be tested; while the priority score of the target box positioning operator is 60 points, which is less than the preset score, and it is not listed as the object of this test for the time being. In other operation themes and corresponding operator categories, screening is also carried out in the same way. For example, in the "data calculation operator category" under the "financial data statistical analysis theme", if the priority score of the mean calculation operator is 75 points, which is greater than the preset score, it becomes the operator to be tested, while the priority score of a certain complex risk assessment calculation operator is 65 points, which is less than the preset score, and it is not selected.

[0083] The operators finally screened out in the embodiments of the present invention for testing are all those with relatively high priorities under their respective platform operation themes, ensuring that subsequent tests can focus on key operators, improving test efficiency and accuracy, and more effectively evaluating the performance and compatibility of operators under different domestically controllable platforms.

[0084] In one embodiment, the descriptions of steps 205 to 208 are as follows:

[0085] Step 205: Construct an operator characteristic topology structure based on the characteristic information of each type of operator.

[0086] Optionally, for the characteristic information of each type of operator, such as matrix operations, convolution operations, and logical operations, etc., the operand elements in these characteristic information are used as nodes to construct an operator characteristic topology structure. The operand elements can be more refined operations. For example, in matrix operations, matrix multiplication, matrix addition, etc. can be used as operand element nodes. The edges between nodes are determined according to the execution order or dependency relationship between the operand elements. For example, if matrix addition may be required after matrix multiplication, then there will be an edge from the matrix multiplication node to the matrix addition node indicating the execution order; if a certain logical operation depends on the result of a convolution operation, then there will be an edge from the convolution operation node to this logical operation node indicating the dependency relationship.

[0087] In one embodiment, an operator is used for image processing, including a convolution operation and subsequent logical judgment (judging whether the image contains specific features). In the convolution operation, it can be further divided into two operator sub-elements: convolution kernel generation and convolution calculation. In the operator characteristic topology structure constructed by the test system, there are three nodes: convolution kernel generation, convolution calculation, and logical judgment. There is an edge connecting the convolution kernel generation node to the convolution calculation node because the convolution calculation depends on the generation of the convolution kernel, which reflects the execution order and dependency relationship; there is an edge connecting the convolution calculation node to the logical judgment node because the logical judgment needs to be based on the result of the convolution calculation.

[0088] Step 206: Based on the dependency relationship and abstraction level among the hardware model, operating system version, and software framework version of the self-controlled and controllable platform, divide the hardware model, operating system version, and software framework version of the self-controlled and controllable platform into different platform environment levels.

[0089] Furthermore, the test system analyzes the dependency relationship and abstraction level among the hardware model, operating system version, and software framework version of the self-controlled and controllable platform. Hardware is the foundation for the operation of the entire platform, and all operating systems and software frameworks rely on the support of the hardware, so the hardware is at the bottom layer. The operating system runs based on the hardware, provides a running environment and resource management services for the software framework, and is at the middle layer. The software framework depends on the operating system to call the hardware resources and execute its functions, and is at the top layer. Based on this, the hardware model, operating system version, and software framework version of the self-controlled and controllable platform are divided into different platform environment levels.

[0090] In one embodiment, taking a certain self-controlled and controllable platform as an example, the hardware model is XX-FT-2000 / 4, the operating system is YY-Desktop Operating System V7.0, and the software framework is TensorFlow2.3.0. The XX-FT-2000 / 4 hardware provides basic resources such as computing and storage, and is at the bottom layer. The YY-Desktop Operating System V7.0 is installed on the XX-FT-2000 / 4 hardware, manages the hardware resources, and provides a running environment for TensorFlow2.3.0, and is at the middle layer. TensorFlow2.3.0 depends on the YY-Desktop Operating System V7.0 to call the computing resources of the XX-FT-2000 / 4 hardware to perform deep learning tasks, and is at the top layer.

[0091] Step 207: Based on the compatibility between the operator sub-elements in the operator characteristic topology structure and the platform environment levels, determine the adaptability of the operator in different self-controlled and controllable platform environments.

[0092] Furthermore, the test system compares the compatibility between the operands in the operator feature topology structure and the platform environment levels. For each operand, check whether it can be executed normally under different platform environment levels. For example, some complex matrix operation operands may require the support of specific hardware instruction sets. If the underlying hardware does not have this instruction set, the compatibility is poor; or the convolution operation operand in a certain software framework has special requirements for the memory management mechanism of the operating system. If the intermediate operating system cannot meet these requirements, it will also cause compatibility problems. By comprehensively considering these factors, determine the adaptability of the operator in the environments of different domestically controllable platforms. The adaptability can be simply divided into three levels: high, medium, and low.

[0093] Continuing with the image processing operator as an example, one of the operands is CUDA-accelerated convolution calculation (requiring specific GPU hardware support). On a domestically controllable platform with XX-FT-2000 / 4-CPU (without GPU hardware), YY-desktop operating system V7.0, and TensorFlow 2.3.0, since the underlying hardware does not support GPU, this CUDA-accelerated convolution calculation operand cannot be executed normally. Therefore, the adaptability of this operator in this platform environment is low. While on another platform equipped with YWD-GPU, Ubuntu operating system, and TensorFlow 2.3.0, this operand can run well, and the adaptability is high.

[0094] Step 208: Determine the input parameters, expected output results, and test environment configuration information based on the adaptability of the operator in the environments of different domestically controllable platforms, and generate test cases.

[0095] Furthermore, the test system determines the input parameters, expected output results, and test environment configuration information according to the adaptability of the operator in the environments of different domestically controllable platforms, and generates test cases. For the platform environment with high adaptability, set relatively complex and comprehensive input parameters to fully test the performance of the operator under good adaptation; the expected output results are determined according to the theoretical function and normal execution of the operator. The test environment configuration information specifies the detailed information such as the hardware model, operating system version, and software framework version of the platform environment corresponding to high adaptability. For the platform environment with low adaptability, set some simple input parameters, mainly used to verify whether the operator can barely run in this environment. The expected output results may be error messages or partially correct results, and the test environment configuration information corresponds to the platform environment with low adaptability.

[0096] In one embodiment, for a platform with high adaptability (such as equipped with YWD-GPU, Ubuntu operating system, TensorFlow 2.3.0, and high adaptability of image processing operators), the input parameters are set to a complex image data with high resolution and multiple channels, and the expected output result is clear image data that has been correctly processed by the image processing operator. The test environment configuration information is detailedly recorded as the YWD-specific model GPU, the specific version of the Ubuntu operating system, and the TensorFlow 2.3.0 software framework. For a platform with low adaptability (such as XX-FT-2000 / 4-CPU, YY-desktop operating system V7.0, TensorFlow 2.3.0, and low adaptability of image processing operators), the input parameters are set to a simple low-resolution single-channel image, and the expected output result may be an error message indicating that CUDA-accelerated convolution calculation cannot be performed. The test environment configuration information is recorded as the XX-FT-2000 / 4 hardware model, the YY-desktop operating system V7.0 version, and the TensorFlow 2.3.0 software framework, generating test cases under different adaptabilities.

[0097] The embodiments of the present invention can generate highly targeted and comprehensive test cases, enabling the finally generated test cases to set reasonable input parameters, expected output results, and test environment configuration information according to different adaptabilities, reducing the influence of human factors on test results, improving the accuracy of testing. At the same time, comprehensively testing the operation of the operator in different independently controllable platform environments helps to accurately evaluate the compatibility between the operator and the platform environment.

[0098] In one embodiment, the descriptions of steps 2081 to 2084 are as follows:

[0099] Step 2081, determine the parameter boundary based on the adaptability of the operator in different independently controllable platform environments, and perform parameter acquisition within the parameter boundary based on the adaptability of the operator in different independently controllable platform environments and the complexity of the characteristic information of the operator to obtain the input parameters.

[0100] Optionally, the test system determines the parameter boundary according to the adaptability of the operator in different independently controllable platform environments. For a platform with high adaptability, since the operator can fully exert its performance, the parameter boundary can be set wider to comprehensively test the operator's performance; for a platform with low adaptability, the parameter boundary is set relatively narrow to mainly verify whether the operator can basically operate. At the same time, considering the complexity of the operator's characteristic information, for an operator with high complexity, parameter acquisition needs to be more detailed to cover more possible situations. Within the determined parameter boundary, the test system performs parameter acquisition to obtain the input parameters. For example, for a matrix operation operator, the parameter boundary may involve the dimension range of the matrix, the range of element data types, etc.

[0101] In one embodiment, there is a matrix operation operator for data analysis. On a platform with high adaptability, such as a platform equipped with a high-performance CPU and a large amount of memory, the test system determines that the matrix dimension parameter boundary is [2, 1000] (indicating that the matrix can be 2-dimensional to 1000-dimensional), and the element data type can be various common types such as integers and floating-point numbers. Due to the high complexity of the operator characteristic information, the test system collects various types of input parameters within the parameter boundary. For example, it generates a 100-dimensional integer matrix with element values between 0 and 100; and then generates a 500-dimensional floating-point matrix with element values being random decimals between 0 and 1. While on a platform with low adaptability, such as an embedded platform with less memory, the test system sets the matrix dimension parameter boundary to [2, 10], and the element data type is only integers. Input parameters are collected within this boundary, such as generating a 3-dimensional integer matrix with element values being integers between 1 and 5.

[0102] Step 2082, for each node corresponding to an operation sub-element in the operator characteristic topology structure, traverse each node. Taking each node as the starting node, perform simulation operations according to its execution sequence relationship and dependency relationship with the input parameters and the platform environment hierarchy, and obtain the expected output result corresponding to the input parameters.

[0103] Furthermore, the test system traverses each node corresponding to an operation sub-element in the operator characteristic topology structure one by one. Taking each node as the starting node, perform simulation operations based on its execution sequence relationship and dependency relationship with the input parameters and the platform environment hierarchy. During the simulation process, combine the characteristics of the platform environment hierarchy, such as hardware computing power and operating system resource management methods, to simulate the operation process of the operator during actual operation, so as to obtain the expected output result corresponding to the input parameters. For example, for an operator characteristic topology structure that includes convolution operations and logical judgments, start from the convolution operation node, according to the input image data, combine the hardware acceleration ability of the platform, simulate the convolution operation process, and obtain the convolution result; then based on this result, start from the logical judgment node, simulate the logical judgment operation, and obtain the final expected output result.

[0104] Continuing with the example of an image processing operator, its operator characteristic topology includes a convolution kernel generation, convolution calculation, and logical judgment nodes. For a platform with high adaptability (such as a platform equipped with YWD-GPU), the input parameter is a high-resolution color image. The test system starts simulating operations from the convolution kernel generation node and generates a suitable convolution kernel according to the GPU acceleration ability of the platform and the image processing algorithm. Then, at the convolution calculation node, using the generated convolution kernel and the input image, it simulates the convolution calculation process under GPU acceleration to obtain the convolved feature map. Finally, starting from the logical judgment node, according to the feature map and the pre-set image feature judgment rules, it simulates the logical judgment operation to obtain the expected output result, that is, the conclusion of whether the image contains specific features. On a platform with low adaptability (such as an ordinary CPU platform without GPU), the input parameter is a low-resolution grayscale image. Similarly starting from the convolution kernel generation node, due to limited hardware computing power, it simulates the generation of a relatively simple convolution kernel. At the convolution calculation node, it simulates the convolution calculation process on an ordinary CPU to obtain the convolution result. Finally, at the logical judgment node, based on the convolution result, it conducts a logical judgment to obtain the expected output result, such as a simple conclusion of whether the image contains specific features.

[0105] Step 2083, convert the hardware model, operating system version, and software framework version into corresponding environment identification strings to obtain the test environment configuration information.

[0106] Furthermore, the test system converts the hardware model, operating system version, and software framework version of the autonomous and controllable platform into corresponding environment identification strings. The hardware model can directly use the model name defined by the manufacturer as the identifier; the operating system version and software framework version also use information such as their official release version numbers as identifiers respectively. These identifiers are combined in a specific format to form the test environment configuration information. For example, if the hardware model is XX-FT-2000 / 4, the operating system version is YY-Desktop Operating System V7.0, and the software framework version is TensorFlow 2.3.0, the combined environment identification string can be "FT-2000 / 4_V7.0_TensorFlow2.3.0".

[0107] In one embodiment, for the autonomous and controllable platform, the hardware model is ZZ-3A5000, the operating system version is Deepin V20, and the software framework version is PyTorch1.8.1. The test system converts it into the environment identification string "3A5000_V20_PyTorch1.8.1". For another platform, the hardware model is MM-SW26010, the operating system version is NN-V10, and the software framework version is MindSpore1.6.0. The generated environment identification string is "SW26010_V10_MindSpore1.6.0". These environment identification strings are used as test environment configuration information to clarify the specific platform environment corresponding to the test cases.

[0108] Step 2084, integrate the input parameters, expected output results, and test environment configuration information to generate test cases.

[0109] Further, the test system integrates the input parameters, expected output results, and test environment configuration information. According to a certain format, these three parts of information are combined together to form a complete test case. For example, presented in tabular form, the first column is the input parameter, the second column is the expected output result, and the third column is the test environment configuration information. In one embodiment, on a platform with high adaptability, the input parameter is a 100*100 floating-point matrix, the expected output result is another 100*100 floating-point matrix after matrix operations, and the test environment configuration information is "FT-2000 / 4_V7.0_TensorFlow2.3.0". The test system integrates this information into a test case as shown in the following table:

[0110]

[0111] Similarly, for a platform with low adaptability, if the input parameter is a 3*3 integer matrix, the expected output result is an error prompt (because the platform may not fully support the matrix operation), and the test environment configuration information is "Embedded Platform A_V5.0_SimpleMatrixLib1.0". The integrated test case is:

[0112] Input parameter Expected output result Test environment configuration information 3*3 integer matrix Error prompt Embedded platform A_V5.0_SimpleMatrixLib1.0

[0113] The embodiments of the present invention can generate more detailed, comprehensive, and targeted test cases, clearly specify the input parameters and expected output results of the operator, reduce the influence of human factors on the test results, improve the accuracy of the test. At the same time, through comprehensive analysis based on the test execution results and performance index data, the compatibility of the operator can be evaluated more accurately.

[0114] In one embodiment, the descriptions of steps 401 to 403 are as follows:

[0115] Step 401: For each test case, determine the compatibility evaluation result of the operator under different domestically controllable platforms according to the degree of difference between the actual test output result and the expected output result.

[0116] Optionally, for each test case, the test system compares the actual test output result obtained by executing each test case with the expected output result, and determines the compatibility evaluation result of the operator under different domestically controllable platforms based on the degree of difference between the two. If the actual test output result is exactly the same as the expected output result, or the difference is within an acceptable minimum range, then it can be considered that the operator has good compatibility under this platform; if there are obvious deviations between the actual test output result and the expected output result, such as incorrect output data format, missing key data, etc., it indicates that there are problems with the compatibility of the operator under this platform.

[0117] In one embodiment, the test system tests an operator for temperature data processing on a certain domestically controllable platform. The expected output result is that after being processed by this operator, the cooling amplitude value corresponding to the current ambient temperature can be accurately output, in the format of a floating-point number, and the precision is reserved to one decimal place. In the actual test output result, there is a large deviation in the cooling amplitude value. For example, the expected cooling amplitude is 5.5 °C, but the actual output is 2.0 °C, and the data format is not the specified floating-point number format, but an integer format. Based on this obvious difference, the test system determines that the compatibility of this operator under this domestically controllable platform is poor. On the contrary, if the actual test output result is 5.4 °C, with a minimal difference from the expected 5.5 °C and the data format is correct, then the system will determine that the operator has good compatibility under this platform.

[0118] Step 402: For each test case, determine the comprehensive performance score of each test case according to the execution time and resource occupancy of the test case.

[0119] Optionally, the resource occupancy includes CPU usage rate, memory occupancy, etc. Therefore, the test system determines the comprehensive performance score of each test case according to the execution time and resource occupancy of the test case. Generally speaking, the shorter the execution time of the test case and the less the resource occupancy, the higher the comprehensive performance score. For example, a short execution time of the test case can indicate that the operator has a fast processing speed, and less resource occupancy means efficient utilization of system resources, both of which reflect good performance.

[0120] In one embodiment, on a domestically controllable platform, an operator for regulating the fan speed is tested. During the test, the execution time of this test case is 2 seconds, the CPU usage rate is stable at 30%, and the memory occupancy is 50MB. Compared with other similar test cases, if the execution time is generally around 5 seconds, the CPU usage rate is around 50%, and the memory occupancy is around 80MB, then this test case is given a relatively high comprehensive performance score by the test system. With a full score of 10 points, it can be given 8 points. On the contrary, if a test case has an execution time as long as 10 seconds, the CPU usage rate reaches 70%, and the memory occupancy is 100MB, then its comprehensive performance score will be relatively low, perhaps only 3 points.

[0121] Step 403: Determine the performance analysis results of the operator on different domestically controllable platforms according to the comprehensive performance scores of each test case.

[0122] Furthermore, the test system determines the performance analysis results of the operator on different domestically controllable platforms by analyzing these scores. If, on a certain platform, the comprehensive performance scores of multiple test cases for this operator are generally high, it indicates that the operator performs well on this platform; if the comprehensive performance scores are generally low, it means that the performance of the operator on this platform needs to be improved. At the same time, the system can also compare the comprehensive performance scores of the same operator on different platforms to determine on which platform the operator can better play its advantages.

[0123] In one embodiment, for an operator for refrigeration mode switching, multiple test cases are respectively tested on three different domestically controllable platforms A, B, and C. On platform A, the average comprehensive performance score of these test cases is 7 points; on platform B, the average score is 5 points; on platform C, the average score is 3 points. Thus, the test system obtains the performance analysis results that the operator performs better on platform A and worse on platform C. This means that in practical applications, if more attention is paid to the performance of the refrigeration mode switching operator, it would be a better option to give priority to platform A.

[0124] The embodiment of the present invention can comprehensively and accurately evaluate the compatibility and performance of the operator on different domestically controllable platforms according to the matching result between the actual output result and the expected output result of the test, as well as the execution time and resource occupancy of the test case.

[0125] Furthermore, the test system for the compatibility and performance of multiple frameworks provided by the present invention will be described below. The test system for the compatibility and performance of multiple frameworks described below can be mutually corresponding and referable to the test method for the compatibility and performance of multiple frameworks described above.

[0126] Optionally, refer to Figure 2 ,Figure 2 It is a schematic structural diagram of a test system for the compatibility and performance of multiple frameworks provided by the present invention. The test system for the compatibility and performance of multiple frameworks includes

[0127] A virtual test environment module 210, configured to create a virtual test environment based on virtualization technology according to the environment information of different domestically controllable platforms; each virtual test environment simulates a combination of different environment information;

[0128] A test case generation module 220, configured to obtain operators to be tested based on the operation information of different domestically controllable platforms, and generate test cases based on the characteristic information of the operators and the environment information of different domestically controllable platforms; the test cases specify input parameters, expected output results, and test environment configuration information;

[0129] A test case execution module 230, configured to distribute each test case to the corresponding virtual test environment for execution based on the test environment configuration information, and obtain test execution results and performance metric data during the execution of each test case;

[0130] A test evaluation module 240, configured to determine the compatibility evaluation results and performance analysis results of the operators under different domestically controllable platforms based on the test execution results and performance metric data.

[0131] In the embodiments of the present invention, test cases are distributed to each virtual test environment for execution through automated scripts, avoiding the cumbersome installation, configuration, and test operations in the manual test process, improving the test efficiency. Multiple virtual test environments are created using virtualization technology, enabling test cases to be executed in parallel in different environments simultaneously, shortening the test time. On the other hand, by generating standardized test cases and clearly specifying the input parameters and expected output results of the operators, the influence of human factors on the test results is reduced, improving the test accuracy. Through comprehensive analysis based on the test execution results and performance metric data, the compatibility of the operators can be evaluated more accurately. Further, virtual test environments simulating different hardware and operating systems are created using virtualization technology, achieving cross-platform compatibility evaluation of the operators and improving the generality of the test.

[0132] Please refer to Figure 3 , Figure 3 which is an embodiment diagram of an electronic device provided by the embodiments of the present invention. As Figure 3 shown, the embodiments of the present invention provide an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored on the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, the following steps are implemented:

[0133] Create a virtual test environment based on virtualization technology according to the environmental information of different domestically controllable platforms; each virtual test environment simulates a combination of different environmental information.

[0134] Obtain the operators to be tested based on the operation information of different domestically controllable platforms, and generate test cases based on the characteristic information of the operators and the environmental information of different domestically controllable platforms; the test cases specify input parameters, expected output results, and test environment configuration information.

[0135] Distribute each test case to the corresponding virtual test environment for execution based on the test environment configuration information, and obtain the test execution results and performance metric data during the execution of each test case.

[0136] Determine the compatibility evaluation results and performance analysis results of the operators under different domestically controllable platforms based on the test execution results and performance metric data.

[0137] Please refer to Figure 4 , Figure 4 which is the embodiment diagram of the computer-readable storage medium provided by the embodiment of the present invention. As Figure 4 shown, this embodiment provides a computer-readable storage medium 400, on which a computer program 311 is stored. When the computer program 311 is executed by a processor, the following steps are implemented:

[0138] Create a virtual test environment based on virtualization technology according to the environmental information of different domestically controllable platforms; each virtual test environment simulates a combination of different environmental information.

[0139] Obtain the operators to be tested based on the operation information of different domestically controllable platforms, and generate test cases based on the characteristic information of the operators and the environmental information of different domestically controllable platforms; the test cases specify input parameters, expected output results, and test environment configuration information.

[0140] Distribute each test case to the corresponding virtual test environment for execution based on the test environment configuration information, and obtain the test execution results and performance metric data during the execution of each test case.

[0141] Determine the compatibility evaluation results and performance analysis results of the operators under different domestically controllable platforms based on the test execution results and performance metric data.

[0142] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the compatibility and performance test methods of multiple frameworks provided by the above various methods. The method includes:

[0143] Create a virtual test environment based on virtualization technology according to the environmental information of different domestically controllable platforms; each virtual test environment simulates a combination of different environmental information.

[0144] Obtain the operators to be tested based on the operation information of different domestically controllable platforms, and generate test cases based on the characteristic information of the operators and the environmental information of different domestically controllable platforms; the input parameters, expected output results, and test environment configuration information are specified in the test cases.

[0145] Distribute each test case to the corresponding virtual test environment for execution based on the test environment configuration information, and obtain the test execution results and performance metric data during the execution of each test case.

[0146] Determine the compatibility evaluation results and performance analysis results of the operators under different domestically controllable platforms based on the test execution results and performance metric data.

[0147] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solutions or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for testing the compatibility and performance of multiple frameworks, characterized in that: Including: Creating a virtual test environment based on virtualization technology according to the environment information of different domestically controllable platforms; Each virtual test environment simulates a combination of different environment information; Obtaining the operators to be tested based on the running information of different domestically controllable platforms, and generating test cases based on the characteristic information of the operators and the environment information of different domestically controllable platforms; The test cases are specified with input parameters, expected output results, and test environment configuration information; Distributing each test case to the corresponding virtual test environment for execution based on the test environment configuration information, and obtaining the test execution results and performance metric data during the execution of each test case; Determining the compatibility evaluation results and performance analysis results of the operator under different domestically controllable platforms based on the test execution results and performance metric data.

2. The test method for the compatibility and performance of multiple frameworks according to claim 1, characterized in that, The test execution results represent whether the actual test output results match the expected output results, and the performance metric data includes the execution time of the test case and the resource occupancy situation; The determining the compatibility evaluation results and performance analysis results of the operator under different domestically controllable platforms based on the test execution results and performance metric data includes: For each test case, determining the compatibility evaluation results of the operator under different domestically controllable platforms according to the degree of difference between the actual test output results and the expected output results; For each test case, determining the comprehensive performance score of each test case according to the execution time of the test case and the resource occupancy situation; Determining the performance analysis results of the operator under different domestically controllable platforms according to the comprehensive performance scores of each test case.

3. The test method for the compatibility and performance of multiple frameworks according to claim 1, characterized in that, The environment information includes the hardware model, operating system version, and software framework version; The creating a virtual test environment based on virtualization technology according to the environment information of different domestically controllable platforms includes: Mining the potential correlation relationships between the hardware model, operating system version, and software framework version based on the compatibility, performance adaptability, and function matching degree of the hardware model, operating system version, and software framework version of the domestically controllable platform in different application scenarios, and constructing a correlation relationship model; Planning an initial virtual environment architecture according to the correlation relationship model in combination with the platform characteristics and test requirements of the domestically controllable platform to determine the organization form and interaction method of the virtual hardware, operating system, and software framework in the virtual environment; the initial virtual environment architecture includes the allocation architecture of virtual hardware resources, the installation and deployment architecture of the operating system, and the running architecture of the software framework; Planning the virtual resource allocation information based on the adaptability requirements of the virtual resources for the combinations between different hardware models, operating system versions, and software framework versions; Creating a virtual test environment based on the initial virtual environment architecture and the virtual resource allocation information.

4. The test method for the compatibility and performance of multiple frameworks according to claim 3, characterized in that, The creating a virtual test environment based on the initial virtual environment architecture and the virtual resource allocation information includes: Planning the optimal deployment path of the operating system and software framework in the initial virtual environment architecture based on the virtual resource allocation information, the correlation relationship model, and the storage structure characteristics of the domestically controllable platform; Updating the initial virtual environment architecture based on the optimal deployment path of the operating system and software framework in the initial virtual environment architecture to obtain the target virtual environment architecture; Based on the requirements of network communication for combinations among different hardware models, operating system versions, and software framework versions, construct a virtual environment network communication model; Based on the target virtual environment architecture and the virtual environment network communication model, perform fusion encapsulation to create a virtual test environment.

5. The testing method for the compatibility and performance of multiple frameworks according to claim 1, characterized in that, Generating test cases based on the characteristic information of operators and the environmental information of different domestically controllable platforms includes: Construct an operator characteristic topology structure based on the characteristic information of each type of operator; the characteristic information includes matrix operation, convolution operation, and logical operation; the nodes in the operator characteristic topology structure represent the operator sub-elements in the characteristic information, and the edges between the nodes represent the execution order or dependency relationship between the operator sub-elements; Based on the dependency relationship and abstraction level among the hardware model, operating system version, and software framework version of the domestically controllable platform, divide the hardware model, operating system version, and software framework version of the domestically controllable platform into different platform environment levels; the platform environment levels include the hardware at the bottom layer, the operating system running based on the hardware at the middle layer, and the software framework depending on the operating system at the top layer; Based on the compatibility between the operator sub-elements in the operator characteristic topology structure and the platform environment levels, determine the adaptability of the operator in the environments of different domestically controllable platforms; Based on the adaptability of the operator in the environments of different domestically controllable platforms, determine the input parameters, expected output results, and test environment configuration information, and generate test cases.

6. The testing method for the compatibility and performance of multiple frameworks according to claim 5, characterized in that The generating test cases based on the adaptability of the operator in the environments of different domestically controllable platforms to determine the input parameters, expected output results, and test environment configuration information includes: Based on the adaptability of the operator in the environments of different domestically controllable platforms, determine the parameter boundaries, and collect parameters within the parameter boundaries based on the adaptability of the operator in the environments of different domestically controllable platforms and the complexity of the characteristic information of the operator to obtain the input parameters; For each node corresponding to an operator sub-element in the operator characteristic topology structure, traverse each node, take each node as the starting node, and perform simulation operations according to its execution order relationship and dependency relationship with the input parameters and the platform environment levels to obtain the expected output results corresponding to the input parameters; Convert the hardware model, operating system version, and software framework version into corresponding environment identification strings to obtain the test environment configuration information; Integrate the input parameters, expected output results, and test environment configuration information to generate test cases.

7. The method for testing the compatibility and performance of multiple frameworks according to any one of claims 1 to 6, characterized in that: The obtaining the operator to be tested based on the running information of different domestically controllable platforms includes: Identify and analyze the running information of the domestically controllable platform to obtain various platform running topics of the domestically controllable platform, and classify the operators in the operator library according to the topic classification system to obtain operator categories; Based on the shared features and dependency metrics between the running topics and the operator categories, establish an association mapping relationship between each type of platform running topic and its corresponding operator category; According to the topic popularity of each type of platform running topic, combined with the computational complexity and application frequency of each operator in the operator category corresponding to each type of platform running topic, determine the priority score of each operator in the operator category under each type of platform running topic; The priority score of each operator is traversed, and the operator whose priority score is greater than or equal to a preset score is determined as the operator to be tested.

8. A compatibility and performance testing system for multiple frameworks, characterized by: A method for testing the compatibility and performance of multiple frameworks as claimed in any one of claims 1 to 7; The compatibility and performance testing system of the various frameworks includes: The virtual test environment module is used to create a virtual test environment based on the environmental information of different autonomous and controllable platforms using virtualization technology; each virtual test environment simulates a combination of different environmental information; The test case generation module is used to obtain operators to be tested based on the operating information of different autonomous and controllable platforms, and generate test cases based on the characteristics of the operators and the environment information of different autonomous and controllable platforms; the test cases specify input parameters, expected output results and test environment configuration information; The test case execution module is used to distribute each test case to the corresponding virtual test environment for execution based on the test environment configuration information, and obtain the test execution results and performance indicator data of each test case during the execution process; The test evaluation module is used to determine the compatibility evaluation results and performance analysis results of the operator under different autonomous and controllable platforms based on the test execution results and performance indicator data.

9. An electronic device, comprising: Memory for storing computer software programs; A processor for reading and executing the computer software program, characterized in that when the processor executes the computer software program, it implements the compatibility and performance testing method of the multiple frameworks described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing a computer software program, characterized in that, When the computer software program is executed by a processor, the method for testing the compatibility and performance of multiple frameworks as claimed in any one of claims 1 to 7 is implemented.

Images