Algorithm feasibility test method, system and device and storage medium

Abstract

The invention discloses an algorithm feasibility test method, system and device and a storage medium, and the method comprises the steps: obtaining a to-be-tested algorithm code and a test data set of the to-be-tested algorithm code, determining the parameters of a running environment, simulating an actual running environment according to the parameters of the running environment, compiling the to-be-tested algorithm code into an executable code, and storing the executable code in a database; in an actual operation environment, a plurality of groups of input data of a test data set are respectively input into an executable code to obtain execution result data, the data volumes of each group of input data are different, performance scores of a plurality of dimensions are calculated according to the execution result data, weighted summation is performed on the performance scores of all the dimensions to obtain a feasibility test result, and the feasibility test result is obtained. The performance scores of the multiple dimensions at least comprise two dimensions of a time evaluation score, a memory evaluation score, an energy consumption evaluation score, a delay evaluation score and a stability evaluation score. The method can meet diversified tests and improve the test accuracy, and can be applied to the technical field of algorithm test.

Description

Technical Field

[0001] The present invention relates to the field of algorithm testing technology, and in particular to an algorithm feasibility testing method, system, device and storage medium. Background Art

[0002] With the development of information technology and the advent of the big data era, algorithms are playing an increasingly important role in various fields. Whether it is scientific research or commercial applications, efficient algorithms are the key to achieving specific tasks. However, the performance of an algorithm not only depends on the quality of its core logic, but is also affected by a variety of external factors.

[0003] The existing algorithm feasibility testing method cannot fully reflect the performance of the algorithm. This one-sided testing method ignores the problems that the algorithm may face in application. The evaluation results are highly subjective and uncertain, resulting in inaccurate test results. In addition, most existing systems lack flexibility and scalability, and are difficult to adapt to the diverse needs of algorithm performance testing in different fields and application scenarios. Summary of the invention

[0004] In order to solve one of the above problems, the purpose of the present invention is to provide an algorithm feasibility testing method, system, device and storage medium, which can meet the needs of diversified testing and improve the accuracy of testing.

[0005] In one aspect, the present invention provides an algorithm feasibility testing method, comprising the following steps:

[0006] Obtaining an algorithm code to be tested and a test data set of the algorithm code to be tested;

[0007] Determining parameters of the operating environment, and simulating the actual operating environment according to the parameters of the operating environment;

[0008] Compile the algorithm code to be tested into executable code, and in the actual running environment, input several groups of input data of the test data set into the executable code respectively to obtain execution result data; the execution result data includes actual execution time, actual memory usage, actual energy consumption, actual delay time or actual number of errors; the data volume of each group of input data is different;

[0009] The performance scores of several dimensions are calculated according to the execution result data, and the performance scores of all dimensions are weighted and summed to obtain the feasibility test result.

[0010] Optionally, the parameters of the operating environment include network status parameters, hardware condition parameters and socio-economic parameters; and determining the parameters of the operating environment and simulating the actual operating environment according to the parameters of the operating environment specifically includes:

[0011] Obtaining the network status parameters, hardware conditions and socio-economic parameters according to the custom parameter settings and / or preset parameter settings of the client;

[0012] Setting the network status of the actual operating environment according to the network status parameter;

[0013] The hardware combination and hardware parameters involved in the actual operating environment are set according to the hardware condition parameters;

[0014] The behavior of the client is identified according to the socio-economic parameters, and the operation habits of the client on the actual operating environment are set according to the behavior of the client.

[0015] Optionally, the execution result data includes actual execution time, actual memory usage, actual energy consumption, actual delay time, and actual number of errors; the performance scores of several dimensions include time evaluation score, memory evaluation score, energy consumption evaluation score, delay evaluation score, and stability evaluation score; the performance scores of several dimensions calculated according to the execution result data specifically include:

[0016] Calculating the time evaluation score according to the actual execution time and the expected execution time;

[0017] Calculate the memory evaluation score according to the actual occupied memory and the expected occupied memory;

[0018] Calculating the energy consumption evaluation score according to the actual energy consumption and the expected energy consumption;

[0019] Calculating the delay evaluation score according to the actual delay time and the expected delay time;

[0020] The stability assessment score is calculated based on the actual number of errors and the maximum number of errors in all groups of input data.

[0021] Optionally, the method further includes: calculating an algorithm efficiency index, where the algorithm efficiency index includes time complexity or space complexity, specifically including:

[0022] The time complexity is obtained according to the data volume of each set of input data and the actual execution time corresponding to each set of input data;

[0023] The time complexity is used as an index of the space complexity, and the space complexity is obtained according to the data volume of each group of input data, the actual occupied memory and the index of the space complexity.

[0024] Optionally, the execution result data also includes the output of the algorithm; the method further includes: calculating the algorithm energy efficiency index, the algorithm energy efficiency index includes the energy efficiency ratio and the energy consumption change rate, specifically including:

[0025] Dividing the actual energy consumption by the actual execution time to obtain the energy consumption change rate;

[0026] The energy efficiency ratio is obtained by dividing the output of the algorithm by the actual energy consumption.

[0027] Optionally, the execution result data also includes the number of crashes and the total number of executions; the method further includes: calculating a stability index, a crash rate or an error rate, specifically including:

[0028] Calculate the error rate according to the actual number of errors and the total number of executions;

[0029] Calculating the crash rate according to the number of crashes and the total number of executions;

[0030] The average execution time is calculated according to the actual execution time and the total number of executions, and the stability index is calculated according to the average execution time.

[0031] Optionally, the method further includes: formulating an algorithm optimization scheme according to the performance scores of several dimensions, specifically including:

[0032] Determine whether there is a performance score that is less than a preset score. If there is a performance score that is less than the preset score, obtain the algorithm optimization solution according to the performance score that is less than the preset score.

[0033] On the other hand, the present invention provides an algorithm feasibility test system, including a test input module, an algorithm environment simulation module, an algorithm execution module and a performance evaluation module, wherein:

[0034] The test input module is used to obtain the algorithm code to be tested and the test data set of the algorithm code to be tested;

[0035] The algorithm environment simulation module is used to determine the parameters of the operating environment and simulate the actual operating environment according to the parameters of the operating environment;

[0036] The algorithm execution module is used to compile the algorithm code to be tested into executable code, and in the actual running environment, input several groups of input data of the test data set into the executable code respectively to obtain execution result data; the execution result data includes actual execution time, actual memory usage, actual energy consumption, actual delay time or actual number of errors; the amount of data in each group of input data is different;

[0037] The performance evaluation module is used to calculate the performance scores of several dimensions according to the execution result data, and perform weighted summation on the performance scores of all dimensions to obtain a feasibility test result.

[0038] In another aspect, the present invention provides an algorithm feasibility testing device, comprising:

[0039] at least one processor;

[0040] at least one memory for storing at least one program;

[0041] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0042] In another aspect, the present invention provides a computer-readable storage medium storing a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to perform the method as described above.

[0043] The implementation of the present invention includes the following beneficial effects: the present invention obtains the algorithm code to be tested and the test data set of the algorithm code to be tested, determines the parameters of the operating environment, and simulates the actual operating environment according to the parameters of the operating environment, compiles the algorithm code to be tested into executable code, and in the actual operating environment, inputs several groups of input data of the test data set into the executable code respectively to obtain execution result data, wherein the execution result data includes actual execution time, actual occupied memory, actual energy consumption, actual delay time or actual number of errors; the data volume of each group of input data is different, and the performance scores of several dimensions are calculated according to the execution result data, and the performance scores of all dimensions are analyzed. The weighted sum of the rows is taken to obtain the feasibility test result; by simulating the actual operating environment of the algorithm, the influence of external factors on the execution of the algorithm can be simulated, the subjectivity and uncertainty of the evaluation results can be reduced, and the accuracy of the test results can be improved; by compiling the algorithm code to be tested into executable code, the flexibility and scalability of the test can be improved to adapt to the diverse needs of algorithm performance testing in different fields and application scenarios; the evaluation results are comprehensively calculated through time evaluation score, memory evaluation score, energy consumption evaluation score, delay evaluation score and stability evaluation score, and multi-dimensional performance evaluation is performed to adapt to the diverse needs of algorithm performance testing in different fields and application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of steps of an algorithm feasibility testing method provided by the present invention;

[0045] Figure 2 It is a flow chart of steps for simulating an actual operating environment provided by the present invention;

[0046] Figure 3 is a flow chart of steps for calculating a performance score provided by the present invention;

[0047] Figure 4It is a flow chart of steps for calculating time complexity and space complexity provided by the present invention;

[0048] Figure 5 It is a flow chart of steps for calculating energy efficiency ratio and energy consumption change rate provided by the present invention;

[0049] Figure 6 It is a flow chart of steps for calculating stability index, crash rate and error rate provided by the present invention;

[0050] Figure 7 It is a structural schematic diagram of an algorithm feasibility testing system provided by the present invention;

[0051] Figure 8 It is a structural schematic diagram of another algorithm feasibility testing system provided by the present invention;

[0052] Fig. 9 It is an execution flow chart of an algorithm feasibility testing system provided by the present invention;

[0053] Fig.10 It is a structural schematic diagram of an algorithm feasibility testing device provided by the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0056] When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it is necessary to understand that the terms "first / S100", "second / S200", "third / S300", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0057] In order to better understand the technical solution of the present invention, the following technical terms are explained:

[0058] Time complexity: It is an indicator to measure the speed of an algorithm. It indicates the trend of algorithm execution time as the amount of input data changes. Time complexity mainly measures the relationship between the number of times the basic operations of the algorithm are repeated and the time.

[0059] Space complexity: It is a measure of the amount of storage space temporarily occupied during the operation of an algorithm. Space complexity mainly measures the relationship between the additional storage space requested during the operation of an algorithm and the amount of input data.

[0060] like Figure 1 As shown, Figure 1 The present invention provides a method for testing the feasibility of an algorithm, comprising the following steps:

[0061] S100: Obtain an algorithm code to be tested and a test data set of the algorithm code to be tested.

[0062] The format of the test data set may include, but is not limited to, comma-separated values ​​(CSV) format, JSON (JavaScript Object Notation) format, and eXtensible Markup Language (XML) format. The test data set may be obtained from, but is not limited to, MySQL, Oracle, or SQL-Server databases.

[0063] Specifically, after the algorithm code to be tested is obtained, the algorithm code to be tested is parsed, and the correctness and executability of the parsed algorithm code to be tested are verified.

[0064] After receiving the test data set of the algorithm code to be tested, the test data sets in various formats can be managed, converted into executable formats, and the integrity and rationality of the test data set can be verified. If the integrity and rationality verification fails, error information of the test data set will be fed back.

[0065] For test data sets, it can also support the functions of uploading, downloading, and editing data sets, that is, test data sets can be uploaded from the database or user input, and can also be downloaded by users. When the integrity and rationality verification fails or the user needs to modify the test data set, the uploaded test data set can be edited to obtain the test data set used for testing.

[0066] S200: Determine parameters of an operating environment, and simulate an actual operating environment according to the parameters of the operating environment.

[0067] Among them, the parameters of the operating environment include but are not limited to network status parameters, hardware condition parameters and socio-economic parameters; network status parameters include but are not limited to one or more of bandwidth, delay or packet loss rate; hardware condition parameters include but are not limited to one or more of CPU performance, memory size or storage speed; socio-economic parameters include but are not limited to one or more of user behavior patterns or market dynamics data.

[0068] Specifically, the parameters of the operating environment can be obtained according to the user's settings or the preset parameter configuration of the operating environment, the network status parameters, hardware conditions and socio-economic parameters can be obtained according to the client's customized parameter settings and / or preset parameter settings, the network status of the actual operating environment can be set according to the network status parameters, the hardware combination involved in the actual operating environment and the hardware parameters can be set according to the hardware condition parameters, the client's behavior can be identified according to the socio-economic parameters, and the client's operating habits for the actual operating environment can be set according to the client's behavior.

[0069] S300, compiling the algorithm code to be tested into executable code, and in an actual running environment, inputting several groups of input data of the test data set into the executable code respectively to obtain execution result data.

[0070] The execution result data includes at least two of the actual execution time, actual memory usage, actual energy consumption, actual delay time and actual number of errors for each set of input data, and the amount of data for each set of input data is different.

[0071] It is worth noting that several groups refer to more than one group, and the data in different groups of input data may overlap.

[0072] Specifically, the coder is used to parse the algorithm code to be tested obtained in step S100 to verify the executableness and correctness of the parsed algorithm code to be tested. If the algorithm code to be tested is not executable, the compiler is used to compile the non-executable algorithm to be tested to obtain executable code.

[0073] In the actual operation environment set up above, several groups of input data with different data volumes are input into the executable code respectively, and the executable code is executed. The execution completion of each group of input data and the actual start time T of each group of input data are recorded. start , actual end time T end 、Actual memory occupied M actual , the total energy consumption E at the end of algorithm execution end , the total energy consumption E at the beginning of algorithm execution start , actual delay time Dactual and the actual number of errors F actual , and calculate the actual execution time T exec = Actual end time T end -Actual start time T start ; Calculate the actual energy consumption ΔE=E end -E start .

[0074] S400: Calculate performance scores of several dimensions according to the execution result data, perform weighted summation on the performance scores of all dimensions, and obtain a feasibility test result.

[0075] Among them, the performance scores of several dimensions include at least two dimensions of time evaluation score, memory evaluation score, energy consumption evaluation score, delay evaluation score and stability evaluation score.

[0076] Specifically, the time evaluation score is calculated based on the actual execution time and the expected execution time, the memory evaluation score is calculated based on the actual memory occupied and the expected memory occupied, the energy consumption evaluation score is calculated based on the actual energy consumption and the expected energy consumption, the delay evaluation score is calculated based on the actual delay time and the expected delay time, and the stability evaluation score is calculated based on the actual number of errors and the maximum number of errors in all groups of input data.

[0077] The time evaluation score, memory evaluation score, energy consumption evaluation score, delay evaluation score and stability evaluation score are combined using the weighted average method to obtain a comprehensive performance score, i.e., the feasibility test result. If only the impact of two or more scores on the algorithm test is considered, only the weighted sum of the two or more scores is taken, and the sum of the weights of the two or more scores is 1.

[0078] In some embodiments, Figure 2 As shown, Figure 2 The process of simulating an actual operating environment is as follows: S200 determines parameters of the operating environment and simulates the actual operating environment according to the parameters of the operating environment, specifically including:

[0079] S210: Obtain network status parameters, hardware conditions, and socioeconomic parameters according to the custom parameter settings and / or preset parameter settings of the client.

[0080] Among them, the preset parameter settings include multiple levels of network status parameters, hardware conditions and socio-economic parameters. For example, for the bandwidth of the network status parameter, several levels can be set from a low-bandwidth narrowband network to a high-bandwidth high-speed network, multiple CPU performance levels, from a single-core processor to a multi-core multi-threaded processor, different memory sizes are configured, including small memory and large memory environments, and storage devices of different types and speeds are simulated to simulate the actual environment of algorithm execution.

[0081] Specifically, the network status parameters, hardware conditions and socio-economic parameters can be determined according to the client's customized parameter settings, or according to the preset parameter settings. If some parameters are set in the user's customized parameter settings, the preset parameter settings can also be used to set some of the parameters customized by the user.

[0082] S220. Setting the network status of the actual operating environment according to the network status parameters.

[0083] Specifically, in terms of network condition simulation, bandwidth limits are accurately set according to network condition parameters, different degrees of network delays are simulated, and packet loss rates are set to simulate unstable network connections.

[0084] S230: Setting the hardware combination and hardware parameters involved in the actual operating environment according to the hardware condition parameters.

[0085] Specifically, for the simulation of hardware conditions, the computing resources of the actual operating environment are allocated according to the hardware condition parameter settings, including but not limited to the number of CPU cores, the amount of memory, the number of disks, as well as the performance of the CPU, the memory size of each memory, the disk space of each disk, etc.

[0086] In some embodiments, for the algorithm code to be tested that supports parallel computing, multiple CPUs can also be used to determine the number of CPUs and the task division of each CPU. For the algorithm code to be tested that requires greater computing resources, a hardware combination of distributed CPU, distributed memory, and distributed disk can be used to distribute computing tasks among multiple computers and achieve load balancing.

[0087] In some embodiments, if network status parameters, hardware conditions and socio-economic parameters are obtained through preset parameter settings, several groups of operating environment parameters can be set, and several levels can be set from low-bandwidth narrowband networks to high-bandwidth high-speed networks, multiple CPU performance levels, from single-core processors to multi-core multi-threaded processors, and different memory sizes can be configured. Each group of operating environment parameters includes a level of network status, a level of CPU performance, a processor distribution method, and a memory size. Then, several actual execution environments are simulated according to the parameters of several groups of operating environments, and executable code is executed in each actual execution environment to obtain execution result data and feasibility test results under different execution environments.

[0088] S240: Identify the behavior of the client according to the socioeconomic parameters, and set the client's operating habits for the actual operating environment according to the behavior of the client.

[0089] Specifically, in the simulation of socioeconomic factors, user behavior and market dynamics data are identified based on socioeconomic parameters to obtain the operating habits, demand fluctuations and changes in the competitive environment of different types of users. The operating habits include but are not limited to frequent operations, random operations, and long periods of idleness. The users' operating habits, demand fluctuations and changes in the competitive environment are simulated.

[0090] In some embodiments, Figure 3 As shown, Figure 3 The flowchart is a step of calculating the performance score. Step S400 calculates the performance scores of several dimensions according to the execution result data, specifically including:

[0091] S410: Calculate a time evaluation score according to the actual execution time and the expected execution time.

[0092] Specifically, substitute the actual execution time and expected execution time into formula (1) to obtain the time evaluation score S t :

[0093]

[0094] Among them, T actual =T exec ; T expected Indicates the expected execution time.

[0095] S420: Calculate a memory evaluation score according to the actual memory usage and the expected memory usage.

[0096] Specifically, substitute the actual memory usage and expected memory usage into formula (2) to obtain the memory evaluation score S: s :

[0097]

[0098] Among them, M actual Indicates the actual memory usage; M expected Indicates expected memory usage.

[0099] S430: Calculate an energy consumption evaluation score according to the actual energy consumption and the expected energy consumption.

[0100] Specifically, the actual energy consumption and expected energy consumption are substituted into formula (3) to obtain the energy consumption evaluation score S: e :

[0101]

[0102] Among them, E actual =ΔE; E expected It represents the expected energy consumption during the execution of the algorithm.

[0103] S440: Calculate a delay evaluation score according to the actual delay time and the expected delay time.

[0104] Specifically, substitute the actual delay time and the expected delay time into formula (4) to obtain the delay evaluation score S d :

[0105]

[0106] Among them, D actual Indicates the actual delay time; D expected Indicates the expected delay time.

[0107] S450, calculating a stability evaluation score according to the actual number of errors and the maximum number of errors in all groups of input data.

[0108] Specifically, the actual number of errors and the maximum number of errors in all groups of input data are substituted into formula (5) to obtain the stability evaluation score S a :

[0109]

[0110] Among them, F actual Indicates the actual number of errors in a set of input data; F max Indicates the maximum number of errors.

[0111] For the performance scores of each dimension obtained, the comprehensive performance score P is calculated by weighted summation, as shown in formula (6):

[0112] P=w1×S t +w2×S s +w3×S e +w4×S d +w5×S a (6)

[0113] Among them, w1 represents S t The weight of S s The weight of S e The weight of S d The weight of S a weight; and w1+w2+w3+w4+w5=1.

[0114] If you only care about the impact of performance of some dimensions on the algorithm, you can also calculate the comprehensive performance score P of some performance, for example, only care about S t +S s , then P = w1 × S t +w2×S s , w1+w2=1.

[0115] Among them, the expected execution time, expected memory usage, expected energy consumption and expected delay time can be customized by the user according to the test requirements, or can be calculated based on the data volume of each set of input data and the algorithm code to be tested.

[0116] In some embodiments, Figure 4 As shown, Figure 4 The method further comprises S500, calculating an algorithm efficiency index, wherein the algorithm efficiency index comprises time complexity or space complexity, and specifically comprises:

[0117] S510 , obtaining time complexity according to the data volume of each group of input data and the actual execution time corresponding to each group of input data.

[0118] Among them, time complexity can include linear time complexity and quadratic time complexity.

[0119] Specifically, if the time complexity is linear, substitute the amount of each set of input data and the actual execution time corresponding to each set of input data into formula (7) for fitting, and get the time complexity S n :

[0120] S n =an+b (7)

[0121] Where a and b are constants, and n is the amount of input data.

[0122] If the time complexity is square, substitute the amount of each set of input data and the actual execution time corresponding to each set of input data into formula (8) for fitting, and get the time complexity S n :

[0123] S n =an 2 +bn+c (8)

[0124] Where a, b, and c are constants, and n is the amount of input data.

[0125] Time complexity refers to the execution time of an algorithm under different data amounts.

[0126] S520. Using the time complexity as an index of the space complexity, the space complexity is obtained according to the data volume of each group of input data, the actual occupied memory and the index of the space complexity.

[0127] Specifically, the time complexity is used as the index of the space complexity, and the amount of data of each set of input data, the actual memory occupied, and the index of the space complexity are substituted into formula (9) to obtain the space complexity S(n):

[0128] S(n)=T(n m ) (9)

[0129] Where T(.) represents a function about n, m = S n , n represents the amount of input data.

[0130] In some embodiments, Figure 5 As shown, Figure 5 It is a flowchart of steps for calculating energy efficiency ratio and energy consumption change rate. The execution result data recorded in the aforementioned step S200 also includes the output of the algorithm.

[0131] The above method further includes S600, calculating an algorithm energy efficiency index, where the algorithm energy efficiency index includes an energy efficiency ratio and an energy consumption change rate, specifically including:

[0132] S610 , divide the actual energy consumption by the actual execution time to obtain an energy consumption change rate.

[0133] Specifically, the actual energy consumption is divided by the actual execution time to obtain the energy consumption change rate, that is, Where Δt = T exec .

[0134] S620: Divide the output of the algorithm by the actual energy consumption to obtain an energy efficiency ratio.

[0135] Specifically, the energy consumed by processing unit input data is evaluated, and the energy efficiency ratio E sfcc Represents the output of the algorithm.

[0136] like Figure 6 As shown, Figure 6 It is a flowchart of steps for calculating stability index, crash rate and error rate. The execution result data recorded in step S200 also includes the number of crashes and the total number of executions. The number of crashes refers to the number of crashes of the algorithm in an iterative process in a set of input data, and the total number of executions refers to the total number of iterations of the algorithm in a set of input data.

[0137] The aforementioned method further includes S700, calculating a stability index σ, a crash rate BKL or an error rate CWL, specifically including:

[0138] S710: Calculate the error rate according to the actual number of errors and the total number of executions.

[0139] Specifically, the error rate is calculated Where E = actual number of errors F actual , N represents the total number of executions.

[0140] S720: Calculate the crash rate according to the number of crashes and the total number of executions.

[0141] Specifically, the crash rate is calculated Where C represents the number of crashes.

[0142] S730. Calculate the average execution time according to the actual execution time and the total number of executions, and calculate the stability index according to the average execution time.

[0143] Specifically, according to the actual execution time Calculate the average execution time by summing the total number of executions N Calculate stability index based on average execution time t i Indicates the time used for the i-th iteration.

[0144] In some embodiments, the algorithm execution module also records the CPU usage Peak memory usage M peak =max(M t ), and form an evaluation report with the performance scores of each dimension, comprehensive performance scores, execution result data, time complexity, space complexity, energy consumption change rate, energy efficiency ratio, error rate, crash rate and stability indicators, among which the aforementioned time complexity and space complexity are used to evaluate the efficiency of the algorithm code to be tested; the energy consumption change rate and energy efficiency ratio are used to evaluate the energy utilization of the algorithm code to be tested; the error rate, crash rate and stability indicators are used to evaluate the stability of the algorithm code to be tested.

[0145] In some embodiments, the aforementioned method further includes S800, formulating an algorithm optimization plan according to the performance scores of several dimensions, specifically including:

[0146] Determine whether there is a performance score less than a preset score. If there is a performance score less than the preset score, obtain an algorithm optimization solution based on the performance score less than the preset score.

[0147] Specifically, each performance score has its own corresponding preset score. For example, the energy consumption evaluation score has its corresponding preset energy consumption score, the time evaluation score has its corresponding preset time score, the memory evaluation score has its corresponding preset memory score; the delay evaluation score has its corresponding preset delay score; and the stability evaluation score has its corresponding preset stability score.

[0148] Find the performance scores that are less than the preset scores, and then formulate algorithm optimization plans based on the performance scores that are less than the preset scores. For example, if the time evaluation score or time complexity is less than the preset time score, recommend more efficient data structures and algorithms; if the stability evaluation score is less than the preset stability score, recommend new exception handling logic and hardware conditions, etc.

[0149] Based on the above methods, it can be seen that the feasibility test results may include but are not limited to one or more of execution result data, performance scores in various dimensions, comprehensive performance scores, time complexity, space complexity, energy efficiency ratio, energy consumption change rate, stability index, crash rate or error rate.

[0150] In some embodiments, the present invention can feed back the algorithm optimization plan to the test input module, and the test input module outputs it to the client. The user adjusts the algorithm code to be tested according to the algorithm optimization plan, inputs the algorithm code to be tested into the test input module, and tests it again according to the aforementioned algorithm feasibility test method to continuously improve the performance and feasibility of the algorithm to be tested.

[0151] The implementation of the present invention includes the following beneficial effects: the present invention can simulate the influence of external factors on the execution of the algorithm by simulating the actual operating environment of the algorithm, can reduce the subjectivity and uncertainty of the evaluation results, and improve the accuracy of the test results; by compiling the algorithm code to be tested into executable code, the flexibility and scalability of the test can be improved to adapt to the diverse needs of algorithm performance testing in different fields and different application scenarios; by comprehensively calculating the evaluation results through time evaluation scores, memory evaluation scores, energy consumption evaluation scores, delay evaluation scores and stability evaluation scores, multi-dimensional performance evaluation is performed, which can adapt to the diverse needs of algorithm performance testing in different fields and different application scenarios. The system can automatically complete a series of complex processes of algorithm compilation, execution, performance evaluation and report generation. This automated testing method not only reduces manual intervention and reduces the risk of human error, but also greatly shortens the test cycle, making the optimization and verification of the algorithm more efficient and quick.

[0152] The present invention can provide a parameter customization function for the operating environment, flexibly combine and adjust the parameters of various environmental factors according to the test requirements, and create a highly customized simulation environment. The feasibility of the algorithm can be evaluated from multiple dimensions and angles.

[0153] The present invention adopts a multi-dimensional scoring mechanism, including multiple indicators of correctness, efficiency, and stability, presents test results and evaluation reports, and provides a feedback mechanism to adjust the algorithm based on the feedback.

[0154] like Figure 7 As shown, Figure 7 The present invention also provides an algorithm feasibility testing system, which includes a test input module, an algorithm environment simulation module, an algorithm execution module and a performance evaluation module, wherein:

[0155] A test input module is used to obtain the algorithm code to be tested and the test data set of the algorithm code to be tested;

[0156] The algorithm environment simulation module is used to determine the parameters of the operating environment and simulate the actual operating environment according to the parameters of the operating environment;

[0157] The algorithm execution module is used to compile the algorithm code to be tested into executable code, and in the actual running environment, input several groups of input data of the test data set into the executable code respectively to obtain execution result data; the execution result data includes at least two of the actual execution time, actual memory usage, actual energy consumption, actual delay time and actual number of errors, and the amount of data in each group of input data is different;

[0158] The performance evaluation module is used to calculate the performance scores of several dimensions based on the execution result data, and to perform weighted summation on the performance scores of all dimensions to obtain the feasibility test results; the performance scores of several dimensions include at least two dimensions of time evaluation score, memory evaluation score, energy consumption evaluation score, delay evaluation score and stability evaluation score.

[0159] Specifically, the test input module is used to implement the above step S100 and its subordinate steps; the algorithm environment simulation module is used to implement the above step S200 and its subordinate steps; the algorithm execution module is used to implement the above step S300 and its subordinate steps; the performance evaluation module is used to implement the above step S400 and its subordinate steps.

[0160] Specifically, the performance evaluation module is also used to implement the above steps S500-S700 and their subordinate steps.

[0161] like Figure 8 As shown, Figure 8 It is a structural diagram of another algorithm feasibility testing system. The algorithm feasibility testing system also includes an algorithm feasibility optimization module, and the algorithm feasibility optimization module is used to implement the aforementioned step S800.

[0162] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0163] like Fig. 9 As shown, Fig. 9It is an execution flow chart of an algorithm feasibility test system. In the specific implementation, the test input module receives the algorithm code to be tested and the corresponding test data set uploaded by the user. The test input module supports direct import of multiple data formats such as CSV, JSON, XML, etc., and can directly import data from databases such as MySQL, Oracle, SQL-Server, etc. After the test data set is uploaded, the test input module will perform format check, size limit and type verification on the test data set to verify the accuracy and completeness of the data in the test data set. In addition, the test input module also includes an algorithm code parsing submodule and a data set management submodule. The algorithm code parsing submodule is responsible for parsing the received algorithm code to be tested, and verifying its correctness and executability, and sending the verified algorithm code to be tested to the algorithm execution module. The data set management submodule is responsible for managing the upload, download and editing of the test data set.

[0164] Next, the algorithm environment simulation module simulates environmental factors in multiple dimensions, including network conditions, hardware conditions, and socio-economic factors, according to the test requirements (custom parameter settings and / or preset parameter settings of the client). In the network condition simulation, the bandwidth limit is accurately set, different degrees of network delay are simulated, and the packet loss rate is set to simulate an unstable network connection. The hardware condition simulation covers different levels of CPU performance, memory size, and storage speed to simulate different hardware environments. The socio-economic factor simulation examines the performance of the algorithm in different application scenarios by simulating user behavior patterns and market dynamics. These highly customized simulation environments provide a solid foundation for the comprehensive evaluation of the algorithm.

[0165] The algorithm execution module is responsible for compiling and executing the algorithm code, and supports parallel computing and distributed computing. After receiving the algorithm code to be tested and the test data set from the test input module, the algorithm execution module will allocate computing resources and monitor the execution process of the algorithm in real time. The module will record the execution result data of the algorithm code execution as well as the key execution steps and time nodes, providing detailed data support for subsequent performance evaluation. During the algorithm execution process, the system also supports real-time monitoring and recording of the algorithm execution results to ensure the transparency and traceability of the test process.

[0166] The performance evaluation module conducts a multi-dimensional evaluation of the results and performance data of the execution of the algorithm code to be tested. The evaluation contents include time complexity evaluation, space complexity evaluation, energy consumption evaluation, delay evaluation and stability evaluation. The time complexity evaluation includes the evaluation of time complexity, space complexity, time evaluation score and memory evaluation score. The performance evaluation module will use mathematical analysis and statistical methods to calculate the time growth rate and space occupancy of the algorithm code to be tested under different data volumes. For energy consumption evaluation, the performance evaluation module will monitor the energy consumption changes of the hardware during the execution of the algorithm. Energy consumption evaluation includes evaluating its energy efficiency ratio, energy consumption change rate and energy consumption evaluation score. Delay evaluation focuses on the response time from the algorithm processing input data to generating output results, especially focusing on the performance in high concurrency and large data processing scenarios. Delay evaluation includes calculating its delay evaluation score. Stability evaluation evaluates its stability and reliability under different conditions through repeated tests and statistical indicators such as the error rate and number of crashes of the algorithm. Stability evaluation includes evaluating its stability evaluation score, stability index, crash rate or error rate.

[0167] It is understandable that the algorithm feasibility test method provided by the present invention can be applied to any computer device with data processing computing capabilities, and this computer device can be various types of terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this.

[0168] The possible application scenarios of the above algorithm feasibility testing method and system in the future include:

[0169] 1. Multi-dimensional testing scenarios of various AI algorithms in the field of intelligent security, such as: Network Video Recorder (NVR), such as Telecom Vision Smart Box, etc.; algorithm testing of target recognition, vehicle recognition, fighting, electric vehicle detection, high-altitude object detection, garbage detection, etc. for products such as intelligent IPC.

[0170] 2. Big data AI algorithm testing scenarios, such as: AI platform products and services require performance testing, including data processing, performance evaluation of machine learning algorithms, etc.

[0171] 3. Scenarios with high energy efficiency requirements, such as: drones and robots require efficient algorithms to optimize energy consumption when performing tasks, data exchange efficiency in distributed computing scenarios, and AI algorithms in self-driving cars need to efficiently process large amounts of sensor data while maintaining low energy consumption.

[0172] Possible values ​​of the present invention include:

[0173] 1. Through a highly automated algorithm testing process, the testing efficiency and accuracy of the algorithm are significantly improved. Users upload the algorithm code and test data set to be tested, and the system can automatically complete a series of complex processes including algorithm compilation, execution, performance evaluation and report generation. This not only reduces manual intervention and the risk of human error, but also greatly shortens the testing cycle.

[0174] 2. The detailed test reports and optimization suggestions provided by the system also provide strong support for users' subsequent algorithm improvement work.

[0175] like Fig.10 As shown, Fig.10 The present invention is a schematic diagram of the structure of an algorithm feasibility testing device. The present invention also provides an algorithm feasibility testing device, including:

[0176] at least one processor;

[0177] at least one memory for storing at least one program;

[0178] When the at least one program is executed by the at least one processor, the at least one processor implements the algorithm feasibility testing method steps described in the above method embodiment.

[0179] Among them, the memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. The memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a remote memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0180] It can be seen that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0181] In addition, the embodiment of the present application also discloses a computer program product or a computer program, and the computer program product or the computer program is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above method. Similarly, the contents in the above method embodiment are all applicable to the storage medium embodiment, and the functions specifically implemented by the storage medium embodiment are the same as those in the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method embodiment.

[0182] The present invention also provides a computer-readable storage medium, which stores a program executable by a processor. The program executable by the processor is used to implement the above method when executed by the processor.

[0183] It is understood that all or some steps and systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0184] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0185] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application. In the present invention, the same or substantially equivalent methods between different embodiments are applicable to other embodiments.

Claims

1. An algorithm feasibility testing method, characterized in that: The following steps are involved: Obtaining an algorithm code to be tested and a test data set of the algorithm code to be tested; Determining parameters of the operating environment, and simulating the actual operating environment according to the parameters of the operating environment; Compile the algorithm code to be tested into executable code, and in the actual running environment, input several groups of input data of the test data set into the executable code respectively to obtain execution result data; the execution result data includes actual execution time, actual memory usage, actual energy consumption, actual delay time or actual number of errors; The amount of data in each set of input data is different; The performance scores of several dimensions are calculated according to the execution result data, and the performance scores of all dimensions are weighted and summed to obtain the feasibility test result.

2. The method according to claim 1, characterized in that The parameters of the operating environment include network status parameters, hardware condition parameters and socio-economic parameters; the step of determining the parameters of the operating environment and simulating the actual operating environment according to the parameters of the operating environment specifically includes: Obtaining the network status parameters, hardware conditions and socio-economic parameters according to the custom parameter settings and / or preset parameter settings of the client; Setting the network status of the actual operating environment according to the network status parameter; The hardware combination and hardware parameters involved in the actual operating environment are set according to the hardware condition parameters; The behavior of the client is identified according to the socio-economic parameters, and the operation habits of the client on the actual operating environment are set according to the behavior of the client.

3. The method according to claim 1, characterized in that The performance scores of several dimensions include a time evaluation score, a memory evaluation score, an energy consumption evaluation score, a delay evaluation score, and a stability evaluation score; the performance scores of several dimensions are calculated according to the execution result data, specifically including: Calculating the time evaluation score according to the actual execution time and the expected execution time; Calculate the memory evaluation score according to the actual occupied memory and the expected occupied memory; Calculating the energy consumption evaluation score according to the actual energy consumption and the expected energy consumption; Calculating the delay evaluation score according to the actual delay time and the expected delay time; The stability assessment score is calculated based on the actual number of errors and the maximum number of errors in all groups of input data.

4. The method according to claim 1, characterized in that: The method further includes: calculating an algorithm efficiency index, wherein the algorithm efficiency index includes time complexity or space complexity, specifically including: The time complexity is obtained according to the data volume of each set of input data and the actual execution time corresponding to each set of input data; The time complexity is used as an index of the space complexity, and the space complexity is obtained according to the data volume of each group of input data, the actual occupied memory and the index of the space complexity.

5. The method according to claim 1, characterized in that: The execution result data also includes the output of the algorithm; the method also includes: calculating the algorithm energy efficiency index, the algorithm energy efficiency index includes the energy efficiency ratio and the energy consumption change rate, specifically including: Dividing the actual energy consumption by the actual execution time to obtain the energy consumption change rate; The energy efficiency ratio is obtained by dividing the output of the algorithm by the actual energy consumption.

6. The method according to claim 1, characterized in that The execution result data also includes the number of crashes and the total number of executions; the method also includes: calculating the stability index, the crash rate or the error rate, specifically including: Calculate the error rate according to the actual number of errors and the total number of executions; Calculating the crash rate according to the number of crashes and the total number of executions; The average execution time is calculated according to the actual execution time and the total number of executions, and the stability index is calculated according to the average execution time.

7. The method according to claim 1, characterized in that The method further includes: formulating an algorithm optimization scheme according to the performance scores of several dimensions, specifically including: Determine whether there is a performance score that is less than a preset score. If there is a performance score that is less than the preset score, obtain the algorithm optimization solution according to the performance score that is less than the preset score.

8. An algorithm feasibility testing system, characterized in that: It includes test input module, algorithm environment simulation module, algorithm execution module and performance evaluation module, among which, The test input module is used to obtain the algorithm code to be tested and the test data set of the algorithm code to be tested; The algorithm environment simulation module is used to determine the parameters of the operating environment and simulate the actual operating environment according to the parameters of the operating environment; The algorithm execution module is used to compile the algorithm code to be tested into executable code, and in the actual running environment, input several groups of input data of the test data set into the executable code respectively to obtain execution result data; the execution result data includes actual execution time, actual memory usage, actual energy consumption, actual delay time or actual number of errors; the data volume of each group of input data is different; The performance evaluation module is used to calculate the performance scores of several dimensions according to the execution result data, and perform weighted summation on the performance scores of all dimensions to obtain a feasibility test result.

9. An algorithm feasibility testing device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

Images