An automated testing method, device and electronic equipment based on multi-intelligent algorithm

Abstract

The present application provides an automated testing method, device and electronic device based on multi-intelligent algorithms; wherein the method comprises: according to a preset protocol, arranging each single intelligent algorithm in the multi-intelligent algorithm into a plurality of test cases for testing the corresponding algorithms; according to a preset correlation table, determining the correlation between any two single intelligent algorithms in the multi-intelligent algorithm; according to the correlation, calculating the correlation metric of each single intelligent algorithm in the multi-intelligent algorithm; the correlation metric of any single intelligent algorithm is a comprehensive index obtained by comprehensive calculation of the correlation between any single intelligent algorithm and other single intelligent algorithms except itself; according to the correlation metric, determining the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm, and scheduling the test cases of each single intelligent algorithm according to the scheduling order to test the corresponding algorithm, and obtaining the test result of the collaborative work of the multi-intelligent algorithm.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an automated testing method, device and electronic equipment based on a multi-intelligent algorithm. Background Art

[0002] Video-based intelligent algorithms are widely used in embedded smart devices that process surveillance videos. The intelligent algorithm automated testing method aims to form test cases through abstract test steps, and use tools such as test frameworks or host computers to execute test cases to achieve automated testing and reduce manual input.

[0003] With the development of embedded applications, a variety of intelligent algorithms can usually be installed in one embedded device. How to improve the testing efficiency of multiple intelligent algorithms has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide an automated testing method, device, and electronic device based on a multi-intelligent algorithm, which are used to improve the efficiency of automated testing of the multi-intelligent algorithm and ensure the reliability and performance of the multi-intelligent algorithm in practical applications.

[0005] In a first aspect, an embodiment of the present application provides an automated testing method based on a multi-intelligent algorithm, comprising:

[0006] According to a preset protocol, each single intelligent algorithm in the multi-intelligent algorithm is arranged into a plurality of test cases for testing the corresponding algorithm;

[0007] According to a preset correlation table, determining the correlation between any two single-intelligent algorithms in the multi-intelligent algorithm; wherein the correlation is used to characterize the probability of parallel operation between the corresponding two single-intelligent algorithms;

[0008] According to the correlation degree, calculating the correlation metric of each single intelligent algorithm in the multi-intelligent algorithm; wherein the correlation metric of any single intelligent algorithm is a comprehensive index obtained by comprehensive calculation based on the correlation degree between any single intelligent algorithm and other single intelligent algorithms except itself;

[0009] According to the association metric, the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm is determined, and the test cases of each single intelligent algorithm are scheduled according to the scheduling order to test the corresponding algorithm to obtain the test results of the collaborative work of the multi-intelligent algorithm.

[0010] In one possible implementation, determining the scheduling order of collaborative work of each single intelligent algorithm in the multi-intelligent algorithm according to the association metric, and scheduling the test cases of each single intelligent algorithm according to the scheduling order to test the corresponding algorithm, and obtaining the test result of the collaborative work of the multi-intelligent algorithm includes:

[0011] Determine a single intelligent algorithm with the highest correlation metric from the multiple intelligent algorithms, prioritize the scheduling of test cases corresponding to the single intelligent algorithm with the highest correlation metric, and use the single intelligent algorithm with the highest correlation metric as the initial scheduling algorithm;

[0012] According to the preset correlation table, from the multi-intelligent algorithm, except the initial scheduling algorithm, the algorithm with the highest correlation with the initial scheduling algorithm is searched and used as the candidate scheduling algorithm, the test case of the candidate scheduling algorithm is scheduled, and the initial scheduling algorithm and the candidate scheduling algorithm are used as the combination algorithm of the current scheduling;

[0013] From the algorithms to be scheduled in the multi-intelligent algorithms except the combined algorithm, select the test case corresponding to the single-intelligent algorithm with the highest correlation metric with the currently scheduled combined algorithm for scheduling until the test cases of all algorithms in the multi-intelligent algorithms reach a preset scheduling number, and test the corresponding algorithms to obtain the test results of the collaborative work of the multi-intelligent algorithms.

[0014] In one possible implementation, before scheduling the test cases of each single intelligent algorithm, the method further includes:

[0015] Obtain the remaining resources of the test equipment currently used to schedule the test cases, and verify them with the resource usage required to run the corresponding test cases;

[0016] According to the verification results, the test cases of the multi-intelligent algorithm are adaptively scheduled and the corresponding algorithms are tested.

[0017] In one possible implementation, adaptively scheduling the test cases of the multi-intelligent algorithm according to the verification result and testing the corresponding algorithm includes:

[0018] If the resource usage required to run the corresponding test case on the test device is greater than the remaining resource, it indicates that the test case does not meet the scheduling condition;

[0019] Adjusting the scheduling order of the test cases in the corresponding algorithm;

[0020] The remaining test cases of the test case in the corresponding algorithm except the test case are traversed in sequence until the resource usage required by the corresponding test case is less than or equal to the remaining resource amount, the corresponding test case is scheduled, and the corresponding algorithm is tested.

[0021] In one possible implementation, after arranging the individual single-intelligent algorithms in the multi-intelligent algorithms into a plurality of test cases for testing the corresponding algorithms according to the preset protocol, the method further includes:

[0022] Running each test case of each single intelligent algorithm in the multi-intelligent algorithm on the test device one by one to check the availability of the corresponding test case;

[0023] Delete the test cases with invalid test results, retain the valid test cases, and update the test cases corresponding to each single intelligent algorithm.

[0024] In one possible implementation, the method of arranging the individual single-intelligent algorithms in the multi-intelligent algorithms into multiple test cases for testing the corresponding algorithms according to a preset protocol includes:

[0025] Based on the test requirements of each single intelligent algorithm in the multi-intelligent algorithm, decompose the operation steps for testers to configure the corresponding algorithm during actual testing;

[0026] The operation steps of each single intelligent algorithm are configured according to the preset protocol to generate multiple test cases for the corresponding algorithm; wherein each test case has a single function, and each test case consists of a set of operation steps, and each operation step in the set of operation steps is abstracted as represented by a set {test data parameters, test step parameters}; the test data parameters cover various characteristics related to the video; and the test step parameters are information related to the operation instructions executed for each intelligent algorithm.

[0027] In a second aspect, an embodiment of the present application provides an automated testing device based on a multi-intelligent algorithm, comprising:

[0028] A use case configuration module, used to arrange each single intelligent algorithm in the multi-intelligent algorithm into multiple test cases for testing the corresponding algorithm according to a preset protocol;

[0029] A use case deployment module, used to determine the correlation between any two single intelligent algorithms in the multi-intelligent algorithm according to a preset correlation table; wherein the correlation is used to characterize the probability of parallel operation between the corresponding two single intelligent algorithms; according to the correlation, calculate the correlation metric of each single intelligent algorithm in the multi-intelligent algorithm; wherein the correlation metric of any single intelligent algorithm is a comprehensive index calculated based on the correlation between any single intelligent algorithm and other single intelligent algorithms except itself;

[0030] The use case scheduling module is used to determine the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm according to the association metric, and schedule the test cases of each single intelligent algorithm according to the scheduling order to test the corresponding algorithm, so as to obtain the test results of the collaborative work of the multi-intelligent algorithm.

[0031] In one possible implementation, the use case deployment module is used to:

[0032] Determine a single intelligent algorithm with the highest correlation metric from the multiple intelligent algorithms, prioritize the scheduling of test cases corresponding to the single intelligent algorithm with the highest correlation metric, and use the single intelligent algorithm with the highest correlation metric as the initial scheduling algorithm;

[0033] According to the preset correlation table, from the multi-intelligent algorithm, except the initial scheduling algorithm, the algorithm with the highest correlation with the initial scheduling algorithm is searched and used as the candidate scheduling algorithm, the test case of the candidate scheduling algorithm is scheduled, and the initial scheduling algorithm and the candidate scheduling algorithm are used as the combination algorithm of the current scheduling;

[0034] From the algorithms to be scheduled in the multi-intelligent algorithms except the combined algorithm, select the test case corresponding to the single-intelligent algorithm with the highest correlation metric with the currently scheduled combined algorithm for scheduling until the test cases of all algorithms in the multi-intelligent algorithms reach a preset scheduling number, and test the corresponding algorithms to obtain the test results of the collaborative work of the multi-intelligent algorithms.

[0035] In one possible implementation, before scheduling the test cases of each single intelligent algorithm, the test case deployment module is further used to:

[0036] Obtain the remaining resources of the test equipment currently used to schedule the test cases, and verify them with the resource usage required to run the corresponding test cases;

[0037] According to the verification results, the test cases of the multi-intelligent algorithm are adaptively scheduled and the corresponding algorithms are tested.

[0038] In one possible implementation, the use case deployment module is specifically used to:

[0039] If the resource usage required to run the corresponding test case on the test device is greater than the remaining resource, it indicates that the test case does not meet the scheduling condition;

[0040] Adjusting the scheduling order of the test cases in the corresponding algorithm;

[0041] The remaining test cases of the test case in the corresponding algorithm except the test case are traversed in sequence until the resource usage required by the corresponding test case is less than or equal to the remaining resource amount, the corresponding test case is scheduled, and the corresponding algorithm is tested.

[0042] In one possible implementation, after arranging the individual single-intelligent algorithms in the multi-intelligent algorithm into a plurality of test cases for testing the corresponding algorithms according to the preset protocol, the case configuration module is further used to:

[0043] Running each test case of each single intelligent algorithm in the multi-intelligent algorithm on the test device one by one to check the availability of the corresponding test case;

[0044] Delete the test cases with invalid test results, retain the valid test cases, and update the test cases corresponding to each single intelligent algorithm.

[0045] In one possible implementation, the use case configuration module is specifically used to:

[0046] Based on the test requirements of each single intelligent algorithm in the multi-intelligent algorithm, decompose the operation steps for testers to configure the corresponding algorithm during actual testing;

[0047] The operation steps of each single intelligent algorithm are configured according to the preset protocol to generate multiple test cases for the corresponding algorithm; wherein each test case has a single function, and each test case consists of a set of operation steps, and each operation step in the set of operation steps is abstracted as represented by a set {test data parameters, test step parameters}; the test data parameters cover various characteristics related to the video; and the test step parameters are information related to the operation instructions executed for each intelligent algorithm.

[0048] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0049] Memory, used to store computer programs;

[0050] The processor is used to implement the method steps described in any one of the above items when executing the computer program stored in the memory.

[0051] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method steps described in any one of the above items.

[0052] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes: a computer program code, and when the computer program code runs on a computer, the computer executes any of the methods described above.

[0053] The beneficial effects of the embodiments of the present application are as follows:

[0054] The embodiment of the present application provides an automated testing method, device and electronic device based on a multi-intelligent algorithm. First, according to a preset protocol, each single intelligent algorithm in the multi-intelligent algorithm is arranged into multiple test cases for testing the corresponding algorithm; illustratively, the multiple test cases can be two, or three or more, which are not limited here. Then, according to a preset correlation table, the correlation between any two single intelligent algorithms in the multi-intelligent algorithm is determined; wherein the correlation is used to characterize the probability of parallel operation between the corresponding two single intelligent algorithms; the parallel here is to start the test of each intelligent algorithm in sequence according to a certain scheduling strategy within a relatively close time range, and they overlap in time and run approximately at the same time. Exemplarily, the specific value of the correlation reflects the degree of mutual correlation and mutual dependence between the two intelligent algorithms in function. The larger the value, the stronger the mutual dependence between the two intelligent algorithms in function realization, and they are more likely to work together in practical applications. Exemplarily, the higher the correlation between the two single intelligent algorithms, the more likely they are to run in parallel and have a greater impact on each other in practical applications. In actual testing, giving priority to arranging their parallel testing can better discover potential problems, such as data interaction errors, resource competition conflicts, etc.

[0055] Then, according to the correlation, the correlation measure of each single intelligent algorithm in the multi-intelligent algorithm is calculated; wherein, the correlation measure of any single intelligent algorithm is a comprehensive index obtained by comprehensive calculation of the correlation between any single intelligent algorithm and other single intelligent algorithms except itself; illustratively, the correlation measure of any single intelligent algorithm reflects the overall correlation degree of the algorithm in the entire multi-intelligent algorithm system. For example, in an intelligent security system, the target detection algorithm has correlation with the behavior analysis algorithm, face recognition algorithm, alarm triggering algorithm, etc., and its correlation measure is the sum of these correlations. The larger the value of the correlation measure, the more extensive and close the connection between the algorithm and other algorithms in the system, and the higher its influence and importance in the overall multi-intelligent algorithm system, and it plays a more critical role in the realization of the entire system function and collaborative work.

[0056] Then, according to the association metric, the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm is determined, and the test cases of each single intelligent algorithm are scheduled according to the scheduling order to test the corresponding algorithm, and the test results of the multi-intelligent algorithm system are obtained. Exemplarily, the algorithm with higher association metric has higher test priority. In this case, by testing these algorithms and their related combinations first, key problems that may exist in the system can be found more quickly, especially problems in algorithm interaction and collaboration. That is to say, in the embodiment of the present application, the relevant metrics of each single intelligent algorithm can be calculated according to the degree of association between any two single intelligent algorithms in the multi-intelligent algorithm, so as to determine the centrality, and the test cases can be scheduled and the next test case can be deployed accordingly, so as to better meet the needs of the multi-intelligent algorithm combination test, and significantly reduce the redundant combination test cases while ensuring the test coverage. In this way, the test efficiency is improved and the test cost is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of one of the methods of the automated testing method based on a multi-intelligent algorithm provided in an embodiment of the present application;

[0058] Figure 2 for Figure 1 A method flow chart of step S104;

[0059] Figure 3 A flowchart of one of the methods of the automated testing method based on a multi-intelligent algorithm provided in an embodiment of the present application;

[0060] Figure 4 for Figure 3 A method flow chart of step S302;

[0061] Figure 5 A schematic diagram of one of the scheduling methods in the automated testing method based on a multi-intelligent algorithm provided in an embodiment of the present application;

[0062] Figure 6 A flowchart of one of the methods of the automated testing method based on a multi-intelligent algorithm provided in an embodiment of the present application;

[0063] Figure 7 For Figure 1 A flow chart of one of the methods after step S101;

[0064] Figure 8 for Figure 1 A method flow chart of step S101 in FIG.

[0065] Fig. 9 A flowchart of one of the methods of the automated testing method based on a multi-intelligent algorithm provided in an embodiment of the present application;

[0066] Fig.10 A flowchart of one of the methods of the automated testing method based on a multi-intelligent algorithm provided in an embodiment of the present application;

[0067] Fig.11 A schematic diagram of a structure of an automated testing device based on a multi-intelligent algorithm provided in an embodiment of the present application;

[0068] Fig.12 A schematic diagram of a structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0070] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0071] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0072] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0073] Before introducing the automated testing method based on a multi-intelligent algorithm provided in the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail.

[0074] In the related art, the existing testing methods for intelligent algorithms have many disadvantages as shown below:

[0075] 1. Running single intelligent test cases one by one may miss the mutual impact of multiple intelligent functions in parallel, resulting in incomplete testing;

[0076] 2. The parallel testing process of multiple intelligent algorithms is abstracted into one test case, which couples the test case with multiple solutions and is not conducive to reuse;

[0077] 3. When combining single-intelligent test cases to form multi-intelligent test cases, the workload of manual arrangement is huge as the number of single-intelligent test cases increases;

[0078] 4. The existing combination test case generation method only considers the coverage breadth, but does not consider the concurrent correlation of multiple schemes, resulting in a large number of redundant combination test cases;

[0079] In addition, due to the diversity of embedded platforms, different devices have different algorithm requirements and test focuses. Only considering the test case combination with coverage breadth may generate a large number of redundant tests, and may generate a large number of redundant tests on different devices, and may need to re-arrange the test case deployment combination on different platform devices. It can be seen that the existing testing methods have problems such as incomplete testing, serious case coupling, large scheduling workload, many redundant tests, and inability to accurately hit the actual test requirements. Especially for the testing of multiple intelligent algorithms, the overall test efficiency is low.

[0080] In view of this, the present application provides an automated testing method, device and electronic device based on a multi-intelligent algorithm, which are used to improve the efficiency of automated testing of the multi-intelligent algorithm and ensure the reliability and performance of the multi-intelligent algorithm in practical applications.

[0081] like Figure 1 As shown, the embodiment of the present application provides an automated testing method based on a multi-intelligent algorithm, including:

[0082] S101: Arranging, according to a preset protocol, each single intelligent algorithm in the multi-intelligent algorithm into a plurality of test cases for testing the corresponding algorithm;

[0083] In the specific implementation process, each single intelligent algorithm in the multi-intelligent algorithm can be arranged into multiple test cases according to the preset protocol, and each test case in the multiple test cases can test the corresponding algorithm. Exemplarily, the multi-intelligent algorithms can be two, or three or more. The number of multi-intelligent algorithms can be set according to the actual application needs, and is not limited here. Exemplarily, in the field of intelligent security, the multi-intelligent algorithm can be a target detection algorithm and a tracking algorithm, or a behavior analysis and alarm algorithm. Exemplarily, in the field of intelligent transportation, the multi-intelligent algorithm can be a traffic flow statistics and vehicle speed detection algorithm, or a path planning and navigation algorithm. Exemplarily, in the field of intelligent medical care, the multi-intelligent algorithm can be an image feature extraction and disease diagnosis algorithm, or a physiological signal monitoring and early warning algorithm. Of course, the type of each single intelligent algorithm in the multi-intelligent algorithm can be set according to the actual application needs, and is not limited here.

[0084] Moreover, the test case can be a verification tool for the corresponding algorithm. Specifically, the test case can verify the functions of the corresponding algorithm one by one through carefully designed test steps and expected output results. For example, in the test of an image recognition algorithm, the test case will provide a series of images containing different objects as test data parameters, and then execute the test steps of the image recognition function, and then compare the recognition results output by the algorithm with the expected correct results, so as to determine whether the function of the algorithm is correct; if the results obtained after the test case is executed are consistent with the expectations, then it can be preliminarily considered that the algorithm is correct at this function point; otherwise, it indicates that the algorithm may have functional defects and needs further analysis and improvement.

[0085] In addition, test cases can also be a measurement tool for the performance indicators of the corresponding algorithms. Specifically, test cases can measure and evaluate the performance indicators of the corresponding algorithms in actual operation. For example, in a scenario where multiple intelligent algorithms run in parallel, test cases can simulate the actual operating environment and record the resource usage (e.g., memory, CPU usage, etc.) and response time and other performance data of each intelligent algorithm during concurrent execution. For example, in an embedded device test that runs a people counting algorithm and a behavior analysis algorithm at the same time, test cases can accurately measure the resource consumption of each algorithm under different load conditions, as well as their impact on the overall performance of the device, providing quantitative data support for evaluating the performance of intelligent algorithms in actual applications, helping developers understand the performance bottlenecks of algorithms under different circumstances, so as to carry out targeted optimization.

[0086] S102: Determine the correlation between any two single-intelligent algorithms in the multi-intelligent algorithm according to a preset correlation table; wherein the correlation is used to characterize the probability of the corresponding two single-intelligent algorithms running in parallel;

[0087] In the specific implementation process, the correlation between any two single intelligent algorithms in the multi-intelligent algorithm can be determined according to the preset correlation table. Exemplarily, the preset correlation table can be a table that presents the correlation degree values ​​between the multi-intelligent algorithms in the form of a matrix, and some values ​​are pre-set by the user and can be adjusted dynamically. The correlation in the preset correlation table is used to characterize the probability of parallel operation between the corresponding two single intelligent algorithms. Specifically, the rows and columns in the table represent different single intelligent algorithms, and each element in the table Indicates i The algorithm and j For example, in a system including three algorithms, namely, target detection, behavior analysis, and face recognition, the correlation table is shown in Table 1:

[0088]

[0089] Table 1

[0090] In the exemplary embodiment shown in Table 1, the values ​​on the diagonal in the table (such as the correlation between target detection and itself is 1) can be set according to the definition, generally indicating an inherent correlation degree of the algorithm itself, or can be defined as other values ​​or not involved in the actual calculation according to the specific situation; while the values ​​on the non-diagonal in the table reflect the correlation between different algorithms. For example, the correlation between the target detection algorithm and the behavior analysis algorithm is 0.6, indicating that the two algorithms have a certain correlation in terms of function, data interaction, etc. The larger the value, the stronger the correlation, and the higher the probability of the two algorithms running in parallel.

[0091] S103: Calculating the association metric of each single intelligent algorithm in the multi-intelligent algorithm according to the association degree; wherein the association metric of any single intelligent algorithm is a comprehensive index obtained by comprehensive calculation based on the association degree between any single intelligent algorithm and other single intelligent algorithms except itself;

[0092] In the specific implementation process, after determining the correlation between any two single intelligent algorithms in the multi-intelligent algorithm according to the preset correlation table, the correlation metric of each single intelligent algorithm in the multi-intelligent algorithm can be calculated. Moreover, the correlation metric of any single intelligent algorithm in the multi-intelligent algorithm is a comprehensive indicator obtained by comprehensive calculation of the correlation between any single intelligent algorithm and other single intelligent algorithms except itself. Exemplarily, the higher the correlation metric, the more important the function of the single intelligent algorithm, the higher the scheduling priority, and the higher the priority of the test case combination scheduling result, the faster it can reflect the algorithm availability of the embedded device. Exemplarily, it can be according to the formula:

[0093]

[0094] The correlation metric between each single intelligent algorithm and other single intelligent algorithms except itself is calculated, which provides a key basis for determining the scheduling order of test cases.

[0095] S104: Determine the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm according to the association metric, and schedule the test cases of each single intelligent algorithm according to the scheduling order to test the corresponding algorithm, so as to obtain the test result of the collaborative work of the multi-intelligent algorithm.

[0096] In the specific implementation process, the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm can be determined according to the correlation metric. Exemplarily, the algorithm with higher correlation metric has higher test priority. For example, the test case of the algorithm with the highest correlation metric can be scheduled first. Accordingly, the test case of each single intelligent algorithm is scheduled according to the scheduling order to test the corresponding algorithm, so as to improve the test effect of the collaborative work of the multi-intelligent algorithm. In this way, the algorithm with higher correlation metric and its related combination can be tested first, and the key problems that may exist in the system can be found more quickly. In the application embodiment, the correlation metric of each single intelligent algorithm can be calculated according to the correlation between any two single intelligent algorithms in the multi-intelligent algorithm, so as to determine the centrality, and the test case can be scheduled and the next test case can be deployed accordingly, which can better meet the needs of the combination test of the multi-intelligent algorithm, and greatly reduce the redundant combination test cases while ensuring the test coverage. In this way, the test efficiency is improved and the test cost is reduced.

[0097] like Figure 2As shown, step S104: according to the association metric, determining the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm, and scheduling the test cases of each single intelligent algorithm according to the scheduling order to test the corresponding algorithm, and obtaining the test results of the collaborative work of the multi-intelligent algorithm, including:

[0098] S201: Determine a single intelligent algorithm with the highest correlation metric from the multiple intelligent algorithms, prioritize the scheduling of test cases corresponding to the single intelligent algorithm with the highest correlation metric, and use the single intelligent algorithm with the highest correlation metric as the initial scheduling algorithm;

[0099] S202: According to the preset correlation table, from the multi-intelligent algorithms other than the initial scheduling algorithm, search for an algorithm with the highest correlation with the initial scheduling algorithm and use it as a candidate scheduling algorithm, schedule test cases of the candidate scheduling algorithm, and use the initial scheduling algorithm and the candidate scheduling algorithm as a combination algorithm for current scheduling;

[0100] S203: From the algorithms to be scheduled in the multi-intelligent algorithms except the combined algorithm, select the test case corresponding to the single intelligent algorithm with the highest correlation metric with the currently scheduled combined algorithm for scheduling, until the test cases of all algorithms in the multi-intelligent algorithms reach the preset scheduling times, and test the corresponding algorithms to obtain the test results of the collaborative work of the multi-intelligent algorithms.

[0101] In the specific implementation process, the specific implementation process of step S201 to step S203 is as follows:

[0102] First, determine the single intelligent algorithm with the highest correlation metric from the multiple intelligent algorithms, and prioritize the scheduling of the test cases corresponding to the single intelligent algorithm with the highest correlation metric, and use the single intelligent algorithm with the highest correlation metric as the initial scheduling algorithm. Assuming that there are three intelligent algorithms to be deployed, namely, target detection, behavior analysis, and face recognition, the correlation table shown in Table 1 is still used as an example, where the correlation metric for target detection is (1+0.6+0.5=2.1); the correlation metric for behavior analysis is (0.6+1+0.7=2.3); and the correlation metric for face recognition is (0.5+0.7+1=2.2). After sorting the algorithms according to the size of the correlation metric, the test case of the behavior analysis algorithm with the highest correlation metric can be prioritized, and the test case can be any of the multiple test cases corresponding to the behavior analysis algorithm.

[0103] Then, according to the preset correlation table, from the other algorithms of the multi-intelligent algorithm except the initial scheduling algorithm, search for the algorithm with the highest correlation with the initial scheduling algorithm, and use it as the scheduling algorithm to be selected. Still taking the correlation table shown in Table 1 as an example, from the target detection algorithm and the face recognition algorithm, search for the face recognition algorithm with the highest correlation with the behavior analysis algorithm as the scheduling algorithm to be selected. Then, schedule the test case of the scheduling algorithm to be selected, and use the initial scheduling algorithm and the scheduling algorithm to be selected as the combination algorithm currently scheduled. In addition, the test case corresponding to the single intelligent algorithm with the highest correlation metric with the currently scheduled combination algorithm can be selected from the algorithms to be scheduled in the multi-intelligent algorithm except the combination algorithm for scheduling, until the test cases of all algorithms in the multi-intelligent algorithm reach the preset scheduling times, and test the corresponding algorithms, so as to obtain the test results of the multi-intelligent algorithm system. That is to say, in the embodiment of the present application, according to the preset correlation table, the next algorithm with the highest correlation with the current scheduling method can be searched as the scheduling algorithm to be selected; the test case of the algorithm with the highest correlation metric is selected from the scheduling algorithm to be selected for scheduling. In practical applications, the next candidate scheduling algorithm can be continuously searched according to the algorithm association table, and the above process can be repeated until the test cases of all algorithms are completed and the preset scheduling times are reached. Specifically, each test case can be scheduled cyclically for a preset number of times. After each scheduling is completed, if the test case has been scheduled for a preset number of times, it can be deleted from the case set corresponding to the algorithm, thereby avoiding the problem of redundant scheduling and improving test efficiency.

[0104] In the embodiments of the present application, Figure 3 As shown, before scheduling the test cases of each single intelligent algorithm, the method further includes:

[0105] S301: Obtain the remaining amount of resources of the test equipment currently used to schedule the test case, and perform a judgment and verification with the resource usage required to run the corresponding test case;

[0106] S302: According to the verification result, adaptively schedule the test cases of the multi-intelligent algorithm and test the corresponding algorithm.

[0107] In the specific implementation process, the specific implementation process of step S301 to step S302 is as follows:

[0108] Before scheduling a test case, obtain the remaining amount of resources of the test device currently used to schedule the test case, wherein the test device includes multiple types of resources, such as OS memory, mmz memory, CPU, etc. The remaining amount of resources may be the remaining amount of resources including various resource types. The remaining amount of resources may be judged and verified with the resource usage required to run the corresponding test case. Exemplarily, for the same type of resources, the remaining amount of resources of the current test device may be obtained and judged and verified with the resource usage required to run the test case.

[0109] Then, according to the verification results, the test cases of the multi-intelligent algorithm are adaptively scheduled and the corresponding algorithms are tested. Exemplarily, the real-time remaining amount of resource types R1, ..., Rn of the test equipment is compared with the resource usage r1, ..., rn required to run the test case. Specifically, the remaining amount of resources of the same resource type is compared with the corresponding resource usage; if the formula is met, , it indicates that the test case meets the scheduling conditions; otherwise, it indicates that the scheduling conditions are not met, and the scheduling order of the test case needs to be moved down to continue checking the next test case in the same algorithm.

[0110] Exemplarily, during the scheduling process, the remaining amount of resources of the test device is continuously monitored. For example, when a test case with high resource requirements (such as a high-definition video real-time processing algorithm test case) is scheduled, if the remaining amount of OS memory of the test device is less than the amount of OS memory required by the test case (such as 200MB OS memory is required, and the device has 150MB remaining), the scheduling order of the test case is moved down according to the scheduling rules, and the next test case in the same algorithm is checked to see whether it meets the scheduling conditions. For example, each test case can be scheduled cyclically for a preset number of tests for 3 times. After each scheduling is completed, if the test case reaches the preset number of runs (such as 2 successful runs), the test case is deleted from the test case set of the algorithm. If all test cases in the entire algorithm do not meet the scheduling conditions, and traversing all test cases still does not meet them, wait for the running cases to end, and schedule the test cases in the algorithm to be scheduled again after the resources are met. If the scheduling of the test cases in a single intelligent algorithm is completed, the algorithm is deleted from the algorithm to be tested. This cycle is repeated until the test cases of all algorithms have reached the preset scheduling times, ensuring the comprehensiveness and effectiveness of the test, and accurately evaluating the collaborative performance of multi-intelligent algorithms in actual application scenarios.

[0111] In the embodiments of the present application, Figure 4 As shown, step S302: according to the verification result, adaptively schedule the test cases of the multi-intelligent algorithm and test the corresponding algorithm, including:

[0112] S401: If the resource usage required to run the corresponding test case on the test device is greater than the remaining resource, it indicates that the test case does not meet the scheduling condition;

[0113] S402: Adjust the scheduling order of the test cases in the corresponding algorithm;

[0114] S403: traverse the remaining test cases of the test case in the corresponding algorithm except the test case in turn until the resource usage required by the corresponding test case is less than or equal to the remaining resource amount, schedule the corresponding test case, and test the corresponding algorithm.

[0115] In the specific implementation process, the specific implementation process of step S401 to step S403 is as follows:

[0116] If the resource usage required to run the corresponding test case on the test device is greater than the remaining resource, for example, if the formula is satisfied: , it indicates that the test case does not meet the scheduling conditions. In practical applications, the scheduling order of the test case in the algorithm can be moved down. Then, the remaining test cases except the test case in the corresponding algorithm are traversed in sequence until the resource usage required by the corresponding test case is less than or equal to the remaining resource amount, the corresponding test case is scheduled, and the corresponding algorithm is tested.

[0117] Combine the following Figure 5 The scheduling diagram shown in the figure explains in detail the specific implementation process of test case scheduling based on the correlation measurement between algorithms.

[0118] The order of multiple intelligent algorithms from high to low according to the correlation metric is S1, S2, ..., Sn. Among them, the test cases corresponding to the intelligent algorithm S1 are F1, F2, F3, ..., Fm, the test cases corresponding to the intelligent algorithm S2 are F1, F2, F3, ..., Fm, ..., and the test cases corresponding to the intelligent algorithm Sn are F1, F2, F3, ..., Fm, where n and m are hyperparameters. After sorting the multiple intelligent algorithms according to the correlation metric, the F1 test case in S1 is first scheduled; then, according to the correlation between S1 and other algorithms, the F1 test case in S2 with the highest correlation metric and correlation is selected in S2, ..., Sn for scheduling; then, according to the correlation metric and correlation between the S1 and S2 algorithms and other algorithms, the remaining algorithms are selected for scheduling in turn. When the equipment resources do not meet the test case scheduling requirements, the other test cases in the algorithm are traversed in turn for testing. If none of them are satisfied, it waits. Until each test case is scheduled and loaded to its preset number of times. Among them, the test cases in the next algorithm of each scheduling are combined with the test cases with the highest correlation with the algorithm in the current scheduling, ensuring the accurate hit test requirements of multi-algorithm test cases. Using this method to complete the test in sequence, the number of test cases will not expand, but gradually converge to the end.

[0119] It should be noted that if the test cases in the entire algorithm do not meet the scheduling conditions, the running test cases will be waited for to end, and the test cases in the candidate scheduling algorithm will be scheduled again after the resources are met. In this way, when deploying the combined multi-algorithm test tasks, the resource usage required by the test cases of a single intelligent algorithm is used as a threshold to verify the resources of the test equipment. The maximum number of parallelism can be adaptively adjusted according to the remaining resources of the test equipment, the number of algorithms deployed for testing can be controlled, and multi-algorithm parallel automated testing can be realized. Regardless of whether the test device is a server or an embedded device, it can reasonably arrange test tasks according to its own resource status, thereby improving the stability and reliability of the test, ensuring that the test process is not affected by resource limitations, effectively avoiding test failures or inaccurate results due to insufficient resources, and improving test efficiency.

[0120] After obtaining the test case set of each single intelligent algorithm in the multi-intelligent algorithm, the following is combined Figure 6 The method flow chart shown provides a detailed explanation of the specific scheduling process for the test case in the embodiment of the present application.

[0121] First, determine the single intelligent algorithm with the highest correlation metric from the multi-intelligent algorithms, and prioritize the test cases corresponding to the single intelligent algorithm with the highest correlation metric. Exemplarily, prioritize the first test case in the single intelligent algorithm with the highest correlation metric. Specifically, based on the correlation table of the algorithm functions to be deployed, calculate the correlation metric of each single intelligent algorithm in the multi-intelligent algorithm with other algorithms except itself according to the following formula:

[0122]

[0123] in, represents the association metric of the i-th single intelligent algorithm in the multi-intelligent algorithm, It indicates the correlation between the i-th single intelligent algorithm and the j-th single intelligent algorithm in the multi-intelligent algorithm. The higher the correlation metric is, the more important the function of the single intelligent algorithm is and the higher the scheduling priority is. The scheduling result of the test case combination with a higher priority can quickly reflect the availability of the algorithm carried by the test equipment.

[0124] Then, execute S1: obtain the association table;

[0125] S2: Determine the scheduling algorithm to be selected;

[0126] Specifically, the scheduling algorithm to be selected is determined according to the following steps:

[0127] S21: Obtain x algorithms in the current schedule; illustratively, x may be one, or may be two or more, and the value thereof changes dynamically.

[0128] S22: Obtain other algorithms to be scheduled; specifically, obtain other algorithms to be scheduled except the x algorithms currently scheduled;

[0129] S23: Calculate the correlation metric between each of the other algorithms to be scheduled and the x algorithms in the current scheduling;

[0130] S24: Select the algorithm with the highest correlation metric as the candidate scheduling algorithm (i.e., the next algorithm to be scheduled). Specifically, calculate the correlation metric of each algorithm in the other algorithms to be scheduled and the x algorithms in the current scheduling according to the following formula:

[0131]

[0132] S3: Select the first test case from the candidate scheduling algorithms for scheduling;

[0133] S4: Extract resource requirements r in test cases;

[0134] S5: extract the remaining resources R of the test equipment;

[0135] S6: Determine whether r is less than R;

[0136] If yes, execute S7: schedule the test case;

[0137] If not, execute S8: determine whether the test case is the last test case in the candidate scheduling algorithm; if so, execute S9: wait for resources; until the resources are sufficient, schedule the test case again. If not, execute S10: schedule the next test case in the candidate scheduling algorithm except the test case. Accordingly, extract the resource requirements of the next test case to determine whether the scheduling conditions are met. In this way, the traversal and judgment verification of all test cases in the candidate scheduling algorithm are realized.

[0138] After each scheduling of the test case is completed, S11 is executed: determining whether the test case has reached a preset number of runs;

[0139] If yes, then execute S12: delete the test case from the set of test cases of the algorithm to be scheduled;

[0140] If not, continue to execute S2.

[0141] In the embodiments of the present application, Figure 7 As shown, in step S101: after arranging the individual single-intelligent algorithms in the multi-intelligent algorithms into a plurality of test cases for testing the corresponding algorithms according to a preset protocol, the method further comprises:

[0142] S701: running each test case of each single intelligent algorithm in the multi-intelligent algorithm on the test device one by one to check the availability of the corresponding test case;

[0143] S702: Delete the test cases with invalid test results, retain the valid test cases, and update the test cases corresponding to each single intelligent algorithm.

[0144] In the specific implementation process, the specific implementation process of step S701 to step S702 is as follows:

[0145] First, on the test equipment, run each test case of each single intelligent algorithm in the multi-intelligent algorithm one by one to verify the availability of the corresponding test case.

[0146] Then, delete the test cases with invalid test results among the multiple test cases of the same single intelligent algorithm. Exemplarily, delete the test cases with failed test results or unsuccessful runs. Then, retain the valid test cases and update the test cases corresponding to each single intelligent algorithm. In this way, the updated test cases of each single intelligent algorithm can be scheduled later. In this way, the availability of the algorithm installed on the embedded device is guaranteed.

[0147] In addition, in the process of running each test case of a single intelligent algorithm, the total amount of resources occupied by the corresponding test case can also be recorded. Among them, the total amount of resources is the sum of the amount occupied by each resource type during the execution (or operation) of the test case. In this way, the resource usage of the corresponding test case is recorded; in this way, when performing multi-intelligent algorithm testing, the test case can be tested in parallel according to the recorded resource usage and the current resource remaining amount of the test device; wherein, the scheduling order of the test case does not need to be manually specified, and the existing test cases are adaptively arranged for testing. Exemplarily, for a test case of a single intelligent algorithm, its total amount of resources includes the os memory size occupied when the test case is executed (in bytes, for example, 100MB of os memory is occupied), mmz memory size (for example, 50MB of mmz memory is occupied) and other resources that may be involved, such as the average value of CPU usage in a certain period of time, such as an average of 30% of CPU resources. The total amount of resources is a comprehensive indicator that can describe the demand and occupancy of system resources by test cases from multiple dimensions. By counting the occupancy of different resource types, we can fully understand the overall impact of a single intelligent algorithm on system resources during runtime.

[0148] In the embodiments of the present application, Figure 8 As shown, step S101: according to a preset protocol, each single intelligent algorithm in the multi-intelligent algorithm is arranged into a plurality of test cases for testing the corresponding algorithm, including:

[0149] S801: Based on the test requirements for each single intelligent algorithm in the multi-intelligent algorithm, decompose the operation steps for the tester to configure the corresponding algorithm during the actual test;

[0150] S802: Configure the operation steps of each single intelligent algorithm according to the preset protocol to generate multiple test cases for the corresponding algorithm; wherein each test case has a single function, and each test case consists of a set of operation steps, and each operation step in the operation step set is abstracted as represented by a set {test data parameters, test step parameters}; the test data parameters cover various characteristics related to the video; and the test step parameters are information related to the operation instructions executed for each intelligent algorithm.

[0151] In the specific implementation process, the specific implementation process of step S801 to step S802 is as follows:

[0152] First, based on the test requirements of each single intelligent algorithm in the multi-intelligent algorithm, the operation steps of the tester to configure the corresponding algorithm during the actual test are decomposed; then, the operation steps of each single intelligent algorithm are configured according to the preset protocol to generate multiple test cases of the corresponding algorithm; exemplary, it can be to first build a single intelligent algorithm test case, and then perform the test deployment of the multi-intelligent algorithm according to the test requirements of the multi-intelligent algorithm. Specifically, firstly, based on multiple requirements for the single intelligent algorithm test, multiple test cases are built; then, based on the test requirements of the multi-intelligent algorithm, a case set for the multi-intelligent algorithm test is built; then, based on the test case set, a multi-algorithm parallel scheduling test is performed. Among them, each test case has a single function, and each test case consists of an operation step set, and each operation step in the operation step set is abstracted as represented by a set {test data parameter, test step parameter}. Exemplarily, each operation step can be abstracted as a cmd expression, and each cmd is expressed by a parameter set {test data parameter, test step parameter}, etc. For example, {video, 1.avi}, {rule, rule.json}. All operation step cmd sets constitute a test case for a single intelligent algorithm. That is to say, a single test case only describes the test steps of a single intelligent algorithm and is not coupled with other algorithms. When testing multiple intelligent algorithms, single test cases are used to combine and construct test cases for multiple intelligent algorithms, which increases the reusability of single test cases.

[0153] Combine the following Fig. 9 The method flow chart shown provides a detailed explanation of the process of constructing test cases for a single intelligent algorithm.

[0154] S20: Obtain the test requirements of the single-intelligence algorithm; for example, the tester deeply analyzes the functional characteristics and test requirements of the single-intelligence algorithm, and formulates detailed test calculations for different types of algorithms (e.g., target detection, behavior analysis, image recognition). For example, taking the target detection algorithm as an example, the test requirements include the accuracy of detecting different types of targets (e.g., people, vehicles, objects, etc.), performance under different lighting conditions and video resolutions, etc.

[0155] S30: Design complete operation steps; specifically, according to the test requirements, decompose the operation steps that the tester needs to configure the single intelligent algorithm during the actual test; for example, select a suitable test video set (including videos of different scenes and target densities), set detection thresholds, determine target categories, and other operation steps.

[0156] S40: split and abstract the operation steps according to the preset protocol; specifically, abstract each operation step into a cmd expression, and each cmd is expressed by a parameter set {test data parameters, test step parameters}, etc. Exemplarily, the test data parameters can be parameters such as the resolution, frame rate, sensitivity, and scene content of the test video (for example, including different numbers and types of targets, different lighting conditions, and backgrounds of varying complexity); the test step parameters can be parameters such as starting target detection and setting detection thresholds. Of course, other parameters can also be used, which are not limited here.

[0157] S50: Generate test cases; illustratively, all cmd sets constitute multiple test cases for a single intelligent algorithm. Similarly, for other single intelligent algorithms, such as behavior analysis algorithms, corresponding test cases can be constructed according to the needs of analyzing behavior types (e.g., walking, running, gathering, etc.), providing diversified test components for subsequent multi-algorithm testing.

[0158] Combine the following Fig.10 The method flow chart shown provides a detailed explanation of the process of constructing test cases for multi-intelligent algorithms.

[0159] S60: Obtain test requirements for multi-intelligent algorithms;

[0160] S70: Obtain a test case set for each single intelligent algorithm in the multiple intelligent algorithms; specifically, extract the test cases for each single intelligent algorithm in the multiple intelligent algorithms to form a test case set. Exemplarily, for the comprehensive test requirements of the multiple intelligent algorithms, relevant test cases are selected from the constructed test cases for each single intelligent algorithm to form a test case set.

[0161] S80: Run the test cases of each single intelligent algorithm in sequence; illustratively, the test cases of the single intelligent algorithms in the set are embedded in the test equipment (eg, intelligent security monitoring equipment, vehicle-mounted intelligent system, etc.) in sequence for execution.

[0162] S90: Obtain operating resource parameters; specifically, during the operation, the test device monitors and records in real time the total amount of resources occupied by each test case in the test device. For example, OS memory usage, mmz memory space occupied, etc. Among them, resource types are divided into multiple situations according to different embedded device chips, generally including OS memory, mmz memory and other device resource parameters that can limit the operation of the algorithm, and r1, ..., rn are used to represent each resource, and recorded in the corresponding test case. For example, when a single test case of an image recognition algorithm occupies 100MB of OS memory and 50MB of mmz memory during operation, these resource occupancy values ​​are recorded in the information of the test case.

[0163] S91: judging the running result of the test case;

[0164] If the operation result fails, such as algorithm crash, serious deviation of detection result or inability to complete the scheduled task, execute S92: delete the test case with failed operation result from the case set of the corresponding algorithm; in this way, the effectiveness of subsequent multi-algorithm testing is ensured.

[0165] If the running result is successful, then execute S93: fill the running resource parameters into the current test case;

[0166] S94: Obtain a set of test cases corresponding to the updated algorithm.

[0167] Based on the same inventive concept, Fig.11 As shown, the embodiment of the present application also provides an automated testing device based on a multi-intelligent algorithm, including:

[0168] A case configuration module 10 is used to arrange each single intelligent algorithm in the multi-intelligent algorithm into a plurality of test cases for testing the corresponding algorithm according to a preset protocol;

[0169] The use case deployment module 20 is used to determine the correlation between any two single intelligent algorithms in the multi-intelligent algorithm according to a preset correlation table; wherein the correlation is used to characterize the probability of parallel operation between the corresponding two single intelligent algorithms; according to the correlation, the correlation metric of each single intelligent algorithm in the multi-intelligent algorithm is calculated; wherein the correlation metric of any single intelligent algorithm is a comprehensive index calculated based on the correlation between any single intelligent algorithm and other single intelligent algorithms except itself;

[0170] The use case scheduling module 30 is used to determine the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm according to the association metric, and schedule the test cases of each single intelligent algorithm according to the scheduling order to test the corresponding algorithm, so as to obtain the test results of the collaborative work of the multi-intelligent algorithm.

[0171] In one possible implementation, the use case deployment module 20 is used to:

[0172] Determine a single intelligent algorithm with the highest correlation metric from the multiple intelligent algorithms, prioritize the scheduling of test cases corresponding to the single intelligent algorithm with the highest correlation metric, and use the single intelligent algorithm with the highest correlation metric as the initial scheduling algorithm;

[0173] According to the preset correlation table, from the multi-intelligent algorithm, except the initial scheduling algorithm, the algorithm with the highest correlation with the initial scheduling algorithm is searched and used as the candidate scheduling algorithm, the test case of the candidate scheduling algorithm is scheduled, and the initial scheduling algorithm and the candidate scheduling algorithm are used as the combination algorithm of the current scheduling;

[0174] From the algorithms to be scheduled in the multi-intelligent algorithms except the combined algorithm, select the test case corresponding to the single-intelligent algorithm with the highest correlation metric with the currently scheduled combined algorithm for scheduling until the test cases of all algorithms in the multi-intelligent algorithms reach a preset scheduling number, and test the corresponding algorithms to obtain the test results of the collaborative work of the multi-intelligent algorithms.

[0175] In one possible implementation, before scheduling the test cases of each single intelligent algorithm, the test case deployment module 20 is further used to:

[0176] Obtain the remaining resources of the test equipment currently used to schedule the test cases, and verify them with the resource usage required to run the corresponding test cases;

[0177] According to the verification results, the test cases of the multi-intelligent algorithm are adaptively scheduled and the corresponding algorithms are tested.

[0178] In one possible implementation, the use case deployment module 20 is specifically used to:

[0179] If the resource usage required to run the corresponding test case on the test device is greater than the remaining resource, it indicates that the test case does not meet the scheduling condition;

[0180] Adjusting the scheduling order of the test cases in the corresponding algorithm;

[0181] The remaining test cases of the test case in the corresponding algorithm except the test case are traversed in sequence until the resource usage required by the corresponding test case is less than or equal to the remaining resource amount, the corresponding test case is scheduled, and the corresponding algorithm is tested.

[0182] In one possible implementation, after arranging the individual single-intelligent algorithms in the multi-intelligent algorithms into a plurality of test cases for testing the corresponding algorithms according to the preset protocol, the case configuration module 10 is further used to:

[0183] Running each test case of each single intelligent algorithm in the multi-intelligent algorithm on the test device one by one to check the availability of the corresponding test case;

[0184] Delete the test cases with invalid test results, retain the valid test cases, and update the test cases corresponding to each single intelligent algorithm.

[0185] In one possible implementation, the use case configuration module 10 is specifically used to:

[0186] Based on the test requirements of each single intelligent algorithm in the multi-intelligent algorithm, decompose the operation steps for testers to configure the corresponding algorithm during actual testing;

[0187] The operation steps of each single intelligent algorithm are configured according to the preset protocol to generate multiple test cases for the corresponding algorithm; wherein each test case has a single function, and each test case consists of a set of operation steps, and each operation step in the set of operation steps is abstracted as represented by a set {test data parameters, test step parameters}; the test data parameters cover various characteristics related to the video; and the test step parameters are information related to the operation instructions executed for each intelligent algorithm.

[0188] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, and the electronic device can realize the functions of the aforementioned automatic testing device based on multi-intelligent algorithm, referring to Fig.12 , the electronic device comprises:

[0189] Memory 100, used for storing computer programs;

[0190] The processor 200 is used to implement the steps of the automatic testing method based on multi-intelligence algorithm as described in any of the above items when executing the computer program stored in the memory.

[0191] The specific connection medium between the processor 200 and the memory 100 is not limited in the embodiment of the present application. Fig.12 In the example, the processor 200 and the memory 100 are connected via a bus 300. The bus 300 is connected to the memory 100 via a bus 300. Fig.12 The bus 300 is represented by a bold line, and the connection between other components is only for schematic illustration and is not intended to be limiting. The bus 300 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the figure, but it does not mean that there is only one bus 300 or one type of bus 300. Alternatively, the processor 200 may also be called a controller, and there is no limitation on the name.

[0192] In the embodiment of the present application, the memory 100 stores instructions that can be executed by the processor 200. When the processor 200 executes the computer program stored in the memory 100, the automated testing method based on the multi-intelligence algorithm discussed above can be executed. The processor 200 can implement Fig.12The functions of each module are shown in the figure.

[0193] Among them, the processor 200 is the control center of the device, and can use various interfaces and lines to connect various parts of the entire control device, and through running or executing computer programs stored in the memory 100 and calling data stored in the memory 100, the various functions of the device and process data, thereby monitoring the device as a whole.

[0194] In one possible design, the processor 200 may include one or more processing units, and the processor 200 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. In some embodiments, the processor 200 and the memory 100 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0195] The memory 100, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 100 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. Alternatively, the memory 100 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 100 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0196] By designing and programming the processor, the code corresponding to the automated testing method based on the multi-intelligent algorithm described in the above embodiment can be fixed into the chip, so that the chip can execute the steps of the automated testing method based on the multi-intelligent algorithm discussed above when running. How to design and program the processor is a technology well known to those skilled in the art and will not be described in detail here.

[0197] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is used to enable a computer to execute the steps of the automated testing method based on the multi-intelligent algorithm discussed above.

[0198] In some possible implementations, various aspects of the automated testing method based on multi-intelligent algorithms provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a device, the program code is used to enable the control device to execute the steps of the automated testing method based on multi-intelligent algorithms according to various exemplary implementations of the present application described above in this specification.

[0199] The embodiment of the present application provides an automated testing method, device and electronic device based on a multi-intelligent algorithm. First, according to a preset protocol, each single intelligent algorithm in the multi-intelligent algorithm is arranged into multiple test cases for testing the corresponding algorithm; illustratively, the multiple test cases can be two, or three or more, which are not limited here. Then, according to a preset correlation table, the correlation between any two single intelligent algorithms in the multi-intelligent algorithm is determined; wherein the correlation is used to characterize the probability of parallel operation between the corresponding two single intelligent algorithms; the parallel here is to start the test of each intelligent algorithm in sequence according to a certain scheduling strategy within a relatively close time range, and they overlap in time and run approximately at the same time. Exemplarily, the specific value of the correlation reflects the degree of mutual correlation and mutual dependence between the two intelligent algorithms in function. The larger the value, the stronger the mutual dependence between the two intelligent algorithms in function realization, and they are more likely to work together in practical applications. Exemplarily, the higher the correlation between the two single intelligent algorithms, the more likely they are to run in parallel and have a greater impact on each other in practical applications. In actual testing, giving priority to arranging their parallel testing can better discover potential problems, such as data interaction errors, resource competition conflicts, etc.

[0200] Then, according to the correlation, the correlation measure of each single intelligent algorithm in the multi-intelligent algorithm is calculated; wherein, the correlation measure of any single intelligent algorithm is a comprehensive index obtained by comprehensively calculating the correlation between any single intelligent algorithm and other single intelligent algorithms except itself; illustratively, the correlation measure of any single intelligent algorithm reflects the overall correlation degree of the algorithm in the entire multi-intelligent algorithm system. For example, in an intelligent security system, the target detection algorithm has correlation with the behavior analysis algorithm, face recognition algorithm, alarm triggering algorithm, etc., and its correlation measure is the combination of these correlations. The larger the value of the correlation measure, the more extensive and close the connection between the algorithm and other algorithms in the system, and the higher its influence and importance in the overall multi-intelligent algorithm system, and it plays a more critical role in the realization of the entire system function and collaborative work.

[0201] Then, according to the association metric, the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm is determined, and the test cases of each single intelligent algorithm are scheduled according to the scheduling order to test the corresponding algorithm, and the test results of the multi-intelligent algorithm system are obtained. Exemplarily, the algorithm with higher association metric has higher test priority. In this case, by testing these algorithms and their related combinations first, key problems that may exist in the system can be found more quickly, especially problems in algorithm interaction and collaboration. That is to say, in the embodiment of the present application, the relevant metrics of each single intelligent algorithm can be calculated according to the degree of association between any two single intelligent algorithms in the multi-intelligent algorithm, so as to determine the centrality, and the test cases can be scheduled and the next test case can be deployed accordingly, so as to better meet the needs of the multi-intelligent algorithm combination test, and significantly reduce the redundant combination test cases while ensuring the test coverage. In this way, the test efficiency is improved and the test cost is reduced.

[0202] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0203] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An automated testing method based on a multi-intelligent algorithm, characterized in that: include: According to a preset protocol, each single intelligent algorithm in the multi-intelligent algorithm is arranged into a plurality of test cases for testing the corresponding algorithm; Determine the correlation between any two single-intelligent algorithms in the multi-intelligent algorithm according to a preset correlation table; wherein the correlation is used to characterize the probability of parallel operation between the corresponding two single-intelligent algorithms; According to the correlation degree, calculating the correlation metric of each single intelligent algorithm in the multi-intelligent algorithm; wherein the correlation metric of any single intelligent algorithm is a comprehensive index obtained by comprehensive calculation based on the correlation degree between any single intelligent algorithm and other single intelligent algorithms except itself; According to the association metric, determine the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm, and schedule the test cases of each single intelligent algorithm according to the scheduling order to test the corresponding algorithm, so as to obtain the test result of the collaborative work of the multi-intelligent algorithm; Among them, the step of determining the scheduling order of collaborative work of each single intelligent algorithm in the multi-intelligent algorithm according to the association metric, and scheduling the test cases of each single intelligent algorithm according to the scheduling order to test the corresponding algorithm, and obtaining the test results of the collaborative work of the multi-intelligent algorithm, includes: Determine a single intelligent algorithm with the highest correlation metric from the multiple intelligent algorithms, prioritize the scheduling of test cases corresponding to the single intelligent algorithm with the highest correlation metric, and use the single intelligent algorithm with the highest correlation metric as the initial scheduling algorithm; According to the preset correlation table, from the multi-intelligent algorithm, except the initial scheduling algorithm, the algorithm with the highest correlation with the initial scheduling algorithm is searched and used as the candidate scheduling algorithm, the test case of the candidate scheduling algorithm is scheduled, and the initial scheduling algorithm and the candidate scheduling algorithm are used as the combination algorithm of the current scheduling; From the algorithms to be scheduled in the multi-intelligent algorithms except the combined algorithm, select the test case corresponding to the single-intelligent algorithm with the highest correlation metric with the currently scheduled combined algorithm for scheduling until the test cases of all algorithms in the multi-intelligent algorithms reach a preset scheduling number, and test the corresponding algorithms to obtain the test results of the collaborative work of the multi-intelligent algorithms.

2. The method according to claim 1, characterized in that Before scheduling the test cases of each single intelligent algorithm, the method further includes: Obtain the remaining resources of the test equipment currently used to schedule the test cases, and verify them with the resource usage required to run the corresponding test cases; According to the verification results, the test cases of the multi-intelligent algorithm are adaptively scheduled and the corresponding algorithms are tested.

3. The method according to claim 2, characterized in that According to the verification result, adaptively scheduling the test cases of the multi-intelligent algorithm and testing the corresponding algorithm, including: If the resource usage required to run the corresponding test case on the test device is greater than the remaining resource, it indicates that the test case does not meet the scheduling condition; Adjusting the scheduling order of the test cases in the corresponding algorithm; The remaining test cases of the test case in the corresponding algorithm except the test case are traversed in sequence until the resource usage required by the corresponding test case is less than or equal to the remaining resource amount, the corresponding test case is scheduled, and the corresponding algorithm is tested.

4. The method according to claim 3, characterized in that After arranging the individual single-intelligent algorithms in the multi-intelligent algorithms into a plurality of test cases for testing the corresponding algorithms according to the preset protocol, the method further comprises: Running each test case of each single intelligent algorithm in the multi-intelligent algorithm on the test device one by one to check the availability of the corresponding test case; Delete the test cases with invalid test results, retain the valid test cases, and update the test cases corresponding to each single intelligent algorithm.

5. The method according to claim 1, characterized in that According to the preset protocol, each single intelligent algorithm in the multi-intelligent algorithm is arranged into multiple test cases for testing the corresponding algorithm, including: Based on the test requirements of each single intelligent algorithm in the multi-intelligent algorithm, decompose the operation steps for testers to configure the corresponding algorithm during actual testing; The operation steps of each single intelligent algorithm are configured according to the preset protocol to generate multiple test cases for the corresponding algorithm; wherein each test case has a single function, and each test case consists of a set of operation steps, and each operation step in the set of operation steps is abstracted as represented by a set {test data parameters, test step parameters}; the test data parameters cover various characteristics related to the video; and the test step parameters are information related to the operation instructions executed for each intelligent algorithm.

6. An automated testing device based on a multi-intelligent algorithm, characterized in that: include: A use case configuration module, used to arrange each single intelligent algorithm in the multi-intelligent algorithm into multiple test cases for testing the corresponding algorithm according to a preset protocol; A use case deployment module, used to determine the correlation between any two single intelligent algorithms in the multi-intelligent algorithm according to a preset correlation table; wherein the correlation is used to characterize the probability of parallel operation between the corresponding two single intelligent algorithms; according to the correlation, calculate the correlation metric of each single intelligent algorithm in the multi-intelligent algorithm; wherein the correlation metric of any single intelligent algorithm is a comprehensive index calculated based on the correlation between any single intelligent algorithm and other single intelligent algorithms except itself; A use case scheduling module is used to determine the scheduling order of the collaborative work of each single intelligent algorithm in the multi-intelligent algorithm according to the association metric, and schedule the test cases of each single intelligent algorithm according to the scheduling order to test the corresponding algorithm, so as to obtain the test results of the collaborative work of the multi-intelligent algorithm; Wherein, the use case scheduling module is specifically used for: Determine a single intelligent algorithm with the highest correlation metric from the multiple intelligent algorithms, prioritize the scheduling of test cases corresponding to the single intelligent algorithm with the highest correlation metric, and use the single intelligent algorithm with the highest correlation metric as the initial scheduling algorithm; According to the preset correlation table, from the multi-intelligent algorithm, except the initial scheduling algorithm, the algorithm with the highest correlation with the initial scheduling algorithm is searched and used as the candidate scheduling algorithm, the test case of the candidate scheduling algorithm is scheduled, and the initial scheduling algorithm and the candidate scheduling algorithm are used as the combination algorithm of the current scheduling; From the algorithms to be scheduled in the multi-intelligent algorithms except the combined algorithm, select the test case corresponding to the single-intelligent algorithm with the highest correlation metric with the currently scheduled combined algorithm for scheduling until the test cases of all algorithms in the multi-intelligent algorithms reach a preset scheduling number, and test the corresponding algorithms to obtain the test results of the collaborative work of the multi-intelligent algorithms.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 5 when executing the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images