A screening method, device and testing platform for algorithm models

Abstract

The present invention discloses a method, device and testing platform for screening an algorithm model, and specifically relates to the field of detection technology. The method comprises obtaining a scene segmentation data set; setting test parameters of the algorithm model; executing a preset test process based on the scene segmentation data set and the test parameters to obtain a test result of the algorithm model; and evaluating and screening the algorithm model according to the test result of the algorithm model. By obtaining a scene segmentation data set and setting parameters for the algorithm model, the algorithm model is tested according to the preset test process, and then the tested algorithm model is evaluated and screened according to the test results, thereby realizing rapid screening of the most comprehensive algorithm model.

Description

Technical Field

[0001] The present invention relates to the field of detection technology, and in particular to a screening method, device and testing platform for an algorithm model. Background Art

[0002] In the prior art, after training the data using deep learning or classification algorithms, a large number of data models will be generated, and in order to be able to screen out the best model for production. For this reason, the method used in the prior art is to execute the test script in the server environment, and based on the GPU or CPU environment of the server, perform a re-injection test on the test data set, calculate the relevant indicators of the model, and then comprehensively compare and select the model with the best indicators, convert the format and apply it to the product, or, execute the test script in the test device terminal, import the model file to be tested into the device for conversion, and put the test data into the device at the same time, use a re-injection test close to the actual operating state of the device, and then collect the test results, manually summarize the results and screen the model.

[0003] However, for the above method, since the operating environment of data training is GPU or CPU, in actual application, the model may need to be transplanted to the corresponding AI chip for operation. Therefore, there is a quantization error between the chip and the host, and its effect is different from that running on the host, which will cause deviations in the screened model; or when the test results are more in line with the actual operating state of the device, its performance indicators are more credible, but the test process and environment are complex, and manual observation is required. The test status cannot be grasped in time to find problems and repair them in time. When the test indicators are obviously poor, it is impossible to skip to the next model test in time, which is a great waste of time and computing resources. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method, device and testing platform for screening an algorithm model to solve the problem that the existing screening method cannot quickly screen out the algorithm model with the best comprehensive performance.

[0005] According to a first aspect, an embodiment of the present invention provides a method for screening an algorithm model, comprising:

[0006] Acquire a scene segmentation data set; set test parameters of the algorithm model; execute a preset test process based on the scene segmentation data set and the test parameters to obtain a test result of the algorithm model; and evaluate and screen the algorithm model according to the test result of the algorithm model.

[0007] The algorithm model screening method provided in the embodiment of the present invention obtains a scene segmentation data set and sets parameters for the algorithm model, implements algorithm model testing according to a preset test process, and then evaluates and screens the tested algorithm model based on the test results, thereby realizing rapid screening of the most comprehensive algorithm model.

[0008] In combination with the first aspect, in a first implementation of the first aspect, obtaining a scene segmentation dataset includes: obtaining a test dataset; and performing scene segmentation based on the test dataset to obtain a scene segmentation dataset.

[0009] In combination with the first implementation of the first aspect, in the second implementation of the first aspect, the scene segmentation based on the test data set to obtain the scene segmentation data set includes: performing weight allocation according to dimensional information based on the test data set to obtain a dimensional data set; and performing weight allocation on the data sets under the same dimension according to the dimensional data set to obtain a sub-data set.

[0010] The algorithm model screening method provided in the embodiment of the present invention performs scenario segmentation on the test data set so that the segmented and sorted data can be quickly and accurately extracted, thereby improving the screening capability and further realizing the rapid screening of the algorithm model with the best comprehensive performance.

[0011] In combination with the first aspect, in the third implementation of the first aspect, setting the test parameters of the algorithm model includes: setting the test execution conditions and configuring the operating equipment, the model to be tested, and the data to be tested.

[0012] The algorithm model screening method provided in the embodiment of the present invention determines the algorithm model that needs to be tested and the device that runs the algorithm model by setting the algorithm model parameters, so as to quickly screen out the algorithm model with the best comprehensive performance.

[0013] In combination with the first aspect, in the fourth implementation of the first aspect, the execution of a preset test process based on the test data set and the test parameters also includes: performing data synchronization based on the test parameters to obtain synchronization data; extracting model data in the test parameters, converting the model data to obtain an execution model; performing data re-injection testing based on the synchronization data and the execution model, and outputting the test results of the algorithm model.

[0014] The algorithm model screening method provided by the embodiment of the present invention improves the screening ability of the screening method by executing a preset test process, and further realizes the rapid screening of the algorithm model with the best comprehensive performance.

[0015] In combination with the first aspect, in the fifth implementation of the first aspect, the evaluation and screening of the algorithm model based on the test results of the algorithm model includes: obtaining at least 2 test results; and performing comparative analysis based on the test results to determine the optimal algorithm model.

[0016] The algorithm model screening method provided in the embodiment of the present invention screens out the algorithm model with the best comprehensive performance by comparing and analyzing multiple test results, thereby realizing rapid screening of the algorithm model with the best comprehensive performance.

[0017] According to the second aspect, an embodiment of the present invention provides a device for screening an algorithm model, comprising: an acquisition module for acquiring a scene segmentation data set; a setting module for setting test parameters of the algorithm model; an execution module for executing a preset test process based on the scene segmentation data set and the test parameters to obtain a test result of the algorithm model; and a screening module for evaluating and screening the algorithm model according to the test result of the algorithm model.

[0018] The screening device for algorithm models provided in the embodiment of the present invention obtains the scene segmentation data set through the acquisition module, and completes the setting of the test parameters of the algorithm model through the setting module, and then uses the execution module to test the scene segmentation data set according to the preset process to obtain the test results, and finally sends the test results to the screening module to determine the optimal screening results, thereby realizing the rapid screening of the comprehensive optimal algorithm model.

[0019] According to the third aspect, an embodiment of the present invention provides a testing platform, including: a screening device for executing the screening method of the algorithm model described in the first aspect or any one of the embodiments of the first aspect.

[0020] According to the third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the algorithm model screening method described in the first aspect or any one of the embodiments of the first aspect by executing the computer instructions.

[0021] According to the fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the algorithm model screening method described in the first aspect or any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0023] Figure 1 It is a flowchart of a method for screening an algorithm model provided by an embodiment of the present invention;

[0024] Figure 2 It is a flowchart of step S10 of an optional algorithm model screening method provided in an embodiment of the present invention;

[0025] Figure 3 It is a structural block diagram of a passenger car overload detection device provided by an embodiment of the present invention;

[0026] Figure 4 It is a structural block diagram of an electronic device provided by an embodiment of the present invention.

[0027] Reference numerals

[0028] 10 - acquisition module; 11 - setting module; 12 - execution module; 13 - screening module; 20 - processor; 21 - memory. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0030] In addition, it should be noted that the algorithm model screening method provided by the present invention can be used in conjunction with a test platform, wherein the test platform can include a test management platform module: the platform includes four basic modules, which respectively implement unified management functions for test processes, models, data, and test reports, and provide a user interaction interface; Jenkins: It is the key node connecting the platform and the test equipment, and performs scheduling of automated tests, and realizes functions such as automated task configuration, triggering, control, and status monitoring; Test equipment instance: Taking into account that the model needs to run in different environments, such as GPU, CPU, NPU chip equipment, ARM chip equipment, etc., in order to achieve the test results that are closest to the real scene, the resources of each test equipment are uniformly managed, and during testing, Jenkins completes the test on the corresponding device according to the parameter configuration. A specific embodiment of the present invention provides a method for screening an algorithm model, such as Figure 1 As shown, the screening method of the algorithm model includes:

[0031] S10, obtaining a scene segmentation dataset.

[0032] In this embodiment, the scene segmentation data set obtained is determined by the algorithm model selected according to the data currently required to be tested. For example, the optimal vehicle detection algorithm needs to be obtained, so the required scene segmentation data set is the video data frame or image data of road vehicles.

[0033] S11, setting the test parameters of the algorithm model.

[0034] In this embodiment, it is necessary to use a test platform to set the test parameters of the algorithm model.

[0035] S12, executing a preset test process based on the scenario segmentation data set and test parameters to obtain the test results of the algorithm model.

[0036] S13, evaluating and screening the algorithm model according to the test results of the algorithm model.

[0037] The algorithm model screening method provided in this embodiment obtains a scene segmentation data set and sets parameters for the algorithm model, implements algorithm model testing according to a preset test process, and then evaluates and screens the tested algorithm model based on the test results, thereby quickly screening out the most comprehensive algorithm model.

[0038] Optional, such as Figure 2 As shown, step S10 includes:

[0039] S101, obtaining a test data set.

[0040] In this embodiment, the test data set is a set of picture data or a set of video frames, and the detection objects in the test data set are determined by the user.

[0041] S102, performing scene segmentation based on the test data set to obtain a scene segmentation data set.

[0042] In this embodiment, the scene segmentation data set is weighted according to the dimensional information based on the test data set to obtain a dimensional data set; and the data sets under the same dimension are weighted according to the dimensional data set to obtain a sub-data set.

[0043] Optionally, the algorithm model can be obtained through deep learning training, but if only a single scenario data set is used for evaluation when it is ultimately applied in a real production environment, the test results will be biased. The performance of the screened model in the real application environment and the test results may not be ideal. Therefore, it is necessary to construct a segmented scenario data set and evaluate the model in detail from multiple dimensions.

[0044] For a certain model application scenario, the scenario dataset can be constructed from n dimensions. For example, for a vehicle detection model, it can be segmented from the perspectives of weather, road conditions, time period, etc. For the i-th dimension, it can be divided into multiple sub-datasets j. For example, weather can be divided into sunny, cloudy, rainy, snowy, and foggy days. At this time, the number of dimensions m is i =5; the road conditions can be divided into urban, highway, and suburban areas. In this case, the dimension is m i =3; and so on. The entire data set constructed is:

[0045] D={d i,j |i∈[1...n],j∈[1...m i ]}

[0046] The number of sub-datasets under the same dimension is represented by j, and d i,j It is the sub-dataset corresponding to the specified dimension. Due to different factors such as data difficulty, data volume, and frequency of scene occurrence, the degree of dependence on its indicators during testing is different. Therefore, different weights are given to sub-datasets:

[0047] W = {w i,j |i∈[1...n],j∈[1...m i ]}

[0048] And there is where w i,j is the weight value of the sub-dataset under the corresponding dimension;

[0049] Similarly, dataset dimensions also have different weights.

[0050] V={v i |i∈[1...n]}

[0051] And there is where v i is the weight value of the corresponding dimension.

[0052] When testing the model based on the above segmented scenario data sets, the final indicator of the test results is the weighted result.

[0053]

[0054] In the above formula, p i,j is the calculation index of the model test result on the segmented data set, and p is the comprehensive evaluation index of the model on the test set. The evaluation index can be precision, recall, AP, etc.

[0055] For example, taking vehicle detection as an example, the vehicle detection data set can be constructed into three sub-datasets according to the three dimensions of road, weather, and time. The weights of the three sub-datasets are different, and the three sub-datasets can be divided and detailed to construct specific data according to the actual situation. It can be found that the data set is actually divided into sub-datasets according to the dimensions, forming a tree structure, and different weights are given to each result to reflect the importance of different sub-datasets in the final indicator. If necessary, it is entirely possible to construct a three-layer or four-layer tree relationship of sub-datasets, but usually the division of two layers of sub-datasets is sufficient, and too complex a relationship is not conducive to data set management and indicator weighted calculation.

[0056] In this embodiment, a structured understanding of the data set can be established through the hierarchical tree structure, and the construction of segmented scenarios of the data set can be purposefully guided. During automated testing, the lowest-level data set indicators are always directly calculated, and then weighted calculations are performed upwards. When the model is finally screened, a comprehensive screening of weighted indicators and sub-data set indicators can be performed to obtain a model with better overall performance.

[0057] In this embodiment, by constructing a segmented scenario data set, the automated test is given greater flexibility. At different iteration stages, automatic calculation and evaluation are performed based on the focused product requirements to screen out the best model that meets actual needs.

[0058] Optionally, step S11 may be to set the test execution conditions and configure the running equipment, the model to be tested, and the data to be tested. When performing specific settings, each of the test execution conditions and the configuration running equipment, the model to be tested, and the data to be tested may be configured, and in actual operation, it may be set to use the default value or the last set parameter value as needed. The parameters to be set may specifically include:

[0059] Trigger conditions: Set the trigger conditions for automated testing, which may be code, model, data set, equipment, etc. When Jenkins detects that the trigger conditions are met, it starts to execute the automated testing steps;

[0060] Device instance: configure the device instance that needs to be run for the current task;

[0061] Model to be tested: configure the model that needs to be tested for the current task;

[0062] Data to be tested: configure the data set that needs to be tested for the current task;

[0063] Other parameters: Set other detailed parameters that need to be used, including but not limited to confidence threshold, iou threshold, nms threshold, etc.

[0064] Optionally, Jenkins itself provides triggering methods including manual triggering, periodic triggering, precondition triggering, and code update triggering. The manual triggering method can also be implemented by sending the triggering instruction to the Jenkins server. Therefore, the code update triggering method can be configured in the Jenkins automated test project. The other automatic triggering methods are implemented through the test management platform module. When there are changes in models, data sets, devices, etc., the instructions for triggering automatic testing are actively discovered to the Jenkins service.

[0065] In this embodiment, in order to provide a friendly user interaction interface to configure relevant parameters, relevant parameters that Jenkins itself does not provide are stored in the platform database, and the parameters are accessed and executed through the interface when the automation code is executed.

[0066] Optionally, step S12 includes: executing a preset test process based on a test data set and test parameters, and also includes: performing data synchronization based on the test parameters to obtain synchronization data; extracting model data in the test parameters, converting the model data to obtain an execution model; performing a data re-injection test based on the synchronization data and the execution model, and outputting the test results of the algorithm model.

[0067] In this embodiment, script code synchronization is synchronization of automated test script code on a specified test device instance through git or other means; model file synchronization is synchronization of checking the model file version of the platform after the test device instance runs and receives the latest script code; model file conversion is conversion of the model file into the format required in the test device environment, such as pytorch, mxnet, caffe, etc.; test data synchronization is version detection and synchronization with the test data of the platform;

[0068] Data re-injection testing is to re-inject the data set using the specified model in the test equipment based on the corresponding software and hardware environment and output the test results.

[0069] In this embodiment, the test time can be long or short depending on the model and the size of the data set. However, since the entire process is automatically triggered, the night time can be fully utilized for calculations, and the operating status and indicators can be observed at any time, allowing for timely manual intervention.

[0070] Optionally, step S13 includes: obtaining at least 2 test results; and performing comparative analysis based on the test results to determine the optimal algorithm model.

[0071] In this embodiment, since the test process is automated, the model can be tested in large quantities, and some early termination thresholds can be set. When the model's indicators perform too poorly, the test process can be terminated early to save test time and resources. Therefore, when all the test processes of a task are completed, the test reports can be compared and analyzed on the platform, and combined with historical indicators, segmented scenario indicators, comprehensive indicators, computing efficiency and other factors, multi-faceted screening can be performed to select the optimal model required.

[0072] In this embodiment, Jenkins is used for deep learning algorithm testing to facilitate the configuration management of models, data, and test tasks, and to complete a large number of algorithm model tests on multiple platforms. By proposing the construction of segmented scenario data sets and a comprehensive indicator calculation method, weighted calculation is used to make the screened models closer to real scenarios and have stronger comprehensive performance. The automated testing process is triggered by Jenkins, the script files are distributed to designated devices, the models and data are automatically synchronized, and then the test is re-injected and the report is uploaded.

[0073] The embodiment of the present invention provides a device for screening an algorithm model, such as Figure 3 As shown, the screening device of the algorithm model includes:

[0074] The acquisition module 10 is used to acquire a scene segmentation dataset. For details, refer to the relevant description of step S10 in the above method embodiment.

[0075] The setting module 11 is used to set the test parameters of the algorithm model. For details, please refer to the relevant description of step S11 of the above method embodiment.

[0076] The execution module 12 is used to execute a preset test process based on the scene segmentation data set and the test parameters to obtain the test results of the algorithm model. For details, please refer to the relevant description of step S12 of the above method embodiment.

[0077] The screening module 13 is used to evaluate and screen the algorithm model according to the test results of the algorithm model. For details, please refer to the relevant description of step S13 of the above method embodiment.

[0078] The screening device for the algorithm model provided in this embodiment obtains the scene segmentation data set through the acquisition module, and completes the setting of the test parameters of the algorithm model through the setting module, and then uses the execution module to test the scene segmentation data set according to the preset process to obtain the test results, and finally sends the test results to the screening module to determine the optimal screening results, thereby realizing the rapid screening of the comprehensive optimal algorithm model.

[0079] A test platform provided by an embodiment of the present invention may include: a screening device, wherein the screening device may also include a test management module, wherein the test management module is connected to a software interface, and the software interface is also connected to a test device. The environment in which the test device runs may be at least one of a GPU, a CPU, an NPU chip device, and an ARM chip device. The optional software interface may be Jenkins. A test process submodule, a model management submodule, a data management submodule, and a test report management submodule are provided in the test management module.

[0080] In this embodiment, by constructing a test data set for segmented scenarios, and setting the algorithm model, test data, and test parameters in the screening device, and then adding automated test tasks through the Jenkins software interface, and configuring a variety of trigger conditions and test processes, the test scripts, models, and data are automatically synchronized to the specified device instance. After the test is automatically completed, the results are uploaded to the platform for comparative analysis, so as to quickly screen out the algorithm model with the best comprehensive performance.

[0081] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, the electronic device may include a processor 20 and a memory 21, wherein the processor 20 and the memory 21 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0082] The processor 20 may be a central processing unit (CPU). The processor 20 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0083] The memory 21 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method for shielding buttons of a vehicle-mounted display device in an embodiment of the present invention (for example, Figure 3 The processor 20 executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory 21, that is, the screening method of the algorithm model in the above method embodiment is implemented.

[0084] The memory 21 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor 20, etc. In addition, the memory 21 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 21 may optionally include a memory remotely arranged relative to the processor 20, and these remote memories may be connected to the processor 20 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0085] The one or more modules are stored in the memory 21, and when executed by the processor 20, the following is performed: Figure 1-2 The screening method of the algorithm model in the illustrated embodiment.

[0086] For details of the above electronic equipment, please refer to Figures 1 to 3 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0087] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0088] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for screening an algorithm model, characterized in that: include: Acquire a scene segmentation dataset, where the scene segmentation dataset is a video data frame or image data of a vehicle traveling on a road; Set the test parameters of the algorithm model; Execute a preset test process based on the scenario segmentation data set and the test parameters to obtain a test result of the algorithm model; Evaluating and screening the algorithm model according to the test result of the algorithm model, wherein the test result is an evaluation index calculated on the scene segmentation data set, and the evaluation index includes precision, recall and average precision; The step of obtaining a scene segmentation dataset includes: Get the test dataset; Performing scene segmentation based on the test data set to obtain a scene segmentation data set; The step of performing scene segmentation based on the test data set to obtain a scene segmentation data set includes: Based on the test data set, weight allocation is performed according to the dimension information to obtain a dimensional data set; According to the dimensional data set, weights are assigned to data sets under the same dimension to obtain sub-data sets; The performing of a preset test process based on the test data set and the test parameters includes: Perform data synchronization based on test parameters to obtain synchronized data; Extracting model data from the test parameters, and converting the model data to obtain an execution model; A data re-injection test is performed according to the synchronization data and the execution model, and a test result of the algorithm model is output.

2. The method according to claim 1, characterized in that The test parameters of the algorithm model are set, including: setting the test execution conditions and configuring the running equipment, the model to be tested, and the data to be tested.

3. The method according to claim 1, characterized in that The step of evaluating and screening the algorithm model according to the test results of the algorithm model includes: Get at least 2 test results; A comparative analysis is performed based on the test results to determine the optimal algorithm model.

4. A device for screening an algorithm model, characterized in that: include: An acquisition module, used to acquire a scene segmentation dataset, wherein the scene segmentation dataset is a video data frame or image data of a vehicle traveling on a road; Setting module, used to set the test parameters of the algorithm model; An execution module, used to execute a preset test process based on the scenario segmentation data set and the test parameters to obtain a test result of the algorithm model; A screening module, used to evaluate and screen the algorithm model according to the test result of the algorithm model, wherein the test result is an evaluation index calculated on the scene segmentation data set, and the evaluation index includes precision, recall and average precision; Wherein, the acquisition module specifically includes: Get the test dataset; Performing scene segmentation based on the test data set to obtain a scene segmentation data set; The acquisition module specifically includes: Based on the test data set, weight allocation is performed according to the dimension information to obtain a dimensional data set; According to the dimensional data set, weights are assigned to data sets under the same dimension to obtain sub-data sets; The performing of a preset test process based on the test data set and the test parameters includes: Perform data synchronization based on test parameters to obtain synchronized data; Extracting model data from the test parameters, and converting the model data to obtain an execution model; A data re-injection test is performed according to the synchronization data and the execution model, and a test result of the algorithm model is output.

5. A test platform, characterized in that: include: A screening device for executing the screening method of the algorithm model described in any one of claims 1-3.

6. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the screening method of the algorithm model described in any one of claims 1 to 3 by executing the computer instructions.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the screening method of the algorithm model described in any one of claims 1-3.

Images