Algorithm model management method and system, electronic device and storage medium

Abstract

The invention relates to an algorithm model management method and system, an electronic device and a storage medium, and the algorithm model management method comprises the steps: obtaining a first algorithm model; the first algorithm model is generated by the algorithm warehouse portal platform based on training of application demand information acquired from the algorithm warehouse platform; the algorithm warehouse portal platform is connected with the algorithm warehouse platform; acquiring equipment information of all end side equipment; determining target equipment in the end side equipment based on the equipment information and a first algorithm model; the end side equipment is connected with the algorithm bin platform; issuing the first algorithm model to a target device for test operation to obtain a test result set; and based on the test result set, generating an evaluation result of the first algorithm model. According to the invention, the problem of low management efficiency of the end-side algorithm model is solved.

Description

Technical Field

[0001] The present application relates to the field of model management technology, and in particular to a management method, system, electronic device and storage medium for an algorithm model. Background Art

[0002] With the development of Internet of Vehicles, smart cities, VR / AR and the advancement of terminal technology, the computing and storage capabilities of terminal devices are constantly improving, and the total volume and diversity of their services are also growing, bringing a large number of computing-intensive and delay-sensitive task requirements. In related technologies, in order to meet the different demand scenarios of customers, there are usually many algorithm vendors for end-side algorithms, and they often lack an efficient and complete management system for end-side algorithms, which makes the introduction process of end-side algorithms more cumbersome, thus affecting the management efficiency of end-side algorithm models.

[0003] Currently, no effective solution has been proposed to address the problem of low efficiency in algorithm model management on the edge in related technologies. Summary of the invention

[0004] The embodiments of the present application provide a method, system, electronic device and storage medium for managing an algorithm model, so as to at least solve the problem of low efficiency of algorithm model management on the terminal side in the related art.

[0005] In a first aspect, an embodiment of the present application provides a method for managing an algorithm model, which is applied to an algorithm warehouse platform; the method comprises:

[0006] Acquire a first algorithm model; the first algorithm model is generated by the algorithm warehouse portal platform based on the application requirement information obtained from the algorithm warehouse platform; the algorithm warehouse portal platform is connected to the algorithm warehouse platform;

[0007] Acquire device information of all end-side devices; determine a target device among the end-side devices based on the device information and the first algorithm model; and connect the end-side devices to the algorithm warehouse platform;

[0008] The first algorithm model is sent to the target device for testing and operation to obtain a test result set; based on the test result set, an evaluation result of the first algorithm model is generated.

[0009] In some embodiments, the first algorithm model carries model metadata; acquiring device information of all end-side devices; and determining a target device among the end-side devices based on the device information and the first algorithm model include:

[0010] Traversing the device information in the device resource pool of the algorithm warehouse platform;

[0011] The model metadata is matched with the traversed device information, and the target device is determined according to the matching result.

[0012] In some embodiments, sending the first algorithm model to the target device for testing and running to obtain a test result set includes:

[0013] Based on the application requirement information, obtaining a corresponding test data set;

[0014] Sending the test data set and the first algorithm model to the target device;

[0015] Among them, the target device tests and runs the first algorithm model based on the test data set, generates a test result set, and reports the test result set to the algorithm warehouse platform.

[0016] In some embodiments, the obtaining of a corresponding test data set based on preset application requirement information includes:

[0017] If the application requirement information matches the historical requirement information stored in the algorithm warehouse platform, obtaining a historical data set of the historical requirement information;

[0018] In response to the detected adjustment instruction, an adjustment parameter for the historical data set is obtained; and according to the historical data set and the adjustment parameter, the test data set is obtained.

[0019] In some embodiments, obtaining the first algorithm model includes:

[0020] Acquire category information of the application requirement information;

[0021] Based on the category information, the application requirement information is sent to the corresponding algorithm warehouse portal platform; wherein, the algorithm warehouse portal platform downloads the corresponding training data from the algorithm warehouse platform based on the application requirement information, uses the training data to train locally to generate the first algorithm model, and uploads the first algorithm model to the algorithm warehouse platform.

[0022] In some embodiments, after generating the evaluation result of the first algorithm model, the method further includes:

[0023] When the evaluation result indicates that the first algorithm model has passed the evaluation, creating application logic based on the application requirement information, and packaging the application logic and the first algorithm model to generate a second algorithm model;

[0024] Sending the second algorithm model to the target device;

[0025] Among them, the target device performs a functional test on the second algorithm model; if the functional test passes, the target device runs the second algorithm model; if the functional test fails, the generated functional test data is reported to the algorithm warehouse platform for iterative processing.

[0026] In some embodiments, after sending the second algorithm model to the target device, the method further includes:

[0027] Get the preset reflux evaluation information;

[0028] Based on the reflow evaluation information, determine the corresponding algorithm model to be reflowed from the second algorithm model, and obtain the reflow data uploaded by the target device; wherein the reflow data is data generated by the target device running the algorithm model to be reflowed;

[0029] Acquire pre-annotated data for the reflow data; generate a reflow test set according to the reflow data and the pre-annotated data;

[0030] Based on the reflow evaluation information, the reflow test set is sent to the reflow device in the terminal side device for testing and running to obtain a reflow test result.

[0031] In a second aspect, an embodiment of the present application provides a management system for an algorithm model, including: an algorithm warehouse platform, an algorithm warehouse portal platform, and a terminal-side device;

[0032] The algorithm warehouse platform connects the algorithm warehouse portal platform and the terminal side device, and is used to execute the algorithm model management method described in the first aspect above.

[0033] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for managing the algorithm model as described in the first aspect above is implemented.

[0034] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the management method of the algorithm model as described in the first aspect above is implemented.

[0035] Compared with the related art, the algorithm model management method, system, electronic device and storage medium provided in the embodiment of the present application are obtained by obtaining a first algorithm model; the first algorithm model is generated by the algorithm warehouse portal platform based on the application requirement information obtained from the algorithm warehouse platform; the algorithm warehouse portal platform is connected to the algorithm warehouse platform; the device information of all end-side devices is obtained; based on the device information and the first algorithm model, the target device in the end-side device is determined; the end-side device is connected to the algorithm warehouse platform; the first algorithm model is sent to the target device for testing and operation to obtain a test result set; based on the test result set, the evaluation result of the first algorithm model is generated. Based on this, a complete process system for unified management of algorithm models of different algorithm vendors is provided, and the automated evaluation of the end-side algorithm is realized, which reduces the input of repeated human labor and effectively solves the problem of low management efficiency of the end-side algorithm model. At the same time, in order to improve the communication accuracy of multiple end-side devices, the target device is determined by matching the device information of the end-side device with the first algorithm model, so that the algorithm model can be accurately transmitted to the corresponding end-side device, thereby improving the accuracy of algorithm model management.

[0036] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 It is a hardware structure block diagram of a terminal of a management method of an algorithm model according to an embodiment of the present application;

[0039] Figure 2 is a flow chart of a management method of an algorithm model according to an embodiment of the present application;

[0040] Figure 3 is a schematic diagram of information interaction in an algorithm introduction process according to an embodiment of the present application;

[0041] Figure 4 It is a schematic diagram of a process of online data reflow according to an embodiment of the present application;

[0042] Figure 5 is a flowchart of another algorithm model management method according to an embodiment of the present application;

[0043] Figure 6 This is a flow chart of an automated evaluation according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0045] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0046] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0047] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 1 is a hardware structure block diagram of a terminal according to an algorithm model management method of an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0048] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the management method of the algorithm model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0049] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0050] This embodiment provides a method for managing an algorithm model, which is applied to an algorithm warehouse platform; in order to facilitate the unified management of artificial intelligence (AI) algorithm models, the algorithm warehouse platform can be deployed in the AI ​​center. Figure 2 is a flow chart of a management method of an algorithm model according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0051] Step S210, obtain the first algorithm model; the first algorithm model is generated by the algorithm warehouse portal platform based on the application requirement information obtained from the algorithm warehouse platform; the algorithm warehouse portal platform is connected to the algorithm warehouse platform.

[0052] The algorithm warehouse portal platform refers to a portal platform operated by algorithm vendors. With the rapid development of the artificial intelligence industry, the demand for different types of end-side algorithms is gradually increasing. Therefore, it is necessary to introduce end-side algorithm models provided by different algorithm vendors.

[0053] In this step, the algorithm manufacturer can receive the application requirement information issued by the algorithm warehouse platform on the algorithm warehouse portal platform; the application requirement information is used to indicate the implementation functions or scenarios of the required end-side algorithm and other information. For example, the application requirement information can be the requirements set by the platform staff for electric vehicle detection or fire monitoring. Next, the algorithm manufacturer conducts local training on the algorithm warehouse portal according to the application requirement information, and finally uploads the trained first algorithm model to the algorithm warehouse platform for testing. Alternatively, where permitted by the embodiment, each algorithm manufacturer can also directly train and generate the first algorithm model on the algorithm warehouse platform; at this time, the training data and training algorithm for the algorithm model are provided by the algorithm warehouse platform. The first algorithm model is an algorithm model that has not yet been automatically tested or has not passed the automated test; the data of the first algorithm model includes algorithm model files, algorithm pre- and post-processing programs, configuration files, and dependency libraries.

[0054] Step S220, obtaining device information of all end-side devices; determining the target device in the end-side devices based on the device information and the first algorithm model; and connecting the end-side devices to the algorithm warehouse platform.

[0055] End-side devices refer to terminal devices that apply end-side algorithms to meet various user needs; end-side devices may include but are not limited to various network cameras, tablet computers or vehicle-mounted terminals and other devices. Among them, the algorithm warehouse platform can connect to multiple end-side devices of various types or models. Based on this, in order to improve the accuracy of data transmission between the algorithm warehouse platform and the end-side devices, the algorithm warehouse platform can query the device information of each end-side device; the device information includes information such as the device model, chip model or version number of the end-side device. Based on the queried device information that matches the first algorithm model, determine the corresponding target device to run the first algorithm model.

[0056] Step S230, sending the first algorithm model to the target device for testing and operation to obtain a test result set; based on the test result set, generating an evaluation result of the first algorithm model.

[0057] Specifically, after the algorithm manufacturer registers the algorithm model to the algorithm warehouse platform, the algorithm evaluation is automatically initiated. The algorithm warehouse platform sends test instructions to the end-side device; the end-side device responds to the received test instructions and uses its onboard scheduling software development kit (Software Development Kit, SDK) to remotely download the first algorithm model from the algorithm warehouse platform and load it into the firmware; and the scheduling SDK notifies the dynamic loading framework in the end-side device to implement algorithm control management, and then the evaluation SDK on the end-side device tests the algorithm model, and uploads the algorithm prediction results, running time and other evaluation reports generated by the custom test, that is, the test result set, to the algorithm warehouse platform. Finally, the algorithm warehouse platform processes the evaluation results and analyzes the automated test results of the current algorithm model, such as function, accuracy, and performance, to determine whether the algorithm model can pass the evaluation. It should be understood that if the automated evaluation results indicate that the current first algorithm model evaluation has not passed, the algorithm warehouse platform can send the data to the algorithm warehouse portal platform for model optimization; after the algorithm manufacturer optimizes the model on the algorithm warehouse portal platform, the optimized model data will be registered to the algorithm warehouse platform to continue the automated evaluation until the algorithm model evaluation passes, thereby realizing the iterative optimization of the first algorithm model without the need for offline manual docking.

[0058] It should be noted that, in the present embodiment, automated testing can be performed on multiple algorithm models. That is, after the algorithm manufacturer registers the model, it submits the model data to be stored in the platform database. The algorithm warehouse platform scans the information in the database and generates different multi-tasks. The scheduling module of the algorithm warehouse platform uses multi-threading and other technologies to perform automated testing on multiple first algorithm models at the same time. Alternatively, in another embodiment, in view of the situation where platform resources are tight, in order to make full use of computing resources, the algorithm warehouse platform can also sort multiple tasks according to a priority strategy, and give priority to automated testing for tasks with high priorities. Among them, the priority strategy can be set in advance; for example, for tasks with high demand importance, their priority is set high; or, for tasks generated based on the algorithm model uploaded by the algorithm manufacturer first, their priority is set high, etc.

[0059] In the above algorithm model management method, the algorithm warehouse platform obtains the first algorithm model uploaded by the algorithm warehouse portal platform and performs automated testing on it, thereby providing a complete process system for unified management of algorithm models of different algorithm vendors, and realizing the construction of automated evaluation of end-side algorithms, reducing the input of repetitive human labor, and effectively solving the problem of low management efficiency of end-side algorithm models. At the same time, in order to improve the communication accuracy of multiple end-side devices, the target device is determined by matching the device information of the end-side device with the first algorithm model, so that the algorithm model can be accurately transmitted to the corresponding end-side device, improving the accuracy of algorithm model management.

[0060] In some embodiments, the first algorithm model carries model metadata; the model metadata refers to data used to describe model attributes. Specifically, the model metadata may include at least data such as the device model of the applicable end-side device; after the algorithm manufacturer generates the first algorithm model, the metadata is packaged together into a model introduction package and uploaded to the algorithm warehouse platform.

[0061] The above-mentioned acquisition of device information of all end-side devices; determining the target device in the end-side device based on the device information and the first algorithm model, also includes the following steps: traversing the device information in the device resource pool of the algorithm warehouse platform; matching the model metadata with the traversed device information, and determining the target device according to the matching result. Among them, the device information of all end-side devices that are connected to the algorithm warehouse platform for communication can be pre-stored in the device resource pool of the algorithm warehouse platform. After the algorithm manufacturer registers the model introduction package, the algorithm warehouse platform traverses and queries the device information in its device resource pool, and matches the traversed device information with the model metadata, and compares whether the currently traversed device information is the same as the metadata such as the device model carried by the model; if they are the same, the end-side device corresponding to the successfully matched device information is used as the above-mentioned target device.

[0062] Through the above-mentioned embodiments, by pre-storing the device information in the device resource pool, the target device can be quickly and accurately found during the automated test, thereby improving the accuracy and efficiency of the algorithm model management.

[0063] In some embodiments, the above-mentioned sending the first algorithm model to the target device for testing and running to obtain a test result set also includes the following steps:

[0064] Based on the application requirement information, obtain the corresponding test data set; send the test data set and the first algorithm model to the target device; wherein the target device tests and runs the first algorithm model based on the test data set, generates a test result set, and reports the test result set to the algorithm warehouse platform. The test data set can be pre-configured on the algorithm warehouse platform by the platform staff based on the proposed application requirement information; specifically, the platform staff uploads and annotates the image set and other data to be tested on the annotation platform of the algorithm warehouse platform. Alternatively, the test data set can also be data such as reflux data or false alarm data set collected during the application process.

[0065] In this step, the terminal device responds to the test instructions of the algorithm warehouse platform, inputs the test data set issued by the algorithm warehouse platform into the corresponding first algorithm model for prediction, and generates corresponding prediction results according to the evaluation specifications provided by the algorithm manufacturer carried by the model, that is, the above-mentioned test data set is obtained, thereby realizing a high-precision and accurate automated evaluation process for the algorithm model.

[0066] In some embodiments, the above-mentioned obtaining of a corresponding test data set based on preset application requirement information further includes the following steps:

[0067] If the application demand information matches the historical demand information stored in the algorithm warehouse platform, a historical data set of the historical demand information is obtained; in response to the detected adjustment instruction, adjustment parameters for the historical data set are obtained; and a test data set is obtained based on the historical data set and the adjustment parameters.

[0068] Among them, the algorithm warehouse platform can also store historical demand information proposed by the platform staff in the historical time period. In this embodiment, the algorithm warehouse platform compares the current application demand information with the historical demand information to determine whether the application demand information is an improvement of a certain historical demand information. If so, the test data set of the corresponding historical demand information, that is, the historical data set, can be extracted. Then the staff detects whether it is necessary to improve the historical data set based on the application demand information; if improvement is required, the staff adjusts or adds annotations, or supplements the data to generate a test data set corresponding to the current application demand information; otherwise, the historical data set can be directly used as the test data set. Taking the target device as a certain model of network camera as an example, if the current application demand information is fixed-point monitoring of fire conditions, and the historical application demand information matching the same category is monitoring of whether a fire has occurred; then the algorithm warehouse platform can extract the historical data set of the historical application information, and perform manual annotations based on the fire points, and finally generate the above-mentioned test data set. Of course, if the current application demand information is not an improvement of the historical demand information, it means that the test data set has not been configured, and the staff can upload the data and annotate normally.

[0069] Through the above embodiments, for the test data set of existing application requirements, when the requirements change, it is only necessary to adjust the existing data set accordingly or directly adopt the data set, so there is no need to repeatedly upload test data, which improves the efficiency of test set production and further improves the efficiency of automated testing.

[0070] In some embodiments, the obtaining of the first algorithm model further includes the following steps:

[0071] Obtain category information of application requirement information; based on the category information, send the application requirement information to the corresponding algorithm warehouse portal platform; wherein the algorithm warehouse portal platform downloads corresponding training data from the algorithm warehouse platform based on the application requirement information, uses the training data to train locally to generate a first algorithm model, and uploads the first algorithm model to the algorithm warehouse platform.

[0072] See also Figure 3 , the party proposing the requirements proposes application requirement information on the algorithm warehouse platform based on the actual situation; the AI ​​product manager classifies the original requirements proposed to obtain the category information of the requirements; based on the requirements classification results, the requirements are supplemented or new algorithm requirements are created, and the supplemented or newly created requirement information is published to the algorithm warehouse portal platform. At this time, the corresponding algorithm manufacturer will claim and accept the requirements. The AI ​​product manager and the partner responsible for requirement refinement in the algorithm manufacturer that accepts the requirements will connect and improve the algorithm requirement content; during the connection process, after claiming the algorithm requirement, the requirement refinement partner will send the claiming partner's information to the AI ​​product manager, and the AI ​​product manager will track and confirm the partner; the requirement refinement partner will then send the confirmed requirement content to the AI ​​product manager, and the AI ​​product manager will issue a solution design to the algorithm engineer operating on the algorithm warehouse portal platform.

[0073] Next, based on the training data corresponding to the application requirement information downloaded from the algorithm warehouse platform, the algorithm engineer will locally implement the algorithm input and output and the classification specification of the algorithm specification design, the preparation of the evaluation data set, and the design of the evaluation plan. The evaluation plan design also includes the selection of evaluation indicators, evaluation indicator weights, scoring rules, etc. The algorithm engineer will send the algorithm specification design, evaluation plan design, etc. to the contracting partner for algorithm adaptation development and algorithm introduction; after the algorithm model training is completed, the algorithm manufacturer will register the algorithm on the algorithm warehouse portal platform and start the automated evaluation of the algorithm model. At the same time, the algorithm engineer will track the evaluation progress with the contracting partner, the contracting partner will track the introduction progress with the AI ​​product manager, and the AI ​​product manager will track the introduction progress of the AI ​​requirements with the party proposing the requirements.

[0074] Through the above embodiments, a complete process system is constructed from proposing requirements, formulating algorithm input and output specifications, designing evaluation plans, publishing requirements, and finally having the end-side algorithm manufacturer accept the requirements, thus achieving unified management and improving the efficiency of algorithm model management.

[0075] In some embodiments, after generating the evaluation result of the first algorithm model, the algorithm model management method further includes the following steps:

[0076] Step S241, when the evaluation result indicates that the first algorithm model has passed the evaluation, create application logic based on the application requirement information, and package the application logic and the first algorithm model to generate a second algorithm model.

[0077] The above-mentioned second algorithm model is an application package that can be distributed and installed on the end-side device for production business use after the algorithm model passes the automated evaluation; the second algorithm model contains data such as algorithm model files, application logic programs, algorithm pre- and post-processing programs, configuration files, and dependency libraries. Among them, the algorithm warehouse platform performs pipeline orchestration on a single or multiple first algorithm models that have passed the evaluation, adds business logic processing, and produces algorithm applications for specific AI business scenarios, that is, packages the above-mentioned second algorithm model.

[0078] Step S242, sending the second algorithm model to the target device; wherein the target device performs a functional test on the second algorithm model; if the functional test passes, the target device runs the second algorithm model; if the functional test fails, the generated functional test data is reported to the algorithm warehouse platform for iterative processing.

[0079] Specifically, the algorithm warehouse portal platform registers the second algorithm model to the algorithm warehouse platform for algorithm application packaging. The algorithm warehouse platform sends the packaged second algorithm model to the corresponding target device for end-to-end functional testing. If the test passes, the second algorithm model can be put into the warehouse to be officially put into operation on the target device. If the test fails, the test data can be fed back to the algorithm warehouse portal platform, and the algorithm manufacturer will modify the error problem on the algorithm warehouse portal platform to complete the algorithm model. The algorithm model with the modified problem is registered on the algorithm warehouse platform again, and functional testing continues, thereby achieving iterative optimization of the algorithm application employee access stage.

[0080] Through the above embodiment, the second algorithm model generated after the evaluation is packaged is functionally tested and iteratively processed, which further improves the management process of the algorithm model. At the same time, a complete end-side algorithm iterative optimization process is established to help end-side algorithm manufacturers to use the algorithm warehouse on-site to iteratively optimize the algorithm using online data.

[0081] In some embodiments, after sending the second algorithm model to the target device, the algorithm model management method further includes the following steps:

[0082] Step S243, obtaining preset reflow evaluation information.

[0083] The above-mentioned reflux evaluation information may include an online data reflux strategy for indicating strategies such as online data acquisition cycles, and preset rules such as periodic automated evaluation rules. The online data reflux strategy and periodic automated evaluation rules may be pre-defined by staff. Among them, the periodic automated evaluation rules include customized algorithms or requirements that require reflux evaluation, and reflux evaluation time.

[0084] Step S244, based on the reflow evaluation information, determine the corresponding algorithm model to be reflowed from the second algorithm model, and obtain the reflow data uploaded by the target device; wherein the reflow data is data generated by the target device running the algorithm model to be reflowed.

[0085] The above-mentioned algorithm model to be reflowed refers to the algorithm model in the second algorithm model that needs to be reflowed and evaluated according to the definition rules of the automated evaluation. The above-mentioned reflow data can be obtained by: the management list set by the platform staff in the algorithm warehouse platform, manually detecting the algorithm misidentification images during the application process; or, it can also be a data set obtained by collecting error data reported by the end-side device customers during use.

[0086] Step S245, obtaining pre-annotated data for the reflow data; generating a reflow test set according to the reflow data and the pre-annotated data; and sending the reflow test set to the reflow device in the terminal device for test operation based on the reflow evaluation information to obtain the reflow test result.

[0087] The above online data reflow is described below in conjunction with specific embodiments. Figure 4 , the process includes the following steps:

[0088] In step S401, the staff configures the online data reflow strategy on the algorithm warehouse platform, and reflows the algorithm false alarm data to the data set management center of the algorithm warehouse platform according to the measured flow.

[0089] In step S402, the data center processes the online data that flows back to the data set management center, including processing the frame extraction data and the alarm data, pre-labeling the data, manually analyzing and generating the data set version, and determining the data set label, such as a data set label suitable for vehicle detection or pedestrian detection, and generating the above-mentioned return test set.

[0090] Step S403, sending the reflow test set that meets the periodic automated evaluation definition rules to the matching reflow device, and performing automated evaluation according to the defined automated evaluation rules; this automated evaluation is similar to the above-mentioned evaluation process of the first algorithm model based on the test data set, and will not be repeated here.

[0091] Step S404, generating an evaluation report based on the automated evaluation results; if the evaluation results are poor, the algorithm warehouse platform customer will send a message to the corresponding product manager via email or text message; the product manager will confirm whether it is necessary to initiate algorithm optimization.

[0092] It should be noted that after the algorithm model is put into production, in order to ensure that the algorithm can always maintain good accuracy and performance, the algorithm needs to be periodically evaluated to ensure that once the algorithm accuracy or performance decreases, optimization can be initiated in a timely manner to ensure the normal operation of online business. Based on this, this application has formulated a data reflow strategy through the above embodiments, which strategically reflows and periodically automatically evaluates the production data of the online end-side algorithm, ensuring the continuous verification of the accuracy and performance of the algorithm model and realizing the timely optimization of the end-side algorithm.

[0093] The embodiments of the present application are described and illustrated below through specific examples. Figure 5 is a flowchart of another algorithm model management method according to an embodiment of the present application, such as Figure 5 As shown, the process includes the following steps:

[0094] During the demand recruitment stage, the Algorithm Warehouse Platform publishes the algorithm application requirements. The Algorithm Warehouse portal platform receives the requirements, including the algorithm application requirements, class definition, model test solution design, annotation specifications, and evaluation SDK specification design, and implements local development.

[0095] During the model access phase, the algorithm warehouse portal platform registers the model introduction package; the model introduction package is a program package that can be sent down and installed on the IPC test device for model accuracy evaluation, which is the first algorithm model mentioned above. The algorithm warehouse platform begins the evaluation and sends the model introduction package to the target device; the target device evaluates the model's functions, accuracy, performance, and specifications; for example, the algorithm warehouse portal locally develops an AI model for video face monitoring based on AI needs and packages it; the algorithm warehouse specifically uses the test set to evaluate whether the AI ​​model can achieve face recognition functions, recognition accuracy, model generalization capabilities, etc., and determines whether it passes the above evaluation. If the evaluation passes, the model is saved on the algorithm warehouse platform; otherwise, the evaluation results are fed back to the algorithm warehouse portal platform for model optimization, and the optimized and packaged model introduction package is registered again.

[0096] During the algorithm application access phase, the algorithm warehouse portal platform registers the algorithm application package, which is the second algorithm model mentioned above. The algorithm warehouse platform packages it and sends it to the target device for end-to-end functional testing; among them, the algorithm warehouse platform can replace the model files in the algorithm application package uploaded by the algorithm manufacturer with the model files that have passed the evaluation and package them into an algorithm application package according to the algorithm application package standard. If the evaluation passes, the algorithm application package will be put into the warehouse; otherwise, the test error data will be fed back to the algorithm warehouse portal platform for problem modification, and the modified packaged algorithm application package will be registered again. For example, after the model evaluation passes, the algorithm warehouse adds application logic to the iteratively optimized AI model data, packages it into an algorithm introduction package and sends it to the network camera; the network camera runs the application for functional testing, and officially runs after the test passes.

[0097] In addition, for the automated evaluation process of the client-side algorithm model, please refer to Figure 6 , the process includes the following steps:

[0098] Step S601: The algorithm manufacturer registers the algorithm to the algorithm warehouse and automatically initiates algorithm evaluation.

[0099] Step S602: Send instructions to the network camera. The scheduling SDK downloads the algorithm from the remote and loads it into the firmware. The scheduling SDK notifies the dynamic loading framework to start the algorithm, and the algorithm runs successfully on the device.

[0100] Step S603: the algorithm warehouse platform sends the evaluation data set to the camera.

[0101] Step S604, predicting the evaluation data set using the algorithm running on the network camera, and generating corresponding prediction results according to the specification;

[0102] Step S605, the evaluation SDK reports the evaluation results to the algorithm warehouse;

[0103] Step S606: the algorithm warehouse processes the evaluation results, analyzes the accuracy and performance of the current algorithm, and determines whether the algorithm passes the evaluation.

[0104] It should be noted that in order to ensure the security of data assets and the credibility between the end-side devices and the platform, the algorithm warehouse platform can authorize the end-side devices to ensure that when the subsequent devices report the evaluation data set results to the platform, they are all credible devices. The authorization process includes the following steps:

[0105] Step S701: The evaluation SDK in the end-side device will use the public key to encrypt the device's online status, timestamp, private key, device code, device MAC and other information when the device is powered on and initialized; the encrypted ciphertext is reported to the platform. After the platform gets the data, it uses the private key to decrypt it and obtains the device code, MAC, timestamp, and evaluation data set decryption key to complete the device identity authentication.

[0106] Step S702: Before the company's evaluation data set for evaluation is sent to the end-side device to be evaluated, the algorithm warehouse platform can encrypt the image data set or video data set based on the security of data assets. The encryption key can be the dedicated key reported by the device during the device and platform identity authentication. The encrypted data is sent to the end-side device, thereby effectively ensuring the security of data transmission from the platform side to the device side. After the target end-side device obtains the data set, it needs to use the key to decrypt it before it can obtain the image or video data of the evaluation data set.

[0107] In step S703, after the end-side device obtains the evaluation data set, the evaluation results are reported to the algorithm warehouse platform using the interface. Before reporting, the device will encrypt the evaluation results to ensure the security of data transmission from the device to the platform. After receiving the results, the platform uses a dedicated key to decrypt the request and obtain the evaluation results.

[0108] Through the above embodiments, a terminal-side algorithm introduction management system has been established, from the release of requirements, to the formulation of algorithm input and output specifications, the design of evaluation plans, the release of requirements, and finally to the acceptance of requirements by the terminal-side algorithm manufacturer, a complete process system has been established to achieve unified management. In the algorithm iteration optimization, the entire process is standardized, and based on the premise of data security, a complete closed-loop process from initiating optimization to the on-site presence of algorithm manufacturers, the use of online data, labeling, training, evaluation, and warehousing is achieved, thereby improving the efficiency of algorithm iteration optimization. Automation is achieved in the algorithm performance and accuracy testing, and multiple algorithm evaluations can be performed at the same time, saving manpower and time. At the same time, the embodiment of the present application strategically reflows the production data of the online terminal-side algorithm and automatically evaluates it to achieve continuous verification of the algorithm accuracy and performance, and ensure timely optimization of the terminal-side algorithm.

[0109] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0110] This embodiment also provides a management system for an algorithm model, which management system includes: an algorithm warehouse platform, an algorithm warehouse portal platform and an end-side device; the algorithm warehouse platform is respectively connected to the algorithm warehouse portal platform and the end-side device, and is used to execute the algorithm model management method described in any of the above embodiments.

[0111] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0112] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0113] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0114] S1, obtain the first algorithm model; the first algorithm model is generated by the algorithm warehouse portal platform based on the application demand information obtained from the algorithm warehouse platform; the algorithm warehouse portal platform is connected to the algorithm warehouse platform.

[0115] S2, obtaining device information of all end-side devices; determining the target device among the end-side devices based on the device information and the first algorithm model; and connecting the end-side devices to the algorithm warehouse platform.

[0116] S3, sending the first algorithm model to the target device for testing and running to obtain a test result set; based on the test result set, generating an evaluation result of the first algorithm model.

[0117] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0118] In addition, in combination with the algorithm model management method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the algorithm model management methods in the above embodiment is implemented.

[0119] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A management method for algorithm models, It is characterized in that Applied to the algorithm warehouse platform; the method comprises: Acquire a first algorithm model; the first algorithm model is generated by the algorithm warehouse portal platform based on the application requirement information obtained from the algorithm warehouse platform; the algorithm warehouse portal platform is connected to the algorithm warehouse platform; Acquire device information of all end-side devices; determine a target device among the end-side devices based on the device information and the first algorithm model; and connect the end-side devices to the algorithm warehouse platform; The first algorithm model is sent to the target device for testing and operation to obtain a test result set; based on the test result set, an evaluation result of the first algorithm model is generated.

2. The management method according to claim 1, It is characterized in that The first algorithm model carries model metadata; the device information of all end-side devices is obtained; Determining a target device in the terminal side device based on the device information and the first algorithm model includes: Traversing the device information in the device resource pool of the algorithm warehouse platform; The model metadata is matched with the traversed device information, and the target device is determined according to the matching result.

3. The management method according to claim 1, It is characterized in that The sending the first algorithm model to the target device for testing and running to obtain a test result set includes: Based on the application requirement information, obtaining a corresponding test data set; Sending the test data set and the first algorithm model to the target device; Among them, the target device tests and runs the first algorithm model based on the test data set, generates a test result set, and reports the test result set to the algorithm warehouse platform.

4. The management method according to claim 3, It is characterized in that The step of obtaining a corresponding test data set based on preset application requirement information includes: If the application requirement information matches the historical requirement information stored in the algorithm warehouse platform, obtaining a historical data set of the historical requirement information; In response to the detected adjustment instruction, an adjustment parameter for the historical data set is obtained; and according to the historical data set and the adjustment parameter, the test data set is obtained.

5. The management method according to claim 1, It is characterized in that The obtaining of the first algorithm model comprises: Acquire category information of the application requirement information; Based on the category information, the application requirement information is sent to the corresponding algorithm warehouse portal platform; wherein, the algorithm warehouse portal platform downloads the corresponding training data from the algorithm warehouse platform based on the application requirement information, uses the training data to train locally to generate the first algorithm model, and uploads the first algorithm model to the algorithm warehouse platform.

6. The management method according to claim 1, It is characterized in that After generating the evaluation result of the first algorithm model, the method further includes: When the evaluation result indicates that the first algorithm model has passed the evaluation, creating application logic based on the application requirement information, and packaging the application logic and the first algorithm model to generate a second algorithm model; Sending the second algorithm model to the target device; Among them, the target device performs a functional test on the second algorithm model; if the functional test passes, the target device runs the second algorithm model; if the functional test fails, the generated functional test data is reported to the algorithm warehouse platform for iterative processing.

7. The management method according to claim 6, It is characterized in that After sending the second algorithm model to the target device, the method further includes: Get the preset reflux evaluation information; Based on the reflow evaluation information, determine the corresponding algorithm model to be reflowed from the second algorithm model, and obtain the reflow data uploaded by the target device; wherein the reflow data is data generated by the target device running the algorithm model to be reflowed; Acquire pre-annotated data for the reflow data; generate a reflow test set according to the reflow data and the pre-annotated data; Based on the reflow evaluation information, the reflow test set is sent to the reflow device in the terminal side device for test operation to obtain a reflow test result.

8. A management system for algorithmic models, It is characterized in that include: Algorithm warehouse platform, algorithm warehouse portal platform and end-side devices; The algorithm warehouse platform connects the algorithm warehouse portal platform and the terminal side device, and is used to execute the algorithm model management method as described in any one of claims 1 to 7.

9. An electronic device comprising a memory and a processor, It is characterized in that A computer program is stored in the memory, and the processor is configured to run the computer program to execute the management method of the algorithm model according to any one of claims 1 to 7.

10. A storage medium, It is characterized in that The storage medium stores a computer program, wherein the computer program is configured to execute the management method of the algorithm model according to any one of claims 1 to 7 when running.

Images