Evaluation method, system and storage medium of algorithm model

Abstract

The present application relates to an evaluation method, system and storage medium for an algorithm model, wherein the evaluation method is applied to an algorithm evaluation platform, the algorithm evaluation platform is connected to a camera device, and the camera device is integrated with an algorithm scheduling module and an algorithm front-end module; the method includes: obtaining the algorithm model to be tested and the first evaluation data set, and obtaining the algorithm type information of the algorithm model to be tested; sending the algorithm model to be tested and the first evaluation data set to the algorithm front-end module via the algorithm scheduling module; determining the reporting configuration structure corresponding to the algorithm type information, and receiving the first inference data reported by the algorithm front-end module based on the reporting configuration structure through the algorithm scheduling module; the first inference data is obtained by inferring the algorithm model to be tested using the first evaluation data set; generating the first evaluation result of the algorithm model to be tested according to the first inference data reported by the algorithm scheduling module. Through the present application, the problem of low efficiency of algorithm model evaluation is solved.

Description

Technical Field

[0001] The present application relates to the field of camera technology, and in particular to an evaluation method, system and storage medium for an algorithm model. Background Art

[0002] With the continuous development of computer hardware technology and software technology, various embedded devices, mobile devices and smart devices have been widely promoted. Cloud artificial intelligence (AI) surveillance cameras are emerging smart home security devices. By deploying AI algorithm applications on the cloud side, users can configure and download algorithms to the camera on demand, such as passenger flow statistics, area intrusion, face capture, etc., and support a single algorithm to adapt to devices with the same chip but different device models.

[0003] Pre-launch testing and verification of AI algorithm applications is necessary, otherwise it is easy to cause uneven quality of online AI application products, and business personnel cannot confirm the performance boundaries of the products. Some people evaluate the quality of algorithms based entirely on the self-test reports provided by the algorithm manufacturers. Due to differences in test environment data, this may cause the algorithm to perform well on a specific data set, but the performance degrades in actual applications, and it cannot meet product requirements, and it is difficult to unify the algorithm accuracy standards. The existing solution is usually to manually build a real-life environment to retest the accuracy, and acceptance can be carried out for business functions, but there are problems such as high labor and time costs for acceptance, limited enumeration of test cases, and easy statistical errors in data, which seriously affects the online progress of AI algorithm applications.

[0004] Currently, no effective solution has been proposed to address the problem of low efficiency in algorithm model evaluation in related technologies. Summary of the invention

[0005] The embodiments of the present application provide an algorithm model evaluation method, system and storage medium to at least solve the problem of low efficiency of algorithm model evaluation in related technologies.

[0006] In a first aspect, an embodiment of the present application provides an algorithm model evaluation method, which is applied to an algorithm evaluation platform; the algorithm evaluation platform is connected to a camera device, and the camera device is integrated with an algorithm scheduling module and an algorithm front-end module; the method includes:

[0007] Obtaining an algorithm model to be tested and a first evaluation data set, and obtaining algorithm type information of the algorithm model to be tested;

[0008] Sending the algorithm model to be tested and the first evaluation data set to the algorithm front-end module via the algorithm scheduling module;

[0009] Determine a reporting configuration structure corresponding to the algorithm type information, and receive, through the algorithm scheduling module, first inference data reported by the algorithm front-end module based on the reporting configuration structure; the first inference data is obtained by inferring the algorithm model to be tested using the first evaluation data set;

[0010] A first evaluation result of the algorithm model to be tested is generated according to the first inference data reported by the algorithm scheduling module.

[0011] In some embodiments, after determining the reporting configuration structure corresponding to the algorithm type information, the method further includes:

[0012] Receiving, by the algorithm scheduling module, second inference data reported by the algorithm front-end module based on the reporting configuration structure; the second inference data is obtained by inferring the algorithm model to be tested using a second evaluation data set;

[0013] The second evaluation data set is generated by the algorithm front-end module dynamically adjusting the first evaluation data set based on the device resource information obtained by the algorithm scheduling module from the camera device and the first inference data generated;

[0014] A second evaluation result of the algorithm model to be tested is generated based on the second inference data.

[0015] In some embodiments, generating the second evaluation data set further includes:

[0016] The algorithm scheduling module calculates the data set complexity according to the first inference data, and obtains the data set occupied space of the first evaluation data set;

[0017] The algorithm scheduling module allocates a first weight coefficient and a second weight coefficient to the complexity of the data set and the space occupied by the data set respectively;

[0018] The algorithm scheduling module performs weighted calculation on the complexity of the data set and the space occupied by the data set based on the first weight coefficient and the second weight coefficient to generate a data set evaluation result;

[0019] The algorithm scheduling module dynamically adjusts the first evaluation data set based on the device resource information and the data set evaluation result to obtain the second evaluation data set.

[0020] In some embodiments, the device resource information includes memory resource usage information; the algorithm scheduling module dynamically adjusts the first evaluation data set based on the device resource information and the first inference data to obtain a second evaluation data set, including:

[0021] The algorithm scheduling module constructs a first device constraint condition based on the memory resource usage information;

[0022] The algorithm scheduling module adjusts the first evaluation data set based on the first device constraint condition and the data set evaluation result, and obtains the second evaluation data set.

[0023] In some embodiments, the device resource information further includes a central processing unit (CPU) resource occupancy rate; the algorithm scheduling module dynamically adjusts the first evaluation data set based on the device resource information and the first inference data to obtain a second evaluation data set, including:

[0024] The algorithm scheduling module constructs a second device constraint condition based on the CPU resource usage information;

[0025] The algorithm scheduling module assigns a third weight coefficient and a fourth weight coefficient to the first device constraint condition and the second device constraint condition respectively according to the first inference data;

[0026] The algorithm scheduling module determines a third device constraint condition based on the first device constraint condition and the third weight coefficient, and determines a fourth device constraint condition based on the second device constraint condition and the fourth weight coefficient;

[0027] The algorithm scheduling module adjusts the first evaluation data set based on the third device constraint condition, the fourth device constraint condition and the data set evaluation result, and obtains the first evaluation result.

[0028] In some embodiments, sending the algorithm model to be tested and the first evaluation data set to the algorithm front-end module via the algorithm scheduling module includes:

[0029] Establishing a first communication connection; the first communication connection is used to connect the algorithm evaluation platform and the algorithm scheduling module;

[0030] Detecting a data download request sent by the algorithm scheduling module based on the first communication connection; in response to the detected data download request, sending the algorithm model to be tested and the first evaluation data set to the algorithm scheduling module based on the first communication connection;

[0031] Establish a second communication connection; the second communication connection is used to connect the algorithm scheduling module and the algorithm front-end module; wherein the algorithm scheduling module sends the algorithm model to be tested and the first evaluation data set to the algorithm front-end module through the second communication connection.

[0032] In some embodiments, the camera device further includes a storage module; the storage module is used to store the first evaluation data set sent by the algorithm scheduling module;

[0033] The first inference data is generated by the algorithm front-end module reading the first evaluation data set from the storage module and inferring the algorithm model to be tested based on the first evaluation data set; wherein the algorithm front-end module is also used to delete the first evaluation data set from the storage module after the inference is completed.

[0034] In some embodiments, the method further comprises:

[0035] Sending the algorithm model to be tested and the first evaluation data set to the algorithm scheduling module; wherein the algorithm scheduling module performs splicing according to the storage path of the first evaluation data set, and encrypts the ID information of the first evaluation data set to obtain encrypted data;

[0036] The algorithm front-end module receives the algorithm model to be tested and the encrypted data, decrypts the encrypted data according to the storage path to obtain decrypted data, and performs reasoning based on the decrypted data and the algorithm model to be tested to obtain the first reasoning data.

[0037] In a second aspect, an embodiment of the present application provides an algorithm model evaluation system, characterized in that the system comprises: an algorithm evaluation platform and a camera device;

[0038] The camera device is integrated with an algorithm scheduling module and an algorithm front module;

[0039] The algorithm evaluation platform is used to execute the evaluation method of the algorithm model as described in the first aspect above.

[0040] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the evaluation method of the algorithm model described in the first aspect above is implemented.

[0041] Compared with the related technologies, the algorithm model evaluation method, system and storage medium provided in the embodiments of the present application interact with the camera device through an algorithm evaluation platform, and infer the algorithm model on the camera device side, effectively solving the problem of low efficiency of algorithm model evaluation caused by the accuracy of re-testing in the artificial real-life environment, and achieving performance and stability evaluation of the AI ​​algorithm model in the actual application of the camera device while testing the model accuracy, thereby optimizing the problem of incomplete performance evaluation of the pure cloud testing algorithm.

[0042] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 It is a hardware structure block diagram of a terminal according to an algorithm model evaluation method of an embodiment of the present application;

[0045] Figure 2 is a flow chart of an evaluation method of an algorithm model according to an embodiment of the present application;

[0046] Figure 3 is a flow chart of another algorithm model evaluation method according to an embodiment of the present application;

[0047] Figure 4 is a schematic diagram of a communication interaction according to an embodiment of the present application;

[0048] Figure 5 It is a structural diagram of an evaluation system for an algorithm model according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0050] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0051] Unless otherwise defined, the technical terms or scientific terms involved in this application shall be understood by persons with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a kind of", "the" and the like involved in this application do not indicate a quantitative limitation and may represent the singular or plural. The terms "include", "include", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" may represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects, and do not represent a specific ordering of the objects.

[0052] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 1 is a hardware structure block diagram of a terminal according to an algorithm model evaluation method of an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0053] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the evaluation method of the algorithm model in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0054] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0055] This embodiment provides an algorithm model evaluation method, which is applied to an algorithm evaluation platform; the algorithm evaluation platform is connected to a camera device, and the camera device is integrated with an algorithm scheduling module and an algorithm front module. Generally, the algorithm scheduling module is implemented through a software development kit (SDK). The algorithm front module is a module built into the camera device for processing data before input to the algorithm. Figure 2 is a flow chart of an evaluation method of an algorithm model according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0056] Step S210, obtaining the algorithm model to be tested and the first evaluation data set, and obtaining the algorithm type information of the algorithm model to be tested.

[0057] The algorithm model to be tested refers to the target detection or other artificial intelligence algorithms that are waiting to be deployed to the camera equipment application after passing the evaluation. The first evaluation data set for the algorithm model to be tested can be completed in advance through experiments or historical data collection. Among them, the staff can operate and register the algorithm model to be tested on the algorithm evaluation platform, create an evaluation data set, and provide the algorithm type information of the algorithm model to be tested, such as target detection, 1V1 comparison, structured output, etc.

[0058] Step S220: Send the algorithm model to be tested and the first evaluation data set to the algorithm front-end module via the algorithm scheduling module.

[0059] It should be noted that the interaction specifications between modules should be pre-configured before the actual evaluation. Specifically, for the data set specifications of the algorithm evaluation platform, the algorithm scheduling module callback function notification method can be used, and the input parameters are agreed to be the reasoning data path DataPath and the JavaScript Object Notation (JSON) configuration information ConfigData required for the operation of different types of algorithms. Taking the target detection algorithm as an example, the configuration information format is as follows:

[0060] {

[0061] "sourceId":"0a4f9efe-21b1404bc898_real.jpg",

[0062] "monitorArea":[[0,0],[0,0.2],[0.3,0.2],[0.3,0]],

[0063] "modelConfig":{}

[0064] }

[0065] Among them, sourceId represents the unique ID of the image that needs to be inferred, monitorArea represents the coordinates of the area in the image where the ID is detected, and modelConfig represents the personalized parameters required for the algorithm to run. The above datasets support common image (such as .JPG format) and video formats (such as .MP4 format).

[0066] Therefore, in this step, the algorithm evaluation platform can issue data based on the above-mentioned issuance specifications.

[0067] Step S230, determine the reporting configuration structure corresponding to the algorithm type information, and receive the first inference data reported by the algorithm front-end module based on the reporting configuration structure through the algorithm scheduling module; the first inference data is obtained by inferring the algorithm model to be tested using the first evaluation data set.

[0068] The reporting specifications for the inference results of the interaction between the algorithm evaluation platform and the algorithm front-end module are adjusted accordingly according to the detection algorithm type. Taking the target detection algorithm as an example, the above reporting configuration structure is as follows:

[0069] {

[0070] "code": 0,

[0071] "msg":" ",

[0072] "data": {

[0073] "sourceId": "0a4f9efe-21b1404bc898_real.jpg",

[0074] "executionTime": 20,

[0075] "frameResult": [

[0076] {

[0077] "bbox": [

[0078] {

[0079] "monitorAreaId": "daba82e6-1509-4b77-bbe1-ddb5a16523d4",

[0080] "conf": 0.931640625,

[0081] "class": "car",

[0082] "point": [

[0083] 606,

[0084] 138,

[0085] 719, 291 ]

[0088] } ]

[0090] } ]

[0092] }

[0093] }

[0094] In the above code example, code indicates whether the data inference is successful, 0 for success and 1 for failure; if it fails, the msg field needs to describe the specific reason for the failure. ourceId indicates the unique identifier of the image that needs to be inferred, and executionTime indicates the time it takes to infer the result. frameResult contains the target result data detected in this frame, where monitorAreaId indicates the area id where the target appears, ponit is the specific area coordinate, conf indicates the detection confidence, and class indicates the type of detected target.

[0095] In the above steps, data sending specifications and data reporting specifications are formulated respectively; the algorithm scheduling module and the algorithm front-end module interact with the algorithm model to be tested according to the specifications, and the algorithm front-end module reports the algorithm reasoning results according to the reporting configuration structure of the above specifications, so that the reasoning results can be scored on the platform side.

[0096] Step S240: Generate a first evaluation result of the algorithm model to be tested according to the first inference data reported by the algorithm scheduling module.

[0097] Taking the target detection algorithm as an example, the algorithm front-end module infers the type of detection target in the image based on the evaluation image data sent by the algorithm evaluation platform, and gives the location area coordinates and confidence. It also generates the first inference data according to the above reporting configuration results and reports it to the algorithm evaluation platform through the algorithm scheduling module. Among them, the algorithm evaluation platform side compares whether the type and quantity inferred by the algorithm are consistent with the number of target types pre-marked by the platform, and detects whether the intersection of Union (IoU) calculated in the area reported by the algorithm reaches the predetermined threshold, and whether the confidence also reaches the threshold. If the above detection passes, it is considered that the image is inferred. Finally, the proportion of images that pass the detection in the overall evaluation data set is statistically analyzed to give the accuracy rate, missed detection rate and other evaluation index values.

[0098] In the above-mentioned algorithm model evaluation method, the algorithm evaluation platform interacts with the camera device and the algorithm model is inferred on the camera device side, which effectively solves the problem of low efficiency of algorithm model evaluation caused by the accuracy of re-testing in the artificial real-life environment. While testing the model accuracy, it also completes the performance and stability evaluation of the AI ​​algorithm model in the actual application of the camera device, and optimizes the problem of incomplete performance evaluation of the pure cloud testing algorithm.

[0099] In some embodiments, after determining the reporting configuration structure corresponding to the algorithm type information, refer to Figure 3 , the evaluation methods of the algorithm model include Figure 2 Steps S210 to S230 in the embodiment further include the following steps:

[0100] Step S310, receiving the second inference data reported by the algorithm front-end module based on the reporting configuration structure through the algorithm scheduling module; the second inference data is obtained by inferring the algorithm model to be tested using the second evaluation data set; wherein the second evaluation data set is generated by the algorithm front-end module dynamically adjusting the first evaluation data set based on the device resource information obtained from the camera device by the algorithm scheduling module and the generated first inference data.

[0101] Specifically, the algorithm scheduling module can respond to the evaluation task notified by the algorithm evaluation platform through signaling, and obtain the TF card mounting path and the occupancy rate and occupancy limit of each module resource in the camera device in real time, that is, obtain the above-mentioned device resource information. More specifically, the device resource information can include the CPU occupancy rate of the camera device, the CPU occupancy limit of the camera device, the remaining memory space of the camera device, and the memory space limit of the camera device.

[0102] Afterwards, the resources acquired by the algorithm scheduling module can be notified to the algorithm front-end module through the socket protocol, and reported to the algorithm evaluation platform through the result reporting interface of the algorithm front-end module. In addition, the algorithm front-end module detects whether the current first evaluation data set is too large based on the acquired device resource data and the first inference data obtained by its first inference, so as to dynamically adjust the first evaluation data set based on the detection result, and then obtain the above-mentioned second evaluation data set.

[0103] Step S320: Generate a second evaluation result of the algorithm model to be tested based on the second inference data.

[0104] In this step, the algorithm evaluation platform analyzes the second inference data obtained by inferring the data set adjusted by the algorithm scheduling module to obtain the corresponding second evaluation result. The evaluation data set transfer, adjustment and processing operations in this process are all performed on the camera device side, and the algorithm evaluation platform is not aware of it, thereby improving processing efficiency, ensuring the flexibility of data processing on the camera device side, and facilitating the expansion of application scenarios.

[0105] It should be understood that the algorithm scheduling module can further adjust the current second evaluation data set based on the currently acquired device resource information and the second inference data generated using the second evaluation data set, and repeat this cycle until a loop expiration condition is reached, such as the data set adjustment duration exceeds a preset duration, or the number of adjustments exceeds a preset number, or a task end instruction is detected, and the inference data generated based on the final adjusted evaluation data set is sent to the algorithm evaluation platform for analysis.

[0106] Through the above embodiments, in addition to the initial inference of the algorithm front module, the evaluation data set that needs to be inferred can be dynamically adjusted based on the output results of the algorithm model inferred by the algorithm front module, effectively avoiding the problem of too much single inference data, resulting in the algorithm inference occupying more device memory resources and causing the camera device to restart, thereby ensuring the stability of the camera device operation during the evaluation phase.

[0107] In some embodiments, generating the second evaluation data set further includes the following steps:

[0108] The above-mentioned algorithm scheduling module calculates the complexity of the data set according to the first inference data, and obtains the data set occupied space of the first evaluation data set. Among them, the algorithm front module can link to libraries such as libpng and libjpeg to obtain the size of each picture in the evaluation data set, so that the size of the evaluation data set can be dynamically calculated according to the evaluation data set input each time. Here, the first evaluation data set is taken as an example, that is, the data set occupied space of the first evaluation data set is calculated according to the size of each picture in the data set. In addition, since the algorithm reasoning performance time of different pictures in the same data set is not much different, in this embodiment, the defined data set complexity refers to a value that is positively correlated with the time spent on the previous algorithm reasoning, and it is believed that the longer the reasoning time, the more complex the data set. For example, the executionTime (the time taken to infer the result) in the first reasoning result first generated by the algorithm front module above can be used as the value of the current data set complexity.

[0109] The algorithm scheduling module allocates a first weight coefficient and a second weight coefficient to the complexity of the data set and the space occupied by the data set, respectively; the algorithm scheduling module performs weighted calculation on the complexity of the data set and the space occupied by the data set based on the first weight coefficient and the second weight coefficient to generate a data set evaluation result; the algorithm scheduling module dynamically adjusts the first evaluation data set based on the device resource information and the data set evaluation result to obtain the second evaluation data set.

[0110] Specifically, the evaluation results of the above dataset D c The calculation process can be shown by the following formula:

[0111]

[0112] In the above formula, and They are used to represent the complexity of the dataset and the space occupied by the dataset respectively. The first weight coefficient and the second weight coefficient It can be set according to the actual evaluation requirements; for example, if the current evaluation scenario focuses on algorithm accuracy, you can increase the If the current evaluation scenario focuses on the algorithm duration, you can increase it accordingly. Finally, by judging whether the data set evaluation results obtained by the above weighted calculation can meet the device resource information, it is determined whether it is necessary to dynamically adjust the data set size and / or data set complexity, so as to realize the dynamic adjustment of the current evaluation data set in the algorithm front module.

[0113] Through the above embodiments, a method for detecting the size and complexity of a data set is provided.

[0114] In some embodiments, the device resource information includes memory resource usage information; the algorithm scheduling module dynamically adjusts the first evaluation data set based on the device resource information and the first inference data to obtain the second evaluation data set, and may also include the following steps:

[0115] The algorithm scheduling module constructs a first device constraint condition based on the memory resource usage information; the algorithm scheduling module adjusts the first evaluation data set based on the first device constraint condition and the data set evaluation result, and obtains a second evaluation data set.

[0116] Specifically, the first device constraint condition can be determined according to the upper limit and lower limit of the memory resource usage rate set by the camera device. One expression form of the first device constraint condition can be shown in the following formula:

[0117] st D c <L MEM -B MEM

[0118] In the above formula, L MEM Used to indicate the upper limit of memory resource usage set by the camera device. MEM It is used to indicate the memory resource usage rate when the camera device has not started the inference algorithm model. st is used to indicate D c It can be seen that based on the above formula, the evaluation data set can be associated with the device resource usage, thereby realizing the dynamic adjustment of the evaluation data set under the device resource constraint.

[0119] In some embodiments, the device resource information further includes CPU resource occupancy rate; the algorithm scheduling module dynamically adjusts the first evaluation data set based on the device resource information and the first inference data to obtain the second evaluation data set, and may further include the following steps:

[0120] The algorithm scheduling module constructs a second device constraint condition based on the CPU resource utilization information. Specifically, the second device constraint condition can be determined according to the upper limit and lower limit of the CPU resource utilization rate set by the camera device. One expression form of the second device constraint condition can be shown in the following formula:

[0121] st D c <L CPU -B CPU

[0122] In the above formula, L CPU Used to indicate the upper limit of CPU resource usage set by the camera device. CPU Used to indicate the CPU resource usage when the camera device has not started inferring the algorithm model.

[0123] In this embodiment, multiple device constraints are provided; in order to accurately integrate the constraints, the processing process of the algorithm scheduling module is as follows: the algorithm scheduling module assigns a third weight coefficient and a fourth weight coefficient to the first device constraint and the second device constraint respectively according to the first inference data; the algorithm scheduling module determines the third device constraint based on the first device constraint and the third weight coefficient, and determines the fourth device constraint based on the second device constraint and the fourth weight coefficient; the algorithm scheduling module adjusts the first evaluation data set based on the third device constraint, the fourth device constraint and the data set evaluation result, and obtains the first evaluation result.

[0124] Based on the above analysis, the corresponding third device constraint condition and fourth device constraint condition are as shown in the following formula:

[0125]

[0126]

[0127] More specifically, the algorithm scheduling module can determine which device, the memory or the CPU in the camera device, consumes more resources based on the first algorithm model reasoning result, and determine the allocation ratio of the third weight coefficient and the fourth weight coefficient based on the judgment result. For example, if the algorithm scheduling module determines that the memory consumes more resources based on the first reasoning data, the camera device may be in a data-intensive task where the algorithm needs to process a large amount of data or store intermediate results, and the memory resource consumption is more, then the third weight coefficient corresponding to the memory device is increased accordingly; conversely, the camera device may be in a computing-intensive task or parallel computing task where the algorithm involves a large amount of mathematical operations, logical judgments, and the CPU resource consumption is more, then the fourth weight coefficient corresponding to the CPU is increased.

[0128] In some embodiments, the above-mentioned sending the algorithm model to be tested and the first evaluation data set to the algorithm front-end module via the algorithm scheduling module may also include the following steps:

[0129] Establish a first communication connection; the first communication connection is used to connect the algorithm evaluation platform and the algorithm scheduling module; detect the data download request sent by the algorithm scheduling module based on the first communication connection; in response to the detected data download request, send the algorithm model to be tested and the first evaluation data set to the algorithm scheduling module based on the first communication connection. Among them, the algorithm scheduling module establishes Hypertext Transfer Protocol Secure (HTTPS) communication with the algorithm evaluation platform, and uses signaling notification to complete the download request processing of the evaluation algorithm package and the ZIP encrypted data set. The ZIP encrypted data set refers to the encrypted data set in .zip format.

[0130] A second communication connection is established; the second communication connection is used to connect the algorithm scheduling module and the algorithm front-end module; wherein the algorithm scheduling module sends the algorithm model to be tested to the algorithm front-end module through the second communication connection.

[0131] Specifically, cross-process communication is established between the algorithm scheduling module and the algorithm front-end module, and the communication adopts TCP socket mode.

[0132] See also Figure 4 The algorithm scheduling module acts as a server, creates a server socket, performs a bind operation, and then continuously listens to a fixed port. The algorithm front-end module acts as a client and actively connects to the algorithm scheduling module. The data packets are uniformly in JSON format. The plain text exchange format of the message between the algorithm scheduling module and the algorithm front-end module and the data packet type code are formulated. The data packet starts with SDC and ends with \r\n. The received data packet needs to be judged by 'head and tail' to determine whether the data packet is complete.

[0133] 3001 / 3000 is formulated as the algorithm front-end module online protocol code pair. If the algorithm front-end module is online, the 3001 online message code is reported according to the protocol and carries the AI ​​algorithm business process ID (procId) tag, such as {"CODE":3001,"MSG":{"procId":"132568"}}; after receiving the online message, the algorithm scheduling module replies with a 3000 message response code, completes the AI ​​algorithm business management with the AI ​​algorithm business process ID, and sends data such as {"CODE":3000,"MSG":{"configPath":" / tmp / config.json"}}. 2501 / 2500 is formulated as the algorithm front-end module to request the TF card data storage path (tfcardPath), CPU occupancy (cpuUsage), CPU usage limit (maxCPU), remaining memory resources (freeMem) and memory resource usage limit (maxMem) information code pair. After the AI ​​algorithm front-end module is successfully launched, it reports the 2501 message {"CODE":2501,"MSG":{"procId":"132568"}} according to the protocol, and queries the device resource status to the algorithm scheduling module. After receiving the query request, the algorithm scheduling module queries the information according to the callback interface agreed with the camera, responds to the AI ​​algorithm front-end module 2500 protocol code, and formulates 2001 / 2000 as the dataset download link of the algorithm evaluation platform and the specific information notification response code pair; specifically, {"CODE":2000,"MSG":{"dataset":" ","procId":"132568"}}, and {"CODE":2001,"MSG":{"resultcode":"0","msg":" "}}; where" " indicates encrypted information.

[0134] Through the above embodiments, communication connections between modules are established, effectively ensuring the efficiency and accuracy of data transmission.

[0135] In some embodiments, the camera device further includes a storage module; the storage module is used to store the first evaluation data set sent by the algorithm scheduling module. The first inference data is generated by the algorithm front module reading the first evaluation data set from the storage module and inferring the algorithm model to be tested based on the first evaluation data set; wherein the algorithm front module is also used to delete the first evaluation data set from the storage module after the inference is completed.

[0136] In this embodiment, the storage module can use the TF card (TransFlash Card) built into the camera device. Specifically, considering the limited FLASH space (flash memory storage space) and memory resources of the camera device, the algorithm scheduling module stores the evaluation data set issued by the algorithm evaluation platform on the TF card storage module. After the algorithm front-end module reads the data set and configuration information stored in the TF card, it inputs the data set into the algorithm module for reasoning, and finally reports the inferred algorithm results to the algorithm evaluation platform according to the above specifications. The algorithm front-end module reports the callback function result based on the received result, and deletes the data set stored on the TF card storage module that has completed the reasoning, so as to effectively utilize the storage space.

[0137] In some embodiments, the evaluation method of the algorithm model further includes the following steps:

[0138] The algorithm model to be tested and the first evaluation data set are sent to the algorithm scheduling module; wherein the algorithm scheduling module splices the evaluation data set according to the storage path of the evaluation data set, and encrypts the ID (Identity document) information of the evaluation data set to obtain encrypted data. Specifically, the algorithm scheduling module completes the data set storage path splicing according to the download task of the algorithm evaluation platform. At the same time, the encryption of the data set ID and other information issued by the algorithm accuracy evaluation platform is completed. The encryption can adopt the Cipher Block Chaining (CBC) mode of the Advanced Encryption Standard (AES) and is sent to the algorithm front-end module through the established socket communication protocol code.

[0139] The algorithm front module receives the algorithm model to be tested and the encrypted data, decrypts the encrypted data according to the storage path to obtain the decrypted data, and performs reasoning based on the decrypted data and the algorithm model to be tested to obtain the first reasoning data. Specifically, after the algorithm front module reads the data set and configuration information stored in the TF card, it completes the decryption of the ZIP encrypted data set according to the storage path, and restores the data set encryption after the image data to be reasoned is read into the memory.

[0140] Through the above embodiments, data is encrypted during the data transmission process of each module, thereby ensuring the security and privacy of data transmission.

[0141] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0142] This embodiment also provides an algorithm model evaluation system, which includes: an algorithm evaluation platform and a camera device; the camera device is integrated with an algorithm scheduling module and an algorithm front-end module; the algorithm evaluation platform is used to execute the steps in any of the above method embodiments.

[0143] The embodiments of the present application are described in detail below in conjunction with actual application scenarios. Figure 5 is a structural diagram of an evaluation system for an algorithm model according to an embodiment of the present application, such as Figure 5 As shown, the system includes an algorithm evaluation platform and an intelligent camera device.

[0144] Among them, the evaluation algorithm warehouse and evaluation data warehouse deployed on the algorithm evaluation platform are used to store AI algorithm models and evaluation data sets respectively. The algorithm evaluation platform sends the current algorithm model to be tested and the first evaluation data set to the algorithm scheduling module, and the algorithm scheduling module handles TF card status monitoring, data set transfer, evaluation data distribution, and evaluation application management. The algorithm front module can read the evaluation data set from the TF card to realize evaluation data download, storage, encryption and decryption, and report the algorithm reasoning results to the algorithm evaluation platform through the algorithm scheduling module according to the reporting configuration structure. Finally, the evaluation module of the algorithm evaluation platform scores the algorithm model based on the reasoning results.

[0145] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0146] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0147] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0148] S1, obtaining an algorithm model to be tested and a first evaluation data set, and obtaining algorithm type information of the algorithm model to be tested.

[0149] S2, sending the algorithm model to be tested and the first evaluation data set to the algorithm front-end module via the algorithm scheduling module.

[0150] S3, determining a reporting configuration structure corresponding to the algorithm type information, and receiving, through the algorithm scheduling module, first inference data reported by the algorithm front module based on the reporting configuration structure; the first inference data is obtained by inferring the algorithm model to be tested using the first evaluation data set;

[0151] S4, generating a first evaluation result of the algorithm model to be tested according to the first inference data reported by the algorithm scheduling module.

[0152] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0153] In addition, in combination with the evaluation method of the algorithm model in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by the processor, the evaluation method of any algorithm model in the above embodiment is implemented.

[0154] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for evaluating an algorithm model, characterized in that: Applied to an algorithm evaluation platform; the algorithm evaluation platform is connected to a camera device, and the camera device is integrated with an algorithm scheduling module and an algorithm front-end module; the method includes: Obtaining an algorithm model to be tested and a first evaluation data set, and obtaining algorithm type information of the algorithm model to be tested; Sending the algorithm model to be tested and the first evaluation data set to the algorithm front-end module via the algorithm scheduling module; Determine the reporting configuration structure corresponding to the algorithm type information, and receive the first inference data reported by the algorithm front-end module based on the reporting configuration structure through the algorithm scheduling module; the first inference data is obtained by inferring the algorithm model to be tested using the first evaluation data set; the reporting configuration structure is a configuration structure of data reporting specifications including whether the inference is successful, a field describing the reason for failure, a unique identifier of the image, the time spent on inferring the result, the target result data detected in this frame, the target appearance area id, the area coordinates, the detection confidence and the detection target type; Generate a first evaluation result of the algorithm model to be tested according to the first inference data reported by the algorithm scheduling module, including: whether the type and quantity inferred by the comparison algorithm are consistent with the number of target types pre-marked by the platform, and detect whether the intersection and union ratio calculated by the area reported by the algorithm reaches a predetermined threshold, and whether the confidence also reaches the threshold. If the above detection passes, it is considered that the image has passed the inference; the proportion of images that have passed the detection in the overall evaluation data set is statistically analyzed to give the accuracy rate, missed detection rate and other evaluation index values; Receiving, by the algorithm scheduling module, second inference data reported by the algorithm front-end module based on the reporting configuration structure; the second inference data is obtained by inferring the algorithm model to be tested using a second evaluation data set; The second evaluation data set is generated by the algorithm front-end module dynamically adjusting the first evaluation data set based on the device resource information obtained by the algorithm scheduling module from the camera device and the first inference data generated; A second evaluation result of the algorithm model to be tested is generated based on the second inference data.

2. The evaluation method according to claim 1, characterized in that: Generating the second evaluation data set also includes: The algorithm scheduling module calculates the data set complexity according to the time taken to infer the result included in the first inference data, and obtains the data set occupied space of the first evaluation data set; the data set complexity refers to a value positively correlated with the time taken for the previous algorithm inference; The algorithm scheduling module allocates a first weight coefficient and a second weight coefficient to the complexity of the data set and the space occupied by the data set respectively; The algorithm scheduling module performs weighted calculation on the complexity of the data set and the space occupied by the data set based on the first weight coefficient and the second weight coefficient to generate a data set evaluation result; The algorithm scheduling module dynamically adjusts the first evaluation data set based on the device resource information and the data set evaluation result to obtain the second evaluation data set.

3. The evaluation method according to claim 2, characterized in that: The device resource information includes memory resource usage information; The algorithm scheduling module dynamically adjusts the first evaluation data set based on the device resource information and the first inference data to obtain a second evaluation data set, including: The algorithm scheduling module constructs a first device constraint condition based on the memory resource usage information; The algorithm scheduling module adjusts the first evaluation data set based on the first device constraint condition and the data set evaluation result, and obtains the second evaluation data set.

4. The evaluation method according to claim 3, characterized in that: The device resource information also includes CPU resource occupancy rate; The algorithm scheduling module dynamically adjusts the first evaluation data set based on the device resource information and the first inference data to obtain a second evaluation data set, including: The algorithm scheduling module constructs a second device constraint condition based on the CPU resource occupancy rate; The algorithm scheduling module assigns a third weight coefficient and a fourth weight coefficient to the first device constraint condition and the second device constraint condition respectively according to the first inference data; The algorithm scheduling module determines a third device constraint condition based on the first device constraint condition and the third weight coefficient, and determines a fourth device constraint condition based on the second device constraint condition and the fourth weight coefficient; The algorithm scheduling module adjusts the first evaluation data set based on the third device constraint condition, the fourth device constraint condition and the data set evaluation result, and obtains the first evaluation result.

5. The evaluation method according to claim 1, characterized in that: The sending the algorithm model to be tested and the first evaluation data set to the algorithm front-end module via the algorithm scheduling module includes: Establishing a first communication connection; the first communication connection is used to connect the algorithm evaluation platform and the algorithm scheduling module; Detecting a data download request sent by the algorithm scheduling module based on the first communication connection; in response to the detected data download request, sending the algorithm model to be tested and the first evaluation data set to the algorithm scheduling module based on the first communication connection; Establish a second communication connection; the second communication connection is used to connect the algorithm scheduling module and the algorithm front-end module; wherein the algorithm scheduling module sends the algorithm model to be tested and the first evaluation data set to the algorithm front-end module through the second communication connection.

6. The evaluation method according to claim 1, characterized in that: The camera device further includes a storage module; the storage module is used to store the first evaluation data set sent by the algorithm scheduling module; The first inference data is generated by the algorithm front-end module reading the first evaluation data set from the storage module and inferring the algorithm model to be tested based on the first evaluation data set; wherein the algorithm front-end module is also used to delete the first evaluation data set from the storage module after the inference is completed.

7. The evaluation method according to any one of claims 1 to 6, characterized in that: The method further comprises: Sending the algorithm model to be tested and the first evaluation data set to the algorithm scheduling module; wherein the algorithm scheduling module performs splicing according to the storage path of the first evaluation data set, and encrypts the ID information of the first evaluation data set to obtain encrypted data; The algorithm front-end module receives the algorithm model to be tested and the encrypted data, decrypts the encrypted data according to the storage path to obtain decrypted data, and performs reasoning based on the decrypted data and the algorithm model to be tested to obtain the first reasoning data.

8. An algorithm model evaluation system, characterized in that: The system includes: an algorithm evaluation platform and a camera device; The camera device is integrated with an algorithm scheduling module and an algorithm front module; The algorithm evaluation platform is used to execute the evaluation method of the algorithm model as described in any one of claims 1 to claim 7.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the evaluation method of the algorithm model described in any one of claims 1 to 7 when running.

Images