A testing method, device, camera access method and medium for algorithm metrics

Abstract

Embodiments of the present disclosure provide a method and apparatus for testing algorithm metrics, a method for accessing a camera, and a medium. The method for testing algorithm metrics includes: obtaining algorithm benchmark metrics; controlling the AI platform to play a video source file multiple times with different numbers of playback channels, and obtaining the algorithm metrics of the data processing algorithm for processing each video source file during each playback of the video source file by the AI platform; and obtaining a test result of the algorithm metrics based on the algorithm benchmark metrics, the number of playback channels of the multiple playbacks of the video source file, and the obtained algorithm metrics. The technical solution provided by the present disclosure overcomes the problems of video stream processing failure and service crash caused by insufficient resources of the AI platform after the camera is accessed to the AI platform.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to, but are not limited to, the field of artificial intelligence technology, and in particular, to a method and device for testing algorithm metrics, a camera access method, and a medium. Background Art

[0002] In recent years, with the development of the field of artificial intelligence, more and more AI (Artificial Intelligence) algorithms have been developed and applied to various industries. There are problems in the existing detection of AI algorithm online (especially in the case of batch online of AI algorithms) that the time for troubleshooting is relatively long and the labor cost is relatively high. After the AI algorithm is online on the AI platform, when the camera is connected to the AI platform, there are often problems such as the failure to process the video stream from the camera or the service hanging up due to insufficient resource configuration of the AI platform. Summary of the Invention

[0003] The problem to be solved by the embodiments of the present disclosure is to provide a method and device for testing algorithm metrics and a storage medium to overcome the problems that the video stream processing fails and the service hangs up due to insufficient resources of the AI platform after the camera is connected to the AI platform.

[0004] To solve the above technical problems, an embodiment of the present disclosure provides a method for testing algorithm metrics, which is set to test the algorithm metrics of a data processing algorithm in an AI platform. The testing method includes:

[0005] Obtain algorithm benchmark metrics;

[0006] Control the AI platform to play the video source file multiple times with different numbers of playback channels, and obtain the algorithm metrics of the data processing algorithm for processing each video source file during each playback of the video source file by the AI platform;

[0007] Obtain the test result of the algorithm metrics according to the algorithm benchmark metrics, the number of playback channels of the multiple playbacks of the video source file, and the obtained algorithm metrics.

[0008] In an exemplary embodiment, the algorithm metrics include at least one of the following: the number of alarm messages, the average processing frame rate, the pixel position of the detection box of the alarm picture, the system resource occupancy rate, and the alarm message file;

[0009] The obtaining of the algorithm benchmark metrics includes: playing the video source file multiple times through an offline algorithm, and obtaining the offline algorithm metrics output by the offline algorithm multiple times, and calculating the average value of each offline algorithm metric obtained multiple times to obtain the algorithm benchmark metrics;

[0010] The algorithm benchmark metrics include at least one of the following: the number of benchmark alarm messages, the benchmark average processing frame rate, the benchmark pixel positions of the detection frames in the alarm pictures, the benchmark occupancy rate of system resources, and the benchmark alarm message files.

[0011] In an exemplary embodiment, controlling the AI platform to play the video source file multiple times with different numbers of playback channels, and obtaining the algorithm metrics of the AI algorithm for processing each video source file during each playback of the video source file by the AI platform, including:

[0012] Set the initial values of the total number of playback channels and the number of playback channels;

[0013] Generate a first configuration file for the AI platform to access the video stream of the current playback channel, and make the first configuration file take effect; the first configuration file includes the video stream address and the algorithm information for processing the video stream;

[0014] Send an instruction to the streaming media service according to the video stream address in the first configuration file, and the instruction includes the current playback channel number;

[0015] The streaming media service receives the instruction, obtains the video source file, converts the video source file into a video stream of the current playback channel according to the instruction, and sends the video stream of the current playback channel to the AI platform; the AI platform includes: a decoding processor and a data processing algorithm;

[0016] The decoding processor in the AI platform converts the video stream of the current playback channel into picture frames, and transmits the picture frames to the data processing algorithm corresponding to the algorithm information for processing the video stream in the first configuration file;

[0017] The data processing algorithm in the AI platform plays the video stream of the current playback channel according to the picture frames, and outputs the processing results of processing each video stream, and uses the processing results as the algorithm metrics;

[0018] Update the value of the current playback channel number, and determine whether the updated value of the current playback channel number exceeds the total number of playback channels. When the updated value of the current playback channel number does not exceed the total number of playback channels, continue to send an instruction to the streaming media service according to the video stream address in the first configuration file.

[0019] In an exemplary embodiment, before the decoding processor in the AI platform converts the video stream of the current playback channel into picture frames and transmits the picture frames to the data processing algorithm corresponding to the algorithm information in the first configuration file, it further includes: controlling the AI platform to decode the video stream of the current playback channel to obtain picture data, and obtaining the average processing speed of the AI platform for decoding each video stream;

[0020] The decoding processor converts the video stream of the current playing channel into picture frames, including: the decoding processor converts the picture data of the video stream of the current playing channel into picture frames.

[0021] In an exemplary embodiment, when the updated value of the current playing channel exceeds the total number of playing channels, the method further includes: obtaining the test result of the AI platform metrics based on the decoding benchmark metrics pre-stored in the AI platform, the number of channels of the video stream during multiple playbacks of the video source file, and the average processing speed of each video stream obtained by the AI platform.

[0022] In an exemplary embodiment, the first configuration file at least includes: the video stream address, the frame rate of playing the video stream, and the algorithm information for processing the video stream;

[0023] The data processing algorithm in the AI platform plays the video stream of the current playing channel according to the picture frame, including: the data processing algorithm in the AI platform plays the video stream of the current playing channel at the frame rate of playing the video stream in the first configuration file according to the picture frame.

[0024] In an exemplary embodiment, making the first configuration file take effect includes: controlling the AI platform to restart to make the first configuration file take effect, or transmitting an activation parameter to the AI platform to make the first configuration file dynamically take effect.

[0025] In an exemplary embodiment, after the streaming service receives the instruction and before converting the video source file into the video stream of the current playing channel according to the instruction, it further includes: the streaming service restarts.

[0026] In an exemplary embodiment, obtaining the test result of the algorithm metrics based on the algorithm benchmark metrics, the number of playing channels during multiple playbacks of the video source file, and the output algorithm metrics includes:

[0027] During each playback of the video source file, compare the number of alarm messages in each channel with the benchmark number of alarm messages, and / or compare the pixel positions of the alarm picture detection frames in each channel with the benchmark pixel positions of the alarm picture detection frames, and count the algorithm accuracy rate of each playback of the video source file according to the comparison results;

[0028] Obtain the test result of the algorithm metrics based on the algorithm accuracy rate statistically obtained from multiple playbacks of the video source file.

[0029] In an exemplary embodiment, before comparing the number of alarm messages in each channel with the benchmark number of alarm messages, and / or comparing the pixel positions of the alarm picture detection frames in each channel with the benchmark pixel positions of the alarm picture detection frames during each playback of the video source file, it further includes:

[0030] Determine whether the average processing frame rate of multiple channels is lower than the reference average processing frame rate during each playback of the video source file. When the average processing frame rate of multiple channels is lower than the reference average processing frame rate, determine that the number of playback channels supported for this playback is not supported. When the average processing frame rate of multiple channels is not lower than the reference average processing frame rate, determine the number of playback channels supported for this playback; compare whether the system resource occupancy rate in each channel is lower than the system resource reference occupancy rate;

[0031] During each playback of the video source file, compare the number of alarm messages in each channel with the reference number of alarm messages, and / or compare the pixel positions of the alarm picture detection frames in each channel with the reference pixel positions of the alarm picture detection frames. According to the comparison results, calculate the algorithm accuracy rate for each playback of the video source file, including:

[0032] Calculate the difference between the number of alarm messages in each channel and the reference number of alarm messages, and the difference between the pixel positions of the alarm picture detection frames in each channel and the reference pixel positions of the alarm picture detection frames during each playback of the video source file, and compare whether the alarm message files in each channel are consistent with the reference alarm message files. According to the judgment, calculation, and comparison results, calculate the algorithm accuracy rate for each playback of the video source file.

[0033] In an exemplary embodiment, the calculating the difference between the number of alarm messages in each channel and the reference number of alarm messages, and the difference between the pixel positions of the alarm picture detection frames in each channel and the reference pixel positions of the alarm picture detection frames during each playback of the video source file, and comparing whether the alarm message files in each channel are consistent with the reference alarm message files includes:

[0034] During each playback of the video source file, calculate the difference between the number of alarm messages in each channel and the reference number of alarm messages. If the difference between the number of alarm messages and the reference number of alarm messages exceeds the preset difference, determine that the algorithm is inaccurate when playing the current playback channel number;

[0035] Calculate the difference between the pixel positions of the alarm picture detection frames in each channel and the reference pixel positions of the alarm picture detection frames. If the difference between the pixel positions of the alarm picture detection frames and the reference pixel positions of the alarm picture detection frames exceeds the preset difference range, determine that the algorithm is inaccurate when playing the current playback channel number;

[0036] Compare whether the alarm message files in each channel are consistent with the reference alarm message files. If they are inconsistent, determine that the algorithm is inaccurate when playing the current playback channel number;

[0037] Comparing whether the system resource occupancy rate in each path is lower than the system resource benchmark occupancy rate includes: comparing whether the system resource occupancy rate in each path is lower than the system resource benchmark occupancy rate. If the system resource occupancy rate is not lower than the system resource benchmark occupancy rate, it is determined that the algorithm is inaccurate when playing the current number of playing paths;

[0038] Statistical algorithm accuracy rate for each play of the video source file according to the judgment, calculation, and comparison results includes: counting the accuracy rate of playing multiple video streams during each play of the video source file, and obtaining the algorithm accuracy rate based on the number of playing paths and the accuracy rate of multiple plays of the video source file.

[0039] In an exemplary embodiment, obtaining the test result of the algorithm metric according to the algorithm benchmark metric, the number of playing paths, and the output algorithm metric during multiple plays of the video source file includes:

[0040] Judging whether the average processing frame rate of multiple paths is lower than the benchmark average processing frame rate during each play of the video source file. When the average processing frame rate of multiple paths is lower than the benchmark average processing frame rate, it is determined that the number of playing paths for this play is not supported. When the average processing frame rate of multiple paths is not lower than the benchmark average processing frame rate, it is determined that the number of playing paths for this play is supported;

[0041] Calculating the difference between the number of alarm messages and the benchmark number of alarm messages in each path, and the difference between the pixel positions of the alarm picture detection frames and the benchmark pixel positions of the alarm picture detection frames during each play of the video source file, and comparing whether the alarm message files in each path are consistent with the benchmark alarm message files, and comparing whether the system resource occupancy rate in each path is lower than the system resource benchmark occupancy rate. Statistical algorithm accuracy rate for each play of the video source file according to the judgment, calculation, and comparison results;

[0042] Obtaining the test result of the algorithm metric based on the algorithm accuracy rate statistically obtained from multiple plays of the video source file.

[0043] In an exemplary embodiment, obtaining the test result of the algorithm metric based on the algorithm accuracy rate statistically obtained from multiple plays of the video source file includes at least one of the following:

[0044] Obtaining a relationship curve graph of the number of playing paths and the average frame rate, a relationship curve graph of the number of playing paths and the system resource occupancy rate, and a relationship curve graph of the number of playing video paths and the algorithm accuracy rate according to the algorithm accuracy rate;

[0045] The test result of the algorithm metric includes at least one of the following: the relationship curve graph of the number of playing paths and the average frame rate, the relationship curve graph of the number of playing paths and the system resource occupancy rate, and the relationship curve graph of the number of playing paths and the algorithm accuracy rate.

[0046] In an exemplary embodiment, during each playback of the video source file, the calculation includes the difference between the number of alarm messages and the reference number of alarm messages in each channel, the difference between the pixel positions of the alarm picture detection frames and the reference pixel positions of the alarm picture detection frames in each channel, and comparing whether the alarm message files in each channel are consistent with the reference alarm message files, and comparing whether the system resource occupancy rate in each channel is lower than the reference system resource occupancy rate. According to the judgment, calculation, and comparison results, the algorithm accuracy rate for each playback of the video source file is statistically calculated, including:

[0047] During each playback of the video source file, calculate the difference between the number of alarm messages and the reference number of alarm messages in each channel. If the difference between the number of alarm messages and the reference number of alarm messages exceeds the preset difference, it is determined that the algorithm is inaccurate when playing the current playback channel number;

[0048] Calculate the difference between the pixel positions of the alarm picture detection frames and the reference pixel positions of the alarm picture detection frames in each channel. If the difference between the pixel positions of the alarm picture detection frames and the reference pixel positions of the alarm picture detection frames exceeds the range of the preset difference, it is determined that the algorithm is inaccurate when playing the current playback channel number;

[0049] Compare whether the alarm message files in each channel are consistent with the reference alarm message files. If they are inconsistent, it is determined that the algorithm is inaccurate when playing the current playback channel number;

[0050] Compare whether the system resource occupancy rate in each channel is lower than the reference system resource occupancy rate. If the system resource occupancy rate is not lower than the reference system resource occupancy rate, it is determined that the algorithm is inaccurate when playing the current playback channel number;

[0051] Statistically calculate the accuracy rate of playing multiple video streams during each playback of the video source file, and obtain the algorithm accuracy rate based on the number of playback channels and the accuracy rate of multiple playbacks of the video source file.

[0052] In an exemplary embodiment, the statistical calculation of the accuracy rate of playing multiple video streams during each playback of the video source file includes:

[0053] Statistically calculate the ratio of the number of channels determined to have an accurate algorithm to the total number of playback channels during each playback of the video source file;

[0054] Among them, during the playback of each channel of the video source file, when the determination result of each algorithm index is accurate, it is determined that the algorithm is accurate when processing this channel of the video source file. When the determination result of any algorithm index is inaccurate, it is determined that the algorithm is inaccurate when processing this channel of the video source file.

[0055] In an exemplary embodiment, the AI platform includes a plurality of AI algorithms, and the plurality of AI algorithms include the data processing algorithm. Before the video source file is played multiple times in different numbers of channels on the AI platform, it further includes: automatically detecting a plurality of the AI algorithms before they are launched on the AI platform.

[0056] In an exemplary embodiment, the automatically detecting a plurality of AI algorithms before they are launched on the AI platform includes:

[0057] Automatically obtaining a plurality of AI algorithm codes to be tested from the algorithm code storage platform, checking various indicators of the plurality of AI algorithm codes, and if any abnormality occurs in the check of any indicator, calling the interface of the information feedback platform to send the corresponding abnormal information to the abnormal information feedback platform;

[0058] When no abnormality occurs in the check of various indicators of the plurality of algorithm codes, checking the startup status of the AI platform. When an abnormality occurs in the startup of the AI platform, sending the AI platform startup abnormal information to the abnormal information feedback platform;

[0059] When no abnormality occurs in the startup of the AI platform, controlling the AI platform to start a plurality of AI algorithms, checking the running status of the plurality of AI algorithms, and when an abnormality occurs in the running status of the plurality of AI algorithms, sending the information on the abnormal running status of the algorithm to the abnormal information feedback platform.

[0060] In an exemplary embodiment, after automatically obtaining a plurality of AI algorithm codes to be tested from the algorithm code storage platform, it further includes: generating a configuration benchmark file according to the plurality of AI algorithm codes, and generating a second configuration file based on the configuration benchmark file;

[0061] The configuration benchmark file includes the algorithm names of a plurality of algorithms, algorithm model parameters, path parameters of the database required for algorithm operation, resource configuration parameters, and video stream information as algorithm inputs. The video stream information includes the algorithm names of the plurality of AI algorithms, algorithm strategy information, and frame rate thresholds.

[0062] In an exemplary embodiment, the generating a second configuration file based on the configuration benchmark file includes: generating the content in the configuration benchmark file into a second configuration file in CSV format. The second configuration file includes algorithm basic information and algorithm input information. The basic information includes the algorithm name, algorithm model parameters, path parameters of the database required for algorithm operation, and resource configuration parameters; the algorithm input information includes the video stream information.

[0063] In an exemplary embodiment, the checking various indicators of the plurality of AI algorithm codes includes:

[0064] Check whether the algorithm name in the algorithm input information in the second configuration file is consistent with the algorithm name in the algorithm basic information. When it is found that the algorithm name in the algorithm input information in the second configuration file is inconsistent with the algorithm name in the algorithm basic information, feedback information about the abnormality of the second configuration file to the exception information feedback platform;

[0065] When it is checked that the algorithm name in the algorithm input information in the second configuration file is consistent with the algorithm name in the algorithm basic information, the detection platform obtains a compilation instruction from the algorithm code storage platform, and automatically calls a compilation interface to compile the multiple AI algorithms according to the compilation instruction; obtain a compilation log, and check whether there are errors in the compilation log. When it is checked that there are errors in the compilation log, feedback information about the compilation exception to the exception information feedback platform;

[0066] After it is checked that the algorithm name in the algorithm input information in the second configuration file is consistent with the algorithm name in the algorithm basic information, it further includes: the test platform checks whether the model files required for the AI algorithms to be launched this time are ready according to the second configuration file. If it is detected that the model files are not ready, feedback information about the model exception to the exception information feedback platform.

[0067] In an exemplary embodiment, checking the startup status of the AI platform includes:

[0068] Start the AI platform, wait for a first preset time, and check whether the process of the AI platform is started. When it is checked that the process of the AI platform is not started, send AI platform startup exception information to the exception information feedback platform; when it is checked that the process of the AI platform is started, start multiple AI algorithms on the AI platform and check the running status of the multiple AI algorithms.

[0069] In an exemplary embodiment, the AI platform starts multiple AI algorithms and checks the running status of the multiple AI algorithms, including:

[0070] Control the AI platform to start a thread group corresponding to multiple AI algorithms;

[0071] Control the AI platform to read the second configuration file, and add the AI algorithms to be detected recorded in the second configuration file to the thread group;

[0072] Control the AI platform to run multiple threads in the thread group. When any one of the AI algorithms in the multiple threads runs abnormally, send information about the abnormal operation of the corresponding AI algorithm to the exception feedback platform;

[0073] Start a summary thread, summarize the detection results, and send the summarized detection results to the exception feedback platform.

[0074] In an exemplary embodiment, when the operating states of multiple AI algorithms do not exhibit anomalies, it further includes: putting the multiple AI algorithm codes online on the AI platform.

[0075] In an exemplary embodiment, before automatically triggering the automated detection of multiple AI algorithms: it further includes triggering periodic detection.

[0076] The embodiments of the present disclosure also provide a test device for algorithm metrics, including a first acquisition module, a second acquisition module, and a third acquisition module;

[0077] The first acquisition module is used to acquire algorithm benchmark metrics;

[0078] The second acquisition module is used to control the AI platform to play the video source file multiple times with different numbers of playback channels, and acquire the algorithm metrics of the AI algorithm for processing each video stream during each playback of the video source file by the AI platform;

[0079] The third acquisition module is used to obtain the test result of the algorithm metrics based on the algorithm benchmark metrics acquired by the first acquisition module, the number of playback channels of the multiple playbacks of the video source file, and the algorithm metrics acquired by the second acquisition module.

[0080] The embodiments of the present disclosure also provide a camera access method, including: accessing one or more cameras on the AI platform according to the test result of the algorithm metrics obtained by the test method described in any of the above embodiments.

[0081] The embodiments of the present disclosure also provide a computer-readable storage medium, which is used to store computer program instructions, wherein the computer program instructions can implement the test method of the algorithm metrics described in any of the above embodiments when running.

[0082] Compared with the related art, a test method, device, camera access method, and medium for algorithm metrics provided by the embodiments of the present disclosure, in the test method of algorithm metrics, control the AI platform to play the video source file multiple times with different numbers of playback channels, and acquire the algorithm metrics of the data processing algorithm for processing each video source file during each playback of the video source file by the AI platform; obtain the test result of the algorithm metrics based on the algorithm benchmark metrics, the number of playback channels of the multiple playbacks of the video source file, and the acquired algorithm metrics. The obtained test result of the algorithm metrics can make a good plan in advance for the number of cameras actually accessed and the resource configuration of the AI platform, and overcome the problem that the video stream processing fails and the service hangs due to insufficient resources of the AI platform after the camera is accessed to the AI platform.

[0083] Other features and advantages of the embodiments of the present disclosure will be described in the subsequent description, and partly will be obvious from the description, or will be understood by implementing the embodiments of the present disclosure. Other advantages of the embodiments of the present disclosure can be realized and obtained by the solutions described in the description and the drawings. Description of the Drawings

[0084] The drawings are used to provide an understanding of the technical solutions of the embodiments of the present disclosure, and constitute a part of the description. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation to the technical solutions of the present disclosure.

[0085] Figure 1a The following shows a flowchart of a test method for algorithm indicators provided by an embodiment of the present disclosure;

[0086] Figure 1b The following shows a flowchart of an algorithm for obtaining and processing algorithm indicators for each video source file provided by an exemplary embodiment of the present disclosure;

[0087] Figure 2a The following shows a schematic diagram of the logical architecture of AI algorithm detection provided by an exemplary embodiment of the present disclosure;

[0088] Figure 2b The following shows a schematic diagram of a Jenkins framework structure provided by an exemplary embodiment of the present disclosure;

[0089] Figure 3 The following shows a flowchart of a method for checking the running status of an AI platform provided by an exemplary embodiment of the present disclosure;

[0090] Figure 4 The following shows a flowchart of a method for checking the running status of an AI algorithm provided by an exemplary embodiment of the present disclosure;

[0091] Figure 5 The following shows a schematic diagram of the logical structure of algorithm indicator testing provided by an exemplary embodiment of the present disclosure;

[0092] Figure 6a The following shows a logical framework diagram of video source processing provided by an exemplary embodiment of the present disclosure;

[0093] Figure 6b The following shows a logical framework diagram of video source processing provided by an exemplary embodiment of the present disclosure;

[0094] Figure 7 The following shows a flowchart of a method for obtaining test results of algorithm indicators provided by an exemplary embodiment of the present disclosure;

[0095] Figure 8 The following shows a flowchart of a method for testing AI algorithm indicators provided by an exemplary embodiment of the present disclosure;

[0096] Figure 9a The figure shows a curve graph of the algorithm accuracy rate and the number of camera channels provided by an exemplary embodiment of the present disclosure;

[0097] Figure 9b The figure shows a curve graph of the algorithm accuracy rate and the number of camera channels provided by an exemplary embodiment of the present disclosure;

[0098] Figure 9c The figure shows a curve graph of the algorithm accuracy rate and the number of camera channels provided by an exemplary embodiment of the present disclosure;

[0099] Figure 10 The figure shows a module diagram of a test device for algorithm metrics provided by an embodiment of the present disclosure. Detailed implementation manners

[0100] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, without conflict, the embodiments and features in the present disclosure can be combined with each other arbitrarily.

[0101] Unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood in the ordinary sense by those of ordinary skill in the art to which the present invention belongs. The terms "first", "second", and similar terms used in the embodiments of the present disclosure do not denote any order, quantity, or importance, but are only used to distinguish different components. The terms "comprising", "including", or similar terms are intended to cover the elements or items listed after such term and their equivalents, without excluding other elements or items.

[0102] In this specification, for convenience, terms indicating orientation or positional relationships such as "middle", "upper", "lower", "front", "rear", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are used to describe the positional relationships of the constituent elements with reference to the accompanying drawings. This is only for the convenience of describing this specification and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation on the present invention. The positional relationships of the constituent elements are appropriately changed according to the directions for describing the constituent elements. Therefore, it is not limited to the terms described in this specification, and can be appropriately replaced according to the circumstances.

[0103] In this specification, unless otherwise clearly defined and limited, the terms "installation", "connection", and "coupling" shall be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate member, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in combination with specific circumstances.

[0104] The AI algorithm runs on the AI platform. In the actual business scenario, one or more cameras may be connected. In practical applications, problems such as video stream processing failure and service crash often occur due to insufficient resources of the AI platform.

[0105] To solve the problems of video stream processing failure and service crash caused by insufficient resources of the AI platform, the embodiments of the present disclosure provide a method for testing algorithm metrics, which is set to test the algorithm metrics of the data processing algorithm in the AI platform. The testing method may include:

[0106] Obtain the algorithm benchmark metrics;

[0107] Control the AI platform to play the video source file multiple times with different numbers of playback channels, and obtain the algorithm metrics of the data processing algorithm for processing each video source file during each playback of the video source file by the AI platform;

[0108] Obtain the test result of the algorithm metrics based on the algorithm benchmark metrics, the number of playback channels of the multiple playbacks of the video source file, and the obtained algorithm metrics.

[0109] The method for testing algorithm metrics provided by the embodiments of the present disclosure controls the AI platform to play the video source file multiple times with different numbers of playback channels, and obtains the algorithm metrics of the data processing algorithm for processing each video source file during each playback of the video source file by the AI platform; the test result of the algorithm metrics is obtained based on the algorithm benchmark metrics, the number of playback channels of the multiple playbacks of the video source file, and the obtained algorithm metrics. The obtained test result of the algorithm metrics can plan in advance the number of actually connected cameras and the resource configuration of the AI platform, and overcome the problems of video stream processing failure and service crash caused by insufficient resources of the AI platform after the cameras are connected to the AI platform.

[0110] As Figure 1a shown, it is a flowchart of a method for testing algorithm metrics provided by an exemplary embodiment of the present disclosure. The method for testing algorithm metrics may include:

[0111] Step S0: Obtain the algorithm benchmark metrics;

[0112] Step S1: Control the AI platform to play the video source file multiple times with different numbers of playback channels, and obtain the algorithm metrics of the data processing algorithm for each channel of the video source file during each playback of the video source file;

[0113] Step S2: Obtain the test results of the algorithm metrics based on the algorithm benchmark metrics, the number of playback channels of the video source file played multiple times, and the obtained algorithm metrics.

[0114] In an exemplary embodiment, the algorithm metrics may include at least one of the following: the number of alarm messages, the average processing frame rate, the pixel positions of the detection frames of the alarm pictures, the system resource occupancy rate, the alarm message files;

[0115] Step S0 may include: playing the video source file multiple times through an offline algorithm, and obtaining the offline algorithm metrics output by the offline algorithm multiple times, calculating the average value of each offline algorithm metric obtained multiple times to obtain the algorithm benchmark metrics; or, Step S0 may include: obtaining the pre-stored algorithm benchmark metrics from a storage device.

[0116] The algorithm benchmark metrics may include at least one of the following: the number of benchmark alarm messages, the benchmark average processing frame rate, the benchmark pixel positions of the detection frames of the alarm pictures, the system resource benchmark occupancy rate, the benchmark alarm message files.

[0117] In the embodiments of the present disclosure, in the above Step S1, controlling the AI platform to play the video source file multiple times with different numbers of playback channels can be executed in two ways: The first way is to control the AI platform to notify the streaming media service to convert the video source file into one or more video streams, and the AI platform plays the one or more video streams; The second way is to control the AI platform to directly obtain the corresponding video source file according to the address of the video source file, and the AI platform plays the video source file without the need for the streaming media service to transcode the video source file. In practical applications, in the scenario of actually accessing a camera, usually the video file of the camera is transcoded by the streaming media to form a video stream, and then handed over to the AI platform to play according to the video stream. Therefore, in the above Step S1, controlling the AI platform to play the video source file multiple times with different numbers of playback channels, processing in the first way is closer to the actual application scenario of the camera accessing the AI platform, which can make the test results more accurate. Processing in the second way does not require the streaming media service, which can reduce the test cost and improve the test efficiency to a certain extent. However, due to the difference from the actual application scenario of the camera accessing, the final test result is not as accurate as the first way.

[0118] In an exemplary embodiment, for the video source file played multiple times in step S1, the same video source file is played to ensure that the video source files played in different channels are the same, thereby making the test results of algorithm metrics more accurate; in step S1, when the number of channels for playing the video source file in one play is multiple, multiple video source files can be played simultaneously, and the multiple video source files played are the same, so as to ensure that the video source files played in multiple channels are the same and make the test results of algorithm metrics more accurate.

[0119] In an exemplary embodiment, step S1 may include:

[0120] Set the initial values of the total number of play channels and the number of play channels;

[0121] Generate a first configuration file for the AI platform to access the video stream of the current number of play channels and make the first configuration file take effect; the first configuration file includes the video stream address and the algorithm information for processing the video stream;

[0122] Send an instruction to the streaming media service according to the video stream address in the first configuration file, and the instruction contains the current number of play channels;

[0123] The streaming media service receives the instruction, obtains the video source file, converts the video source file into a video stream of the current number of play channels according to the instruction, and sends the video stream of the current number of play channels to the AI platform; the AI platform includes: a decoding processor and a data processing algorithm;

[0124] The decoding processor in the AI platform converts the video stream of the current number of play channels into picture frames and transmits the picture frames to the data processing algorithm corresponding to the algorithm information for processing the video stream in the first configuration file;

[0125] The data processing algorithm in the AI platform plays the video stream of the current number of play channels according to the picture frames and outputs the processing results of each video stream, and uses the processing results as algorithm metrics;

[0126] Update the value of the current number of play channels, and determine whether the updated value of the current number of play channels exceeds the total number of play channels. When the updated value of the current number of play channels does not exceed the total number of play channels, continue to send an instruction to the streaming media service according to the video stream address in the first configuration file.

[0127] In an exemplary embodiment, as Figure 1b shown, step S1 may include:

[0128] Step A1: Set the initial values of the total number of play channels and the number of play channels;

[0129] Step A2: Generate a first configuration file for the AI platform to access the video stream of the current number of play channels and make the first configuration file take effect; the first configuration file includes the video stream address and the algorithm information for processing the video stream;

[0130] Step A3: Send an instruction to the streaming media service according to the video stream address in the first configuration file. The instruction includes the current number of playing channels;

[0131] Step A4: The streaming media service receives the instruction, obtains the video source file, converts the video source file into a video stream of the current number of playing channels according to the instruction, and sends the video stream of the current number of playing channels to the AI platform; The AI platform includes: a decoding processor and a data processing algorithm;

[0132] Step A5: The decoding processor in the AI platform converts the video stream of the current number of playing channels into picture frames, and transmits the picture frames to the data processing algorithm corresponding to the algorithm information for processing the video stream in the first configuration file;

[0133] Step A6: The data processing algorithm in the AI platform plays the video stream of the current number of playing channels according to the picture frames, and outputs the processing results of each video stream, and uses the processing results as algorithm metrics;

[0134] Step A7: Update the value of the current number of playing channels, and determine whether the updated value of the current number of playing channels exceeds the total number of playing channels. When the updated value of the current number of playing channels does not exceed the total number of playing channels, execute Step A3.

[0135] In an exemplary embodiment, before Step A5, it further includes: controlling the AI platform to decode the video stream of the current number of playing channels to obtain picture data, and obtaining the average processing speed of the AI platform for decoding each video stream;

[0136] In Step A5, the decoding processor converts the video stream of the current number of playing channels into picture frames, which may include: the decoding processor converts the picture data of the video stream of the current number of playing channels into picture frames.

[0137] In the embodiments of the present disclosure, the average processing speed of the AI platform for decoding each video stream can be understood as the average frame rate. The frame rate is the number of picture frames transmitted in 1 second, and can also be understood as the number of times the graphics processor can refresh per second, usually expressed in fps (Frames Per Second, the number of picture updates per second).

[0138] In an exemplary embodiment, when it is determined in Step A7 that the updated value of the current number of playing channels exceeds the total number of playing video streams, it further includes: obtaining the test result of the AI platform index according to the decoding benchmark index pre-stored in the AI platform, the number of video stream channels during multiple playbacks of the video source file, and the average processing speed of the AI platform for each video stream.

[0139] In an exemplary embodiment, according to the decoding benchmark metrics of the AI platform, the number of video streams during multiple plays of the video source file, and the average processing speed of the AI platform for each video stream obtained, the test result of the AI platform metrics may include steps E1 - E2:

[0140] Step E1: Determine whether the average decoding processing frame rate of the AI platform for each video source file is lower than the decoding benchmark metric during each play of the video source file. If so, it is determined that the video source file for the current play number is not supported; otherwise, it is determined that the video source file for the current play number is supported.

[0141] Step E2: Based on the result of whether the AI platform supports the play number of the video source file for each play and the play number for each play, obtain the test result of the AI platform metrics.

[0142] In an exemplary embodiment, the test result of the AI platform metrics may include a relationship curve graph of the average decoding processing frame rate of the AI platform for each video stream and the number of play paths.

[0143] In an exemplary embodiment, the first configuration file at least includes: the above video stream address, the frame rate of playing the video stream, and the algorithm information for processing the video stream.

[0144] In the above step A6, the data processing algorithm in the AI platform plays the video stream of the current play number according to the picture frame, which may include: the data processing algorithm in the AI platform plays the video stream of the current play number at the frame rate of playing the video stream in the first configuration file according to the picture frame.

[0145] In an exemplary embodiment, to make the configuration file take effect, it may include: controlling the AI platform to restart to make the first configuration file take effect, or transmitting the activation parameter to the AI platform to make the first configuration file dynamically take effect.

[0146] In an exemplary embodiment, in step A4, before the streaming media service receives the instruction and converts the video source file into the video stream of the current play number according to the instruction, it further includes: restarting the streaming media service.

[0147] In an exemplary embodiment, the above step S2 may include: during each play of the video source file, comparing the number of alarm messages in each path with the benchmark number of alarm messages, and / or comparing the pixel positions of the alarm picture detection frames in each path with the benchmark pixel positions of the alarm picture detection frames, and statistically calculating the algorithm accuracy rate of each play of the video source file according to the comparison results; obtaining the test result of the algorithm metrics based on the algorithm accuracy rates statistically calculated from multiple plays of the video source file.

[0148] In an exemplary embodiment, before comparing the number of alarm messages in each channel with the reference number of alarm messages and / or comparing the pixel positions of the alarm picture detection frames in each channel with the reference pixel positions of the alarm picture detection frames during each playback of the video source file, the following steps may further be included:

[0149] Determine whether the average processing frame rate of multiple channels is lower than the reference average processing frame rate during each playback of the video source file. When the average processing frame rate of multiple channels is lower than the reference average processing frame rate, determine that the number of playback channels supported for this playback is not supported. When the average processing frame rate of multiple channels is not lower than the reference average processing frame rate, determine that the number of playback channels supported for this playback is supported; compare whether the system resource occupancy rate in each channel is lower than the reference system resource occupancy rate;

[0150] During each playback of the video source file, when comparing the number of alarm messages in each channel with the reference number of alarm messages and / or comparing the pixel positions of the alarm picture detection frames in each channel with the reference pixel positions of the alarm picture detection frames, the algorithm accuracy rate for each playback of the video source file can be statistically calculated based on the comparison results, which may include: calculating the difference between the number of alarm messages in each channel and the reference number of alarm messages, and the difference between the pixel positions of the alarm picture detection frames in each channel and the reference pixel positions of the alarm picture detection frames during each playback of the video source file, and comparing whether the alarm message files in each channel are consistent with the reference alarm message files. Based on the judgment, calculation, and comparison results, the algorithm accuracy rate for each playback of the video source file is statistically calculated.

[0151] In an exemplary embodiment, the above step S2 may include:

[0152] Determine whether the average processing frame rate of multiple channels is lower than the reference average processing frame rate during each playback of the video source file. When the average processing frame rate of multiple channels is lower than the reference average processing frame rate, determine that the number of playback channels supported for this playback is not supported. When the average processing frame rate of multiple channels is not lower than the reference average processing frame rate, determine that the number of playback channels supported for this playback is supported;

[0153] Calculate the difference between the number of alarm messages in each channel and the reference number of alarm messages, and the difference between the pixel positions of the alarm picture detection frames in each channel and the reference pixel positions of the alarm picture detection frames during each playback of the video source file, and compare whether the alarm message files in each channel are consistent with the reference alarm message files and whether the system resource occupancy rate in each channel is lower than the reference system resource occupancy rate. Based on the judgment, calculation, and comparison results, the algorithm accuracy rate for each playback of the video source file is statistically calculated;

[0154] Obtain the test result of the algorithm index based on the algorithm accuracy rate statistically calculated from multiple playbacks of the video source file.

[0155] In an exemplary embodiment, the test results of the algorithm metrics obtained based on the algorithm accuracy rate statistically calculated for multiple plays of the video source file may include at least one of the following: a relationship curve graph of the number of playing channels and the average frame rate, a relationship curve graph of the number of playing channels and the system resource occupancy rate, a relationship curve graph of the number of playing channels and the algorithm accuracy rate; the test results of the algorithm metrics may include at least one of the following: a relationship curve graph of the number of playing channels and the average frame rate, a relationship curve graph of the number of playing channels and the system resource occupancy rate, a relationship curve graph of the number of playing channels and the algorithm accuracy rate.

[0156] In an exemplary embodiment, the above calculation may include, during each play of the video source file, calculating the difference between the number of alarm messages and the reference number of alarm messages in each channel, the difference between the pixel positions of the alarm picture detection frames and the reference pixel positions of the alarm picture detection frames in each channel, and comparing whether the alarm message files in each channel are consistent with the reference alarm message files:

[0157] During each play of the video source file, calculate the difference between the number of alarm messages and the reference number of alarm messages in each channel. If the difference between the number of alarm messages and the reference number of alarm messages exceeds the preset difference, it is determined that the algorithm is inaccurate when playing the current playing channel.

[0158] And / or, calculate the difference between the pixel positions of the alarm picture detection frames and the reference pixel positions of the alarm picture detection frames in each channel. If the difference between the pixel positions of the alarm picture detection frames and the reference pixel positions of the alarm picture detection frames exceeds the range of the preset difference, it is determined that the algorithm is inaccurate when playing the current playing channel.

[0159] And / or, compare whether the alarm message files in each channel are consistent with the reference alarm message files. If they are inconsistent, it is determined that the algorithm is inaccurate when playing the current playing channel.

[0160] The above comparison of whether the system resource occupancy rate in each channel is lower than the reference system resource occupancy rate may include: comparing whether the system resource occupancy rate in each channel is lower than the reference system resource occupancy rate. If the system resource occupancy rate is not lower than the reference system resource occupancy rate, it is determined that the algorithm is inaccurate when playing the current playing channel.

[0161] The above statistical calculation of the algorithm accuracy rate for each play of the video source file based on the judgment, calculation, and comparison results may include: statistically calculating the accuracy rate of playing multiple video streams during each play of the video source file, and obtaining the algorithm accuracy rate based on the number of playing channels and the accuracy rate for multiple plays of the video source file.

[0162] In an exemplary embodiment, the system resource occupancy rate may be the occupancy rate of the CPU, GPU, etc.

[0163] In an exemplary embodiment, to statistically analyze the accuracy rate of playing multiple video streams during each playback of a video source file, it may include:

[0164] Statistically analyze the ratio of the number of accurately determined paths by the determination algorithm to the total number of played paths during each playback of the video source file;

[0165] Among them, during the playback of each video source file, when the determination result of each algorithm index is that the algorithm is accurate, it is determined that the algorithm is accurate when processing this video source file; when the determination result of any algorithm index is that the algorithm is inaccurate, it is determined that the algorithm is inaccurate when processing this video source file.

[0166] When an AI algorithm is launched on an Artificial Intelligence (AI) platform, it usually has the following characteristics: when launched in batches, the number of algorithms is uncertain, sometimes there are more than a dozen; the launch frequency is high, and developers may be ready to launch at any time during a day; the algorithm development sites and personnel are relatively dispersed, such as in sites in Shanghai, Beijing, or overseas. Based on the characteristics of algorithm launch, using the manual launch detection method has the defects of a relatively long time for troubleshooting problems, a relatively high labor cost, a long problem feedback cycle, and a long problem regression feedback cycle.

[0167] In an exemplary embodiment, to solve the problems of a long troubleshooting time and a high labor cost in the existing AI algorithm launch, before playing the video source file multiple times with different numbers of paths on the AI platform, it may further include: performing automated detection on multiple AI algorithms before they are launched on the AI platform, where the AI platform includes multiple AI algorithms, and the multiple AI algorithms include the above data processing algorithms.

[0168] In an exemplary embodiment, performing automated detection on multiple AI algorithms before they are launched on the AI platform may include:

[0169] Automatically obtain the code of multiple AI algorithms to be tested from the algorithm code storage platform, perform checks on various indicators of the multiple AI algorithm codes, and if any abnormality is found in any indicator check, call the interface of the information feedback platform to send the corresponding abnormal information to the abnormal information feedback platform;

[0170] When no abnormality is found in the checks on various indicators of the multiple algorithm codes, check the startup status of the AI platform. When an abnormality is detected in the startup of the AI platform, send the AI platform startup abnormal information to the abnormal information feedback platform;

[0171] When no abnormality is found in the startup of the AI platform, control the AI platform to start multiple AI algorithms, check the running status of the multiple AI algorithms, and when an abnormality occurs in the running status of the multiple AI algorithms, send the information about the abnormal running status of the algorithms to the abnormal information feedback platform.

[0172] In an exemplary embodiment, after automatically obtaining multiple AI algorithm codes to be tested from an algorithm code storage platform, it may further include: generating a configuration benchmark file according to the multiple AI algorithm codes, and generating a second configuration file based on the configuration benchmark file;

[0173] The configuration benchmark file includes the algorithm names of multiple algorithms, algorithm model parameters, path parameters of the database required for algorithm operation, resource configuration parameters, and video stream information serving as algorithm input. The video stream information includes the algorithm names of multiple AI algorithms, algorithm strategy information, and frame rate thresholds.

[0174] In an exemplary embodiment, generating a second configuration file based on the configuration benchmark file may include: generating the content in the configuration benchmark file into a second configuration file in CSV format. The second configuration file includes algorithm basic information and algorithm input information. The basic information includes algorithm names, algorithm model parameters, path parameters of the database required for algorithm operation, and resource configuration parameters; the algorithm input information includes video stream information.

[0175] In an exemplary embodiment, performing checks on various indicators of multiple AI algorithm codes may include:

[0176] Checking whether the algorithm name in the algorithm input information in the second configuration file is consistent with the algorithm name in the algorithm basic information. When it is checked that the algorithm name in the algorithm input information in the second configuration file is inconsistent with the algorithm name in the algorithm basic information, feedback the information of the second configuration file exception to the exception information feedback platform;

[0177] When it is checked that the algorithm name in the algorithm input information in the second configuration file is consistent with the algorithm name in the algorithm basic information, the detection platform obtains a compilation instruction from the algorithm code storage platform, and automatically calls a compilation interface to compile multiple AI algorithms according to the compilation instruction; obtains a compilation log, and checks whether there are errors in the compilation log. When it is checked that there are errors in the compilation log, feedback the information of compilation exception to the exception information feedback platform;

[0178] After it is checked that the algorithm name in the algorithm input information in the second configuration file is consistent with the algorithm name in the algorithm basic information, it further includes: the test platform checks whether the model files required for the AI algorithms to be launched this time are prepared correctly according to the second configuration file. If it is detected that the model files are not prepared well, feedback the information of model exception to the exception information feedback platform.

[0179] In an exemplary embodiment, checking the startup status of the AI platform may include:

[0180] Start the AI platform and wait for the first preset time. Check whether the process of the AI platform has started. When it is checked that the process of the AI platform has not started, send the AI platform startup exception information to the exception information feedback platform; when it is checked that the process of the AI platform has started, start multiple AI algorithms on the AI platform and check the running status of the multiple AI algorithms.

[0181] In an exemplary embodiment, starting multiple AI algorithms on the AI platform and checking the running status of the multiple AI algorithms includes:

[0182] Control the AI platform to start the thread groups corresponding to multiple AI algorithms;

[0183] Control the AI platform to read the second configuration file and add the AI algorithms recorded in the second configuration file that need to be detected to the thread groups;

[0184] Control the AI platform to run multiple threads in the thread groups. When any one of the multiple AI algorithms runs abnormally, send the information of the corresponding AI algorithm running abnormally to the exception feedback platform;

[0185] Start the summary thread, summarize the detection results, and send the summarized detection results to the exception feedback platform.

[0186] In the embodiments of the present disclosure, by automatically checking the running status of multiple AI algorithms (i.e., batch algorithms), during the detection process before the batch algorithms are put on the line, there is no need for manual online detection, avoiding the defects of long time for algorithm detection and troubleshooting, long problem feedback cycle, and long problem regression feedback cycle, and reducing the labor cost, which greatly improves the detection efficiency.

[0187] In an exemplary embodiment, when the running status of multiple AI algorithms does not show any abnormality, it further includes: putting the multiple AI algorithm codes on the line on the AI platform.

[0188] In an exemplary embodiment, before automatically triggering the automated detection of multiple AI algorithms: it further includes triggering periodic detection.

[0189] In the embodiments of the present disclosure, the above operation mode can be controlled by a test script, and the test script can run on a test platform.

[0190] In an exemplary embodiment, as Figure 2a shown, it is a schematic diagram of the logical architecture of AI algorithm detection. The jenkins periodic trigger automated detection platform (hereinafter referred to as the detection platform) can be an operation and maintenance platform or a test platform, or other platforms for testing or online detection. In Figure 2aIn the logical framework shown, after the R & D personnel complete the model development and strategy development, they submit the algorithm to the code storage server. The operation and maintenance platform or the test platform periodically triggers the automated detection of the algorithm in the code storage server through Jenkins. The algorithm detection content can include configuration detection, compilation detection, model detection, startup status detection of the AI platform, and algorithm operation status detection.

[0191] As Figure 2b shown, the Jenkins framework can include six configuration modules:

[0192] General module: Some basic configurations for building tasks, discarding old builds, setting the save policy for build history; Select to set the parameterized build process, and different parameters can be configured to facilitate the reference of these parameters during the build.

[0193] Source code management module: Select GIT and set the corresponding GIT parameters. In an exemplary embodiment, setting the GIT parameters can be setting the GIT address, and the GIT address can be the SVN address for accessing the code storage server.

[0194] Build trigger module: Select timed build and set the corresponding time parameters. After the build trigger module, the test can be triggered periodically.

[0195] Build environment module: Select the build tool of Delete workspace before build starts.

[0196] Build module: Usually, the build module environment is to write an execution file. The present disclosure embodiment does not make settings for this module.

[0197] Post-build operation module: It is implemented by adopting the method of designing call commands and writing scripts.

[0198] According to the period set by the build trigger module and the GIT parameters set by the source code management module, the algorithm code is periodically pulled from the GIT address for testing.

[0199] The following details the algorithm detection method:

[0200] (1) Jenkins automatically pulls the git code: When the test period arrives, the periodic automated detection is triggered, and the detection platform automatically pulls the algorithm code corresponding to the git address from the code storage server through Jenkins. Each algorithm code in the code storage server corresponds to a git address, and the detection platform can access the corresponding algorithm code in the code storage server through the git address. In an exemplary embodiment, the code storage server can be referred to as the code storage platform.

[0201] In an exemplary embodiment, the operation and maintenance platform obtains the corresponding algorithm code through the git address. When batch algorithms are launched or batch algorithm detections are performed, jenkins can obtain the git address corresponding to the batch algorithm from the code storage server, and obtain the corresponding multiple algorithm codes according to the git address, so as to achieve the launch of batch algorithms or batch algorithm detections. In an exemplary embodiment, the same git address can correspond to multiple algorithms in the batch algorithm, or each algorithm corresponds to a git address.

[0202] (2) Generate a configuration benchmark file: Generate a configuration benchmark file according to the algorithm code.

[0203] In an exemplary embodiment, the configuration benchmark file may include the algorithm names, algorithm model parameters, path parameters of the database required for algorithm operation, resource configuration parameters, and video stream information as the algorithm input in multiple algorithms in the batch algorithm. The video stream information includes information such as the corresponding algorithm name, algorithm policy information, and frame rate threshold. In an exemplary embodiment, the configuration benchmark file may also include relevant information of the algorithm R & D person in charge and the platform person in charge.

[0204] In an exemplary embodiment, the resource configuration parameters may include occupying resources such as CPU and GPU. For example, an algorithm needs to occupy 100M of CPU space and 50M of GPU space.

[0205] (3) Generate a configuration file in CSV format based on the configuration benchmark file.

[0206] In an exemplary embodiment, in order to meet the format requirements of the detection platform for the configuration file, a configuration file in CSV format is generated according to the configuration benchmark file, and the configuration file in CSV format is used as the standard in the subsequent detection process.

[0207] In an exemplary embodiment, the configuration file in CSV format may include two parts arranged in sequence. The first part may include the basic information of multiple algorithms. The basic information may include the above-mentioned algorithm names, algorithm model parameters, and path parameters of the database required for algorithm operation. The second part may include algorithm input information, and the algorithm input information includes the above-mentioned video stream information. Among them, the basic information of multiple algorithms in the first part may be arranged in sequence, and the algorithm input information of multiple algorithms in the second part may be arranged in sequence.

[0208] (4) CSV generation check: Check whether the configuration file in CSV format is in the standard format specified by the detection platform. If it is not in the standard format specified by the detection platform, call the JIRA interface to feedback the configuration bug of the corresponding algorithm.

[0209] In an exemplary embodiment, comma-separated values (CSV), sometimes also referred to as character-separated values because the separator character can be other than a comma, store tabular data (numbers and text) in plain text form. Plain text means that the file is a sequence of characters and does not contain data that must be interpreted like binary digits. A CSV file consists of any number of records separated by some line break character; each record consists of fields separated by other characters or strings, most commonly a comma or a tab.

[0210] In an exemplary embodiment, CSV checking may include checking whether a configuration file in CSV format conforms to the format requirements of a standard configuration file. For example, if the format of the standard configuration file is that records are separated by commas, and it is found that records in the CSV format configuration file are separated by semicolons, then an exception occurs in CSV generation.

[0211] (5) Configuration checking may include: checking whether the algorithm name in the basic information of the configuration file is the same as the algorithm name in the algorithm input information. If they are not the same, then call the JIRA interface to report the configuration bug of the corresponding algorithm.

[0212] In an exemplary embodiment, for some algorithms that do not require an input video stream, the algorithm name in the basic information can be marked as having no input video stream information. When it is detected that the algorithm name is marked as having no input video stream information, it can be determined according to this marking that no exception has occurred, and the JIRA interface may not be called to report the bug of the corresponding algorithm.

[0213] In an exemplary embodiment, for some multi-algorithms, even though they do not use video stream information as input during actual operation, video stream information is configured during model development and strategy development, but the corresponding video stream resources are not used during operation. In this case, if the video stream information corresponding to the algorithm name is not detected during configuration checking, the JIRA interface can be called to report the bug of the corresponding algorithm.

[0214] In an exemplary embodiment, a bug is a general term for vulnerabilities, defects, and error problems in software, programs, code, algorithms, and computer systems.

[0215] (6) Compile code: Jenkins calls the compilation interface to compile the algorithm code according to the compilation instructions.

[0216] In an exemplary embodiment, Jenkins obtains the compilation instructions for the corresponding algorithm from the git address and automatically calls the compilation interface to compile the algorithm code, which can reduce manual deployment of the compilation environment and the manual compilation process, thereby reducing labor costs and improving efficiency.

[0217] (7) Compilation checks may include: checking whether there are any errors during the algorithm compilation process and whether the algorithm compilation result is successful. If there are errors during the compilation process or the compilation result is not successful, the JIRA interface is called to report the compilation bugs of the corresponding algorithm. In an exemplary embodiment, checking whether there are any errors during the algorithm compilation process and whether the algorithm compilation result is successful may include: obtaining the jenkins compilation log and checking whether there are any errors in the compilation log. For example, checking whether there is information such as "error" in the compilation log.

[0218] (8) Model checks may include: according to the configuration file, checking whether the model files required for the algorithm to be launched this time are ready correctly. If it is detected that the model files are not ready, the JIRA interface is called to report the model bugs of the corresponding algorithm.

[0219] In an exemplary embodiment, checking whether the model files required for the algorithm to be launched this time are ready correctly may include: finding whether the model files corresponding to the corresponding algorithm exist according to the model parameters in the configuration file.

[0220] In an exemplary embodiment, during the above CSV generation check, configuration check, and model check, once an exception occurs, the interface service of JIRA is called to automatically submit the corresponding bug to the JIRA server. The JIRA server displays the corresponding bug to the user through the browser, and the corresponding developer can view the corresponding bug through the corresponding browser.

[0221] During the manual go-live process, after problems occur in operation and maintenance or testing, it is usually the operation and maintenance or testing personnel who communicate with the developers. Moreover, the operation and maintenance or testing personnel do not fully understand where the bugs in the development occur, resulting in a high communication cost. In the embodiments of the present disclosure, the bug information is uploaded to the JIRA server through the JIRA interface, and the developers, platform leaders, test or operation and maintenance personnel can view the corresponding bug information by logging in to the JIRA account, which greatly reduces the communication cost. In an exemplary embodiment, Jenkins is an open-source continuous integration (CI) tool with a friendly operation interface, mainly used for continuously and automatically building / testing software projects and monitoring the operation of external tasks.

[0222] (8) AI platform operation status check.

[0223] As Figure 3 shown, the AI platform operation status check may include the following steps:

[0224] Step 11: Start the AI platform and wait for the first preset time to execute Step 12.

[0225] In an exemplary embodiment, the first preset time may be from 1 minute to 5 minutes. For example, the first preset time may be 3 minutes.

[0226] In an exemplary embodiment, the AI platform may be started after the code compilation is completed. After the compilation check and model check are executed, the check of the running state of the AI platform is started. From the start of the execution of the running state check of the AI platform, wait for the first preset time to execute step 12.

[0227] Step 12: Check whether the AI platform service exists. If it exists, the check is completed; otherwise, execute step 13.

[0228] In an exemplary embodiment, to check whether the AI platform service exists, it may be checked whether the process of the AI platform is started. If the process is not started, execute step 13; if the process has been started, the check is completed.

[0229] Step 13: Link the JIRA interface and submit a bug.

[0230] In step 13, the abnormal start of the AI platform is submitted to the JIRA server through the JIRA interface. The user (the person in charge of the AI platform or the developer) can log in to the JIRA server to view the corresponding bug to solve the corresponding problem. In the embodiment of the present disclosure, the JIRA server may be the above-mentioned abnormal information feedback platform.

[0231] (9) Check the running state of the algorithm.

[0232] As Figure 4 shown, the check of the running state of the AI algorithm may include the following steps:

[0233] Step 21: Start the thread group corresponding to the algorithm.

[0234] In an exemplary embodiment, after the process in the AI platform is started, the operation of starting the algorithm can be executed.

[0235] In an exemplary embodiment, starting the algorithm may start threads corresponding to the number of algorithms after the AI platform process is started. When starting multiple algorithms, each algorithm corresponds to a thread, and a thread group of multiple threads is started in the process.

[0236] Step 22: Read the configuration file and add the algorithms marked as needing to be detected in the configuration file to the thread group of the AI platform.

[0237] In an exemplary embodiment, during batch algorithm testing or the process of rolling out a batch algorithm, due to limited thread group resources, only a part of the algorithms can be added to the current thread group, and the remaining algorithms can be added to other thread groups or tested during the next test. In an exemplary embodiment, the algorithms recorded in the configuration file can be defaulted to require detection, and no flag indicating whether detection is required is set.

[0238] In an exemplary embodiment, each algorithm is loaded into one of the threads in the thread group, that is, each algorithm can correspond to one thread.

[0239] Step 23: Run multiple threads in the thread group. When any one of the AI algorithms in the multiple threads runs abnormally, send information about the abnormal operation of the corresponding AI algorithm to the exception information feedback platform.

[0240] In an exemplary embodiment, when an exception occurs in the detection algorithm, it will link with the JIRA interface, submit a bug and feedback it to the JIRA service platform (i.e., the exception feedback platform). The algorithm responsible person can log in to the JIRA server, view the JIRA bug, and handle the corresponding algorithm exception.

[0241] In an exemplary embodiment, after the algorithm detection is completed, the output result of the algorithm can be obtained when no exception occurs in the detection result.

[0242] Step 24: Start the summary thread, summarize the detection results and feedback them to the JIRA platform through the JIRA interface.

[0243] In an exemplary embodiment, the configuration file can include the email addresses of the R & D responsible person and the AI platform responsible person. After the JIRA platform receives the corresponding bug, it can send the corresponding bug information to the corresponding R & D responsible person or AI platform responsible person through the email address.

[0244] In an exemplary embodiment, the summary thread feeds the total detection results back to the JIRA server through the JIRA interface. The AI platform responsible person logs in to the JIRA server to obtain the detection results, and confirms whether the algorithm rollout result meets the expectations according to the detection results. In an exemplary embodiment, the total detection results can include: how many algorithm codes are detected in total, the number of successfully tested algorithms, the number of failed algorithm tests, the success list, and the failure list.

[0245] In an exemplary embodiment, if there is a bug during the process of the thread executing the algorithm detection, it is considered that the test is unsuccessful, and the corresponding bug information is uploaded to the JIRA platform through JIRA.

[0246] In an exemplary embodiment, the success list contains a list of algorithms that have passed the algorithm test, and the failure list contains a list of algorithms that have failed the algorithm test.

[0247] In an exemplary embodiment, the person in charge of the AI platform can confirm whether the result of the algorithm going online meets the expectation according to the detection result, and can judge according to the type of the algorithm with detection failure or success. For example, there are a total of 21 algorithms for batch testing. If there is an abnormality in the testing of one algorithm, and through the evaluation of the person in charge of the platform, the algorithm with the abnormality is not the one that must go online this time, then only 20 algorithms with successful detection can be put online, and the algorithm testing this time meets the expectation; if there are a total of 21 algorithms being detected and 10 algorithms that must go online are detected with abnormalities, then it does not meet the expectation and cannot be put online. The corresponding R & D person in charge needs to resolve the corresponding bugs and then retest, that is, repeat the detection process from (1) to (9) above until it meets the test expectation before going online.

[0248] In an exemplary embodiment, automatically submitting the detection result to the JIRA platform server can achieve the role of a pipeline, eliminating the need for manual operation and saving labor costs.

[0249] In an exemplary embodiment, the detection result can include a detection log and an exception record. The detection log can include the detection time and the above-mentioned success list and failure list. For example, the detection log is as follows:

[0250] 2021-10-18 16:10:25[model_repository2]auto test end!total:16failed:7

[0251] FAILED LIST:['highway_lowspeed','drive_without_license',

[0252] drive_license_without_permission’,'drive_inout','driver_car_match',

[0253] 'station_leave','wandering_alarm']

[0254] NEW JIRA LIST:[]

[0255] YF2021430-131

[0256] The above detection log records that the detection end time is 16:10:25 on October 18, 2021, the total number of detections is 16, and the number of failures is 7. The algorithms with detection failures in the failure list include:

[0257] 'highwayjowspeed','drive_withoutjicense','drive_license_without_permission',

[0258] 'drivejncut','driver_car_match','stationjeave','wandering_alarm'

[0259] The exception record summary includes:

[0260] [AI300OnlineCheck:C-Video][check.ConfigCheckLog]ERRORBUG exists invehiclebreakin

[0261] [AI300OnlineCheck:C-Video][CHECK_CompileCheckLog]ERRORBUG exists inNonVehiclelllegalParkingDetect

[0262] [Al300OnlineCheck:C-Video][check_ConfigCheckLog]ERRORBUG exists invehiclebreakin

[0263] In an exemplary embodiment, the Jenkins integration can be set up to run automatic online detection at a scheduled time to improve detection efficiency. For example, the online detection service can be set up in Jenkins to run periodically and automatically at 11:30 a.m. and 4:30 p.m. every weekday to facilitate the algorithm to go online in the morning or afternoon.

[0264] In the embodiments of the present disclosure, the algorithm runs on an AI platform to provide a message interface for services. In actual business scenarios, one or more cameras may need to be connected. If the platform resources are insufficient, problems such as video stream processing failure and service crash may occur. To avoid problems such as video stream processing failure and service crash caused by insufficient platform resources after connection, before connecting multiple cameras to the AI platform after the algorithm detection shows no abnormality and the algorithm is successfully launched, the algorithm metrics when multiple cameras are connected to the AI platform can be tested. In an exemplary embodiment, the algorithm metrics when N cameras are connected to the AI platform under a single-card / single-machine configuration can be tested, and a curve relationship diagram between the algorithm metric values and the number of camera channels under the existing service configuration of the platform can be obtained, which has data significance for the early planning and design of product implementation and resource allocation. In the embodiments of the present disclosure, a single card may refer to a Graphics Processing Unit (GPU), also known as a display core, visual processor, or display chip, and a single machine may be a physical machine configured with multiple GPU cards.

[0265] In the embodiments of the present disclosure, the logic of algorithm metric testing is as Figure 5 shown. The following describes the video stream, the AI service platform, and the metric item data:

[0266] Video stream: The input source of the AI platform service. Multiple video streams can be simulated by using video files, or one video stream can be converted into multiple streams for simulation.

[0267] In an exemplary embodiment, one video file can be copied into N copies, and each of the N video files can be transcoded to form N video streams; or one video file can be transcoded to form a video stream, and the video stream can be copied into N copies to form N video streams.

[0268] AI platform service: An algorithm service based on the AI platform framework. Its input is one or multiple video streams; the output includes frame rate, number of processed messages, message files, system resource occupancy (such as CPU / GPU occupancy rate), etc. The AI platform service includes functions such as video stream decoding, algorithm processing, recording, and output of metric data.

[0269] Metric item data: The metric item output required when the AI platform service processes N streams. Taking the perimeter intrusion algorithm as an example, the required output needs to include the number of alarm messages, average processing frame rate (fps), pixel positions of the detection boxes in the alarm pictures, and system resource occupancy rate (CPU / GPU).

[0270] In an exemplary embodiment, as Figure 6a and Figure 6b shown, the following are the logic framework diagrams for processing two video sources. As Figure 6a shown, using a video file as the video source:

[0271] Streaming media service: Provides video file transcoding service. It can transcode a video file into N video streams according to specified requirements. The transcoded video streams are used as the video stream input for the AI platform service.

[0272] AI platform service: For Figure 5 the services provided by the AI service platform shown, for specific services, refer to the description of the above AI service platform and will not be elaborated here.

[0273] Result data processing: Perform data processing on the output of the AI platform service to obtain the corresponding index relationship diagram.

[0274] As Figure 6b shown, for a real camera as the video source input:

[0275] Streaming media service: Provides transcoding service. It can transcode the video stream of a camera into N video streams according to specified requirements. The transcoded video streams are used as the video stream input for the AI platform service.

[0276] AI platform service: For Figure 5 the services provided by the AI service platform shown, for specific services, refer to the description of the above AI service platform and will not be elaborated here.

[0277] Result data processing: Perform data processing on the output of the AI platform service to obtain the corresponding index relationship diagram.

[0278] In an exemplary embodiment, the obtained index relationship diagram may include a precision - number of camera channels curve graph.

[0279] In the embodiments of the present disclosure, Figure 6a and Figure 6b for the result data processing described above, the implementation manner may adopt the form described in step S2 above, and the final result of the result data processing is the test result of the algorithm index.

[0280] In the embodiments of the present disclosure, video streams can be generated in a simulation manner, which has the following advantages compared with the video streams of real cameras:

[0281] (1) It can ensure that the input sources are consistent, and the obtained index conclusions are comparable.

[0282] (2) It can ensure that the density of a single - frame image meets specific requirements. For example, the number of people in a single - frame image needs to reach 30 people, and the index values for capacity testing can be obtained; while it is very difficult for a real camera to ensure the density of a single - frame image.

[0283] (3) It is easy to expand and build. It can compare the index values for N channels (such as 8 channels, 16 channels, 32 channels, 100 channels) according to actual demand changes.

[0284] Based on the above three points, when there are many routes to compare, it is difficult to quickly achieve the number of cameras, procurement, installation and crowd density simulation using real cameras.

[0285] In an exemplary embodiment, Figure 6a and Figure 6b In the two video stream simulation methods, the above Figure 6a The video stream obtained by using the video file is the same as the above Figure 6b Compared with the video stream simulated by a real camera, it is convenient to formulate scene videos that meet the single-frame image density.

[0286] In an exemplary embodiment, if Figure 7 As shown in the figure, taking the perimeter intrusion algorithm as an example, before N cameras are connected to the AI ​​platform, obtaining the algorithm indicator test results may include the following steps:

[0287] Step 31: Prepare the video files and benchmark values ​​required for the test.

[0288] In an exemplary embodiment, the video file may be a video in mp4 format or other playable formats.

[0289] In an exemplary embodiment, the benchmark indicator values ​​may include: the number of alarm messages, alarm picture frames, average processing frame rate (fps), pixel position of the alarm picture detection frame, and system resource occupancy (CPU / GPU).

[0290] In an exemplary embodiment, the method for obtaining the benchmark indicator value may include: directly inputting the video file into the offline algorithm, obtaining the number of alarm messages, the average processing frame rate, the pixel position of the alarm picture detection frame, and the system resource occupancy rate (CPU / GPU) output by the offline algorithm. This process may be performed multiple times and manually checked to obtain more accurate results.

[0291] Step 32: Provide the video file as a video source to a streaming service.

[0292] In an exemplary embodiment, the conversion of a video file into a stream may support the output of N video streams.

[0293] Step 33: Set the corresponding playback times and file storage rules.

[0294] In an exemplary embodiment, the corresponding number of playback times may be set to be played only once, without looping.

[0295] In an exemplary embodiment, setting the file storage rule may include setting the alarm message files to be all stored separately.

[0296] Step 34: Configure 1 to N cameras, start the AI platform to run and test the algorithm metrics of the AI platform to obtain output data.

[0297] Step 35: Process the output data.

[0298] In an exemplary embodiment, step 34 can be understood as testing the algorithm metrics for N cameras accessing the AI platform. As Figure 8 shown, the algorithm metric testing method may include:

[0299] Step 401: Initialize the number of channels to be tested, and set the initial value of the number of channels to be tested to 32.

[0300] In the embodiments of the present disclosure, the number of channels to be tested may be the number of video stream playback channels described above.

[0301] Step 402: Set the loop variable i of the number of video streams to 1.

[0302] Step 403: Determine whether the value of the loop variable i is greater than 32. If so, end; otherwise, execute step 404.

[0303] Step 404: Generate a configuration file for the AI platform to access i video streams and make the AI platform configuration file effective.

[0304] Step 405: Send a command to the streaming media service to generate i video streams.

[0305] Step 406: The streaming media service restarts and generates i video streams, and sends the i video streams to the AI platform.

[0306] Step 407: The AI platform receives the i video streams and performs a decoding operation to obtain the average frame rate of the AI platform processing the i video streams.

[0307] In step 407, the AI platform outputs the single-channel processing speed (i.e., the average frame rate).

[0308] Step 408: The AI platform parses the i video streams and converts them into frames and inputs them to the algorithm.

[0309] Step 409: The algorithm outputs the average frame rate, alarm information, and alarm files for processing the video streams.

[0310] In step 409, what the algorithm outputs is the single-channel processing speed (i.e., the average frame rate), single-channel alarm information, and single-channel alarm files.

[0311] Step 410: Update the loop variable with the data of adding 1 to the loop variable i to get i = i + 1, and execute step 403.

[0312] In an exemplary embodiment, the output data obtained in step 34 may include the average frame rate of each video stream processed by the AP platform in step 407, the average frame rate of each video stream processed by the algorithm in step 409, alarm information, and alarm files. In the embodiments of the present disclosure, the alarm information may include the picture frame corresponding to the thing in the video stream, the time, and the coordinates of the alarm picture detection box, and the alarm file may contain the alarm information. In the embodiments of the present disclosure, the thing may be a person or other physical object that appears in the video stream.

[0313] In an exemplary embodiment, during the execution of the above steps 401 - 410, process control may be performed through a script, and the output data is stored in the form of logs and files.

[0314] In an exemplary embodiment, step 35 may include:

[0315] Step 351: Determine whether the average frame rate of the algorithm for each camera among the N cameras meets the given frame rate threshold F (the value of F is the preset average frame rate); if it is lower than the frame rate threshold F, it is considered that the N cameras are not supported; if it is higher than the frame rate threshold F, it is considered supported, calculate the accuracy rate of the algorithm at the current moment, record the accuracy rate, and execute step 352. In an exemplary embodiment, the frame rate threshold may be the average processing frame rate set in step 31.

[0316] In an exemplary embodiment, calculating the algorithm accuracy rate at the current moment may include: reading the output data, comparing the difference between the number of alarms in the output data and the number of alarm messages of the reference metric value; comparing whether the deviation of the coordinates of the alarm picture detection box from the reference value coordinates is within the coordinate threshold range (i.e., pixel deviation, and the reference value coordinates may be the pixel positions of the alarm picture detection box in the above step 31); checking whether the alarm pictures are missed alarms or false alarms (which can be compared one by one with the alarm picture frames in the above step 31). According to the comparison and inspection results, the algorithm accuracy rate is statistically calculated. The curve of the algorithm accuracy rate and the number of camera channels is as Figure 9a - Figure 9c shown, in Figure 9a - Figure 9c the said curve graph, the abscissa is the number of camera channels accessed (i.e., the number of video streams played), and the ordinate is the algorithm accuracy rate (unit: %), and as the number of camera channels accessed increases, the algorithm accuracy rate decreases. When actually accessing, the curve of the number of camera channels accessed and the algorithm accuracy rate can be used to measure how many camera channels are more appropriate to access, that is, taking into account accessing more camera channels while achieving a certain accuracy rate. According to the resource configuration of the AI platform, the number of camera channels accessed and the algorithm accuracy rate are also different, as Figure 9a shown, when the number of camera channels accessed is 10, the algorithm accuracy rate is about 30%, and when the number of camera channels accessed is 19, the algorithm accuracy rate is about 10%; as Figure 9bAs shown, when the number of access cameras is 100, the algorithm accuracy rate is about 30%. When the number of access cameras is about 20, the algorithm accuracy rate is about 90%. As Figure 9c shown, when the number of access cameras is 700, the algorithm accuracy rate is about 30%. When the number of access cameras is about 200, the algorithm accuracy rate is about 70%.

[0317] In an exemplary embodiment, the coordinate threshold represents the pixel deviation. The default coordinate threshold can be 6 to 12. For example, the default coordinate threshold maxD = 9, that is, the deviation threshold between the coordinates of the warning picture detection box and the reference value coordinates is 9.

[0318] Step 352: According to the calculation results, obtain the curve graphs of the number of supported cameras, average frame rate, and algorithm accuracy rate.

[0319] From Step 351 and Step 352, the relationship curve of algorithm accuracy rate - number of cameras, the relationship curve of CPU / GPU utilization rate - number of cameras, and the relationship curve of frame rate - number of cameras can be obtained.

[0320] The verification of algorithm indicators (detection / recognition type) usually adopts the method of manual verification, which has high accuracy, but when the supported cameras are dozens or hundreds or thousands of cameras, the labor cost is too high. The algorithm indicator test method provided by the embodiments of the present disclosure can automatically obtain the curve relationship between algorithm indicators and the number of cameras, greatly saving labor costs. In the embodiments of the present disclosure, the above method of accessing video streams with different numbers of channels simulates accessing different numbers of cameras. The number of camera channels mentioned above can be understood as the number of video streams when playing the video source file.

[0321] The embodiments of the present disclosure also provide a test device for algorithm indicators, as Figure 10 shown, which may include a first acquisition module 01, a second acquisition module 02, and a third acquisition module 03;

[0322] The first acquisition module 01 is used to acquire algorithm benchmark indicators;

[0323] The second acquisition module 02 is used to control the AI platform to play the video source file multiple times with different playback channels, and acquire the algorithm indicators of the AI algorithm for processing each video stream during each playback of the video source file by the AI platform;

[0324] The third acquisition module 03 is used to obtain the test results of algorithm indicators according to the algorithm benchmark indicators acquired by the first acquisition module 01, the playback channels of playing the video source file multiple times, and the algorithm indicators acquired by the second acquisition module 02.

[0325] An embodiment of the present disclosure also provides a method for accessing a camera, including: accessing one or more cameras on an AI platform according to the test results of the algorithm metrics obtained by the test method of any of the above embodiments.

[0326] In an exemplary embodiment, accessing one or more cameras on an AI platform according to the test results of the algorithm metrics can avoid problems such as video stream processing failure and service crash caused by excessive number of camera channels.

[0327] An embodiment of the present disclosure also provides a computer-readable storage medium for storing computer program instructions, wherein the computer program instructions, when running, can implement the test method of the algorithm metrics of any of the above embodiments.

[0328] A test method, device, camera access method, and medium for algorithm metrics provided by an embodiment of the present disclosure. In the test method of the algorithm metrics, the AI platform is controlled to play the video source file multiple times with different playback channels, and the algorithm metrics of the data processing algorithm for processing each video source file during each playback of the video source file by the AI platform are obtained; according to the algorithm benchmark metrics, the playback channels of the multiple playbacks of the video source file, and the obtained algorithm metrics, the test results of the algorithm metrics are obtained. The obtained test results of the algorithm metrics can make a good plan in advance for the actual number of cameras to be accessed and the resource configuration of the AI platform, and overcome the problems of video stream processing failure and service crash caused by insufficient resources of the AI platform after the camera is accessed to the AI platform.

[0329] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, and the functional modules / units in systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations. In the hardware implementation, the division of functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component can have multiple functions, or a function or step can be executed by several physical components in cooperation. Some or all components can be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cartridges, tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that a communication medium typically contains computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0330] The accompanying drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures can refer to the general design.

[0331] Without conflict, the embodiments of the present invention, that is, the features in the embodiments, can be combined with each other to obtain new embodiments.

[0332] Although the disclosed embodiments of the present invention are as above, the content is only an embodiment adopted for the convenience of understanding the present invention and is not intended to limit the present invention. Any person skilled in the art within the scope of the present invention can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed by the present invention. However, the scope of patent protection of the present invention shall still be subject to the scope defined by the appended claims.

Claims

1. A test method for algorithm metrics, characterized in that, Set to test the algorithm metrics of the data processing algorithm in the AI platform. The test method includes: Obtain algorithm benchmark metrics, which include at least one of the following: the number of benchmark alarm messages, the benchmark average processing frame rate, the benchmark pixel positions of the detection frames of alarm pictures, the benchmark occupancy rate of system resources, and the benchmark alarm message files; Control the AI platform to play the video source file multiple times with different numbers of playback channels, and obtain the algorithm metrics of the data processing algorithm for each video source file during each playback of the video source file by the AI platform. The algorithm metrics include at least one of the following: the number of alarm messages, the average processing frame rate, the pixel positions of the detection frames of alarm pictures, the occupancy rate of system resources, and the alarm message files; Obtain the test results of the algorithm metrics based on the algorithm benchmark metrics, the number of playback channels for playing the video source file multiple times, and the obtained algorithm metrics. The test results are to make a good plan for the actual number of camera accesses and the resource configuration of the AI platform in advance. The test results of the algorithm metrics include at least one of the following: the relationship curve graph between the number of playback channels and the average frame rate, the relationship curve graph between the number of playback channels and the occupancy rate of system resources, and the relationship curve graph between the number of playback channels and the algorithm accuracy rate.

2. The test method according to claim 1, wherein The obtaining of the algorithm benchmark metrics includes: playing the video source file multiple times through an offline algorithm, obtaining the offline algorithm metrics output by the offline algorithm multiple times, and calculating the average value of each offline algorithm metric obtained multiple times to obtain the algorithm benchmark metrics.

3. The test method according to claim 1, wherein The controlling the AI platform to play the video source file multiple times with different numbers of playback channels and obtaining the algorithm metrics of the AI algorithm for each video source file during each playback of the video source file by the AI platform includes: Set the initial values of the total number of playback channels and the number of playback channels; Generate a first configuration file for the AI platform to access the video stream of the current number of playback channels and make the first configuration file take effect; the first configuration file includes the video stream address and the algorithm information for processing the video stream; Send an instruction to the streaming media service according to the video stream address in the first configuration file, and the instruction contains the current number of playback channels; The streaming media service receives the instruction, obtains the video source file, converts the video source file into a video stream of the current number of playback channels according to the instruction, and sends the video stream of the current number of playback channels to the AI platform; the AI platform includes: a decoding processor and a data processing algorithm; The decoding processor in the AI platform converts the video stream of the current number of playback channels into picture frames, and transmits the picture frames to the data processing algorithm corresponding to the algorithm information for processing the video stream in the first configuration file; The data processing algorithm in the AI platform plays the video stream of the current number of playback channels according to the picture frames, outputs the processing results for each video stream, and uses the processing results as the algorithm metrics; Update the value of the current playback channel number, and determine whether the updated value of the current playback channel number exceeds the total number of playback channels. When the updated value of the current playback channel number does not exceed the total number of playback channels, continue to send an instruction to the streaming media service according to the video stream address in the first configuration file.

4. The test method according to claim 3, characterized in that, Before the decoding processor in the AI platform converts the video stream of the current playback channel number into picture frames and transmits the picture frames to the data processing algorithm corresponding to the algorithm information in the first configuration file, it further includes: controlling the AI platform to decode the video stream of the current playback channel number to obtain picture data, and obtaining the average processing speed of the AI platform for decoding each video stream. The decoding processor converts the video stream of the current playback channel number into picture frames, including: the decoding processor converts the picture data of the video stream of the current playback channel number into picture frames.

5. The test method according to claim 4, characterized in that When the updated value of the current playback channel number exceeds the total number of playback channels, the method further includes: obtaining the test result of the AI platform index based on the decoding benchmark index pre-stored in the AI platform, the number of video stream channels during multiple playbacks of the video source file, and the average processing speed of the AI platform for each video stream.

6. The test method according to claim 3, wherein, The first configuration file at least includes: the video stream address, the frame rate of playing the video stream, and the algorithm information for processing the video stream. The data processing algorithm in the AI platform plays the video stream of the current playback channel number according to the picture frames, including: the data processing algorithm in the AI platform plays the video stream of the current playback channel number at the frame rate of playing the video stream in the first configuration file according to the picture frames.

7. The test method according to claim 3, wherein Making the first configuration file take effect includes: controlling the AI platform to restart to make the first configuration file take effect, or transmitting an effective parameter to the AI platform to make the first configuration file take effect dynamically.

8. The test method according to claim 3, characterized in that, Before the streaming media service receives the instruction and before converting the video source file into the video stream of the current playback channel number according to the instruction, it further includes: restarting the streaming media service.

9. The test method according to claim 2, wherein Obtaining the test result of the algorithm index according to the algorithm benchmark index, the number of playback channels during multiple playbacks of the video source file, and the output algorithm index includes: During each playback of the video source file, compare the number of alarm messages in each channel with the reference number of alarm messages, and / or compare the pixel positions of the alarm picture detection frames in each channel with the reference pixel positions of the alarm picture detection frames, and count the algorithm accuracy rate of each playback of the video source file according to the comparison results. Obtain the test result of the algorithm index according to the algorithm accuracy rate statistically obtained from multiple playbacks of the video source file.

10. The test method according to claim 9, characterized in that, Before comparing the number of alarm messages in each channel with the reference number of alarm messages, and / or comparing the pixel positions of the alarm picture detection frames in each channel with the reference pixel positions of the alarm picture detection frames during each playback of the video source file, it further includes: Determine whether the average processing frame rate of multiple channels is lower than the reference average processing frame rate during each playback of the video source file. When the average processing frame rate of multiple channels is lower than the reference average processing frame rate, determine that the number of playback channels supported for this playback is not supported. When the average processing frame rate of multiple channels is not lower than the reference average processing frame rate, determine the number of playback channels supported for this playback; compare whether the system resource occupancy rate in each channel is lower than the system resource reference occupancy rate; during each playback of the video source file, compare the number of alarm messages in each channel with the reference number of alarm messages, and / or compare the pixel positions of the alarm picture detection frames in each channel with the reference pixel positions of the alarm picture detection frames. According to the comparison results, calculate the algorithm accuracy rate for each playback of the video source file, including: Calculate the difference between the number of alarm messages in each channel and the reference number of alarm messages, and the difference between the pixel positions of the alarm picture detection frames in each channel and the reference pixel positions of the alarm picture detection frames during each playback of the video source file, and compare whether the alarm message files in each channel are consistent with the reference alarm message files. According to the judgment, calculation, and comparison results, calculate the algorithm accuracy rate for each playback of the video source file.

11. The test method according to claim 9 or 10, characterized in that, The test results of the algorithm metrics obtained based on the algorithm accuracy rates statistically obtained from multiple playbacks of the video source file include at least one of the following: Obtain the relationship curve graph of the number of playback channels and the average frame rate, the relationship curve graph of the number of playback channels and the system resource occupancy rate, and the relationship curve graph of the number of playback channels and the algorithm accuracy rate according to the algorithm accuracy rate.

12. The testing method according to claim 10, wherein The calculation of the difference between the number of alarm messages in each channel and the reference number of alarm messages, and the difference between the pixel positions of the alarm picture detection frames in each channel and the reference pixel positions of the alarm picture detection frames during each playback of the video source file, and the comparison of whether the alarm message files in each channel are consistent with the reference alarm message files include: During each playback of the video source file, calculate the difference between the number of alarm messages in each channel and the reference number of alarm messages. If the difference between the number of alarm messages and the reference number of alarm messages exceeds the preset difference, determine that the algorithm is inaccurate when playing the current playback channel. Calculate the difference between the pixel positions of the alarm picture detection frames in each channel and the reference pixel positions of the alarm picture detection frames. If the difference between the pixel positions of the alarm picture detection frames and the reference pixel positions of the alarm picture detection frames exceeds the preset difference range, determine that the algorithm is inaccurate when playing the current playback channel. Compare whether the alarm message files in each channel are consistent with the reference alarm message files. If they are inconsistent, determine that the algorithm is inaccurate when playing the current playback channel. The comparison of whether the system resource occupancy rate in each channel is lower than the system resource reference occupancy rate includes: comparing whether the system resource occupancy rate in each channel is lower than the system resource reference occupancy rate. If the system resource occupancy rate is not lower than the system resource reference occupancy rate, determine that the algorithm is inaccurate when playing the current playback channel. The algorithm accuracy rate for counting each play of the video source file according to the judgment, calculation, and comparison results includes: counting the accuracy rate of playing multiple video streams during each play of the video source file, and obtaining the algorithm accuracy rate based on the number of played paths and the accuracy rate of multiple plays of the video source file.

13. The test method according to claim 12, characterized in that, The counting of the accuracy rate of playing multiple video streams during each play of the video source file includes: counting the ratio of the number of paths where the algorithm is determined to be accurate to the total number of played paths during each play of the video source file; Among them, during the process of playing each video source file, when the determination result of each algorithm index is that the algorithm is accurate, it is determined that the algorithm is accurate when processing this video source file. When the determination result of any algorithm index is that the algorithm is inaccurate, it is determined that the algorithm is inaccurate when processing this video source file.

14. The test method according to claim 1, characterized in that The AI platform includes multiple AI algorithms, and the multiple AI algorithms include the data processing algorithm. Before playing the video source file multiple times with different numbers of paths on the AI platform, it also includes: performing automated detection on the multiple AI algorithms before they are launched on the AI platform.

15. The test method according to claim 14, wherein The performing of automated detection on the multiple AI algorithms before they are launched on the AI platform includes: automatically obtaining multiple AI algorithm codes to be tested from the algorithm code storage platform, performing inspections on various indicators of the multiple AI algorithm codes, and if any abnormality occurs in any indicator inspection, calling the interface of the information feedback platform to send the corresponding abnormal information to the abnormal information feedback platform; when no abnormality occurs in the inspections of various indicators of the multiple algorithm codes, checking the startup status of the AI platform, and when an abnormality occurs in the startup of the AI platform, sending the AI platform startup abnormal information to the abnormal information feedback platform; when no abnormality occurs in the startup of the AI platform, controlling the AI platform to start multiple AI algorithms, checking the running status of the multiple AI algorithms, and when an abnormality occurs in the running status of the multiple AI algorithms, sending the information of the abnormal algorithm running status to the abnormal information feedback platform.

16. The test method according to claim 15, wherein After automatically obtaining multiple AI algorithm codes to be tested from the algorithm code storage platform, it also includes: generating a configuration benchmark file based on the multiple AI algorithm codes, and generating a second configuration file based on the configuration benchmark file; The configuration benchmark file includes the algorithm names of multiple algorithms, algorithm model parameters, path parameters of the database required for algorithm operation, resource configuration parameters, and video stream information as algorithm input. Among them, the video stream information contains the algorithm names of the multiple AI algorithms, algorithm strategy information, and frame rate thresholds.

17. The test method according to claim 16, characterized in that, The generating of the second configuration file based on the configuration benchmark file includes: generating the content in the configuration benchmark file into a second configuration file in CSV format. The second configuration file includes algorithm basic information and algorithm input information. The basic information includes the algorithm name, algorithm model parameters, path parameters of the database required for algorithm operation, resource configuration parameters; the algorithm input information includes the video stream information.

18. The test method according to claim 17, wherein The performing of inspections on various indicators of the multiple AI algorithm codes includes: Check whether the algorithm name in the algorithm input information in the second configuration file is consistent with the algorithm name in the algorithm basic information. When it is found that the algorithm name in the algorithm input information in the second configuration file is inconsistent with the algorithm name in the algorithm basic information, feedback the information of the abnormal second configuration file to the abnormal information feedback platform; When it is checked that the algorithm name in the algorithm input information in the second configuration file is consistent with the algorithm name in the algorithm basic information, the detection platform obtains the compilation instruction from the algorithm code storage platform, and automatically calls the compilation interface to compile the multiple AI algorithms according to the compilation instruction; obtain the compilation log, and check whether there is an error in the compilation log. When it is checked that there is an error in the compilation log, feedback the information of the compilation exception to the abnormal information feedback platform; After it is checked that the algorithm name in the algorithm input information in the second configuration file is consistent with the algorithm name in the algorithm basic information, it further includes: the test platform checks whether the model files required for the AI algorithms to be launched this time are prepared correctly according to the second configuration file. If it is detected that the model files are not prepared, feedback the information of the model exception to the abnormal information feedback platform.

19. The test method according to claim 15, wherein Checking the startup status of the AI platform includes: Start the AI platform, wait for the first preset time, and check whether the process of the AI platform is started. When it is checked that the process of the AI platform is not started, send the abnormal startup information of the AI platform to the abnormal information feedback platform; when it is checked that the process of the AI platform is started, start multiple AI algorithms on the AI platform and check the running status of the multiple AI algorithms.

20. The test method according to claim 16, characterized in that The AI platform starts multiple AI algorithms and checks the running status of the multiple AI algorithms, including: Control the AI platform to start the thread groups corresponding to multiple AI algorithms; Control the AI platform to read the second configuration file, and add the AI algorithms recorded in the second configuration file that need to be detected to the thread group; Control the AI platform to run multiple threads in the thread group. When any one of the AI algorithms in the multiple threads runs abnormally, send the information of the abnormal running of the corresponding AI algorithm to the abnormal feedback platform; Start the summary thread, summarize the detection results, and send the summarized detection results to the abnormal feedback platform.

21. The test method according to claim 15, wherein When the running status of the multiple AI algorithms does not show an abnormality, it further includes: going online the multiple AI algorithm codes on the AI platform.

22. The test method according to claim 15, characterized in that, Before the automated detection: It further includes triggering periodic detection.

23. A test device for algorithm metrics, characterized in that, It includes a first acquisition module, a second acquisition module, and a third acquisition module; The first acquisition module is used to acquire algorithm benchmark metrics, and the algorithm benchmark metrics include at least one of the following: the number of benchmark alarm messages, the benchmark average processing frame rate, the benchmark pixel positions of the detection frames of the alarm pictures, the benchmark occupancy rate of system resources, and the benchmark alarm message files; The second acquisition module is configured to control the AI platform to play the video source file multiple times with different numbers of playback channels, and acquire the algorithm metrics for each video stream processed by the AI algorithm during each playback of the video source file by the AI platform. The algorithm metrics include at least one of the following: the number of alarm messages, the average processing frame rate, the pixel positions of the detection frames of the alarm pictures, the system resource occupancy rate, and the alarm message files. The third acquisition module is configured to obtain the test results of the algorithm metrics based on the algorithm benchmark metrics acquired by the first acquisition module, the number of playback channels for playing the video source file multiple times, and the algorithm metrics acquired by the second acquisition module. The test results are used to make advance plans for the actual number of cameras connected and the resource configuration of the AI platform. The test results of the algorithm metrics include at least one of the following: the relationship curve between the number of playback channels and the average frame rate, the relationship curve between the number of playback channels and the system resource occupancy rate, and the relationship curve between the number of playback channels and the algorithm accuracy rate.

24. A camera access method, characterized in that, Including: Based on the test results of the algorithm metrics obtained by the test method according to any one of claims 1 to 22, one or more cameras are connected to the AI platform.

25. A computer-readable storage medium, characterized in that, The storage medium is used to store computer program instructions, wherein the computer program instructions can implement the test method of the algorithm metrics according to any one of claims 1 to 22 when running.

Images