Method, device and system for testing algorithm performance of network camera

Abstract

The invention provides an algorithm performance test method, device and system for a network camera, and relates to the technical field of computers. According to the specific scheme, test samples and test tasks are allocated to a plurality of network cameras based on test requirements; wherein the plurality of network cameras are provided with the same target recognition algorithm, and the test task is used for indicating the algorithm performance of the test network camera; obtaining test data returned by the plurality of network cameras; wherein the test data comprises a test sample identification number, targets identified in a test sample corresponding to the test sample identification number, and a credibility score of each target; and integrating and analyzing the test data corresponding to the plurality of network cameras to obtain an algorithm performance test result of the target recognition algorithm. According to the scheme disclosed by the invention, a tester can easily maintain and update the sample library, and test the algorithm installed in the network camera by using the recorded video sample, so that the test efficiency and accuracy are greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to an algorithm performance testing method, apparatus, and system for a network camera. Background Art

[0002] In the related art, the algorithm performance testing of Internet Protocol Camera (IPC) devices usually adopts a live test method. Testers must manually simulate various usage scenarios and evaluate the algorithm effect by observing the reactions of IPC devices. This testing method has multiple problems. For example, it requires a large amount of manpower and resources to simulate different scenarios, the recognition effect of the algorithm is easily affected by light and environmental complexity, and each test requires reconstructing the scenario again and it is difficult to ensure the consistency of the scenario. In addition, when users feedback that the algorithm recognizes incorrectly in a certain scenario, it is difficult for testers to accurately reproduce the abnormal scenario feedback by users, resulting in low efficiency and accuracy of IPC device testing. Summary of the Invention

[0003] The present disclosure provides an algorithm performance testing method, apparatus, and system for a network camera.

[0004] According to a first aspect of the present disclosure, there is provided an algorithm performance testing method for a network camera, including:

[0005] Allocating test samples and test tasks to multiple network cameras based on test requirements; wherein, the multiple network cameras are installed with the same target recognition algorithm, the test task is used to indicate the algorithm performance of the test network camera, and the test samples are extracted from a sample library;

[0006] Obtaining test data respectively returned by the multiple network cameras; wherein, the test tasks and test samples corresponding to different network cameras are different, and the test data includes a test sample identification number, a target identified in the test sample corresponding to the test sample identification number, and a confidence score for each target;

[0007] Integrating and analyzing the test data corresponding to each of the multiple network cameras to obtain an algorithm performance test result of the target recognition algorithm.

[0008] According to a second aspect of the present disclosure, there is provided an algorithm performance testing apparatus for a network camera, including:

[0009] An allocation unit, configured to allocate test samples and test tasks to multiple network cameras based on test requirements; wherein, the multiple network cameras are installed with the same target recognition algorithm, the test task is used to indicate the algorithm performance of the test network camera, and the test samples are extracted from a sample library;

[0010] A first acquisition unit, configured to acquire test data respectively returned by a plurality of network cameras; wherein, the test tasks and test samples corresponding to different network cameras are different, and the test data includes a test sample identification number, a target identified in the test sample corresponding to the test sample identification number, and a confidence score for each target.

[0011] An integration and analysis unit, configured to integrate and analyze the test data respectively corresponding to the plurality of network cameras to obtain an algorithm performance test result of the target recognition algorithm.

[0012] According to a third aspect of the present disclosure, there is provided an algorithm performance test automation system, including:

[0013] A control device, configured to allocate test samples and test tasks to a plurality of network cameras based on test requirements; wherein, the plurality of network cameras are installed with the same target recognition algorithm, the test task is used to indicate the algorithm performance of the test network camera, and the test sample is extracted from a sample library; acquire the test data respectively of the plurality of network cameras; wherein, the test data includes a test sample identification number, a target identified in the test sample corresponding to the test sample identification number, and a confidence score for each target; integrate and analyze the test data respectively corresponding to the plurality of network cameras to obtain an algorithm performance test result of the target recognition algorithm.

[0014] A plurality of network cameras, configured to receive the test tasks and test samples issued by the control device, and execute the respective received test tasks based on the respective received test samples; wherein, the test tasks and test samples corresponding to different network cameras are different.

[0015] A storage device, configured to store the sample library; store the test data respectively of the plurality of network cameras.

[0016] According to a fourth aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute any method in the embodiments of the present disclosure.

[0017] According to a fifth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute any method in the embodiments of the present disclosure.

[0018] According to a sixth aspect of the present disclosure, there is provided a computer program product, including a computer program stored on a storage medium, where the computer program, when executed by a processor, implements any method in the embodiments of the present disclosure.

[0019] According to the technical solution of the present disclosure, testers can easily maintain and update the sample library, and use the recorded video samples to test the algorithms installed in network cameras, greatly improving the efficiency and accuracy of testing.

[0020] The above summary is only for the purpose of the specification and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In the drawings, unless otherwise specified, the same reference numerals throughout the several views denote the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0022] Figure 1 is a schematic flowchart of a method for testing the algorithm performance of a network camera according to an embodiment of the present disclosure;

[0023] Figure 2 is a schematic architecture diagram of a method for testing the algorithm performance of a network camera according to an embodiment of the present disclosure;

[0024] Figure 3 is a schematic flowchart of a process for generating a test report according to an embodiment of the present disclosure;

[0025] Figure 4 is a schematic flowchart of a process for discovering a test device according to an embodiment of the present disclosure;

[0026] Figure 5 is a schematic flowchart of an algorithm test according to an embodiment of the present disclosure;

[0027] Figure 6 shows a schematic framework diagram of an algorithm performance test system for a network camera according to an embodiment of the present disclosure;

[0028] Figure 7 is a schematic structural diagram of an algorithm performance test device for a network camera according to an embodiment of the present disclosure;

[0029] Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The present disclosure will be described in further detail below with reference to the drawings. The same reference numerals in the drawings denote components or elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0031] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can also be implemented without certain specific details. In some instances, methods, means, elements, circuits, etc. that are well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0032] The term "and / or" in this article represents three possible relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The term "at least one" in this article represents any one of multiple types or any combination of at least two of multiple types. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set composed of A, B, and C. The terms "first" and "second" in this article represent referring to multiple similar technical terms and distinguishing them, and do not mean to limit the order or limit to only two. For example, the first feature and the second feature refer to two categories / two features. The first feature can be one or more, and the second feature can also be one or more.

[0033] In the related art, the algorithm performance test of IPC devices is usually carried out by building a real scene for testing. However, this method requires testers to manually simulate various usage scenarios, observe the reactions of IPC devices, and evaluate the algorithm effects. Currently, the algorithm performance test of IPC devices adopts the real scene test method. By simulating different scenarios and user usage scenarios, testers evaluate the algorithm effects.

[0034] This method has the following defects and deficiencies:

[0035] 1. High consumption of human and material resources: The real scene test requires a large amount of human and material resources to simulate various scenarios, making the test process expensive and time-consuming.

[0036] 2. Limited test scenarios: The coverage of the real scene test is limited and cannot fully cover all potential usage scenarios.

[0037] 3. Inconsistency of scenarios: Each test requires rebuilding the scenario, and it is difficult to ensure the consistency of the scenarios, resulting in relatively low reliability of the test results.

[0038] 4. Difficulty in reproducing abnormal scenarios: When users feedback that the algorithm recognition is incorrect in a certain scenario, it is difficult for testers to accurately reproduce these abnormal scenarios, making it difficult to conduct in-depth analysis and improvement.

[0039] To at least partially solve one or more of the above problems and other potential problems, the present disclosure proposes a test method for an algorithm installed in a network camera, which can input a large number of test samples into multiple IPC devices, and the algorithm of the IPC device identifies the test samples and outputs the results; moreover, the test data of all network cameras are integrated and analyzed to generate an algorithm performance test report. In this way, testers can easily maintain and update the sample library, and use the recorded video samples to test the algorithm installed in the network camera, greatly improving the test efficiency and accuracy.

[0040] An embodiment of the present disclosure provides a method for testing the algorithm performance of a network camera. Figure 1 FIG. is a schematic flowchart of a method for testing the algorithm performance of a network camera according to an embodiment of the present disclosure. The method for testing the algorithm performance of the network camera can be applied to a device for testing the algorithm performance of the network camera. The device for testing the algorithm performance of the network camera is located in an electronic device. The electronic device includes, but is not limited to, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc. In some possible implementation manners, the method for testing the algorithm performance of the network camera can also be implemented by a processor calling computer-readable instructions stored in a memory. As Figure 1 shown, the method for testing the algorithm performance of the network camera includes:

[0041] S101: Allocate test samples and test tasks to multiple network cameras based on test requirements; wherein, the same target recognition algorithm is installed on the multiple network cameras, the test task is used to indicate the algorithm performance of the test network camera, and the test samples are extracted from a sample library.

[0042] S102: Obtain the test data respectively returned by the multiple network cameras; wherein, the test tasks and test samples corresponding to different network cameras are different, and the test data includes a test sample identification number, the targets identified in the test sample corresponding to the test sample identification number, and the confidence score of each target.

[0043] S103: Integrate and analyze the test data respectively corresponding to the multiple network cameras to obtain the algorithm performance test result of the target recognition algorithm.

[0044] In the embodiments of the present disclosure, the test requirements are for testing the performance, stability, and reliability of network cameras. Exemplarily, the test requirements may be for testing the performance of the object recognition algorithm installed in the network camera, that is, testing the performance of the object recognition algorithm installed in the network camera when processing image and video data, including indicators such as processing speed; the test requirements may also be for testing the accuracy of the object recognition algorithm installed in the network camera, that is, testing the accuracy of the object recognition algorithm installed in the network camera in performing recognition tasks; the test requirements may also be for testing the robustness of the object recognition algorithm installed in the network camera, that is, testing the robustness of the object recognition algorithm installed in the network camera under various environmental conditions, including interference factors such as light changes, occlusion, and noise; the test requirements may also be for testing the stability of the object recognition algorithm installed in the network camera, that is, testing the stability of the object recognition algorithm installed in the network camera at different time periods and in different scenarios, and ensuring that the algorithm can maintain stable performance under different conditions.

[0045] In the embodiments of the present disclosure, the IPC device is a new generation of camera produced by combining traditional cameras with network technology. A network camera generally consists of parts such as a lens, an image sensor, a sound sensor, a signal processor, an analog-to-digital converter (A / D converter), an encoding chip, a main control chip, a network, and a control interface.

[0046] In the embodiments of the present disclosure, the test samples may be samples stored in a sample library. Specifically, obtaining the test samples may include: obtaining test samples through abnormal scenarios feedback by users; obtaining test samples according to special scenarios set by testers; obtaining test samples by intercepting videos recorded by network cameras.

[0047] In the embodiments of the present disclosure, the test tasks refer to the test content of the object recognition algorithm installed in multiple network cameras. Exemplarily, the test tasks may be to test the performance of the object recognition algorithm installed in the network camera; the test tasks may also be to test the accuracy of the object recognition algorithm installed in the network camera; the test tasks may be to test the robustness of the object recognition algorithm installed in the network camera; the test tasks may be to test the stability of the object recognition algorithm installed in the network camera.

[0048] In the embodiments of the present disclosure, the target recognition algorithm refers to the real-time recognition of targets such as people, animals, and vehicles in the video / images recorded by a network camera. The core working principle of the target recognition algorithm in the network camera is to utilize computer vision and deep learning technologies to analyze visual features, shapes, textures, colors, etc. in the image and the associations between the features to determine the target objects in the image. The algorithm extracts key features such as edges, corner points, and color distributions in the image for the recognition and classification of targets. Some target recognition algorithms require large-scale labeled image data for training, build models through machine learning or deep learning, and learn the feature patterns of different targets. After feature extraction and model training, the algorithm can accurately locate and classify the targets in the image.

[0049] In the embodiments of the present disclosure, the network camera equipped with the target recognition algorithm can be applied to implement a video doorbell for recognizing people, animals, etc. passing by / staying in front of the door; the network camera equipped with the target recognition algorithm can be applied to outdoor scenarios such as hospitals, villas, and science and technology museums. Exemplarily, if the network camera equipped with the target recognition algorithm is applied to a hospital, it can be used to implement hospital personnel identity authentication, item tracking, task tracking, vehicle tracking, and environmental detection, etc.

[0050] In the embodiments of the present disclosure, the algorithm performance can include the accuracy of the algorithm, the stability of the algorithm, the robustness of the algorithm, etc.

[0051] In the embodiments of the present disclosure, a large number of test samples can be stored in the sample library. The sample library can be located in the database of the test electronic device; it can also be located in the database of the tester's electronic device. The above is only an exemplary illustration and does not limit all possible storage locations included in the sample library, and only exhaustive listing is not done here.

[0052] In the embodiments of the present disclosure, the test data refers to the test results returned by multiple network cameras equipped with the target recognition algorithm based on the test samples. The test data can include the test sample identification number (Identity document, ID), the network camera ID, the targets recognized in the test sample corresponding to the test sample identification number, and the confidence score for each target. The confidence score can be a specific value or a scoring level, such as excellent, good, or poor.

[0053] Here, the file format of the test data is the Comma-Separated Values (CSV) file format.

[0054] In the embodiments of the present disclosure, the difference in the test tasks and test samples corresponding to different network cameras means that the recognition tasks of different IPC devices are different. Exemplarily, IPC device 1 receives test task 1, which is to test the first type of target, and the first type of target is "person"; IPC device 2 receives test task 2, which is to test the second type of target such as an animal, and the second type of target is "animal". The number of test samples received by each network camera can be one or multiple.

[0055] In the embodiments of the present disclosure, the algorithm performance test result refers to the test result of a network camera installed with a target recognition algorithm based on test samples and test tasks. Exemplarily, if the test task is to test the accuracy of the target recognition algorithm installed in network camera 1, test sample 1 is "a girl in a cheongsam in an ancient-style courtyard", the test data of network camera 1 is test sample 1, the test device is network camera 1, the recognition target is the girl in the cheongsam, and the confidence level of the recognition target is 80%; then by analyzing the test data of the network camera, the accuracy of the target recognition algorithm installed in network camera 1 is obtained as "accurate".

[0056] In the embodiments of the present disclosure, the functions of the target recognition algorithm may include: moving tracking function, human body detection function, anomaly detection function, etc.

[0057] Specifically, the mobile tracking function for testing the target recognition algorithm installed in network cameras may include: allocating test samples and test tasks to multiple network cameras based on test requirements, where the test sample is a recorded video and the test task is to test the performance of the mobile tracking function; obtaining the test data returned by the multiple network cameras; integrating and analyzing the test data corresponding to each of the multiple network cameras to obtain the performance test results of the mobile tracking function of the target recognition algorithm. Exemplarily, the test devices are IPC device 1, IPC device 2, and IPC device 3; among them, the test sample of IPC device 1 is a recorded video of target person 1 driving into the courtyard on a cloudy day, the test sample of IPC device 2 is a recorded video of target person 1 driving into the courtyard at dusk, and the test sample of IPC device 3 is a recorded video of target person 1 driving into the courtyard when the street lights are on at night; IPC device 1, IPC device 2, and IPC device 3 respectively test the mobile tracking function of the target recognition algorithm based on their respective test samples; obtain the test data returned by IPC device 1, IPC device 2, and IPC device 3 respectively; integrate and analyze the test data of IPC device 1, IPC device 2, and IPC device 3 respectively to obtain the performance test results of the mobile tracking function of the target recognition algorithm. The test data of the mobile tracking of the network camera may include indicators such as tracking sensitivity, accuracy, and response time; integrate the test data of IPC device 1, IPC device 2, and IPC device 3, and analyze the performance of the mobile tracking function of the network camera to evaluate its applicability and reliability in actual applications.

[0058] Specifically, the human detection function for testing the target recognition algorithm installed in a network camera may include: allocating test samples and test tasks for multiple network cameras based on test requirements, where the test sample is a recorded video or image, and the test task is to test the performance of the human detection function; obtaining the test data returned by multiple network cameras; integrating and analyzing the test data corresponding to each of the multiple network cameras to obtain the performance test result of the human detection function of the target recognition algorithm. Exemplarily, the test devices are IPC device 1, IPC device 2, and IPC device 3; among them, the test sample of IPC device 1 is a recorded video of a target person walking into the courtyard, the test sample of IPC device 2 is a recorded video of a target person running into the courtyard, and the test sample of IPC device 3 is a recorded video of a target person jumping into the courtyard; IPC device 1, IPC device 2, and IPC device 3 respectively test the human detection function of the target algorithm based on their respective test samples; obtain the test data returned by IPC device 1, IPC device 2, and IPC device 3 respectively; integrate and analyze the test data of IPC device 1, IPC device 2, and IPC device 3 respectively to obtain the performance test result of the human detection function of the target recognition algorithm. The test data for human detection of the network camera may include indicators such as recognition accuracy, response time, etc., as well as problems such as false alarms and missed detections; integrate the test data of IPC device 1, IPC device 2, and IPC device 3, and analyze the performance of the human detection function of the network camera to evaluate its applicability and reliability in actual applications.

[0059] Specifically, the anomaly detection function for testing the target recognition algorithm installed in network cameras may include: allocating test samples and test tasks to multiple network cameras based on test requirements, where the test sample is a recorded video or image, and the test task is to test the performance of the anomaly detection function; obtaining the test data returned by multiple network cameras; integrating and analyzing the test data corresponding to each of the multiple network cameras to obtain the performance test result of the anomaly detection function of the target recognition algorithm. Exemplarily, the test devices are IPC device 1, IPC device 2, and IPC device 3; among them, the test sample of IPC device 1 is a recorded video of target person 1 breaking into the monitoring area, the test sample of IPC device 2 is a recorded video of items being stolen in the monitoring area, and the test sample of IPC device 3 is a recorded video of dangerous target person 1 appearing; IPC device 1, IPC device 2, and IPC device 3 respectively test the anomaly detection function of the target algorithm based on their respective test samples; obtain the test data returned by IPC device 1, IPC device 2, and IPC device 3 respectively; integrate and analyze the test data of IPC device 1, IPC device 2, and IPC device 3 respectively to obtain the performance test result of the anomaly detection function of the target recognition algorithm. The test data for the anomaly detection of the network camera may include indicators such as the accuracy of anomaly detection, response time, etc., as well as problems such as false alarms and missed alarms; integrate the test data of IPC device 1, IPC device 2, and IPC device 3, and analyze the performance of the anomaly detection function of the network camera to evaluate its applicability and reliability in actual applications.

[0060] In the embodiments of the present disclosure, the algorithm testing method for the network camera further includes: allocating test samples and test tasks to multiple network cameras based on test requirements; among them, the same target recognition algorithm is installed in multiple network cameras, the test task is used to indicate the joint use performance of the algorithm of the test network camera with other algorithms or tools, and the test sample is extracted from the sample library; obtaining the test data returned by each of the multiple network cameras; among them, the test tasks and test samples corresponding to different network cameras are different, and the test data includes the test sample identification number, the targets identified in the test sample corresponding to the test sample identification number, the identification numbers of other algorithms or tools, and the confidence score of each target; integrating and analyzing the test data corresponding to each of the multiple network cameras to obtain the algorithm performance test result of the target recognition algorithm. Among them, the other algorithms may include target tracking algorithms, anomaly detection algorithms, behavior analysis algorithms, video encryption algorithms, etc.; the other tools may include: map tools, Global Positioning System (GPS), etc. The above is only an exemplary illustration and does not limit all possible contents included in other algorithms or tools, but only does not list them exhaustively here.

[0061] Figure 2 The architecture schematic diagram of the algorithm performance test method for a network camera is shown. As Figure 2 shown, the algorithm test method for the network camera includes: a control unit (host), an execution unit (IPC device), and a storage module (Network File System (NFS) server). The control unit is used to remotely log in to multiple IPC devices; the storage module NFS is mounted on multiple IPC devices. The control unit and the storage module can communicate, but when the NFS server and the control unit are located on the same host, this process can be omitted.

[0062] Figure 3 The flow schematic diagram for generating a test report is shown. As Figure 3 shown, this process may include:

[0063] S301: Read the configuration file, and then execute S302;

[0064] S302: Transcode the test samples, and then execute S303; that is, transcode the test samples into formats such as Portable Network Graphics / Joint Photographic Experts Group / Moving Picture Experts Group - Part 14 / A Multimedia Container Format (png\jpg\mp4\mkv) and transcode them into the YUV Color Space (YUV) file format;

[0065] S303: Discover local area network devices (i.e., IPC devices), and then execute S304; that is, automatically discover and connect to IPC devices on the local area network;

[0066] S304: Connect via the Telnet protocol, and then execute S305;

[0067] S305: Distribute test tasks, and then execute S306;

[0068] S306: Statistically analyze the test results, and then execute S307;

[0069] S307: Generate a test report.

[0070] Figure 4 The flow schematic diagram for discovering test devices is shown. As Figure 4 shown, this process may include:

[0071] S401: Start, and then execute S402;

[0072] S402: Broadcast local area network messages, and then execute S403;

[0073] S403: Determine whether a reply is received. If so, execute S404; if not, execute S406;

[0074] S404: Determine whether to stop broadcasting. If yes, execute S405; if no, execute S402;

[0075] S405: Wait for 30 seconds, and then execute S402;

[0076] S406: End.

[0077] Figure 5 The flow diagram of the algorithm test is shown, as Figure 5 shown, this process may include:

[0078] S501: Start, and then execute S501;

[0079] S502: Read a yuv file, and then execute S503;

[0080] S503: Call the algorithm interface for recognition, and then execute S504;

[0081] S504: Save the result to a CSV file, and then execute S505;

[0082] S505: Determine whether the yuv files in the specified directory have been recognized. If yes, execute S506; if no, execute S502;

[0083] S506: End.

[0084] In the embodiments of the present disclosure, the work process of the algorithm test may include:

[0085] 1. Executable program startup: After the algorithm test process inside the IPC device starts, it can execute the algorithm test tasks specified by the user.

[0086] 2. Sample data reading: The algorithm test program reads the test sample files in YUV format from the specified directory, and these sample files have been prepared during the transcoding process of the control module.

[0087] 3. Algorithm recognition: The algorithm test program calls the algorithm recognition interface to perform algorithm recognition on the sample data. This includes the execution of the camera algorithm to generate corresponding test results.

[0088] 4. Result writing: The test results (usually in CSV format) are written to a file, including performance metrics such as image processing speed and accuracy for further analysis and report generation.

[0089] The technical solution of the embodiment of the present disclosure allocates test samples and test tasks to multiple network cameras based on test requirements; wherein, the same target recognition algorithm is installed on the multiple network cameras, the test task is used to indicate the algorithm performance of the test network camera, and the test samples are extracted from the sample library; obtain the test data respectively returned by the multiple network cameras; wherein, the test tasks and test samples corresponding to different network cameras are different, and the test data includes the test sample identification number, the targets identified in the test sample corresponding to the test sample identification number, and the confidence score of each target; integrate and analyze the test data corresponding to each of the multiple network cameras to obtain the algorithm performance test result of the target recognition algorithm. In this way, testers can easily maintain and update the sample library, and use the recorded video samples to test the algorithm installed in the network camera, greatly improving the efficiency and accuracy of the test.

[0090] In some embodiments, the method for testing the algorithm performance of the network camera further includes: obtaining a plurality of sample videos recorded by the network camera, the plurality of sample videos including sample videos in different scenarios; determining normal test samples and abnormal test samples based on the plurality of sample videos; storing the normal test samples and abnormal test samples in the sample library.

[0091] In the embodiment of the present disclosure, the sample videos in different scenarios may refer to sample videos of night scenes and day scenes; may also refer to sample videos of indoor scenes and outdoor scenes; may also refer to sample videos of sunny weather, rainy weather, snowy weather, etc.

[0092] In the embodiment of the present disclosure, the network camera for recording the plurality of sample videos may be a network camera in a test state, or a network camera installed in an actual scenario and operating normally. Exemplarily, IPC device 1, IPC device 2, and IPC device 3 are respectively installed in a hospital, a villa, and a school, and the videos recorded by IPC device 1 and IPC device 2 in a normal operating state may be used as sample videos.

[0093] The technical solution of the embodiment of the present disclosure obtains a plurality of sample videos recorded by the network camera, the plurality of sample videos including sample videos in different scenarios; determines normal test samples and abnormal test samples based on the plurality of sample videos; stores the normal test samples and abnormal test samples in the sample library. In this way, by applying the plurality of sample videos in different scenarios to the test of the network camera, the authenticity and diversity of the test samples can be improved, thereby improving the authenticity and diversity of the test data, and further improving the accurate algorithm performance test result.

[0094] In some embodiments, the method for testing the algorithm performance of the network camera further includes: obtaining abnormal videos reported by a user, where the abnormal videos include videos for which the network camera fails to recognize; determining abnormal test samples based on the abnormal videos; and storing the abnormal test samples in a sample library.

[0095] In the embodiments of the present disclosure, the abnormal video may refer to a video affected by abnormal factors during recording, such as abnormal weather, abnormal light, etc. The abnormal video may also refer to videos in which the target recognition algorithm installed in the network camera makes incorrect or failed identifications.

[0096] In the embodiments of the present disclosure, the user deploys the network camera in the hospital to identify vehicles in the hospital. In sunny weather, vehicle identification is normal, while in heavy rain weather, there are situations such as missed reports and failed identifications in vehicle identification. The recorded video during this heavy rain weather is determined as an abnormal test sample, and the abnormal test sample is stored in the sample library.

[0097] The technical solution of the embodiments of the present disclosure is to obtain abnormal videos reported by a user, where the abnormal videos include videos for which the network camera fails to recognize; determine abnormal test samples based on the abnormal videos; and store the abnormal test samples in the sample library. In this way, the abnormal videos uploaded by the user can be determined as abnormal test samples, which helps to comprehensively cover various test scenarios, thereby helping to improve the accuracy and efficiency of testing.

[0098] In some embodiments, integrating and analyzing the test data corresponding to multiple network cameras respectively to obtain the algorithm performance test result of the target recognition algorithm includes: analyzing the first test data returned by the first network camera among the multiple network cameras to obtain a first analysis result of the first recognition function of the target recognition algorithm; where the first test data is obtained by testing based on a first test task, and the first test task is used to indicate testing the first recognition function of the target recognition algorithm; analyzing the second test data returned by the second network camera among the multiple network cameras to obtain a second analysis result of the second recognition function of the target recognition algorithm; where the second test data is obtained by testing based on a second test task, and the second test task is used to indicate testing the second recognition function of the target recognition algorithm; and integrating based on the first analysis result of the first recognition function and the second analysis result of the second recognition function to obtain the algorithm performance test result of the target recognition algorithm.

[0099] In the embodiments of the present disclosure, the target recognition algorithm includes a first recognition function and a second recognition function; the first recognition function is to recognize "moving objects"; the second recognition function is to recognize "stationary objects"; the test task of network camera 1 is to test the accuracy of the first recognition function, the test sample is a video of "a vehicle driving out of the garage", and based on the test task and the test sample, the first test data returned by network camera 1 is "a white car is driving out of the garage"; the test task of network camera 2 is to test the accuracy of the second recognition function, the test sample is an image of "a kitten lying on the sofa", and based on the test task and the test sample, the second test data returned by network camera 2 is "there is a kitten on the sofa"; by analyzing the first test data and the second test data, the accuracies of the first recognition function and the second recognition function of the target recognition algorithm are obtained.

[0100] For the technical solution of the embodiments of the present disclosure, the first test data returned by the first network camera among multiple network cameras is analyzed to obtain a first analysis result of the first recognition function of the target recognition algorithm; the second test data returned by the second network camera among multiple network cameras is analyzed to obtain a second analysis result of the second recognition function of the target recognition algorithm; based on the first analysis result of the first recognition function and the second analysis result of the second recognition function, an algorithm performance test result of the target recognition algorithm is obtained. In this way, by simultaneously testing multiple functions of the target recognition algorithm installed in the network camera, it helps to improve the test efficiency.

[0101] In some embodiments, integrating and analyzing the test data corresponding to each of the multiple network cameras includes: identifying the sample category of the test sample based on the test sample identification number of each test record in the test data, and the sample category is divided into positive samples and negative samples; if the sample category is a positive sample, analyzing the first category of targets in the positive sample and the recognized targets, and combining the confidence scores of each target to obtain the recognition result of the target recognition algorithm for the first category of targets; if the sample category is a negative sample, comparing the second category of targets in the negative sample with the recognized targets, and combining the confidence scores of each target to obtain the recognition result of the target recognition algorithm for the second category of targets; based on the weight ratio of the positive samples and the negative samples, analyzing the recognition result of the target recognition algorithm to obtain the algorithm performance test result of the target recognition algorithm.

[0102] In the embodiments of the present disclosure, the recognition result may include at least one of the accuracy rate of the target recognition algorithm, the false alarm rate of the target recognition algorithm, the miss rate of the target recognition algorithm, and the recall rate of the target recognition algorithm.

[0103] In the embodiments of the present disclosure, the positive samples include successfully recognized target objects; the negative samples include target objects with recognition errors or recognition failures. In actual tests, the test samples sent to the network camera 1 can all be positive samples; the test samples sent to a certain network camera 2 can all be negative samples; the test samples sent to the network camera 3 include both positive samples and negative samples.

[0104] In the embodiments of the present disclosure, if the confidence score in the test data is relatively high and the first type of target in the recognition result is incorrect, it can be concluded that there are serious errors (bugs) in the target recognition algorithm installed in the network camera; if the confidence score in the test data is low and the first type of target in the recognition result is incorrect, it can be concluded that there are minor errors in the target recognition algorithm installed in the network camera; if the confidence score in the test data is high and the first type of target in the recognition result is correct, it can be concluded that the performance of the target recognition algorithm installed in the network camera is good; if the confidence score in the test data is relatively low and the first type of target in the recognition result is correct, it can be concluded that the target recognition algorithm installed in the network camera needs further optimization.

[0105] The technical solution of the embodiments of the present disclosure identifies the sample category of the test sample based on the test sample identification number of each test record in the test data. The sample category is divided into positive samples and negative samples; if the sample category is a positive sample, analyze the first type of target in the positive sample and the recognized target, and combine the confidence score of each target to obtain the recognition result of the target recognition algorithm for the first type of target; if the sample category is a negative sample, compare the second type of target in the negative sample with the recognized target, and combine the confidence score of each target to obtain the recognition result of the target recognition algorithm for the second type of target; based on the weight ratio of the positive samples and negative samples, analyze the recognition result of the target recognition algorithm to obtain the algorithm performance test result of the target recognition algorithm. In this way, the target recognition algorithm can be tested through positive samples and negative samples, and the performance of the algorithm can be evaluated more comprehensively. At the same time, the generalization ability of the algorithm can be better evaluated.

[0106] In some embodiments, the algorithm performance test method of the network camera further includes: marking the targets with confidence scores lower than the preset value in the test samples to obtain new test samples, where the targets to be recognized in the new test samples are the targets with confidence scores lower than the preset value; storing the new test samples in the sample library.

[0107] In the embodiments of the present disclosure, the preset value can be set and adjusted according to actual requirements and the requirements of testers.

[0108] In an embodiment of the present disclosure, if the preset value is 80% and the confidence score of the network camera 1 for the test sample A is 60%, then the confidence score of the network camera 1 for the test sample A is lower than the preset value, and the test sample A is marked to obtain a new test sample; the new test sample is stored in the sample library.

[0109] The technical solution of the embodiment of the present disclosure marks the target with a confidence score lower than the preset value in the test sample to obtain a new test sample; the new test sample is stored in the sample library. In this way, the quality of the test sample can be improved, and the performance of the algorithm can be better evaluated.

[0110] In some embodiments, the algorithm performance test method of the network camera further includes: determining the optimization direction of the target recognition algorithm based on the algorithm performance test result; extracting test samples matching the optimization direction from the sample library based on the optimization direction.

[0111] In an embodiment of the present disclosure, if the target recognition algorithm has three recognition functions, the test results of the first recognition function and the second recognition function are good, and the test result of the third recognition function is poor, then the optimization direction of the algorithm should be to optimize the third recognition function.

[0112] The technical solution of the embodiment of the present disclosure can provide a direction for testers to optimize the target recognition algorithm installed in the network camera, which helps to improve the accuracy of the target recognition algorithm.

[0113] The embodiment of the present disclosure provides an algorithm performance test automation system, as Figure 6 shown, including a control device 10, a plurality of network cameras 20, and a storage device 30; wherein,

[0114] The control device 10 is configured to allocate test samples and test tasks to a plurality of network cameras based on test requirements; wherein, the same target recognition algorithm is installed on the plurality of network cameras, the test task is used to indicate the algorithm performance of the test network camera, and the test sample is extracted from the sample library; obtain the test data of each of the plurality of network cameras; wherein, the test data includes a test sample identification number, the target identified in the test sample corresponding to the test sample identification number, and the confidence score of each target; integrate and analyze the test data corresponding to each of the plurality of network cameras to obtain the algorithm performance test result of the target recognition algorithm;

[0115] The plurality of network cameras 20 are configured to receive the test tasks and test samples sent by the control device, and execute the test tasks received by each of them based on the test samples received by each of them; wherein, the test tasks and test samples corresponding to different network cameras are different;

[0116] The storage device 30 is configured to store the sample library; store the test data of each of the plurality of network cameras.

[0117] In the embodiments of the present disclosure, the working process of the control device 10 may include:

[0118] 1. Configuration file reading: When the program starts, the control device reads the configuration file, which includes the following information:

[0119] (1) Sample path: Specifies the path where the test samples are stored.

[0120] (2) Test result storage path: Specifies the path where the test results of the IPC device are stored.

[0121] (3) NFS configuration information: Configuration parameters required for the IPC device to mount the server.

[0122] 2. Sample transcoding: The control device performs transcoding operations on all files under the path according to the path in the configuration file, and converts the configuration file into the yuv format recognizable by the IPC device to ensure the consistency of the sample data.

[0123] 3. LAN device discovery: The control device enables UDP broadcast message listening to listen for broadcast messages in the LAN. The broadcast messages are used to discover available IPC devices. Whenever the control device receives a broadcast message, it identifies the IPC device information in the message.

[0124] 4. Telnet connection: For each IPC device discovered through the broadcast message, the control device uses Telnet connection to establish a communication channel with the IPC device. Through the Telnet connection, the control device can send commands and instructions to the IPC device.

[0125] 5. Test task distribution: The control device sends test tasks to the connected IPC devices. The tasks include specifying the test samples to be used and the commands for algorithm execution.

[0126] 6. Test result statistics: The control device waits for all connected IPC devices to complete the algorithm recognition tasks. When all tasks are completed, the control device collects the test results of each device, including performance metrics such as image processing speed, accuracy, and stability.

[0127] 7. Test report generation: Based on the collected test results, the control device can generate a detailed test report to display the performance data of each IPC device for testers to analyze and evaluate.

[0128] In the embodiments of the present disclosure, the working process of the multiple IPC devices 20 may include:

[0129] Multiple IPC devices can include two processes: 1. IPC device discovery program and algorithm test program; 2. The IPC device discovery program runs automatically after the IPC device is started, and the algorithm test program is remotely executed by the control module.

[0130] Start the device discovery process: After the IPC device is started, it will automatically start a device discovery process, and the task of this process is to broadcast messages to the local area network at regular intervals.

[0131] Content of the broadcast message: Each broadcast message contains the following key information:

[0132] (1) Unique identification code: Used to distinguish this device from other IPC devices.

[0133] (2) Telnet login information: Includes username and password, so that the control device can connect to the device through Telnet.

[0134] (3) Software version: The software version number of the IPC device, used to track device updates and compatibility.

[0135] (4) Algorithm model version: The version of the algorithm model used, so that the control device can coordinate test tasks.

[0136] Reply message processing: After the control device receives the broadcast message, it will reply to confirm the device availability. The IPC device determines whether to continue broadcasting based on the reply message from the control module to ensure that there are no conflicts among devices in the local area network.

[0137] With the technical solution of the embodiment of the present disclosure, testers can easily maintain and update the sample library, and use the recorded video samples to test the algorithms installed in the network camera, greatly improving the efficiency and accuracy of the test.

[0138] It should be understood that Figure 2 、 Figure 3 、 Figure 4 and Figure 5 The schematic diagrams shown are merely exemplary rather than restrictive, and they are extensible. Those skilled in the art can make various obvious changes and / or substitutions based on the examples of Figure 2 、 Figure 3 、 Figure 4 and Figure 5 The obtained technical solutions still fall within the scope of the disclosure of the embodiment of the present disclosure.

[0139] The embodiment of the present disclosure provides an algorithm performance test device for a network camera, as shown in Figure 5 The algorithm performance test device for the network camera includes:

[0140] An allocation unit 710 for allocating test samples and test tasks to multiple network cameras based on test requirements; wherein, the same target recognition algorithm is installed on the multiple network cameras, the test task is used to indicate the algorithm performance of the test network camera, and the test samples are extracted from the sample library;

[0141] A first acquisition unit 720 for acquiring the test data respectively returned by the multiple network cameras; wherein, the test tasks and test samples corresponding to different network cameras are different, and the test data includes a test sample identification number, the targets identified in the test sample corresponding to the test sample identification number, and the confidence score of each target;

[0142] An integration and analysis unit 730 for integrating and analyzing the test data corresponding to each of the multiple network cameras to obtain the algorithm performance test result of the target recognition algorithm.

[0143] In some embodiments, the algorithm performance test device of the network camera further includes: a second acquisition unit ( Figure 7 not shown in the figure) for acquiring a plurality of sample videos recorded by the network camera, the plurality of sample videos including sample videos in different scenarios; a first determination unit ( Figure 7 not shown in the figure) for determining normal test samples and abnormal test samples based on the plurality of sample videos; a first storage unit ( Figure 7 not shown in the figure) for storing the normal test samples and abnormal test samples into the sample library.

[0144] In some embodiments, the algorithm performance test device of the network camera further includes: a third acquisition unit ( Figure 7 not shown in the figure) for acquiring the abnormal videos reported by the user, the abnormal videos including the videos in which the network camera fails to recognize; a second determination unit ( Figure 7 not shown in the figure) for determining abnormal test samples based on the abnormal videos; a second storage unit ( Figure 7 not shown in the figure) for storing the abnormal test samples into the sample library.

[0145] In some embodiments, the integration analysis unit 730 includes: a first analysis subunit, configured to analyze first test data returned by a first network camera among a plurality of network cameras to obtain a first analysis result of a first recognition function of a target recognition algorithm; wherein, the first test data is obtained by testing based on a first test task, and the first test task is used to indicate testing the first recognition function of the target recognition algorithm; a second analysis subunit, configured to analyze second test data returned by a second network camera among a plurality of network cameras to obtain a second analysis result of a second recognition function of the target recognition algorithm; wherein, the second test data is obtained by testing based on a second test task, and the second test task is used to indicate testing the second recognition function of the target recognition algorithm; an integration subunit, configured to perform integration based on the first analysis result of the first recognition function and the second analysis result of the second recognition function to obtain an algorithm performance test result of the target recognition algorithm.

[0146] In some embodiments, the integration analysis unit 730 includes: a recognition subunit, configured to recognize a sample category of a test sample based on a test sample identification number of each test record in the test data, and the sample category is divided into positive samples and negative samples; a third analysis subunit, configured to, if the sample category is a positive sample, analyze the first category of targets in the positive sample and the recognized targets, and combine the confidence score of each target to obtain a recognition result of the target recognition algorithm for the first category of targets; a comparison subunit, configured to, if the sample category is a negative sample, compare the second category of targets in the negative sample with the recognized targets, and combine the confidence score of each target to obtain a recognition result of the target recognition algorithm for the second category of targets; a fourth analysis subunit, configured to analyze the recognition result of the target recognition algorithm based on the weight ratio of the positive samples and the negative samples to obtain an algorithm performance test result of the target recognition algorithm.

[0147] In some embodiments, the algorithm performance test device of the network camera further includes: a marking unit ( Figure 7 not shown in the figure), configured to mark the targets in the test samples with confidence scores lower than a preset value to obtain new test samples, wherein the targets to be recognized in the new test samples are the targets with confidence scores lower than the preset value; a third storage unit ( Figure 7 not shown in the figure), configured to store the new test samples into a sample library.

[0148] In some embodiments, the algorithm performance test device of the network camera further includes: a third determination unit ( Figure 7 not shown in the figure), configured to determine an optimization direction of the target recognition algorithm based on the algorithm performance test result; an extraction module ( Figure 7 not shown in the figure), configured to extract test samples matching the optimization direction from the sample library based on the optimization direction.

[0149] Those skilled in the art should understand that the functions of the various processing modules in the algorithm performance testing device of the network camera according to the embodiments of the present disclosure can be understood with reference to the relevant descriptions of the aforementioned algorithm performance testing method of the network camera. The various processing modules in the algorithm performance testing device of the network camera according to the embodiments of the present disclosure can be implemented by an analog circuit that implements the functions described in the embodiments of the present disclosure, or can be implemented by the operation of software that executes the functions described in the embodiments of the present disclosure on an electronic device.

[0150] In the algorithm performance testing device of the network camera according to the embodiments of the present disclosure, testers can easily maintain and update the sample library and perform algorithm testing using the recorded video samples, greatly improving the efficiency and accuracy of testing.

[0151] In the technical solution of the present disclosure, the acquisition, storage, and application of the personal information of the target object involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0152] Figure 8 It is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As Figure 8 shown, the electronic device includes: a memory 810 and a processor 820. The memory 810 stores a computer program that can run on the processor 820. The number of the memory 810 and the processor 820 can be one or more. The memory 810 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided in the above method embodiment. The electronic device may further include: a communication interface 830, configured to communicate with external devices and perform data interaction and transmission.

[0153] If the memory 810, the processor 820, and the communication interface 830 are implemented independently, the memory 810, the processor 820, and the communication interface 830 can be interconnected through a bus and complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 8 only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0154] Optionally, in a specific implementation, if the memory 810, the processor 820, and the communication interface 830 are integrated on a single chip, the memory 810, the processor 820, and the communication interface 830 can communicate with each other through an internal interface.

[0155] It should be understood that the above-mentioned processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the advanced reduced instruction set machine (ARM) architecture.

[0156] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may further include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0157] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present disclosure are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center to another website, a computer, a server, or a data center in a wired manner (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or a wireless manner (such as infrared, Bluetooth, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a Digital Versatile Disc (DVD)), or a semiconductor medium (such as a Solid State Disk (SSD)), etc. It should be noted that the computer-readable storage medium mentioned in the present disclosure can be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0158] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by hardware, or can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium, and the above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, or the like.

[0159] In the description of the embodiments of the present disclosure, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0160] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means "or". For example, A / B may mean A or B. The "and / or" herein is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0161] In the description of the embodiments of the present disclosure, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, unless otherwise specified, "a plurality of" means two or more.

[0162] The above are only exemplary embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. An algorithm performance testing method for a network camera, comprising: Allocating test samples and test tasks to multiple network cameras based on test requirements; wherein, the multiple network cameras are installed with the same target recognition algorithm, the test tasks are used to indicate the algorithm performance of the test network cameras, and the test samples are extracted from a sample library; Obtaining the test data respectively returned by the multiple network cameras; wherein, the test tasks and the test samples corresponding to different network cameras are different, and the test data includes a test sample identification number, the targets identified in the test sample corresponding to the test sample identification number, and the confidence score of each target; Integrating and analyzing the test data respectively corresponding to the multiple network cameras to obtain the algorithm performance test result of the target recognition algorithm.

2. The method according to claim 1, characterized in that, The method further includes: Obtaining multiple sample videos recorded by the network camera, the multiple sample videos including sample videos in different scenarios; Determining normal test samples and abnormal test samples based on the multiple sample videos; Storing the normal test samples and the abnormal test samples into the sample library.

3. The method according to claim 1, wherein The method further includes: Obtaining abnormal videos reported by users, the abnormal videos including videos in which the network camera fails to recognize; Determining abnormal test samples based on the abnormal videos; Storing the abnormal test samples into the sample library.

4. The method according to claim 1, wherein The integrating and analyzing the test data respectively corresponding to the multiple network cameras to obtain the algorithm performance test result of the target recognition algorithm includes: Analyzing the first test data returned by a first network camera among the multiple network cameras to obtain a first analysis result of the first recognition function of the target recognition algorithm; wherein, the first test data is obtained by testing based on a first test task, and the first test task is used to indicate testing the first recognition function of the target recognition algorithm; Analyzing the second test data returned by a second network camera among the multiple network cameras to obtain a second analysis result of the second recognition function of the target recognition algorithm; wherein, the second test data is obtained by testing based on a second test task, and the second test task is used to indicate testing the second recognition function of the target recognition algorithm; Integrating based on the first analysis result of the first recognition function and the second analysis result of the second recognition function to obtain the algorithm performance test result of the target recognition algorithm.

5. The method according to claim 1, wherein The integrating and analyzing the test data respectively corresponding to the multiple network cameras to obtain the algorithm performance test result of the target recognition algorithm includes: Identifying the sample category of the test sample based on the test sample identification number of each test record in the test data, and the sample category is divided into positive samples and negative samples; If the sample category is a positive sample, analyzing based on the first category of targets in the positive sample and the identified targets, and combining the confidence score of each target to obtain the recognition result of the target recognition algorithm for the first category of targets; If the sample category is a negative sample, compare the second-category targets in the negative sample with the identified targets, and combine the confidence scores of each target to obtain the recognition result of the target recognition algorithm for the second-category targets; Analyze the recognition results of the target recognition algorithm based on the weight ratio of positive samples and negative samples to obtain the algorithm performance test result of the target recognition algorithm.

6. The method according to claim 1, wherein The method further includes: Mark the targets with confidence scores lower than the preset value in the test samples to obtain new test samples, where the targets to be recognized in the new test samples are the targets with confidence scores lower than the preset value; Store the new test samples in the sample library.

7. The method according to claim 1, characterized in that, The method further includes: Determine the optimization direction of the target recognition algorithm based on the algorithm performance test result; Extract test samples matching the optimization direction from the sample library based on the optimization direction.

8. An algorithm performance test device for a network camera, comprising: An allocation unit for allocating test samples and test tasks to multiple network cameras based on test requirements; wherein, the same target recognition algorithm is installed on the multiple network cameras, the test tasks are used to indicate the algorithm performance of the test network cameras, and the test samples are extracted from the sample library; A first acquisition unit for acquiring the test data respectively returned by the multiple network cameras; wherein, the test tasks and the test samples corresponding to different network cameras are different, and the test data includes test sample identification numbers, the targets identified in the test samples corresponding to the test sample identification numbers, and the confidence scores of each target; An integration and analysis unit for integrating and analyzing the test data corresponding to each of the multiple network cameras to obtain the algorithm performance test result of the target recognition algorithm.

9. The device according to claim 8, wherein The device further includes: A second acquisition unit for acquiring multiple sample videos recorded by the network camera, where the multiple sample videos include sample videos in different scenarios; A first determination unit for determining normal test samples and abnormal test samples based on the multiple sample videos; A first storage unit for storing the normal test samples and the abnormal test samples in the sample library.

10. The device according to claim 8, characterized in that, The device further includes: A third acquisition unit for acquiring abnormal videos reported by users, where the abnormal videos include videos in which the network camera fails to recognize; A second determination unit for determining abnormal test samples based on the abnormal videos; A second storage unit for storing the abnormal test samples in the sample library.

11. The device according to claim 8, characterized in that, The integration and analysis unit includes: A first analysis subunit for analyzing the first test data returned by the first network camera among the multiple network cameras to obtain a first analysis result of the first recognition function of the target recognition algorithm; wherein, the first test data is obtained by testing based on a first test task, and the first test task is used to indicate testing the first recognition function of the target recognition algorithm; A second analysis subunit, configured to analyze second test data returned by a second network camera among the multiple network cameras, so as to obtain a second analysis result of a second recognition function of the target recognition algorithm; wherein, the second test data is obtained by testing based on a second test task, and the second test task is used to indicate testing the second recognition function of the target recognition algorithm; An integration subunit, configured to perform integration based on the first analysis result of the first recognition function and the second analysis result of the second recognition function, so as to obtain an algorithm performance test result of the target recognition algorithm.

12. The device according to claim 8, characterized in that, The integration and analysis unit includes: A recognition subunit, configured to recognize a sample category of a test sample based on a test sample identification number of each test record in the test data, and the sample category is divided into positive samples and negative samples; A third analysis subunit, configured to, if the sample category is a positive sample, analyze the first category of targets in the positive sample and the recognized targets, and combine the confidence scores of each target to obtain a recognition result of the target recognition algorithm for the first category of targets; A comparison subunit, configured to, if the sample category is a negative sample, compare the second category of targets in the negative sample with the recognized targets, and combine the confidence scores of each target to obtain a recognition result of the target recognition algorithm for the second category of targets; A fourth analysis subunit, configured to analyze the recognition result of the target recognition algorithm based on the weight ratio of the positive samples and the negative samples, so as to obtain an algorithm performance test result of the target recognition algorithm.

13. The device according to claim 8, characterized in that, The apparatus further includes: A marking unit, configured to mark targets with confidence scores lower than a preset value in a test sample to obtain a new test sample, wherein the target to be recognized in the new test sample is the target with a confidence score lower than the preset value; A third storage unit, configured to store the new test sample into the sample library.

14. The device according to claim 8, characterized in that, The apparatus further includes: A third determination unit, configured to determine an optimization direction of the target recognition algorithm based on the algorithm performance test result; An extraction module, configured to extract a test sample matching the optimization direction from the sample library based on the optimization direction.

15. An automated system for testing algorithm performance, characterized in that, It includes: A control device, configured to allocate test samples and test tasks to multiple network cameras based on test requirements; wherein, the same target recognition algorithm is installed on the multiple network cameras, the test task is used to indicate testing the algorithm performance of the network cameras, and the test samples are extracted from a sample library; obtain the test data of each of the multiple network cameras; wherein, the test data includes a test sample identification number, the recognized targets in the test sample corresponding to the test sample identification number, and the confidence score of each target; perform integration and analysis on the test data corresponding to each of the multiple network cameras to obtain an algorithm performance test result of the target recognition algorithm; Multiple network cameras, configured to receive the test tasks and test samples sent by the control device, and execute the received test tasks based on the received test samples; wherein, the test tasks and test samples corresponding to different network cameras are different; A storage device for storing the sample library; storing the test data of each of the multiple network cameras.

16. An electronic device, characterized in that, Comprising: At least one processor; And A memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-7.

Images