Camera chip testing method and related equipment

Abstract

The present disclosure provides a camera chip testing method and related equipment, and relates to the field of intelligent monitoring technology. The method includes, according to a unified message interaction protocol, sending a connection protocol, connecting with multiple cameras to be tested; sending chip test data to each camera to be tested, so that each camera to be tested performs a test according to the chip test data, and obtains the test results of each chip, wherein the cameras to be tested are set with the same algorithm model; receiving the chip test result information returned by each camera to be tested, and determining the chip test result table. The present disclosure combines a unified message interaction protocol with a unified setting algorithm model, which can be compatible with chip architectures and interfaces of different manufacturers and shield the impact of the algorithm on chip performance, overcome the problem of inconsistent test calculation standards for camera chips in related technologies, and realize horizontal comparison of chip performance.

Description

Background Art

[0002] With the rapid development of chips, deep learning algorithms, big data and cloud computing, artificial intelligence technology has been gradually put into use in many industries. With the integration of artificial intelligence (AI), AI smart cameras are widely used in my country, with a large number of users and a large market. At present, AI front-end cameras have accounted for nearly 70% of the market for intelligent video surveillance applications, showing an attractive prospect. In the next 20 years, the total market volume of AI front-end cameras is likely to exceed 1 trillion units. The core of AI smart surveillance cameras lies in AI intelligent algorithms, and its main functions include: pattern classification, pattern recognition, AI mobile trajectory tracking and shooting, etc. However, the quality of AI cameras depends largely on the computing power of AI chips and the accuracy of pattern recognition. Now there are thousands of AI camera products on the market with varying computing power performance.

[0003] In the existing technology, the industry has different interpretations of the product capabilities of AI camera products, and each manufacturer has its own interpretation of the performance of its products. The calculation standards of the performance indicators marked by manufacturers on their products are not unified, and they have no reference value for horizontal comparison.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a camera chip testing method and related equipment, which at least to a certain extent overcome the problem of inconsistent testing and calculation standards for camera chips in the related art.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a method for testing a camera chip is provided, comprising: sending a connection protocol according to a unified message interaction protocol to connect to multiple cameras to be tested; sending chip test data to each camera to be tested, so that each camera to be tested performs a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be tested are set with the same algorithm model; receiving chip test result information returned by each camera to be tested, and determining a chip test result table.

[0008] In some embodiments, the method further includes: if the chip test result of a camera to be detected and a preset chip performance value meet a preset condition, then determining that the camera to be detected is a cheating camera to be detected; performing an anti-cheating detection on the cheating camera to be detected by a preset anti-cheating method, and determining an anti-cheating detection result of the cheating camera to be detected.

[0009] In some embodiments, the method further includes: calculating a first offset value between the chip test result and a preset chip performance value; if the first offset value exceeds a preset threshold, determining that a preset condition is met.

[0010] In some embodiments, the preset anti-cheating method includes: modifying the algorithm model in the cheating camera chip to be detected, and / or modifying the chip test data.

[0011] In some embodiments, when the preset anti-cheating method includes modifying the algorithm model in the chip of the cheating camera to be detected, the anti-cheating detection is performed on the cheating camera to be detected by the preset anti-cheating method, and determining the anti-cheating detection result of the cheating camera to be detected includes: increasing the number of layers of the algorithm model in the chip of the cheating camera to be detected from a first value to a second value, and / or increasing the parameter information of the algorithm model from a third value to a fourth value, testing the chip test data again, and determining the first inspection test result information; if the second offset value between the first inspection test result information and the chip test result information exceeds the preset threshold, it is determined that the cheating camera to be detected is not cheating.

[0012] In some embodiments, when the preset anti-cheating method includes modifying chip test data, the anti-cheating detection is performed on the cheating camera to be detected by the preset anti-cheating method, and determining the anti-cheating detection result of the cheating camera to be detected includes: modifying the data in the chip test data according to a preset ratio; sending the modified chip test data to the cheating camera to be detected, and determining second inspection test result information; if a third offset value between the second inspection test result information and the chip test result information exceeds a preset threshold, it is determined that the cheating camera to be detected is not cheating.

[0013] In some embodiments, before sending chip test data to each camera to be detected, the method also includes: polling to obtain the status of the camera to be detected; when the camera to be detected is in an idle state, sending chip test data to each camera to be detected; when the camera to be detected is in a non-idle state, after a preset time interval, obtaining the status of the camera to be detected again.

[0014] In some embodiments, before sending a preset connection protocol to connect to multiple cameras to be detected, the method further includes: retrieving cameras in a preset Internet Protocol IP segment; adding the retrieved cameras to a device list and determining them as cameras to be detected.

[0015] In some embodiments, sending a connection protocol according to a unified message interaction protocol to connect with multiple cameras to be detected includes: sending a hello message; and when receiving an information message returned by the camera to be detected, connecting with the camera to be detected.

[0016] In some embodiments, sending the chip test data to each camera to be detected includes: sending a test request message, wherein the test request message includes the chip test data; and receiving a return message after the camera to be detected confirms the request.

[0017] In some embodiments, receiving the test result information of the chip returned by each camera to be detected and determining the chip test result table includes: sending a test result acquisition message; acquiring a test result message, wherein the test result message includes test result information; and determining the chip test result table based on multiple test result messages.

[0018] According to another aspect of the present disclosure, a camera chip testing device is also provided, characterized in that it includes: a sending connection protocol module, used to send a connection protocol according to a unified message interaction protocol to connect to multiple cameras to be tested; a chip test data sending module, used to send chip test data to each camera to be tested, so that each camera to be tested performs a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be tested are set with the same algorithm model; a chip test set determination module, used to receive the chip test result information returned by each camera to be tested, and determine the chip test result table.

[0019] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned camera chip testing methods by executing the executable instructions.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for testing a camera chip described in any one of the above is implemented.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned camera chip testing methods.

[0022] The camera chip testing method provided in the embodiment of the present disclosure sends a connection protocol according to a unified message interaction protocol to connect with multiple cameras to be tested; sends chip test data to each camera to be tested, so that each camera to be tested performs a test according to the chip test data to obtain the test results of each chip, wherein the cameras to be tested are set with the same algorithm model; receives the chip test result information returned by each camera to be tested, and determines the chip test result table. The present disclosure uses a unified message interaction protocol combined with a unified setting of an algorithm model to be compatible with chip architectures and interfaces of different manufacturers and shield the impact of the algorithm on chip performance, overcomes the problem of inconsistent test calculation standards for camera chips in related technologies, and realizes horizontal comparison of chip performance.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0025] Figure 1 A schematic diagram showing an exemplary application system architecture of a camera chip testing method according to an embodiment of the present disclosure;

[0026] Figure 2 A flow chart of a camera chip testing method in an embodiment of the present disclosure is shown;

[0027] Figure 3 A flowchart showing a specific example of a camera chip testing method in an embodiment of the present disclosure;

[0028] Figure 4 A flowchart showing another specific example of a camera chip testing method according to an embodiment of the present disclosure;

[0029] Figure 5 A flowchart showing another specific example of a camera chip testing method in an embodiment of the present disclosure;

[0030] Figure 6 A flow chart showing a unified message interface protocol transmission method in an embodiment of the present disclosure is shown;

[0031] Figure 7 A flow chart of an anti-cheating test method for a camera chip in an embodiment of the present disclosure is shown;

[0032] Figure 8 A schematic diagram showing an exemplary device architecture of a camera chip testing method according to an embodiment of the present disclosure;

[0033] Fig. 9 A schematic diagram of a camera chip testing device in an embodiment of the present disclosure is shown;

[0034] Fig.10 A structural block diagram of a computer device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0036] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0037] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0038] IP: Internet Protocol, Internet interconnection protocol;

[0039] Linux: Linux Is No UniX, operating system kernel;

[0040] PC:Personal Computer,personal computer;

[0041] AI: Artificial Intelligence.

[0042] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0043] Figure 1FIG. 1 is a schematic diagram showing an exemplary application system architecture to which the camera chip testing method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include an application interaction layer 101 , a logic test layer 102 , a test physical layer 103 and a network 104 .

[0044] The network 104 is a medium for providing a communication link (using a unified message interaction protocol) between the logic test layer 102 and the test physical layer 103, and may be a wired network or a wireless network.

[0045] The application interaction layer 101 includes an intelligent discovery and connection module 1011 , an algorithm model library 1012 , a test progress monitoring module 1013 and a visualization report module 1014 .

[0046] The logic test layer 102 includes a test logic module 1021 and a data analysis module 1022 . The test logic module 1021 includes a device inspection logic 10211 , a test data set 10212 and a test logic 10213 .

[0047] The test physical layer 103 includes camera 1031 of manufacturer A, camera 1032 of manufacturer B, camera 1033 of manufacturer C, and camera 1034 of manufacturer D. All cameras have a unified exchange message format, use socket technology for network data transmission, and preset algorithm test models.

[0048] Optionally, a unified message interaction protocol: compatible with most mainstream camera chips in China;

[0049] Intelligent discovery and connection module: realize batch terminal search based on IP and IP segment;

[0050] Device detection logic: terminal detection and connection;

[0051] Test logic module: uniformly adopts common classic algorithm models (such as YOLO target detection algorithm model, MobileNet computer vision model, ResNet residual neural network model), shields the impact of algorithm differences on chip performance, and directly obtains the comprehensive performance of chip software and hardware;

[0052] Test data set: unify the test benchmark to achieve horizontal comparison;

[0053] Data analysis module: adapt to the Linux chip architecture and obtain the underlying chip test data.

[0054] Those skilled in the art will know that Figure 1 The number of cameras and networks in the example is only for illustration, and any number of cameras and networks may be provided according to actual needs. This disclosure embodiment does not limit this.

[0055] Figure 2 A flow chart of a camera chip testing method in an embodiment of the present disclosure is shown. Figure 2 As shown, the camera chip testing method provided in the embodiment of the present disclosure includes the following steps:

[0056] S202, sending a connection protocol according to a unified message interaction protocol to connect with multiple cameras to be detected.

[0057] It should be noted that the unified message interaction protocol can be a unified interface message format, and routing and data transmission are performed between hosts, for example, the PC end and the detected camera end (equivalent to the above-mentioned camera to be detected) perform routing and data transmission in a unified interface message format. The above-mentioned connection protocol can be a Hello message. The above-mentioned camera can be a camera with an AI algorithm, for example, an AI front-end camera, an AI smart surveillance camera, or an AI camera.

[0058] For example, a hello message is sent; when an information message returned by the camera to be detected is received, a connection is established with the camera to be detected.

[0059] S204, sending chip test data to each camera to be detected, so that each camera to be detected performs a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are set with the same algorithm model.

[0060] It should be noted that the above chip test data can be the same test data set. The above chip can be an AI chip of the camera to be detected. The above test result information can be the performance test result data of the AI ​​chip, for example, the chip performance measurement index data, such as computing power, power consumption, area, throughput and latency, accuracy, scalability and flexible applicability. The above algorithm model can be an AI algorithm for target detection, for example, a YOLO target detection algorithm model, a MobileNet computer vision model, a ResNet residual neural network model, specifically, the fifth generation YOLO target detection algorithm model of YOLO_v5s, an improved computer vision model of MobileNet_v2, and a new deep residual network model of ResNet_v2-50.

[0061] For example, a test request message is sent, the test request message includes chip test data; and a return message is received after the camera to be tested confirms the request.

[0062] S206, receiving chip test result information returned by each camera to be tested, and determining a chip test result table.

[0063] It should be noted that the above chip test result table includes the test result information of the chip returned by each camera to be tested.

[0064] For example, a test result acquisition message is sent; a test result message is acquired, the test result message includes test result information; and a chip test result table is determined according to a plurality of test result messages.

[0065] The present invention adopts a unified (same) message interaction protocol combined with a unified (same) algorithm model, which can be compatible with chip architectures and interfaces of different manufacturers and shield the impact of algorithms on chip performance, overcome the problem of inconsistent test and calculation standards for camera chips in related technologies, and realize horizontal comparison of chip AI performance.

[0066] In one embodiment of the present disclosure, the testing method of the camera chip further includes: if a chip test result of a camera to be detected and a preset chip performance value meet a preset condition, then a camera to be detected is determined to be a cheating camera to be detected; performing anti-cheating detection on the cheating camera to be detected by a preset anti-cheating method, and determining an anti-cheating detection result of the cheating camera to be detected.

[0067] In one embodiment of the present disclosure, Figure 3 As shown, the camera chip testing method provided in the embodiment of the present disclosure can determine whether the chip needs to be tested for anti-cheating through the following steps, and can check whether the chip actually runs the model imported by the test system (the algorithm model is uniformly set), effectively avoiding the falsification of chip performance data:

[0068] S302, calculating a first offset value between a chip test result and a preset chip performance value;

[0069] S304: If the first offset value exceeds a preset threshold, it is determined that a preset condition is met.

[0070] It should be noted that the above offset value can be used to indicate the degree of difference between the two data of the calculated chip test result and the preset chip performance value. For example, the offset value is determined by percentage change (calculation), the difference between the two numbers (chip test result and preset chip performance value) is calculated, and then the difference is divided by the average value of the original value, and finally the result is multiplied by 100 to obtain the percentage change; the offset value is determined by standard deviation, the standard deviation of the two sets of data (chip test result and preset chip performance value) is calculated, and then the standard deviation is divided by the average value of the average values ​​of the two sets of data; the offset value is determined by relative error, the difference between the two numbers (chip test result and preset chip performance value) is calculated, and then the difference is divided by the absolute value of the average value of the two numbers. The above preset threshold value can be a value preset according to the calculation method adopted by the offset value, such as a percentage number or a real number. The above preset chip performance value can be set according to the performance of the current mainstream chip.

[0071] In one example, the preset anti-cheating method includes: modifying the algorithm model in the cheating camera chip to be detected, and / or modifying the chip test data.

[0072] In a specific embodiment, when the preset anti-cheating method includes modifying the algorithm model in the chip of the cheating camera to be detected, the cheating camera to be detected is subjected to anti-cheating detection through the preset anti-cheating method, and determining the anti-cheating detection result of the cheating camera to be detected includes: increasing the number of layers of the algorithm model in the chip of the cheating camera to be detected from a first value to a second value, and / or increasing the parameter information of the algorithm model from a third value to a fourth value, testing the chip test data again, and determining the first inspection test result information; if the second offset value between the first inspection test result information and the chip test result information exceeds the preset threshold value, it is determined that the cheating camera to be detected is not cheating.

[0073] For example, modify the parameters Conf_Thres (Confidence Threshold) and Iou_Thres (Intersect over Union Threshold) of the model (equivalent to the algorithm model with the above unified settings), and then use the normal test data set for testing. The test frame rate will drop by 8% (equivalent to the above preset threshold) to be normal; if the frame rate drops abnormally (the percentage of drop is less than 8%), there is suspicion of cheating, and the code can be further investigated.

[0074] For example, change the number of layers of the algorithm model from three to five, and then use the normal test data set for testing.

[0075] Alternatively, the parameters of the number of layers of the algorithm model may be modified at the same time, and then the normal test data set may be used for testing.

[0076] In a specific embodiment, when the preset anti-cheating method includes modifying chip test data, an anti-cheating detection is performed on the cheating camera to be detected through the preset anti-cheating method, and determining the anti-cheating detection result of the cheating camera to be detected includes: modifying the data in the chip test data according to a preset ratio; sending the modified chip test data to the cheating camera to be detected, and determining second inspection test result information; if the third offset value between the second inspection test result information and the chip test result information exceeds a preset threshold, it is determined that the cheating camera to be detected is not cheating.

[0077] For example, change 10% (equivalent to the above preset threshold) of the classification result table in the test data set classification result table to error values, and then run the test model (equivalent to the above uniformly set algorithm model) to check whether the measured accuracy (equivalent to the above offset value) has dropped by 8%-10%. If so, cheating suspicion is ruled out; if not, cheating suspicion is present and the code can be further investigated.

[0078] In one embodiment of the present disclosure, the method of modifying the number of layers or parameter information of the algorithm model in the cheating camera chip to be detected for anti-cheating detection can be combined with the method of modifying the chip test data for anti-cheating detection (superimposed combination method) to improve the accuracy of cheating detection.

[0079] For example, the number of layers or parameter information of the algorithm model in the cheating camera chip to be detected is first modified to perform anti-cheating detection. When it is determined that there is suspicion of cheating (the offset value does not exceed the preset threshold), the chip test data is modified to perform anti-cheating detection.

[0080] Alternatively, the chip test data may be modified first to perform anti-cheating detection. When it is determined that cheating is suspected (the offset value does not exceed a preset threshold), the number of layers or parameter information of the algorithm model in the cheating camera chip to be detected may be modified to perform anti-cheating detection.

[0081] In one embodiment of the present disclosure, Figure 4 As shown, the camera chip testing method provided in the embodiment of the present disclosure can determine the timing of sending chip test data through the following steps. Before sending chip test data to each camera to be tested, it can determine whether to perform chip testing according to the device state (when the device is idle) to avoid different chip test conditions affecting the test results:

[0082] S402, polling to obtain the status of the camera to be detected;

[0083] S404, when the camera to be detected is in an idle state, sending chip test data to each camera to be detected;

[0084] S406, when the camera to be detected is in a non-idle state, the state of the camera to be detected is obtained again after a preset time interval.

[0085] In one embodiment of the present disclosure, before sending a preset connection protocol to connect to multiple cameras to be detected, the testing method of the above-mentioned camera chip also includes: retrieving cameras in a preset Internet Protocol IP segment; adding the retrieved cameras to a device list and determining them as cameras to be detected.

[0086] Figure 5 A flowchart showing another specific example of a method for testing a camera chip in an embodiment of the present disclosure is shown as follows: Figure 5 As shown, the camera chip testing method provided in the embodiment of the present disclosure includes the following steps:

[0087] S501: The camera returns device information to the control platform;

[0088] S502: The control platform receives device information returned by the camera and discovers the device;

[0089] S503: The control platform sends the acquired device status to the camera;

[0090] S504: The camera returns the current device status;

[0091] S505: The control platform determines whether the current device status is idle, if so, jump to S507, if not, jump to S506;

[0092] S506: Wait for ten seconds and jump to S503;

[0093] S507: The control platform obtains the last inference result;

[0094] S508: The control platform sends an inference instruction to the camera;

[0095] S509: The camera starts AI inference and sets the device status to busy;

[0096] S510: Camera inference ends, and the device status is idle;

[0097] S511: The camera caches the result, overwriting the last inference result, and jumps to S507.

[0098] In one embodiment of the present disclosure, whether there is a camera device in the preset IP network segment is retrieved, and if the retrieval is successful, the device is added to the device list; a connection protocol connection is sent from the device in the device list, and after a successful handshake, the chip test module is jumped to; the test script, test model, and test data are sent to the test device storage unit; the test model is selected, and the start test button is clicked to send the test task to the camera device; the camera device is polled to obtain the idle state, and the test result is obtained when the camera device test task is completed; data analysis and processing are performed on the test results, and the results are stored in a data table; the history record page is opened to view the previous test data comparison results.

[0099] Figure 6 A flow chart of a unified message interface protocol transmission method in an embodiment of the present disclosure is shown. Figure 6 As shown, the unified message interface protocol transmission method provided in the embodiment of the present disclosure includes the following steps:

[0100] S601: Device discovery, the PC sends a Hello message to the device under test;

[0101] S602: The device under test returns an Info message to the PC;

[0102] S603: Start the test, and the PC sends a test request message (Request) to the device under test;

[0103] S604: The device under test confirms the request and returns a message (Ack);

[0104] S605: Query status, the PC sends a status query message (GetStatus);

[0105] S606: The device under test returns a device status message (Status);

[0106] S607: Get the result. The PC sends a Get Result message.

[0107] S608: Test result message (Result).

[0108] Figure 7 A flowchart of an anti-cheating test method for a camera chip in an embodiment of the present disclosure is shown. Figure 7 As shown, the anti-cheating test method for the camera chip provided in the embodiment of the present disclosure includes the following steps:

[0109] S701: Loading anti-cheating detection algorithm model;

[0110] S702: Import a normal test data set;

[0111] S703: Determine whether the test result is consistent with the estimated value, if so, jump to S704, if not, jump to S707;

[0112] S704: Loading a normal algorithm model;

[0113] S705: Importing anti-cheating detection data set;

[0114] S706: Determine whether the test result is consistent with the estimated value, if so, jump to S708, if not, jump to S707;

[0115] S707: Determine cheating and conduct in-depth inspection;

[0116] S708, determining that there is no cheating.

[0117] Among them, the anti-cheating detection algorithm model modifies the number of layers or algorithm parameters of the general model, and checks whether the test results are consistent with the modified estimated results to determine whether there is a possibility of cheating. If the test results are not within the estimated range, there is a possibility of cheating, which can be further investigated.

[0118] For example: modify the model Conf_Thres and Iou_Thres parameters, and then use the normal test data set for testing. The test frame rate will drop by more than 8% to be normal. If the frame rate drops abnormally, there is a suspicion of cheating, and you can further investigate the code.

[0119] The anti-cheating detection data set uses the normal algorithm model to modify the test data set so that the chip test results using the anti-cheating data set are significantly different from the original data set. If the test results are not within the expected range, there is a possibility of cheating, which can be further investigated.

[0120] For example: manually change 10% of the classification result tables in the test data set classification result table to error values, then run the test model to check whether the actual measured accuracy has dropped by 8%-10%. If not, there is suspicion of cheating and you can further investigate the code.

[0121] The present invention discloses a camera chip AI performance detection method and system with a unified interface message format, a unified AI algorithm test model and test data set, and a unified evaluation standard to achieve horizontal comparison of chip AI performance; an anti-cheating detection method can check whether the chip actually runs the model imported by the test system, effectively avoiding the falsification of chip performance data, and can check whether the chip test process runs the specified model and data set as required to improve the authority of the test.

[0122] This paper defines a set of test systems and chip interface message protocols suitable for camera chips, solving the problem of difficulty in adaptation due to different formats among manufacturers. The test process uses a unified classic and representative neural network algorithm model to test chip performance, unify the test benchmark, shield the impact of the algorithm model on chip performance evaluation, and unify the test data set and test standards to achieve horizontal comparison of chip AI performance.

[0123] The present disclosure provides a chip performance anti-cheating detection method, which dynamically modifies the parameters of the general algorithm model and the test data set, so that the test results are significantly different from the general model test results. By evaluating whether the calculation results after the chip is loaded with the anti-cheating detection model and the data set match the estimate, checking whether the chip actually runs the test system import model, effectively avoiding the falsification of chip performance data.

[0124] Figure 8 FIG. 1 is a schematic diagram showing an exemplary device architecture to which the camera chip testing method in the embodiment of the present disclosure can be applied. Figure 8 As shown, the device architecture may include a network AI camera 801, and an AI camera AI test application 802.

[0125] When the device is running, the AI ​​camera AI test application 802 sends test scripts, modes and data sets, and sends test script execution instructions to the network AI camera; the network AI camera returns test data to the AI ​​camera AI test application.

[0126] Based on the same inventive concept, the present disclosure also provides a camera chip testing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0127] Fig. 9 A schematic diagram of a camera chip testing device in an embodiment of the present disclosure is shown. Fig. 9 As shown, the device includes: a connection protocol sending module 91, a chip test data sending module 92, a chip test set determination module 93, an anti-cheating detection module 94, a camera status polling module 95 and an IP network segment retrieval module 96.

[0128] A connection protocol sending module 91 is used to send a connection protocol according to a unified message interaction protocol to connect with multiple cameras to be detected;

[0129] The chip test data sending module 92 is used to send chip test data to each camera to be detected, so that each camera to be detected performs a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are set with the same algorithm model;

[0130] The chip test set determination module 93 is used to receive the test result information of the chip returned by each camera to be detected and determine the chip test result table.

[0131] In one embodiment of the present disclosure, the testing device for the camera chip further includes an anti-cheating detection module 94, which is used to determine that a camera to be detected is a cheating camera to be detected if a chip test result of the camera to be detected and a preset chip performance value meet preset conditions; perform anti-cheating detection on the cheating camera to be detected by a preset anti-cheating method, and determine the anti-cheating detection result of the cheating camera to be detected.

[0132] In one embodiment of the present disclosure, the anti-cheating detection module 94 is further used to: calculate a first offset value between a chip test result and a preset chip performance value; if the first offset value exceeds a preset threshold, it is determined that a preset condition is met.

[0133] In one embodiment of the present disclosure, the anti-cheating method preset in the anti-cheating detection module 94 includes modifying the algorithm model in the cheating camera chip to be detected, and / or modifying the chip test data.

[0134] In one embodiment of the present disclosure, the anti-cheating detection module 94 is also used to: increase the number of layers of the algorithm model in the cheating camera chip to be detected from the first value to the second value, and / or increase the parameter information of the algorithm model from the third value to the fourth value, test the chip test data again, and determine the first inspection test result information; if the second offset value between the first inspection test result information and the chip test result information exceeds a preset threshold, it is determined that the cheating camera to be detected is not cheating.

[0135] In one embodiment of the present disclosure, the anti-cheating detection module 94 is also used to: modify the data in the chip test data according to a preset ratio; send the modified chip test data to the cheating camera to be detected, and determine the second inspection test result information; if the third offset value between the second inspection test result information and the chip test result information exceeds a preset threshold, it is determined that the cheating camera to be detected is not cheating.

[0136] In one embodiment of the present disclosure, the testing device of the above-mentioned camera chip also includes a polling camera status module 95, which is used to poll and obtain the status of the camera to be detected; when the camera to be detected is in an idle state, the chip test data is sent to each camera to be detected; when the camera to be detected is in a non-idle state, the status of the camera to be detected is obtained again after a preset time interval.

[0137] In one embodiment of the present disclosure, the camera chip test device further includes an IP segment retrieval module 96 for retrieving cameras in a preset Internet Protocol IP segment; adding the retrieved cameras to a device list and determining them as cameras to be tested.

[0138] In one embodiment of the present disclosure, the sending connection protocol module 91 is specifically used for: sending a hello message; and when receiving an information message returned by the camera to be detected, connecting with the camera to be detected.

[0139] In one embodiment of the present disclosure, the chip test data sending module 92 is specifically used to: send a test request message, the test request message includes chip test data; and receive a return message after the camera to be tested confirms the request.

[0140] In one embodiment of the present disclosure, the chip test set determination module 93 is specifically used to: send a test result acquisition message; acquire a test result message, the test result message includes test result information; and determine a chip test result table according to multiple test result messages.

[0141] It should be noted that the above-mentioned sending connection protocol module 91, sending chip test data module 92 and chip test set determination module 93 correspond to S202 to S206 in the method embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0142] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0143] Refer to the following Fig.10 The electronic device 1000 according to this embodiment of the present disclosure is described. Fig.10 The electronic device 1000 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0144] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include but are not limited to: the at least one processing unit 1010, the at least one storage unit 1020, and a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010).

[0145] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 executes the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure.

[0146] For example, the processing unit 1010 can execute the following steps of the above-mentioned method embodiment: according to the unified message interaction protocol, send a connection protocol to connect with multiple cameras to be detected; send chip test data to each camera to be detected, so that each camera to be detected performs a test according to the chip test data to obtain the test results of each chip, wherein the cameras to be detected are set with the same algorithm model; receive the chip test result information returned by each camera to be detected, and determine the chip test result table.

[0147] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202 , and may further include a read-only storage unit (ROM) 10203 .

[0148] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0149] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0150] The electronic device 1000 may also communicate with one or more external devices 1040 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0151] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0152] In particular, according to an embodiment of the present disclosure, the process described with reference to the flowchart above may be implemented as a computer program product, which includes: a computer program, which implements the above-mentioned camera chip testing method when executed by a processor.

[0153] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0154] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0156] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0157] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0158] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0159] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0160] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0161] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A camera chip testing method, characterized in that: include: According to the unified message interaction protocol, send the connection protocol to connect with multiple cameras to be detected; Sending chip test data to each camera to be tested, so that each camera to be tested performs a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be tested are set with the same algorithm model; Receive the chip test result information returned by each camera to be tested, and determine the chip test result table; If the chip test result of a camera to be detected and a preset chip performance value meet a preset condition, then the camera to be detected is determined to be a cheating camera to be detected; Performing anti-cheating detection on the cheating camera to be detected by a preset anti-cheating method, and determining the anti-cheating detection result of the cheating camera to be detected; wherein the preset anti-cheating method includes: modifying the algorithm model in the chip of the cheating camera to be detected, and / or modifying the chip test data; when modifying the algorithm model in the chip of the cheating camera to be detected, keeping the chip test data unchanged; when modifying the chip test data, keeping the algorithm model unchanged; When the preset anti-cheating method includes modifying chip test data, determining the anti-cheating detection result of the cheating camera to be detected includes: modifying data in the chip test data according to a preset ratio; sending the modified chip test data to the cheating camera to be detected, and determining second inspection test result information; if a third offset value between the second inspection test result information and the chip test result information exceeds a preset threshold, it is determined that the cheating camera to be detected is not cheating.

2. The camera chip testing method according to claim 1, characterized in that: The method further comprises: Calculating a first offset value between the chip test result and a preset chip performance value; If the first offset value exceeds a preset threshold, it is determined that a preset condition is met.

3. The camera chip testing method according to claim 1, characterized in that: When the preset anti-cheating method includes modifying the algorithm model in the chip of the cheating camera to be detected, the anti-cheating detection is performed on the cheating camera to be detected by the preset anti-cheating method, and determining the anti-cheating detection result of the cheating camera to be detected includes: Increasing the number of layers of the algorithm model in the cheating camera chip to be detected from a first value to a second value, and / or increasing the parameter information of the algorithm model from a third value to a fourth value, testing the chip test data again, and determining first inspection test result information; If the second offset value between the first verification test result information and the chip test result information exceeds a preset threshold, it is determined that the cheating camera to be detected is not cheating.

4. The camera chip testing method according to claim 1, characterized in that: Before sending the chip test data to each camera to be tested, the method further includes: Polling to obtain the status of the camera to be detected; When the camera to be detected is in an idle state, sending chip test data to each camera to be detected; When the camera to be detected is in a non-idle state, the state of the camera to be detected is obtained again after a preset time interval.

5. The camera chip testing method according to claim 1, characterized in that: Before connecting with a plurality of cameras to be detected, the method further includes: Retrieve the camera of the preset Internet Protocol IP segment; The retrieved camera is added to the device list and determined as the camera to be detected.

6. The camera chip testing method according to any one of claims 1 to 5, characterized in that: The sending of the connection protocol according to the unified message interaction protocol to connect with the multiple cameras to be detected includes: Send a hello message; When receiving the information message returned by the camera to be detected, a connection is established with the camera to be detected.

7. The camera chip testing method according to any one of claims 1 to 5, characterized in that: Sending chip test data to each camera to be tested includes: Sending a test request message, wherein the test request message includes chip test data; Receive the return message after the camera to be detected confirms the request.

8. The camera chip testing method according to any one of claims 1 to 5, characterized in that: The receiving the chip test result information returned by each camera to be detected and determining the chip test result table includes: Send a test result message; Obtaining a test result message, wherein the test result message includes test result information; A chip test result table is determined according to a plurality of test result messages.

9. A camera chip testing device, characterized in that: include: A sending connection protocol module is used to send a connection protocol according to a unified message interaction protocol to connect with multiple cameras to be detected; A chip test data sending module is used to send chip test data to each camera to be detected, so that each camera to be detected performs a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are set with the same algorithm model; A chip test set determination module is used to receive the test result information of the chip returned by each camera to be detected and determine the chip test result table; The anti-cheating detection module is used for determining that a camera to be detected is a cheating camera to be detected if the chip test result of the camera to be detected and the preset chip performance value meet the preset conditions; performing anti-cheating detection on the cheating camera to be detected by a preset anti-cheating method to determine the anti-cheating detection result of the cheating camera to be detected; wherein the preset anti-cheating method includes: modifying the algorithm model in the chip of the cheating camera to be detected, and / or modifying the chip test data; when modifying the algorithm model in the chip of the cheating camera to be detected, keeping the chip test data unchanged; when modifying the chip test data, keeping the algorithm model unchanged; The anti-cheating detection module is also used to, when the preset anti-cheating method includes modifying the chip test data, determine the anti-cheating detection result of the cheating camera to be detected, including: modifying the data in the chip test data according to a preset ratio; sending the modified chip test data to the cheating camera to be detected, and determining second inspection test result information; if the third offset value between the second inspection test result information and the chip test result information exceeds a preset threshold, it is determined that the cheating camera to be detected is not cheating.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the camera chip testing method according to any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the camera chip testing method according to any one of claims 1 to 8 is implemented.

Images