Test methods and apparatuses and related products

Abstract

The present disclosure relates to a test method and device and related products, which include a combination processing device including a computing processing device, an interface device, other processing devices and a storage device. The computing processing device can include one or more computing devices. The computing processing device can be configured to perform user-specified operations, and the computing processing device can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor, or one or more computing devices included in the computing processing device can be implemented as an artificial intelligence processor core or part of the hardware structure of an artificial intelligence processor core. By using the above combination processing device, the present disclosure can improve the operation efficiency of related products when performing operations on neural network models.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a test method and device and related products. BACKGROUND

[0002] In the process of system testing, test data can be input into a to-be-tested system to obtain test results, and the test data can be input into a standard system to obtain standard results, and then the distribution difference between the standard results and the test results can be determined. When determining the test results, the distribution difference can be compared with a set threshold value, if the distribution difference is less than the set threshold value, the test passes, if the distribution difference is greater than or equal to the set threshold value, the test fails. With the expansion of the data scale, some errors will appear cumulative phenomenon, causing the error to be large, which can make the to-be-tested data without errors and the test of the to-be-tested system fail, and can also make the to-be-tested data with errors and the test of the to-be-tested system pass, thus possibly reducing the accuracy of the test, and it is difficult to achieve the purpose of reducing system errors or calculation errors.

[0003] Alternatively, the accuracy of two systems can also be compared, for example, the two systems are systems with the same function, but there are certain differences in the structure or algorithms used. In the comparison process, the comparison result can also be inaccurate due to factors such as the scale of the data, the algorithm used, and the structure of the system. SUMMARY

[0004] The present disclosure provides a test method and device and related products.

[0005] According to an aspect of the present disclosure, a test method is provided, comprising: determining a first loss between a to-be-tested system and a standard system based on test information; determining a loss threshold value between the standard system and a comparison system based on the test information, wherein the processing accuracy of the standard system is higher than that of the to-be-tested system and the comparison system; and determining a test result of the to-be-tested system relative to the comparison system according to the first loss and the loss threshold value.

[0006] In a possible implementation, the loss threshold value between the standard system and the comparison system is determined based on the test information, comprising: inputting the test information into the standard system to obtain first reference information; inputting the test information into the comparison system to obtain second reference information; and obtaining the loss threshold value according to the first reference information and the second reference information.

[0007] In a possible implementation, the first loss between the to-be-tested system and the standard system is determined based on the test information, including: inputting the test information into the to-be-tested system to obtain first output information; inputting the test information into the standard system to obtain first reference information; and obtaining the first loss according to the first output information and the first reference information.

[0008] In a possible implementation, the loss threshold or the first loss includes any one of the following: an N-norm between the first reference information and the second reference information or an N-norm between the first reference information and the first output information, where N is a positive integer; a cosine distance between the first reference information and the second reference information or a cosine distance between the first reference information and the first output information; a relative entropy between the first reference information and the second reference information or a relative entropy between the first reference information and the first output information.

[0009] In a possible implementation, the test result of the to-be-tested system relative to the comparison system is determined according to the first loss and the loss threshold, including: in a case where the first loss is less than or equal to the loss threshold, determining the test result as test passed.

[0010] In a possible implementation, the test result of the to-be-tested system relative to the comparison system is determined according to the first loss and the loss threshold, including: in a case where the order of magnitude of the first loss is less than or equal to the order of magnitude of the loss threshold, determining the test result as test passed.

[0011] In a possible implementation, in a case where the order of magnitude of the first loss is less than or equal to the order of magnitude of the loss threshold, the test result is determined as test passed, including: taking a logarithm of the first loss with base 10 to obtain a first comparison result; taking a logarithm of the loss threshold with base 10 to obtain a second comparison result; and determining the test result as test passed if a difference between the first comparison result and the second comparison result is less than 1.

[0012] In a possible implementation, the test information includes at least one of image information, audio information, text information, and data information.

[0013] In a possible implementation, the to-be-tested system, the standard system, and the comparison system are software systems and / or hardware systems.

[0014] In a possible implementation, the to-be-tested system and the comparison system support operation of 32-bit floating-point data, and the standard system supports operation of 64-bit floating-point data.

[0015] According to another aspect of the present disclosure, a testing device is provided, comprising: a first loss determining module configured to determine a first loss between a to-be-tested system and a standard system based on testing information; a loss threshold value determining module configured to determine a loss threshold value between the standard system and a comparison system based on the testing information, wherein a processing accuracy of the standard system is higher than that of the to-be-tested system and the comparison system; and a testing result determining module configured to determine a testing result of the to-be-tested system relative to the comparison system according to the first loss and the loss threshold value.

[0016] In a possible implementation, the loss threshold value determining module is further configured to: input the testing information into the standard system to obtain first reference information; input the testing information into the comparison system to obtain second reference information; and obtain the loss threshold value according to the first reference information and the second reference information.

[0017] In a possible implementation, the first loss determining module is further configured to: input the testing information into the to-be-tested system to obtain first output information; input the testing information into the standard system to obtain first reference information; and obtain the first loss according to the first output information and the first reference information.

[0018] In a possible implementation, the first loss determining module or the loss threshold value determining module is further configured to: an N-norm between the first reference information and the second reference information or an N-norm between the first reference information and the first output information, wherein N is a positive integer; a cosine distance between the first reference information and the second reference information or a cosine distance between the first reference information and the first output information; a relative entropy between the first reference information and the second reference information or a relative entropy between the first reference information and the first output information.

[0019] In a possible implementation, the testing result determining module is further configured to: determine the testing result as test passed in a case where the first loss is less than or equal to the loss threshold value.

[0020] In a possible implementation, the testing result determining module is further configured to: determine the testing result as test passed in a case where an order of magnitude of the first loss is less than or equal to an order of magnitude of the loss threshold value.

[0021] In a possible implementation, the test result determination module is further configured to: take a logarithm of the first loss with base 10 to obtain a first comparison result; take a logarithm of the loss threshold with base 10 to obtain a second comparison result; and determine that the test result is a test pass if a difference between the first comparison result and the second comparison result is less than 1.

[0022] In a possible implementation, the test information includes at least one of image information, audio information, text information, and data information.

[0023] In a possible implementation, the system to be tested, the standard system, and the comparison system are software systems and / or hardware systems.

[0024] In a possible implementation, the system to be tested and the comparison system support 32-bit floating-point data operation, and the standard system supports 64-bit floating-point data operation.

[0025] According to another aspect of the present disclosure, there is provided an artificial intelligence chip, the chip comprising the test device.

[0026] According to another aspect of the present disclosure, there is provided an electronic device, the electronic device comprising the artificial intelligence chip.

[0027] According to another aspect of the present disclosure, there is provided a board card, comprising: a storage device, an interface device, a control device, and the artificial intelligence chip; wherein the artificial intelligence chip is connected with the storage device, the control device, and the interface device respectively; the storage device is configured to store data; the interface device is configured to realize data transmission between the artificial intelligence chip and an external device; and the control device is configured to monitor a state of the artificial intelligence chip.

[0028] According to another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory configured to store processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the test method.

[0029] According to another aspect of the present disclosure, there is provided a computer-readable storage medium, the computer-readable storage medium storing computer program instructions, wherein the computer program instructions are executed by a processor to implement the test method.

[0030] According to the test method of the embodiments of the present disclosure, the loss threshold can be determined for each test information to test the first loss between the system to be tested and the standard system, and the use of the loss threshold for testing can avoid the accumulation of inherent differences as the data size increases, reduce the possibility of misjudgment caused by the accumulation of inherent differences, improve the test accuracy, and improve the interception efficiency of errors.

[0031] According to an aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored by the memory to perform the above method.

[0032] According to an aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions, which, when executed by a processor, implement the above method.

[0033] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become apparent based on the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure together with the description.

[0035] Figure 1 A flow chart of a test method according to an embodiment of the present disclosure is shown;

[0036] Figure 2 An application schematic diagram of a test method according to an embodiment of the present disclosure is shown;

[0037] Figure 3 A block diagram of a test device according to an embodiment of the present disclosure is shown;

[0038] Figure 4 is a structural diagram of a combination processing device according to an embodiment of the present disclosure;

[0039] Figure 5 is a structural schematic diagram of a board card according to an embodiment of the present disclosure;

[0040] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present disclosure.

[0042] It should be understood that the terms "first", "second", etc. in the claims, specification, and drawings of the disclosure are used to distinguish different objects, and are not used to describe a particular order. The terms "include" and "contain" used in the specification and claims of the disclosure indicate the presence of the described features, whole, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0043] It should also be understood that the terms used in the specification of the present disclosure herein are only for the purpose of describing specific embodiments, and are not intended to limit the disclosure. As used in the specification and claims of the disclosure, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise. It should be further understood that the term "and / or" used in the specification and claims of the disclosure means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0044] As used in the specification and claims of the disclosure, the term "if" can be interpreted as "when" or "upon" or "in response to a determination" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0045] The test method according to the embodiments of the disclosure can be applied to a processor. The processor can be a general-purpose processor, such as a CPU (Central Processing Unit), or an artificial intelligence processor (IPU) for performing artificial intelligence operations. The artificial intelligence operations can include machine learning operations, brain-like operations, etc. Among them, the machine learning operations include neural network operations, k-means operations, support vector machine operations, etc. The artificial intelligence processor may, for example, include one or a combination of a GPU (Graphics Processing Unit), a NPU (Neural-Network Processing Unit), a DSP (Digital Signal Process), and a FPGA (Field-Programmable Gate Array) chip. The disclosure does not limit the specific type of processor.

[0046] In a possible implementation, the processor mentioned in the present disclosure can include a plurality of processing units, each of which can independently run various tasks assigned thereto, such as convolution operation tasks, pooling tasks, or full connection tasks, and the like. The present disclosure does not limit the processing units and the tasks run by the processing units.

[0047] Figure 1 A flowchart of a test method according to an embodiment of the present disclosure is shown, which includes:

[0048] In step S11, based on the test information, a first loss between the to-be-tested system and the standard system is determined;

[0049] In step S12, based on the test information, a loss threshold between the standard system and the comparison system is determined, wherein the processing accuracy of the standard system is higher than the processing accuracy of the to-be-tested system and the comparison system.

[0050] In step S13, according to the first loss and the loss threshold, a test result of the to-be-tested system relative to the comparison system is determined.

[0051] The test method according to an embodiment of the present disclosure does not use a fixed threshold, and can determine a loss threshold based on the comparison system and the standard system for each test information, that is, an appropriate threshold can be generated for each test information, avoiding the error accumulation phenomenon caused by the increase in data size, reducing the probability of false positives for errors, and improving the effect of avoiding errors. Further, in the process of comparing the accuracies of two systems (the to-be-tested system and the comparison system), the standard system can be provided as a reference, and by comparing the errors between the two systems and the standard system, the comparison error caused by factors such as data size, algorithm, and system structure in the direct comparison process can be reduced.

[0052] In a possible implementation, before a system (for example, a software system and / or a hardware system) is used, the system can be tested, for example, the system can be used as a to-be-tested system and tested through one or more test information. The difference between the output result of the to-be-tested system (that is, the processing result of the to-be-tested system on the test information) and the output result of the standard system (that is, the processing result of the standard system on the test information) can be compared with a threshold to determine whether the to-be-tested system has an error, for example, if the difference is greater than the threshold, it indicates that the difference between the output result of the to-be-tested system and the accurate result output by the standard system is large, indicating that the to-be-tested system may have an error, otherwise, it indicates that the to-be-tested system has no error.

[0053] In a possible implementation, in the related art, the threshold value is a preset fixed value, that is, the threshold value is constant during the test. However, as the data scale expands, the error of the output result of the system can accumulate, causing the error to increase with the data scale, so that the fixed threshold value cannot adapt to larger-scale data. For example, the error of the test result of a single to-be-tested data is a, and the threshold value is A. When a≤A, the test passes. However, as the data scale increases, the accumulated error can exceed A, but the operation of these data is not wrong. Therefore, using the fixed threshold value can determine that the system that does not have errors is wrong, causing a false judgment.

[0054] In a possible implementation, to address the above problem, a non-fixed threshold value can be used, which can be changed based on the test information input into the to-be-tested system each time, and the threshold value is generated each time, without accumulating errors. The to-be-tested system can be used to process data, images, audio, text, and the like. The test information includes at least one of image information, audio information, text information, and data information.

[0055] In a possible implementation, as described above, the difference between the processing result of the test information by the to-be-tested system and the processing result of the test information by the standard system can be compared with the threshold value. The threshold value can be determined based on a comparison system. For example, the standard system is a standard error-free system, and the calculation accuracy of the standard system is relatively high. The comparison system can be a system that has the same function as the to-be-tested system (which can be different from the to-be-tested system in structure or algorithm used), and the calculation accuracy of the comparison system is lower than that of the standard system. The calculation accuracy of the to-be-tested system can also be lower than that of the standard system. For example, the calculation accuracy of the to-be-tested system can be equal to that of the comparison system, for example, the to-be-tested system and the comparison system support 32-bit floating-point data operation, and the standard system supports 64-bit floating-point data operation. The disclosure does not limit the accuracy of the to-be-tested system and the comparison system. The disclosure does not limit the data type and data bit number of the output result of the to-be-tested system, the standard system, and the comparison system.

[0056] In a possible implementation, the test method of the non-fixed threshold value described above can be used in the test of various systems, for example, the to-be-tested system, the standard system, and the comparison system each include a software system and / or a hardware system. In an example, the types of the to-be-tested system, the standard system, and the comparison system can be the same, for example, the to-be-tested system, the standard system, and the comparison system are all software systems, or the to-be-tested system, the standard system, and the comparison system are all hardware systems. The disclosure does not limit the types of the to-be-tested system, the standard system, and the comparison system.

[0057] In a possible implementation, in step S11, when testing, a difference between the to-be-tested system and the standard system, i.e., the first loss, can be determined based on the test information. For example, the test information can be respectively input into the to-be-tested system and the standard system, and output results of the two systems are respectively obtained, and a difference between the two output results is determined.

[0058] In a possible implementation, step S11 can include: inputting the test information into the to-be-tested system to obtain first output information; inputting the test information into the standard system to obtain first reference information; and obtaining the first loss according to the first output information and the first reference information.

[0059] In a possible implementation, the first output information is an output result of the to-be-tested system after processing the test information, and the first reference information is an output result of the standard system after processing the test information. The difference between the two, i.e., the first loss, can be determined.

[0060] In an example, the difference between the two can be determined. Alternatively, the difference between the two, i.e., the first loss, can include any one of the following: an N-norm between the first reference information and the first output information, where N is a positive integer; a cosine distance between the first reference information and the first output information; and a relative entropy between the first reference information and the first output information.

[0061] In an example, the first loss can be determined according to the following formula (1):

[0062] Diff1 = calculate_L2 (o1, o0) (1)

[0063] where Diff1 is the first loss, o1 is the first output information, o0 is the first reference information, and calculate_L2 (o1, o0) is a 2-norm between the first output information and the first reference information. The disclosure does not limit the determination manner of the first loss.

[0064] In a possible implementation, after determining the first loss, a threshold value for comparison with the first loss can be determined. As described above, the threshold value is not a preset fixed threshold value, but is determined for each input of test information of the to-be-tested system, i.e., determined based on the test information, the standard system, and the comparison system. In step S12, a difference (i.e., a loss threshold value) between the standard system and the comparison system can be determined based on the test information. For example, a system (e.g., a system that passes the test) that has the same function as the to-be-tested system can be taken as the comparison system. And a difference between the comparison system and the labeled system is determined. If the difference (the first loss) between the to-be-tested system and the standard system is less than or equal to the difference (the loss threshold value) between the comparison system and the labeled system, it can be considered that the error of the to-be-tested system is within a controllable range, i.e., the to-be-tested system passes the test. The condition for determining whether the to-be-tested system passes the test can also include other conditions, for example, the order of magnitude of the first loss is less than or equal to the order of magnitude of the loss threshold value, and the like. The disclosure does not limit the determination condition.

[0065] In a possible implementation, step S12 can include: inputting the test information into the standard system to obtain first reference information; inputting the test information into the comparison system to obtain second reference information; and obtaining the loss threshold value according to the first reference information and the second reference information.

[0066] In a possible implementation, the comparison system can be a system that has the same function as the to-be-tested system, but the comparison system can be a system that has passed the test. If the difference between the to-be-tested system and the standard system is less than or equal to the difference between the comparison system and the standard system, it can be considered that the calculation error of the to-be-tested system is small, which is acceptable and will not be regarded as a calculation error. Or, if the order of magnitude of the difference between the to-be-tested system and the standard system is less than or equal to the order of magnitude of the difference between the comparison system and the standard system (the difference between the two is at least one order of magnitude), it can also be considered that the calculation error of the to-be-tested system is small and will not be regarded as a calculation error.

[0067] In a possible implementation, the first reference information is an output result of the standard system after processing the test information, and the second reference information is an output result of the comparison system after processing the test information. A difference between the two can be determined, i.e., a loss threshold value (a threshold value determined for the test information). In an example, the difference between the two can be determined by subtracting the two. Or, the loss threshold value includes any one of the following: an N-norm between the first reference information and the second reference information, where N is a positive integer; a cosine distance between the first reference information and the second reference information; and a relative entropy between the first reference information and the second reference information.

[0068] In an example, the loss threshold can be determined according to the following formula (2):

[0069] Diff2 = calculate_L2 (o2, o0) (2)

[0070] wherein Diff2 is the loss threshold, o2 is the second reference information, o0 is the first reference information, and calculate_L2 (o2, o0) is the 2-norm between the second reference information and the first reference information. The disclosure does not limit the manner of determining the loss threshold.

[0071] In this way, the loss threshold can be determined for each input of the test information of the system under test, and the test using the loss threshold can reduce the possibility of misjudgment caused by the accumulation of inherent differences, and improve the test accuracy.

[0072] In a possible implementation, in step S13, the test result of the system under test relative to the comparison system is determined according to the first loss and the loss threshold. For example, the first loss and the loss threshold can be compared. As described above, the two values can be compared in size, or the two values can be compared in order of magnitude.

[0073] In a possible implementation, the two values can be compared in size directly, and step S13 can include: in the case where the first loss is less than or equal to the loss threshold, determining that the test result is a test pass. That is, when the first loss (i.e., the difference between the system under test and the standard system) is less than or equal to the loss threshold (i.e., the difference between the comparison system and the standard system), it can be indicated that the calculation error of the system under test is very small, and the error is acceptable and will not be regarded as a calculation error, that is, the test result can be considered as a test pass.

[0074] In a possible implementation, the two values can be compared in order of magnitude, and step S13 can include: in the case where the order of magnitude of the first loss is less than or equal to the order of magnitude of the loss threshold, determining that the test result is a test pass. That is, when the order of magnitude of the first loss (i.e., the difference between the system under test and the standard system) is less than or equal to the order of magnitude of the loss threshold (i.e., the difference between the comparison system and the standard system), that is, the first loss and the loss threshold are at least of the same order of magnitude, it can be indicated that the calculation error of the system under test is very small, and the error is acceptable and will not be regarded as a calculation error, that is, the test result can be considered as a test pass.

[0075] In an example, the orders of magnitude of the two can be determined by taking the logarithm with base 10, and then compared, which can include: taking the logarithm with base 10 of the first loss to obtain a first comparison result; taking the logarithm with base 10 of the loss threshold to obtain a second comparison result; and if the difference between the first comparison result and the second comparison result is less than 1, determining that the test result is a test pass.

[0076] In an example, the orders of magnitude of the two can be compared by the following formula (3):

[0077] log 10 diff1-log 10 diff2<1 (3)

[0078] That is, the order of magnitude of the first loss can be represented by log 10 diff1, the order of magnitude of the second loss can be represented by log 10 diff2, and the difference between the two being less than 1 indicates that the two are at least in the same order of magnitude, or the order of magnitude of the first loss is less than the order of magnitude of the loss threshold.

[0079] According to the test method of the embodiments of the present disclosure, the loss threshold can be determined for each test information to test the first loss between the to-be-tested system and the standard system, and using the loss threshold for testing can avoid the inherent difference increasing with the data size, reduce the possibility of misjudgment caused by the accumulation of the inherent difference, improve the test accuracy, and improve the interception efficiency of errors.

[0080] Figure 2 An application schematic diagram of the test method according to the embodiments of the present disclosure is shown, as shown in Figure 2 The standard system, the to-be-tested system, and the comparison system can all be hardware systems, wherein the standard system has the highest calculation accuracy and can output 64-bit floating-point data, and the calculation accuracy of the to-be-tested system and the comparison system is lower than that of the standard system, and both can output 32-bit floating-point data.

[0081] In a possible implementation, when testing the to-be-tested system, the difference between the to-be-tested system and the standard system, i.e., the first loss, can be determined. The test information (for example, an image, text, data, etc.) is input into the to-be-tested system to obtain first output information o1, and the test information is input into the standard system to obtain first reference information o0. Then, the first loss, i.e., the 2-norm diff1 between the two, can be obtained according to formula (1).

[0082] In a possible implementation, the loss threshold value can be obtained for the to-be-tested information. For example, a system that has passed the test is taken as a comparison system, and a difference between the output results of the comparison system and a standard system is taken as the threshold value. The to-be-tested system can pass the test when the error of the to-be-tested system is smaller than that of the comparison system or the error of the to-be-tested system is of the same order of magnitude as that of the comparison system. The test information is input into the comparison system to obtain second reference information o2, and then the loss threshold value, that is, a two-norm diff2 between the first reference information o0 and the second reference information o2, can be obtained according to formula (2).

[0083] In a possible implementation, the first loss and the order of magnitude of the loss threshold value can be compared. For example, the first loss and the second loss can be calculated through formula (3). If the first loss and the second loss make formula (3) hold, that is, the logarithmic difference is smaller than 1, it can be indicated that the first loss is at least of the same order of magnitude as the loss threshold value or the order of magnitude of the first loss is smaller. Thus, it can be determined that the error between the to-be-tested system and the standard system is smaller, and it is determined that the test passes.

[0084] In a possible implementation, the test method can be used in the test of various systems, for example, the test of a hardware system and / or a software system, and can also be used in processing of comparison with other systems, for example, determining whether the error of the to-be-tested system is smaller than that of a comparison system that has passed the test. The application field of the test method is not limited in the present disclosure.

[0085] Figure 3 A block diagram of a test device according to an embodiment of the present disclosure is shown as follows. Figure 3 As shown, the device includes a first loss determination module 11 configured to determine a first loss between a to-be-tested system and a standard system based on test information; a loss threshold value determination module 12 configured to determine a loss threshold value between the standard system and a comparison system based on the test information, where the processing accuracy of the standard system is higher than that of the to-be-tested system and the comparison system; and a test result determination module 13 configured to determine a test result of the to-be-tested system relative to the comparison system according to the first loss and the loss threshold value.

[0086] In a possible implementation, the loss threshold value determination module is further configured to: input the test information into the standard system to obtain first reference information; input the test information into the comparison system to obtain second reference information; and obtain the loss threshold value according to the first reference information and the second reference information.

[0087] In a possible implementation, the first loss determination module is further configured to: input the test information into the system under test to obtain first output information; input the test information into the standard system to obtain first reference information; and obtain the first loss according to the first output information and the first reference information.

[0088] In a possible implementation, the first loss determination module or the loss threshold determination module is further configured to: an N-norm between the first reference information and the second reference information or an N-norm between the first reference information and the first output information, where N is a positive integer; a cosine distance between the first reference information and the second reference information or a cosine distance between the first reference information and the first output information; a relative entropy between the first reference information and the second reference information or a relative entropy between the first reference information and the first output information.

[0089] In a possible implementation, the test result determination module is further configured to: determine the test result as test pass in a case where the first loss is less than or equal to the loss threshold.

[0090] In a possible implementation, the test result determination module is further configured to: determine the test result as test pass in a case where an order of magnitude of the first loss is less than or equal to an order of magnitude of the loss threshold.

[0091] In a possible implementation, the test result determination module is further configured to: obtain a first comparison result by taking a logarithm of the first loss with base 10; obtain a second comparison result by taking a logarithm of the loss threshold with base 10; and determine the test result as test pass in a case where a difference between the first comparison result and the second comparison result is less than 1.

[0092] In a possible implementation, the test information includes at least one of image information, audio information, text information, and data information.

[0093] In a possible implementation, the system under test, the standard system, and the comparison system are software systems and / or hardware systems.

[0094] In a possible implementation, the system under test and the comparison system support 32-bit floating-point data operation, and the standard system supports 64-bit floating-point data operation.

[0095] Figure 4 FIG. 12 is a structural diagram of a combination processing apparatus 1200 according to an embodiment of the present disclosure. As shown in FIG. 12, the combination processing apparatus 1200 includes a system under test 1201, a standard system 1202, a comparison system 1203, a test information input module 1204, a first loss determination module 1205, a loss threshold determination module 1206, and a test result determination module 1207. Figure 4As shown in FIG. 12, the combination processing device 1200 includes a computing processing device 1202, an interface device 1204, other processing devices 1206, and a storage device 1208. Depending on different application scenarios, the computing processing device can include one or more computing devices 1210, which can be configured to perform the operations described herein in connection with the appended drawings. Figure 2 The operations described.

[0096] In different embodiments, the computing processing device of the present disclosure can be configured to perform user-specified operations. In an exemplary application, the computing processing device can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included in the computing processing device can be implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core, the computing processing device of the present disclosure can be considered to have a single-core structure or a homogeneous multi-core structure.

[0097] In an exemplary operation, the computing processing device of the present disclosure can interact with other processing devices through the interface device to jointly complete user-specified operations. Depending on the implementation, the other processing devices of the present disclosure can include one or more types of processors selected from general-purpose and / or special-purpose processors such as a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence processor, etc. These processors can include, but are not limited to, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., and the number thereof can be determined according to actual needs. As mentioned earlier, only in terms of the computing processing device of the present disclosure, it can be considered to have a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and the other processing devices are considered together, they can be considered to form a heterogeneous multi-core structure.

[0098] In one or more embodiments, the other processing device can serve as an interface for the computing processing device (which can be embodied as a related operation device for artificial intelligence such as neural network operation) to external data and control, and perform basic control including but not limited to data transfer, start and / or stop of the computing device, etc. In additional embodiments, the other processing device can also cooperate with the computing processing device to jointly complete an operation task.

[0099] In one or more embodiments, the interface device can be used to transmit data and control instructions between the computing processing device and the other processing device. For example, the computing processing device can obtain input data from the other processing device via the interface device and write the data into the storage device (or memory) on the computing processing device chip. Further, the computing processing device can obtain control instructions from the other processing device via the interface device and write the control instructions into the control buffer on the computing processing device chip. Alternatively or optionally, the interface device can also read data from the storage device of the computing processing device and transmit the data to the other processing device.

[0100] Additionally or optionally, the combined processing device of the present disclosure can further include a storage device. As shown in the figure, the storage device is connected to the computing processing device and the other processing device, respectively. In one or more embodiments, the storage device can be used to save data of the computing processing device and / or the other processing device. For example, the data can be data that cannot be completely saved in the internal or on-chip storage device of the computing processing device or the other processing device.

[0101] In some embodiments, the present disclosure further discloses an artificial intelligence chip (such as the chip 1302 shown in Figure 5 In an implementation, the chip is a system on chip (SoC), and integrates one or more combined processing devices. The chip can be connected to other related components through an external interface device (such as the external interface device 1306 shown in Figure 5 The related components can be, for example, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. In some application scenarios, other processing units (such as video codecs) and / or interface modules (such as DRAM interfaces) can be integrated on the chip. In some embodiments, the present disclosure further discloses a chip package structure including the above chip. In some embodiments, the present disclosure further discloses a board including the above chip package structure. The board will be described in detail below. Figure 5

[0102] Figure 5 is a structural schematic diagram of a board 1300 according to an embodiment of the present disclosure. As shown in​Figure 5 As shown in FIG. 13, the board card includes a memory device 1304 for storing data, which includes one or more memory units 1310. The memory device can be connected and transmit data with the controller device 1308 and the artificial intelligence chip 1302 described above by means of a bus or the like. Further, the board card also includes an external interface device 1306 configured for data relay or switching function between the chip (or the chip in the chip package structure) and an external device 1312 (such as a server or a computer, etc.). For example, the data to be processed can be delivered by the external device to the chip through the external interface device. For another example, the calculation result of the chip can be transmitted back to the external device via the external interface device. According to different application scenarios, the external interface device can have different interface forms, for example, it can adopt a standard PCIE interface or the like.

[0103] Each group of the memory units is connected with the artificial intelligence chip through a bus. It can be understood that each group of the memory units can be a DDR SDRAM (English: Double Data Rate SDRAM, double data rate synchronous dynamic random access memory).

[0104] DDR does not need to increase the clock frequency to double the speed of SDRAM. DDR allows reading data on the rising and falling edges of the clock pulse. The speed of DDR is twice that of standard SDRAM. In an embodiment, the memory device can include 4 groups of the memory units. Each group of the memory units can include a plurality of DDR4 particles (chips). In an embodiment, the artificial intelligence chip can internally include 4 72-bit DDR4 controllers, of which 64 bits are used for data transmission and 8 bits are used for ECC verification. It can be understood that when DDR4-3200 particles are used in each group of the memory units, the theoretical bandwidth of data transmission can reach 25600 MB / s.

[0105] In an embodiment, each group of the memory units includes a plurality of double data rate synchronous dynamic random access memories arranged in parallel. DDR can transmit data twice in one clock cycle. A controller for controlling DDR is arranged in the chip, for controlling data transmission and data storage of each of the memory units.

[0106] The interface device is electrically connected with the artificial intelligence chip. The interface device is used to realize data transmission between the artificial intelligence chip and an external device (for example, a server or a computer). For example, in an embodiment, the interface device can be a standard PCIE interface. For example, data to be processed is transmitted by the server to the chip through the standard PCIE interface, so as to realize data transfer. Preferably, when PCIE 3.0X 16 interface transmission is adopted, the theoretical bandwidth can reach 16000 MB / s. In another embodiment, the interface device can also be other interfaces, and the disclosure does not limit the specific forms of the above-mentioned other interfaces. The interface unit can only realize the switching function. In addition, the calculation result of the artificial intelligence chip is still transmitted back to the external device (for example, the server) by the interface device.

[0107] The control device is electrically connected with the artificial intelligence chip. The control device is used to regulate the state of the artificial intelligence chip. Specifically, the artificial intelligence chip and the control device can be electrically connected through an SPI interface. The control device can include a single-chip microcomputer (MCU). As the artificial intelligence chip can include multiple processing chips, multiple processing cores or multiple processing circuits, it can drive multiple loads. Therefore, the artificial intelligence chip can be in different working states such as multiple loads and light loads. Through the control device, the working states of the multiple processing chips, the multiple processing cores and / or the multiple processing circuits in the artificial intelligence chip can be regulated.

[0108] Those skilled in the art can understand that the disclosure also discloses an electronic device or apparatus, which can include one or more of the above-mentioned board cards, one or more of the above-mentioned chips and / or one or more of the above-mentioned combined processing devices.

[0109] According to different application scenarios, the electronic device or apparatus of the disclosure can include a server, a cloud server, a server cluster, a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a vehicle recording device, a navigation device, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an automatic driving terminal, a vehicle, a household appliance and / or a medical device. The vehicle includes an airplane, a ship and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, an exhaust hood; the medical device includes a nuclear magnetic resonance instrument, a B-ultrasound instrument and / or an electrocardiograph.

[0110] The embodiment of the present disclosure further provides a computer readable storage medium, which has computer program instructions stored thereon, and the computer program instructions are executed by a processor to implement the method.

[0111] The embodiment of the present disclosure further provides an electronic device, which comprises a processor, and a memory for storing processor-executable instructions, wherein the processor is configured to invoke the instructions stored in the memory to execute the method.

[0112] The electronic device can be provided as a terminal, a server or other forms of devices.

[0113] Figure 6 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server. Referring to Figure 6 , the electronic device 1900 comprises a processing component 1922, which further comprises one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can comprise one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the method described above.

[0114] The electronic device 1900 can further comprise a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS XTM , Unix TM , Linux TM , FreeBSD TM or the like.

[0115] In an exemplary embodiment, a non-volatile computer readable storage medium, such as the memory 1932 comprising computer program instructions executable by the processing component 1922 of the electronic device 1900 to complete the method described above, is also provided.

[0116] It should be noted that, for the purpose of simplicity, the present disclosure expresses some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art can understand that the solutions of the present disclosure are not limited by the order of the actions described. Therefore, those skilled in the art can understand that some of the steps can be performed in other orders or at the same time according to the disclosure or teaching of the present disclosure. Further, those skilled in the art can understand that the embodiments described in the present disclosure can be regarded as optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions of the present disclosure. In addition, the description of some embodiments in the present disclosure also focuses on different solutions. Therefore, those skilled in the art can understand that the parts not described in detail in one embodiment of the present disclosure can also refer to the relevant description of other embodiments.

[0117] In terms of specific implementation, those skilled in the art can understand that the several embodiments disclosed in the present disclosure can also be implemented in other ways not disclosed herein based on the disclosure and teaching of the present disclosure. For example, as for each unit in the foregoing electronic device or apparatus embodiments, the units are divided herein based on the logical functions, and there can be another division manner in actual implementation. For another example, a plurality of units or components can be combined or integrated into another system, or some features or functions of the units or components can be selectively disabled. As for the connection relationship between different units or components, the foregoing connections discussed in conjunction with the drawings can be direct or indirect coupling between the units or components.

[0118] In the present disclosure, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units. The foregoing components or units can be located in the same place or distributed on a plurality of network units. In addition, according to actual needs, some or all of the units can be selected to achieve the purpose of the solutions described in the embodiments of the present disclosure. In addition, in some scenarios, a plurality of units in the embodiments of the present disclosure can be integrated into one unit or each unit physically exists separately.

[0119] In the foregoing embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in one embodiment can refer to the relevant description of other embodiments. Each technical feature of the foregoing embodiments can be combined arbitrarily, and in order to make the description concise, not all possible combinations of the technical features in the foregoing embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, they should be regarded as the scope disclosed in the present disclosure.

[0120] The electronic device or apparatus of the present disclosure can also be applied to the fields of Internet, Internet of Things, data center, energy, transportation, public management, manufacturing, education, power grid, telecommunications, finance, retail, construction site, medical treatment, etc. Further, the electronic device or apparatus of the present disclosure can also be used in cloud, edge, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, the electronic device or apparatus with high computing power according to the present disclosure scheme can be applied to a cloud device (e.g., a cloud server), and the electronic device or apparatus with small power consumption can be applied to a terminal device and / or an edge device (e.g., a smart phone or a camera). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or the edge device are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud device according to the hardware information of the terminal device and / or the edge device to simulate the hardware resources of the terminal device and / or the edge device, so as to complete unified management, scheduling and collaborative work of end-cloud integration or cloud-edge integration.

[0121] The foregoing can be better understood in light of the following clauses:

[0122] For example, clause A1, a test method, comprising: determining a first loss between a to-be-tested system and a standard system based on test information; determining a loss threshold between the standard system and a comparison system based on the test information, wherein the processing accuracy of the standard system is higher than the processing accuracy of the to-be-tested system and the comparison system; and determining a test result of the to-be-tested system relative to the comparison system according to the first loss and the loss threshold.

[0123] Clause A2, the method of clause A1, determining a loss threshold between the standard system and a comparison system based on the test information, comprising: inputting the test information into the standard system to obtain first reference information; inputting the test information into the comparison system to obtain second reference information; and obtaining the loss threshold according to the first reference information and the second reference information.

[0124] Clause A3, the method of clause A1, determining a first loss between a to-be-tested system and a standard system based on test information, comprising: inputting the test information into the to-be-tested system to obtain first output information; inputting the test information into the standard system to obtain first reference information; and obtaining the first loss according to the first output information and the first reference information.

[0125] Clause A4, the method according to any one of clauses A2 or A3, the loss threshold or the first loss comprises any one of: an N-norm between the first reference information and the second reference information or an N-norm between the first reference information and the first output information, where N is a positive integer; a cosine distance between the first reference information and the second reference information or a cosine distance between the first reference information and the first output information; a relative entropy between the first reference information and the second reference information or a relative entropy between the first reference information and the first output information.

[0126] Clause A5, the method according to clause A1, determining the test result of the to-be-tested system relative to the comparison system according to the first loss and the loss threshold comprises: in a case where the first loss is less than or equal to the loss threshold, determining the test result as test passed.

[0127] Clause A6, the method according to clause A1, determining the test result of the to-be-tested system relative to the comparison system according to the first loss and the loss threshold comprises: in a case where an order of magnitude of the first loss is less than or equal to an order of magnitude of the loss threshold, determining the test result as test passed.

[0128] Clause A7, the method according to clause A6, in a case where the order of magnitude of the first loss is less than or equal to the order of magnitude of the loss threshold, determining the test result as test passed comprises: taking a logarithm of the first loss with base 10 to obtain a first comparison result; taking a logarithm of the loss threshold with base 10 to obtain a second comparison result; and if a difference between the first comparison result and the second comparison result is less than 1, determining the test result as test passed.

[0129] Clause A8, the method according to clause A1, the test information comprises at least one of image information, audio information, text information, and data information.

[0130] Clause A9, the method according to clause A1, the to-be-tested system, the standard system, and the comparison system are software systems and / or hardware systems.

[0131] Clause A10, the method according to clause A1, the to-be-tested system and the comparison system support operations of 32-bit floating-point data, and the standard system supports operations of 64-bit floating-point data.

[0132] Clause A11, a testing apparatus comprising: a first loss determining module configured to determine a first loss between a to-be-tested system and a standard system based on testing information; a loss threshold determining module configured to determine a loss threshold between the standard system and a comparison system based on the testing information, wherein a processing accuracy of the standard system is higher than processing accuracies of the to-be-tested system and the comparison system; and a test result determining module configured to determine a test result of the to-be-tested system relative to the comparison system according to the first loss and the loss threshold.

[0133] Clause A12, the apparatus according to clause A11, the loss threshold determining module is further configured to: input the testing information into the standard system to obtain first reference information; input the testing information into the comparison system to obtain second reference information; and obtain the loss threshold according to the first reference information and the second reference information.

[0134] Clause A13, the apparatus according to clause A11, the first loss determining module is further configured to: input the testing information into the to-be-tested system to obtain first output information; input the testing information into the standard system to obtain first reference information; and obtain the first loss according to the first output information and the first reference information.

[0135] Clause A14, the apparatus according to clause A12 or A13, the first loss determining module or the loss threshold determining module is further configured to: an N-norm between the first reference information and the second reference information or an N-norm between the first reference information and the first output information, wherein N is a positive integer; a cosine distance between the first reference information and the second reference information or a cosine distance between the first reference information and the first output information; a relative entropy between the first reference information and the second reference information or a relative entropy between the first reference information and the first output information.

[0136] Clause A15, the apparatus according to clause A11, the test result determining module is further configured to: determine the test result as test passed in a case that the first loss is less than or equal to the loss threshold.

[0137] Clause A16, the apparatus according to clause A11, the test result determining module is further configured to: determine the test result as test passed in a case that an order of magnitude of the first loss is less than or equal to an order of magnitude of the loss threshold.

[0138] Clause A17, the apparatus of clause A16, the test result determination module is further configured to: take a logarithm of the first loss with base 10 to obtain a first comparison result; take a logarithm of the loss threshold with base 10 to obtain a second comparison result; and determine that the test result is passed if a difference between the first comparison result and the second comparison result is less than 1.

[0139] Clause A18, the apparatus of clause A11, the test information comprises at least one of image information, audio information, text information, and data information.

[0140] Clause A19, the apparatus of clause A11, the system under test, the standard system, and the comparison system are software systems and / or hardware systems.

[0141] Clause A20, the apparatus of clause A11, the system under test and the comparison system support 32-bit floating point data operations, and the standard system supports 64-bit floating point data operations.

[0142] While several embodiments of the disclosure have been shown and described herein, it is to be understood that the embodiments are merely devised by way of example. Those skilled in the art can conceive many alterations, changes and alternative ways without departing from the idea and spirit of the disclosure. It should be understood that various alternatives to the embodiments of the disclosure described herein can be employed in practicing the disclosure. The appended claims are intended to define the scope of protection of the disclosure and thus cover equivalents or alternatives within the scope of the claims.

Claims

1. A test method characterized by, The method comprises: determining a first loss between a to-be-tested system and a standard system based on test information; determining a loss threshold between the standard system and a comparison system based on the test information, wherein the processing accuracy of the standard system is higher than that of the to-be-tested system and the comparison system, and the loss threshold represents the difference between the standard system and the comparison system; determining a test result of the to-be-tested system relative to the comparison system according to the first loss and the loss threshold; and The method comprises: inputting the test information into the standard system to obtain first reference information; inputting the test information into the comparison system to obtain second reference information; obtaining the loss threshold according to the first reference information and the second reference information. The method comprises: inputting the test information into the to-be-tested system to obtain first output information; inputting the test information into the standard system to obtain first reference information; obtaining the first loss according to the first output information and the first reference information. The loss threshold or the first loss comprises any one of the following: N-norm between the first reference information and the second reference information or N-norm between the first reference information and the first output information, wherein N is a positive integer; cosine distance between the first reference information and the second reference information or cosine distance between the first reference information and the first output information; relative entropy between the first reference information and the second reference information or relative entropy between the first reference information and the first output information.

2. The method of claim 1, wherein, The method comprises: in a case where the first loss is less than or equal to the loss threshold, determining that the test result is test passed.

3. The method of claim 1, wherein, The method comprises: in a case where the order of magnitude of the first loss is less than or equal to the order of magnitude of the loss threshold, determining that the test result is test passed.

4. The method of claim 3, wherein, The method comprises: taking the logarithm of the first loss with base 10 to obtain a first comparison result; taking the logarithm of the loss threshold with base 10 to obtain a second comparison result; if the difference between the first comparison result and the second comparison result is less than 1, determining that the test result is test passed.

5. The method of claim 1, wherein, The test information comprises at least one of image information, audio information, text information and data information.

6. The method of claim 1, wherein, The to-be-tested system, the standard system and the comparison system are software systems and / or hardware systems.

7. The method of claim 1, wherein, The to-be-tested system and the comparison system support 32-bit floating-point data operation, and the standard system supports 64-bit floating-point data operation.

8. A test device, characterized by The method comprises: The first loss determination module is configured to determine a first loss between the to-be-tested system and a standard system based on test information. The loss threshold determination module is configured to determine a loss threshold between the standard system and a comparison system based on the test information, wherein a processing accuracy of the standard system is higher than that of the to-be-tested system and the comparison system, and the loss threshold represents a difference between the standard system and the comparison system. The test result determination module is configured to determine a test result of the to-be-tested system relative to the comparison system according to the first loss and the loss threshold. The loss threshold determination module is further configured to: input the test information into the standard system to obtain first reference information; input the test information into the comparison system to obtain second reference information; and obtain the loss threshold according to the first reference information and the second reference information. The first loss determination module is further configured to: input the test information into the to-be-tested system to obtain first output information; input the test information into the standard system to obtain first reference information; and obtain the first loss according to the first output information and the first reference information. The loss threshold or the first loss includes any one of the following: an N-norm between the first reference information and the second reference information or an N-norm between the first reference information and the first output information, wherein N is a positive integer; a cosine distance between the first reference information and the second reference information or a cosine distance between the first reference information and the first output information; relative entropy between the first reference information and the second reference information or relative entropy between the first reference information and the first output information.

9. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method of any one of claims 1 to 7.

Images