An algorithmic provenance method and system for fully online automated execution

Abstract

The disclosure discloses an online automatic algorithm tracing method, which combines metrology technology and software testing technology, performs CA authentication on the algorithm software to be traced to form an authentication digital certificate, verifies the validity of the certificate by a CA server, generates algorithm tracing server identity verification information, establishes a communication mechanism between the algorithm software to be traced and the algorithm tracing server, and then the algorithm software to be traced can realize automatic algorithm tracing based on the system.

Description

Technical Field

[0001] This disclosure relates to the field of metrology, and in particular to an algorithm tracing method for fully automated online execution, in order to ensure and evaluate the accuracy and reliability of software algorithms in terms of measurement values. Background Technology

[0002] Software, as an extension of human thought, has significant advantages such as low cost, flexibility, and functional complexity. However, from a metrological perspective, software also has disadvantages such as susceptibility to cheating and tampering, difficulty in detecting bugs, and poor compliance with laws and regulations, which have exposed some problems in the use of measuring instruments.

[0003] The biggest challenge currently facing the field of metrology is how to ensure and assess the impact of software algorithms in measuring instruments on the accuracy and reliability of measurement values. For example, inputting the same data into programs with the same functionality written by different programmers may produce different outputs, making it difficult to determine the accuracy of the two software algorithms in terms of metrological requirements. Even if the outputs are the same, it is difficult to determine the software algorithm's compliance with metrological regulations.

[0004] Ensuring and evaluating the accuracy and reliability of software algorithms in terms of measurement values ​​is a focus of global metrology. However, to date, research on the accuracy and reliability of software algorithms is still at the level of algorithm evaluation, which is a qualitative judgment.

[0005] In view of this, it is essential for those skilled in the art to study how to automate quantitative measurement and evaluation software in a trustworthy environment, so that the measurement values ​​are accurate and reliable, and the algorithms can be traced. Summary of the Invention

[0006] To address the aforementioned technical issues, this disclosure innovatively provides a fully online automated algorithm tracing method and system to automate quantitative measurement and evaluation of software in a trusted environment.

[0007] This disclosure reveals a fully automated online algorithm tracing method, including the following steps:

[0008] S100. The algorithm software to be traced establishes communication and interacts with the CA server. During the interaction, the CA server verifies the validity of the digital certificate, wherein the digital certificate is issued to the algorithm software to be traced in advance by the CA server.

[0009] S200. After successful verification, the CA server generates authentication information for the algorithm to be traced. If the verification fails, the CA server performs software initialization of the algorithm to be traced to issue a digital certificate and generate authentication information. The authentication information is ATMS authentication information, which is used for authentication of the algorithm tracing service.

[0010] S300, the algorithm software to be traced establishes communication and interacts with the algorithm tracing server. During the interaction, the algorithm tracing server authenticates the algorithm software to be traced in order to verify whether its ATMS authentication information is valid.

[0011] S400. After verification, the algorithm software to be traced submits a traceability application to the algorithm traceability server. After the algorithm traceability server approves the application, it automatically submits the input data and the calculation results of the algorithm software to be traced to the algorithm traceability server.

[0012] S500: The algorithm tracing server generates algorithm standards based on the tracing application;

[0013] S600. Based on the algorithm standard, the algorithm tracing server further calculates the algorithm standard output result according to the input data submitted by the algorithm software to be traced.

[0014] S700: Based on the calculation results of the algorithm software to be traced and the standard output results of the algorithm, the algorithm tracing server calculates the indication error between the two.

[0015] S800 and the algorithm tracing server further evaluate the measurement uncertainty of the indicated value error;

[0016] S900. Based on the evaluation, the algorithm traceability server generates a digital / electronic certificate or digital / electronic report with measurement uncertainty evaluation as the algorithm traceability result.

[0017] Preferably, the following steps are also included:

[0018] S1000, the algorithm tracing server transmits the algorithm tracing results to the algorithm software to be traced, so that the algorithm software to be traced can obtain the digital / electronic certificate or digital / electronic report.

[0019] Furthermore, this disclosure discloses a fully online automated algorithm tracing system to implement the above method, including:

[0020] The parties involved are the CA server, the algorithm software to be traced, and the algorithm tracing server.

[0021] A CA server is used to verify the validity of digital certificates, wherein the digital certificates are pre-issued by the CA server to the algorithm software to be traced; and,

[0022] After successful verification, the CA server generates authentication information for the algorithm to be traced. If the verification fails, the CA server initializes the software of the algorithm to be traced to issue a digital certificate and generate authentication information. The authentication information is ATMS authentication information, which is used for authentication of the algorithm tracing service.

[0023] The algorithm software to be traced establishes communication and interacts with the algorithm tracing server. During the interaction, the algorithm tracing server authenticates the algorithm software to verify whether its ATMS authentication information is valid.

[0024] The algorithm software to be traced is used to submit a traceability application to the algorithm traceability server after passing the verification, and automatically submit the input data and the calculation results of the algorithm software to be traced after the algorithm traceability server approves the application;

[0025] Algorithm tracing server, used for:

[0026] The algorithm standard is generated based on the aforementioned source tracing application;

[0027] Based on the aforementioned algorithm standard, the algorithm standard output result is further calculated according to the input data submitted by the algorithm software to be traced.

[0028] Based on the calculation results of the algorithm software to be traced and the standard output results of the algorithm, the algorithm tracing server calculates the indication error between the two.

[0029] Further assess the measurement uncertainty of the indicated value error; and,

[0030] Based on the assessment, a digital / electronic certificate or digital / electronic report with measurement uncertainty assessment is generated as the result of algorithm tracing.

[0031] Preferred,

[0032] Algorithm tracing server is also used for:

[0033] The algorithm tracing results are transmitted to the algorithm software to be traced, so that the algorithm software to be traced can obtain the digital / electronic certificate or digital / electronic report.

[0034] In summary, this disclosure closely integrates metrology technology with software testing technology, performs CA certification on the algorithm software to be traced to form a digital certificate, and verifies the validity of the certificate by the CA server, generates identity verification information for the algorithm traceability server, establishes a communication mechanism between the algorithm software to be traced and the algorithm traceability server, and thus enables the algorithm software to be traced to achieve automated algorithm traceability based on this system. Attached Figure Description

[0035] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0036] Figure 1 This is a flowchart illustrating an algorithm tracing method in one embodiment of this disclosure;

[0037] Figure 2 This is a schematic diagram of the configuration process of the algorithm standard in one embodiment of this disclosure. Detailed Implementation

[0038] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0039] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0041] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0042] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0043] For descriptive purposes, this disclosure may use spatial relative terms such as “below,” “under,” “below,” “down,” “above,” “above,” “higher,” and “side (e.g., in a “sidewall”)” to describe the relationship between one component and another component as shown in the accompanying drawings. In addition to the orientations depicted in the drawings, the spatial relative terms are also intended to include different orientations of the device during use, operation, and / or manufacture. For example, if the device in the drawings is flipped, a component described as “below” or “under” another component or feature would subsequently be positioned “above” said other component or feature. Thus, the exemplary term “below” can include both “above” and “below” orientations. Furthermore, the device may be otherwise positioned (e.g., rotated 90 degrees or in other orientations), thus interpreting the spatial relative descriptive terms used herein accordingly.

[0044] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0045] To facilitate a further understanding of this disclosure, the execution entities involved in this disclosure will first be explained. In this disclosure, the following execution entities are mainly involved in the entire algorithm tracing process:

[0046] 1. Algorithm Traceability Master Service (ATMS) refers to a server that allows algorithm standard setters to upload algorithm standards and configure algorithm traceability software modules (providing algorithm traceability calculation services) and can respond to applications for traceable algorithm software.

[0047] 2. A CA (Certificate Authority) server refers to an authoritative CA organization that provides authentication for the algorithm software to be traced. Based on the applicant's application (e.g., in this disclosure, the applicant is the algorithm software to be traced), the authoritative CA organization issues a legally valid and authoritative authentication certificate to the applicant. The certificate contains the signature of the authoritative CA organization's root certificate. The validity of the authentication certificate can be verified through the verification service provided by the CA organization. If the verification is successful, it proves that the certificate has legal effect. Simultaneously, the authentication certificate contains a public key, which can be used to verify the applicant's electronic signature. Successful verification proves that the electronic signature has legal effect, is protected by law, and the signatory cannot deny the content of the signature.

[0048] 3. The source-tracing algorithm software refers to the party that possesses the source-tracing algorithm and can perform related calculations.

[0049] In one embodiment, see Figure 1 This disclosure reveals a fully automated online algorithm tracing method, comprising the following steps:

[0050] S100. The algorithm software to be traced establishes communication and interacts with the CA server. During the interaction, the CA server verifies the validity of the digital certificate, wherein the digital certificate is issued to the algorithm software to be traced in advance by the CA server.

[0051] This is understandable. The CA server needs to check if the authentication certificate for the algorithm software to be traced exists. If it doesn't exist, it needs to reapply for a digital certificate from the authoritative CA. If it exists, it needs to verify whether the digital certificate is valid. If the authentication certificate is invalid, it needs to regenerate the public-private key pair and reapply for the authentication certificate from the authoritative CA. The CA and the CA server can be the same CA server, or the CA can issue digital certificates through a separate server.

[0052] S200. After successful verification, the CA server generates authentication information for the algorithm to be traced. If the verification fails, the CA server performs software initialization of the algorithm to be traced to issue a digital certificate and generate authentication information. The authentication information is ATMS authentication information, which is used for authentication of the algorithm tracing service.

[0053] It is understood that the certificate carries the signature of the CA server's root certificate. The CA server can verify the validity of the digital certificate. The specific verification method can be preset and is not specifically limited in this disclosure. If the certificate passes the verification, it means that the certificate has legal effect; if it fails, the CA server will implement the traceability algorithm software initialization to issue a digital certificate and generate identity verification information.

[0054] S300, the algorithm software to be traced establishes communication and interacts with the algorithm tracing server. During the interaction, the algorithm tracing server authenticates the algorithm software to be traced in order to verify whether its ATMS authentication information is valid.

[0055] S400. After verification, the algorithm software to be traced submits a traceability application to the algorithm traceability server. After the algorithm traceability server approves the application, it automatically submits the input data and the calculation results of the algorithm software to be traced to the algorithm traceability server.

[0056] S500: The algorithm tracing server generates algorithm standards based on the tracing application;

[0057] S600. Based on the algorithm standard, the algorithm tracing server further calculates the algorithm standard output result according to the input data submitted by the algorithm software to be traced.

[0058] S700: Based on the calculation results of the algorithm software to be traced and the standard output results of the algorithm, the algorithm tracing server calculates the indication error between the two.

[0059] S800 and the algorithm tracing server further evaluate the measurement uncertainty of the indicated value error;

[0060] S900. Based on the evaluation, the algorithm traceability server generates a digital / electronic certificate or digital / electronic report with measurement uncertainty evaluation as the algorithm traceability result.

[0061] It is understood that the software used for tracing algorithm origins can be of various known software types, such as 32-bit Win software, 64-bit Win software, Linux application software, client-server architecture software, browser-server architecture software, software running in a physical machine environment, software running in a virtual machine environment, cross-platform container software, etc. When the software used for tracing algorithm origins has its own network communication capabilities, it can directly establish communication and interact with the CA server or algorithm tracing server; when the software itself does not have network communication capabilities, it can establish communication and interact with the CA server or algorithm tracing server through other software.

[0062] Whether communication between the two is achieved directly or indirectly, generating authentication information for the algorithm tracing server for the verified algorithm software to be traced is obviously a preparation for establishing a communication link between the algorithm software to be traced and the algorithm tracing server; establishing a communication link between the algorithm software to be traced and the algorithm tracing server allows the two to communicate automatically without human intervention.

[0063] After successful authentication, the algorithm software to be traced inputs the input data to obtain the calculation result, and submits the input data and calculation result to apply for algorithm traceability. It should be noted that the input data here is provided to the algorithm traceability server by the algorithm software according to the specifications pre-defined by the algorithm standard setter. The calculation time of the algorithm to be traced must be within the validity period of authentication, which can be pre-defined; this disclosure does not specify a specific validity period. Furthermore, this embodiment prioritizes that the algorithm traceability server does not involve the algorithm calculations of the algorithm software to be traced; the relevant calculations are completed by the algorithm software itself. However, it is understood that after passing ATMS verification, the algorithm software to be traced can clone its own execution software to the algorithm traceability server to enable reliable operation on the server. The algorithm traceability server then needs to have the runtime environment to run the algorithm software to be traced, which naturally helps to further ensure trustworthiness. After receiving an application submitted by the algorithm software to be traced, the algorithm tracing server generates a corresponding algorithm standard based on the algorithm information, inputs the received input data into the generated algorithm standard, calculates the corresponding output result, and then calculates the indication error between the two based on the calculation result of the algorithm to be traced and the output result of the algorithm standard.

[0064] It is obvious that this disclosure has the following advantages:

[0065] On the one hand, it provides an algorithm tracing method that is different from algorithm evaluation methods. This method quantitatively evaluates the accuracy and reliability of algorithm calculation results by tracing the source.

[0066] On the other hand, based on the above-mentioned algorithm tracing method, this disclosure can achieve fully online automation and one-click tracing, greatly reducing or even eliminating human intervention, and improving the effectiveness, accuracy and reliability of algorithm tracing.

[0067] In another embodiment, the method further includes the following steps:

[0068] S1000, the algorithm tracing server transmits the algorithm tracing results to the algorithm software to be traced, so that the algorithm software to be traced can obtain the digital / electronic certificate or digital / electronic report.

[0069] In addition, it should be noted that,

[0070] In S500, before the algorithm tracing server generates the algorithm standard based on the tracing application:

[0071] Algorithm standard setters fill in basic algorithm information and configure the algorithm quantification software module on the algorithm tracing server. This module contains the algorithm standard and the corresponding standard calculations used in the algorithm tracing process.

[0072] The algorithm-based data transfer software module refers to the module that performs data transfer calculations for the algorithm to be traced; among which,

[0073] The aforementioned measurement calculation refers to calculating two types of datasets using statistical or other mathematical methods to ultimately obtain the evaluation index Y of the algorithm to be traced and the corresponding measurement uncertainty; wherein, one type of the two types of datasets is the calculation result dataset of the algorithm software to be traced, and the other type is the output result dataset of the algorithm standard calculated based on the input data submitted by the algorithm software to be traced.

[0074] See Figure 2 It illustrates the entire process from logging in as an algorithm standard setter, to identity verification, to filling in basic algorithm information, and configuring the algorithm data transmission software module.

[0075] To measure and evaluate specific algorithms in a particular sub-domain, the methods disclosed in this disclosure are applied, wherein...

[0076] Algorithm standard setters log in to the system, are authenticated, and fill in basic information about the algorithm, including: algorithm standard name, algorithm standard type, algorithm standard setter's name and contact information, affiliated organization, and a brief description of the algorithm; it is understood that algorithm standard setters can be experts from the National Institute of Metrology of China, experts in relevant industry fields, etc., and should be authorized by relevant departments;

[0077] As for configuring the algorithm measurement software module, it can be to configure specific statistical or other mathematical methods for the two types of datasets, which serve as the key to the algorithm standard for measurement. The purpose is to obtain the evaluation index Y of the algorithm software to be traced and the corresponding measurement uncertainty through such methods. Among them, one type of the two types of datasets is the calculation result dataset of the algorithm software to be traced, and the other type is the output result dataset of the algorithm standard calculated based on the input data submitted by the algorithm software to be traced.

[0078] Furthermore, this disclosure provides a more detailed explanation of the algorithm standards:

[0079] Definition of algorithm standard:

[0080] It has a definite value and associated measurement uncertainty, and implements a reference object with the same given quantity definition as similar algorithms.

[0081] Example of algorithm tracing using algorithm standards as the tracing criteria:

[0082] In the field of coordinate measurement, coordinate measuring machines (CMMs) can quickly and accurately evaluate dimensional data, providing operators with useful information about the production process. Their working principle is as follows: The operator places the object to be measured in the measurement space of the CMM. The probe measures the coordinate positions of each measuring point on the object. These spatial coordinate values ​​are then transmitted to the CMM's software module. Relevant software algorithms process the coordinate data to achieve coordinate fitting, ultimately providing the geometric dimensions, shape, and position of the measured object.

[0083] Fitting basic geometric elements is a crucial function of the coordinate measuring machine (CMM) software module in coordinate measurement. The accuracy and reliability of the software algorithm's fitting of coordinate data directly affect the judgment of the size and shape of the measured object. Inaccurate measurements of size and shape can be disastrous for manufacturing plants; therefore, it is necessary to trace the algorithmic origins of the software used for fitting. The fitting of the software algorithm mainly includes the fitting of basic geometric elements such as three-dimensional lines, planes, three-dimensional circles, spheres, cylinders, and cones.

[0084] The following example will use a coordinate fitting algorithm software for a three-dimensional straight line in space as an example to trace the origin of the algorithm.

[0085] 1. The reference standard for traceability adopts the algorithm standard.

[0086] How to formulate the algorithm: By analyzing the theoretical characteristics of a straight line in three-dimensional space, the straight line is parameterized, the distances from spatial coordinates to the line are defined, and the algorithm aims to minimize the sum of the distances from all coordinate points to the line. The minimum fitting is achieved using mathematical methods such as least squares and singular value decomposition. The standard input data for this algorithm is a set of three-dimensional coordinate points in space, and the output data consists of six elements: a point (x0, y0, z0) passing through the three-dimensional straight line and the direction cosine of the line at that point (a, b, c).

[0087] Who developed it: The algorithm standard was developed by experts from the National Institute of Metrology of China and authorized by relevant departments.

[0088] Accuracy: The algorithm has a standard measurement uncertainty of 1.5 μm.

[0089] Input data specifications: In addition, to facilitate algorithm tracing, relevant experts from the National Institute of Metrology of China studied and formulated input data specifications for this example.

[0090] 2. Traceability Process

[0091] The reference standard for this algorithm tracing is the algorithm standard, and the tracing process is as follows:

[0092] 1) After a manufacturer's software for spatial three-dimensional line fitting has passed CA certification and ATMS authentication, it submits input data that meets the requirements and the calculation results of the algorithm corresponding to the input data, in accordance with the input data specifications pre-defined by the algorithm standard setter.

[0093] 2) The algorithm tracing server receives input data and the calculation results of the algorithm software to be traced, based on the application, and generates an algorithm standard for the algorithm software to be traced. Simultaneously, the algorithm tracing server inputs the received input data into the algorithm standard to calculate the corresponding results.

[0094] 3) The two types of calculation results are transmitted to the algorithm measurement software module in the algorithm traceability server, where the software processes the data and obtains the algorithm evaluation value Y and its corresponding measurement uncertainty;

[0095] 4) Based on the evaluation results, the algorithm traceability server generates a digital / electronic certificate or digital / electronic report with measurement uncertainty evaluation as the algorithm traceability result;

[0096] 5) The algorithm traceability server transmits the algorithm traceability results to the algorithm software to be traced, and the algorithm software to be traced obtains a digital / electronic certificate or digital / electronic report with measurement uncertainty assessment.

[0097] 3. Source tracing results

[0098] The algorithm tracing results for the software developed by a certain manufacturer for spatial three-dimensional line fitting are as follows:

[0099] 1) The algorithm evaluation value Y of the algorithm software to be traced is 95.2%, and its corresponding relative standard measurement uncertainty is 3.52%;

[0100] 2) The algorithm tracing results show that the algorithm software to be traced can achieve a good fitting function, and its fitting results meet the needs of production and manufacturing.

[0101] Furthermore, in another embodiment, this disclosure discloses a fully online automated algorithm tracing system to implement the above method, including:

[0102] The parties involved are the CA server, the algorithm software to be traced, and the algorithm tracing server.

[0103] A CA server is used to verify the validity of digital certificates, wherein the digital certificates are pre-issued by the CA server to the algorithm software to be traced; and,

[0104] After successful verification, the CA server generates authentication information for the algorithm to be traced. If the verification fails, the CA server initializes the software of the algorithm to be traced to issue a digital certificate and generate authentication information. The authentication information is ATMS authentication information, which is used for authentication of the algorithm tracing service.

[0105] The algorithm software to be traced establishes communication and interacts with the algorithm tracing server. During the interaction, the algorithm tracing server authenticates the algorithm software to verify whether its ATMS authentication information is valid.

[0106] The algorithm software to be traced is used to submit a traceability application to the algorithm traceability server after passing the verification, and automatically submit the input data and the calculation results of the algorithm software to be traced after the algorithm traceability server approves the application;

[0107] Algorithm tracing server, used for:

[0108] The algorithm standard is generated based on the aforementioned source tracing application;

[0109] Based on the aforementioned algorithm standard, the algorithm standard output result is further calculated according to the input data submitted by the algorithm software to be traced.

[0110] Based on the calculation results of the algorithm software to be traced and the standard output results of the algorithm, the algorithm tracing server calculates the indication error between the two.

[0111] Further assess the measurement uncertainty of the indicated value error; and,

[0112] Based on the assessment, a digital / electronic certificate or digital / electronic report with measurement uncertainty assessment is generated as the result of algorithm tracing.

[0113] In another embodiment,

[0114] Algorithm tracing server is also used for:

[0115] The algorithm tracing results are transmitted to the algorithm software to be traced, so that the algorithm software to be traced can obtain the digital / electronic certificate or digital / electronic report.

[0116] For example, the final result of an algorithm tracing might look like this:

[0117] The evaluation index Y ranges from [0, 100%]. When the Y value approaches 100%, it indicates that the output of the traceability algorithm is sufficiently accurate and reliable. When the Y value approaches 0, it indicates that the implementation process of the traceability algorithm does not meet production requirements. The corresponding measurement uncertainty indicates the dispersion of the Y value.

[0118] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0119] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0120] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A fully online automated algorithm tracing method, characterized in that, The method described above performs CA certification on the algorithm software to be traced to form a digital certificate, and the CA server verifies the validity of the certificate, generates identity verification information for the algorithm tracing server, establishes a communication mechanism between the algorithm software to be traced and the algorithm tracing server, and then enables the algorithm software to achieve automated algorithm tracing. The method includes the following steps: S100. The algorithm software to be traced establishes communication and interacts with the CA server. During the interaction, the CA server verifies the validity of the digital certificate, wherein the digital certificate is issued to the algorithm software to be traced in advance by the CA server. S200. After successful verification, the CA server generates authentication information for the algorithm software to be traced. If the verification fails, the CA server initializes the algorithm software to be traced to issue a digital certificate and generate authentication information. The authentication information is the algorithm tracing server authentication information, which is used for authentication of the algorithm tracing service. S300: The algorithm software to be traced establishes communication and interacts with the algorithm tracing server. During the interaction, the algorithm tracing server authenticates the algorithm software to be traced in order to verify whether its algorithm tracing server authentication information is valid. S400. After verification, the algorithm software to be traced submits a traceability application to the algorithm traceability server. After the algorithm traceability server approves the application, it automatically submits the input data and the calculation results of the algorithm software to be traced to the algorithm traceability server. S500, The algorithm tracing server adopts the algorithm standard according to the tracing application; S600. Based on the algorithm standard, the algorithm tracing server further calculates the algorithm standard output result according to the input data submitted by the algorithm software to be traced. S700: Based on the calculation results of the algorithm software to be traced and the standard output results of the algorithm, the algorithm tracing server calculates the indication error between the two. S800 and the algorithm tracing server further evaluate the measurement uncertainty of the indicated value error; S900. Based on the evaluation, the algorithm traceability server generates a digital certificate or digital report with measurement uncertainty evaluation as the algorithm traceability result. in, In S500, before the algorithm tracing server adopts the algorithm standard based on the tracing application: Algorithm standard setters fill in basic algorithm information and configure the algorithm data transfer software module on the algorithm traceability server. This module contains the algorithm standard and the corresponding data transfer calculations used in the algorithm traceability process. The algorithm-based data transfer software module refers to the module that performs data transfer calculations for the algorithm to be traced; among which, The aforementioned measurement calculation refers to calculating two types of datasets using statistical or other mathematical methods to ultimately obtain the evaluation index Y of the algorithm to be traced and the corresponding measurement uncertainty; wherein, one type of the two types of datasets is the calculation result dataset of the algorithm software to be traced, and the other type is the output result dataset of the algorithm standard calculated based on the input data submitted by the algorithm software to be traced.

2. The method according to claim 1, further comprising the following steps: S1000, the algorithm tracing server transmits the algorithm tracing results to the algorithm software to be traced, so that the algorithm software to be traced can obtain the digital certificate or digital report.

3. A fully online automated algorithm tracing system for implementing the method described in any one of claims 1 to 2, comprising: The parties involved are the CA server, the algorithm software to be traced, and the algorithm tracing server. A CA server is used to verify the validity of digital certificates, wherein the digital certificates are pre-issued by the CA server to the algorithm software to be traced; and, After successful verification, the CA server generates authentication information for the algorithm software to be traced. If the verification fails, the CA server initializes the algorithm software to be traced to issue a digital certificate and generate authentication information. The authentication information is the algorithm tracing server authentication information, which is used for authentication of the algorithm tracing service. The algorithm software to be traced establishes communication and interacts with the algorithm tracing server. During the interaction, the algorithm tracing server authenticates the algorithm software to verify whether its algorithm tracing server authentication information is valid. The algorithm software to be traced is used to submit a traceability application to the algorithm traceability server after passing the verification, and automatically submit the input data and the calculation results of the algorithm software to be traced after the algorithm traceability server approves the application; Algorithm tracing server, used for: The algorithm standard adopted in the aforementioned tracing application; Based on the aforementioned algorithm standard, the algorithm standard output result is further calculated according to the input data submitted by the algorithm software to be traced. Based on the calculation results of the algorithm software to be traced and the standard output results of the algorithm, the algorithm tracing server calculates the indication error between the two. Further assess the measurement uncertainty of the indicated value error; and, Based on the assessment, a digital certificate or digital report with measurement uncertainty assessment is generated as the result of algorithm tracing.

4. The system according to claim 3, wherein, Algorithm tracing server is also used for: The algorithm tracing results are transmitted to the algorithm software to be traced, so that the algorithm software to be traced can obtain the digital certificate or digital report.

Images