Algorithm tracing method and system for whole-course online automatic execution
Through algorithm traceability methods and systems that are automatically executed online throughout the process, using multi-threading technology and hashing algorithms and other means, the problems of low traceability efficiency and inauspicious data collection are solved, and efficient and accurate algorithm traceability and data management are achieved. , enhances the transparency and interpretability of the algorithm.
Patent Information
- Application Number
- CN202411517720.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-01-07
AI Technical Summary
Traditional algorithms rely on manual traceability, are inefficient and prone to errors, and lack automatic mechanisms for data acquisition, making it difficult to ensure the integrity and timeliness of data.
A method and system for automatic execution of algorithmic traceability for full-process online automation is designed, including data acquisition module, data marking module, data storage module, algorithm analysis module, algorithm marking module, association establishment module and traceability query module, through multi-threading technology, Hash algorithm, digital signature technology, distributed database, static and dynamic analysis, association rule mining and other technologies realize the automatic collection, marking, storage, analysis and association of data and algorithms, and provide visual query results.
It greatly reduces manual intervention, improves the efficiency and accuracy of algorithm traceability, enhances the transparency and interpretability of the algorithm, and ensures the integrity and timeliness of the data.
Smart Images
Figure CN119271897A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of algorithm tracing technology, and specifically to an algorithm tracing method and system for fully online automated execution. Background Art
[0002] In today's digital age, algorithms are increasingly used in various fields, from financial transactions to medical diagnosis, from smart transportation to social media.
[0003] At present, although there are some methods and tools for algorithm tracing, most of these methods and tools have the following problems: traditional algorithm tracing relies on manual work, which is inefficient and error-prone, and data collection lacks an automatic mechanism, making it difficult to ensure that the data is complete and timely. To this end, the present invention provides an algorithm tracing method and system that is fully online and automatically executed. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention provides an algorithm tracing method and system with fully online automated execution, which solves the problems that traditional algorithm tracing relies on manual labor, is inefficient and prone to errors, and lacks an automatic mechanism for data collection, making it difficult to ensure data integrity and timeliness.
[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method and system for tracing the algorithm with full online automatic execution, specifically comprising the following steps:
[0006] S1. Set up a data acquisition module to automatically collect input data, intermediate data and output data during the algorithm execution process through a preset data interface.
[0007] S2: Use the data labeling module to label the collected data. The labeling content includes the source of the data, timestamp and algorithm version information.
[0008] S3: Use the data storage module to store the marked data in the distributed database.
[0009] S4: Start the algorithm analysis module to analyze the algorithm and extract the structure and parameter information of the algorithm.
[0010] S5: Mark the algorithm through the algorithm marking module, and the marking content includes the algorithm name, version number and developer information.
[0011] S6: Use the association building module to establish the association relationship between data and algorithms.
[0012] S7: Provides a traceability query module, where users can quickly query the algorithm execution process and related data by entering query conditions.
[0013] S8: The query results are displayed to the user in a visual manner with the help of the result display module.
[0014] Preferably, when collecting data, the data collection module sets different collection priorities according to the importance and real-time nature of the data, and gives priority to collecting data with high importance and strong real-time nature; and delays collecting data with low importance and weak real-time nature.
[0015] Preferably, the data marking module uses a hash algorithm to mark the data.
[0016] Preferably, when storing data, the data storage module adopts different storage strategies according to the access frequency and storage period of the data. For data with high access frequency, high-speed storage devices are used for storage; for data with long storage period, low-cost storage devices are used for storage.
[0017] Preferably, when performing algorithm analysis, the algorithm analysis module adopts different analysis techniques according to the complexity and security requirements of the algorithm. For complex algorithms, deep learning technology is used for analysis; for algorithms with high security requirements, formal verification technology is used for analysis.
[0018] Preferably, the algorithm labeling module uses a multiple labeling method when labeling the algorithm.
[0019] Preferably, when establishing the association relationship between data and algorithms, the association establishment module automatically updates the association relationship according to changes in the data and algorithms. When the data or algorithm changes, the association establishment module detects the change in a timely manner and re-establishes the association relationship to ensure the accuracy and timeliness of the association relationship.
[0020] Preferably, the traceability query module provides personalized query suggestions based on the user's query history and preferences when performing a query. When the user performs a query, the traceability query module analyzes the user's query history and preferences and automatically recommends relevant query conditions and query results.
[0021] The present invention also discloses a traceability system for an algorithm traceability method that is fully online and automatically executed, including a data acquisition module, a data marking module, a storage module, an algorithm analysis module, an algorithm marking module, an association establishment module, a traceability query module and a display module;
[0022] After the data acquisition module collects data through a preset data interface, the data is transmitted to the data marking module. The data marking module marks the data and then transmits it to the data storage module. The algorithm analysis module analyzes the algorithm and then transmits the information to the algorithm marking module and the association establishment module. The algorithm marking module marks the algorithm and then transmits it to the association establishment module. After the association establishment module receives the data information of the data marking module and the algorithm information of the algorithm marking module, it provides the association information to the traceability query module. The traceability query module obtains data from the data storage module, and performs a query in combination with the association information provided by the association establishment module. The query result is transmitted to the result display module.
[0023] Beneficial Effects
[0024] The present invention provides an algorithm tracing method and system for full online automated execution. Compared with the prior art, it has the following beneficial effects:
[0025] (1) This algorithm traceability method and system with fully online automated execution, through the data acquisition module using multi-threading technology and preset data interface, automatically collects various types of data during the algorithm execution process, and monitors the algorithm operation status in real time. Once the data changes, the collection program is immediately started, and the collection method is flexibly selected according to the characteristics of different data. At the same time, all links are closely coordinated, from data marking using hash algorithm, digital signature technology and encryption technology, to data storage in a high-reliability distributed database, to algorithm parsing using a combination of static analysis and dynamic analysis, algorithm marking using digital signature technology and generating a unique identifier, as well as the collaborative work of modules such as association establishment, traceability query and result display, which greatly reduces manual intervention and improves the efficiency and accuracy of algorithm traceability.
[0026] (2) This algorithm traceability method and system, which is fully automated and executed online, establishes a relationship between data and algorithms through detailed labeling of data and algorithms, and displays query results in a visual manner, thereby providing algorithm users and regulators with a clear algorithm execution path and related information, enhancing the transparency and explainability of the algorithm, and effectively addressing the challenge of reduced transparency and explainability due to increased algorithm complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a flow chart of the algorithm tracing method of the present invention;
[0028] Figure 2 This is a principle block diagram of the algorithm tracing system of the present invention. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] See also Figure 1-2 , the present invention provides a technical solution:
[0031] A method for tracing the provenance of an algorithm with full online automated execution, specifically comprising the following steps:
[0032] S1. Set up a data acquisition module to automatically collect input data, intermediate data, and output data during the algorithm execution process through a preset data interface, monitor the algorithm operation status in real time, start the acquisition program immediately when the data changes, use multi-threading technology to collect data from multiple data sources, and select streaming or scheduled acquisition methods based on data characteristics.
[0033] S2. Use the data marking module to combine the hash algorithm and digital signature technology to mark the collected data, including information such as data source, timestamp and algorithm version. Custom marking can be performed. First, use the hash algorithm to generate a hash value, and then use the digital signature technology to sign the hash value.
[0034] S3. Use the data storage module to store the marked data in a distributed database. The database has a high-reliability storage architecture, redundant backup and automatic recovery functions. Different storage methods are selected according to the characteristics of the data, and the data can be encrypted.
[0035] S4. Start the algorithm analysis module and use a combination of static analysis and dynamic analysis to analyze the algorithm, extract information such as the algorithm's logical structure and key parameters, and perform reverse engineering analysis on complex algorithms.
[0036] S5. Through the algorithm marking module, use digital signature technology to mark the algorithm, including information such as the algorithm name, version number and developer, to generate a unique identifier for the algorithm. Different marking methods can be used according to the purpose of the algorithm.
[0037] S6. Use the association establishment module to establish the association relationship between data and the algorithm through the association rule mining algorithm. Different association rules and association strengths can be set, and the association information is stored in the association database.
[0038] S7. Provides a traceability query module where users can enter multiple query conditions for query. It supports multiple query methods and can perform fuzzy query and advanced query according to needs.
[0039] S8. With the help of the result display module, the query results are displayed in a visual way in various forms. Data can be screened, filtered and aggregated, and data download and export functions are provided.
[0040] In the embodiment of the present invention, the data acquisition module automatically collects various data during the execution of the algorithm through a preset data interface, monitors the algorithm operation status in real time, and immediately starts the acquisition program once the data changes. In order to improve efficiency, multi-threading technology is adopted, and multiple data sources can be collected at the same time. The collection time interval can be determined according to the algorithm execution frequency. The formula is T=1 / f, where T represents the collection time interval, and f represents the algorithm execution frequency. For different types of data, the collection method can be flexibly selected according to their characteristics and real-time requirements.
[0041] In an embodiment of the present invention, a data marking module marks the collected data, and the marking content includes information such as the source of the data, timestamp, and algorithm version. The marking method can use a hash algorithm to perform a hash operation on the key information of the data to generate a unique marking value, thereby ensuring the identifiability and traceability of the data.
[0042] In an embodiment of the present invention, the data storage module stores the marked data in a distributed database. The database adopts a distributed storage architecture and has data redundant backup and automatic recovery functions. Different storage methods are selected according to data characteristics and requirements, and the stored data can be encrypted to ensure data security.
[0043] In an embodiment of the present invention, an algorithm analysis module analyzes the algorithm and extracts the structure and parameter information of the algorithm. The analysis method can adopt a combination of static analysis and dynamic analysis. Static analysis mainly analyzes the source code of the algorithm to extract the structure and parameter information of the algorithm; dynamic analysis mainly monitors the state of the algorithm during the execution of the algorithm to extract the runtime information of the algorithm.
[0044] In an embodiment of the present invention, an algorithm marking module marks the algorithm, and the marking content includes information such as the name, version number and developer of the algorithm. The marking method can use digital signature technology to digitally sign the key information of the algorithm and generate a unique marking value, which can ensure the identifiability and traceability of the algorithm.
[0045] In an embodiment of the present invention, an association establishment module establishes an association relationship between data and an algorithm so that the execution process of the algorithm can be quickly traced when necessary. The association method can adopt an association rule mining algorithm to mine the tag information of the data and the algorithm to find out the association relationship between them. The association establishment module can set different association rules and association strengths according to user needs. For example, a direct association, indirect association, and weak association between the data and the algorithm can be set. After the association is established, the information of the data and the algorithm is stored in an association database for convenient and quick query and tracing.
[0046] In an embodiment of the present invention, a traceability query module provides a user interface, and a user can quickly query the execution process and related data of an algorithm by inputting query conditions. The query conditions may include information such as the source of the data, timestamp, algorithm version, and algorithm name; a result display module displays the query results to the user in a visual manner, facilitating the user to analyze and judge. The visualization method may take the form of charts, reports, etc., to intuitively display the execution process and related data of the algorithm.
[0047] In an embodiment of the present invention, the present invention further discloses a traceability system for an algorithm traceability method that is fully online and automatically executed, including a data acquisition module, a data marking module, a storage module, an algorithm analysis module, an algorithm marking module, an association establishment module, a traceability query module, and a display module;
[0048] After the data acquisition module collects data through the preset data interface, the data is transmitted to the data marking module. The data marking module marks the data and then transmits it to the data storage module. The algorithm analysis module analyzes the algorithm and then transmits the information to the algorithm marking module and the association establishment module. The algorithm marking module marks the algorithm and then transmits it to the association establishment module. After the association establishment module receives the data information of the data marking module and the algorithm information of the algorithm marking module, it provides the association information to the traceability query module. The traceability query module obtains data from the data storage module and performs a query based on the association information provided by the association establishment module. The query results are transmitted to the result display module.
[0049] In summary, the data acquisition module uses multi-threading technology and preset data interfaces to automatically collect various types of data during the algorithm execution process, and monitors the algorithm operation status in real time. When the data changes, the acquisition program is immediately started, and the appropriate acquisition method is selected according to the characteristics of the data; then, the data marking module uses a combination of hash algorithm, digital signature technology and encryption technology to mark the collected data, including source, timestamp, algorithm version and other information, and can also perform custom marking. Then, the data storage module stores the marked data in a highly reliable distributed database, selects the storage method according to the characteristics of the data and can encrypt it. After that, the algorithm analysis module uses a combination of static analysis and dynamic analysis. The algorithm is parsed; the algorithm marking module uses digital signature technology to mark the algorithm name, version number and other information and generate a unique identifier. Different marking methods are used according to the purpose of the algorithm. The association establishment module then uses the association rule mining algorithm to establish the association relationship between the data and the algorithm. Different association rules and strengths can be set, and it can be automatically updated according to changes; users can enter a variety of query conditions for query through the traceability query module, which supports multiple query methods and provides suggestions based on user history and preferences. Finally, the result display module displays the query results in a visual way in a variety of forms. It also provides data download and export functions to facilitate users to further analyze and process. The entire process achieves efficient and accurate traceability of the algorithm.
[0050] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.
[0051] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0052] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A fully online automated algorithm tracing method, characterized by: The specific steps include: S1. Set up a data acquisition module to automatically collect input data, intermediate data and output data during the algorithm execution process through a preset data interface. S2: Use the data labeling module to label the collected data. The labeling content includes the source of the data, timestamp and algorithm version information. S3: Use the data storage module to store the marked data in the distributed database. S4: Start the algorithm analysis module to analyze the algorithm and extract the structure and parameter information of the algorithm. S5: Mark the algorithm through the algorithm marking module, and the marking content includes the algorithm name, version number and developer information. S6: Use the association building module to establish the association relationship between data and algorithms. S7: Provides a traceability query module, where users can quickly query the algorithm execution process and related data by entering query conditions. S8: The query results are displayed to the user in a visual manner with the help of the result display module.
2. According to claim 1, the algorithm tracing method for full online automated execution is characterized by: When the data collection module collects data, different collection priorities are set according to the importance and real-time nature of the data, and data with high importance and strong real-time nature are collected first; For data with low importance and weak real-time nature, collection is delayed.
3. The algorithm tracing method for full online automated execution according to claim 1 is characterized by: The data marking module uses a hash algorithm to mark the data.
4. The algorithm tracing method for full online automated execution according to claim 1 is characterized by: When storing data, the data storage module adopts different storage strategies according to the access frequency and storage period of the data. For data with high access frequency, high-speed storage devices are used for storage; for data with long storage period, low-cost storage devices are used for storage.
5. The algorithm tracing method for full online automated execution according to claim 1 is characterized by: When performing algorithm analysis, the algorithm analysis module adopts different analysis technologies according to the complexity and security requirements of the algorithm. For complex algorithms, deep learning technology is used for analysis; for algorithms with high security requirements, formal verification technology is used for analysis.
6. The algorithm tracing method for full online automated execution according to claim 1 is characterized by: The algorithm marking module uses a multiple marking method when marking the algorithm.
7. The algorithm tracing method for full online automated execution according to claim 1 is characterized by: When establishing the association relationship between data and algorithms, the association establishment module automatically updates the association relationship according to changes in the data and algorithms. When the data or the algorithm changes, the association establishment module detects the change in time and re-establishes the association relationship to ensure the accuracy and timeliness of the association relationship.
8. The algorithm tracing method for full online automated execution according to claim 1 is characterized by: When performing a query, the traceability query module provides personalized query suggestions based on the user's query history and preferences. When the user performs a query, the traceability query module analyzes the user's query history and preferences and automatically recommends relevant query conditions and query results.
9. A traceability system for an algorithm traceability method for full online automatic execution according to any one of claims 1 to 8, characterized in that: include: Data collection module, data marking module, storage module, algorithm analysis module, algorithm marking module, association establishment module, traceability query module and display module; After the data acquisition module collects data through a preset data interface, the data is transmitted to the data marking module. The data marking module marks the data and then transmits it to the data storage module. The algorithm analysis module analyzes the algorithm and then transmits the information to the algorithm marking module and the association establishment module. The algorithm marking module marks the algorithm and then transmits it to the association establishment module. After the association establishment module receives the data information of the data marking module and the algorithm information of the algorithm marking module, it provides the association information to the traceability query module. The traceability query module obtains data from the data storage module, and performs a query in combination with the association information provided by the association establishment module. The query result is transmitted to the result display module.