Data acquisition and data and algorithm management system

Abstract

The invention discloses a data acquisition and data and algorithm management system, which belongs to the technical field of data acquisition and management and comprises a data acquisition module, a data processing module, a data storage module, an algorithm management module and a data and algorithm linkage module. According to the invention, data acquisition, storage, processing and algorithm management are integrated into an intelligent whole, the intelligent level is high, the system can adaptively adjust and optimize data processing and algorithm application, intelligent scheduling and optimization are carried out according to the actual environment and demand, the flexibility and efficiency of the system are improved, and the system is suitable for popularization and application. Through the intelligent linkage mechanism, the system can better cope with complex and changeable application scenes, and the value of data and the value of an algorithm are fully exerted.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection and management, and in particular to a data collection and data and algorithm management system. Background Art

[0002] In modern society, data collection, management and utilization play an increasingly important role in various fields. From industrial manufacturing, agricultural production, to medical health and smart homes, various sensors and smart devices continuously generate a large amount of data. The effective collection, storage, processing and analysis of these data are of great significance to improving production efficiency, optimizing resource allocation and improving the quality of life. In addition, due to the different programming languages ​​familiar to R&D personnel, algorithm management is difficult, and there are huge differences in the use of data, which brings great difficulties to the subsequent use, development and upgrading of algorithms. The unified storage, call and upgrade of algorithms will greatly improve the efficiency of R&D personnel and the user experience of customers.

[0003] The existing data collection and data and algorithm management systems have the following deficiencies in practical applications:

[0004] 1. Data silos: In the existing data collection and management system, data between different devices and systems cannot be effectively communicated, resulting in serious data silos. This phenomenon makes it difficult to integrate the data generated and stored in each system, forming isolated data units, which limits the comprehensive use and in-depth analysis of the data. At the same time, data silos also hinder cross-industry and cross-platform data integration and application, further limiting the value mining of data and the realization of commercial potential.

[0005] 2. Poor real-time performance: The data acquisition and processing system has poor real-time performance and cannot meet certain scenarios that require timely response. For example, real-time data is required during the operation of the equipment, and the algorithm is used for real-time processing. However, if the delay of the data acquisition and processing system is large, it will not be possible to detect and correct problems in the operation of the equipment in a timely manner, which may lead to low equipment operation efficiency and equipment errors. Data processing systems with poor real-time performance will find it difficult to respond to emergencies and rapidly changing situations, affecting the reliability and stability of the overall system.

[0006] 3. Difficulty in algorithm management: With the widespread application of artificial intelligence and machine learning technologies, the number and complexity of data processing algorithms continue to increase, and existing systems have difficulties in algorithm version management, updating, scheduling, and data application. Specifically, different versions of algorithms are prone to confusion during management and switching, resulting in inaccurate algorithm version control. Secondly, algorithm updates need to be closely integrated with data and application scenarios, and existing systems find it difficult to achieve flexible updates and seamless deployment of algorithms. In addition, with the diversification and complexity of application scenarios, algorithm scheduling and optimization have become increasingly difficult. Existing systems lack intelligent scheduling mechanisms and are unable to dynamically select and adjust the optimal algorithm based on actual needs, limiting the actual application effect of the algorithm.

[0007] 4. Security issues: Data is vulnerable to attacks during transmission, storage, and processing, and existing systems are deficient in data security and privacy protection. For example, data may be intercepted and tampered with during network transmission, data stored in the cloud or locally may be subject to hacker attacks and data leakage, and the processing process may face the risk of malicious code and internal threats. Existing systems have deficiencies in data encryption, access control, and security auditing, making it difficult to provide comprehensive and effective security protection. Especially in data processing scenarios involving sensitive information and personal privacy, security issues are particularly prominent, directly affecting user trust and the reliable operation of the system.

[0008] 5. Insufficient intelligence: The current data collection and data and algorithm management systems usually operate as independent modules, lacking collaborative work and intelligent linkage mechanisms. This decentralized architecture results in a low overall intelligence level of the system, and is unable to achieve efficient collaboration and dynamic optimization of data and algorithms. In practical applications, in the face of complex and changing environments and demands, it is difficult for existing systems to adaptively adjust and optimize, limiting the potential of intelligent applications. In addition, the lack of unified intelligent management makes it difficult for the system to exert its due efficiency and effectiveness when processing large-scale data and complex algorithms. Summary of the invention

[0009] In view of the above-mentioned problems of data islands, poor real-time performance, difficulty in algorithm management, security issues and insufficient intelligence of traditional data collection and data and algorithm management systems in the prior art, the present invention provides a data collection and data and algorithm management system, aiming to achieve efficient data collection, management and utilization, and provide a safe, real-time and effective data processing and algorithm management solution.

[0010] The technical solution of the present invention to solve the above technical problems is as follows:

[0011] A data collection and data and algorithm management system, comprising: a data collection module, a data processing module, a data storage module, an algorithm management module and a data and algorithm linkage module;

[0012] The data acquisition module periodically collects data from the device and transmits the collected data to the data processing module;

[0013] The data processing module receives the data transmitted by the data acquisition module, processes the data, and transmits the processed data to the data storage module for storage;

[0014] The data storage module is used to store data in a structured or unstructured manner for subsequent retrieval and analysis, and the data storage module uploads the stored data to a database;

[0015] The algorithm management module is responsible for managing various data analysis algorithms and uploading the algorithms to the algorithm part in the database, wherein the database includes a data part and an algorithm part; the data analysis algorithms include but are not limited to machine learning models, deep learning algorithms, statistical analysis algorithms, and optimization algorithms;

[0016] The data and algorithms stored in the database interact through the data and algorithm linkage module. The algorithm linkage module matches and selects the algorithms in the database with the required data information to ensure the normal operation of the algorithm.

[0017] Furthermore, the data acquisition module sends a data acquisition signal to the device through the Modbus-TCP protocol, and obtains corresponding data through the Modbus point table.

[0018] Furthermore, each algorithm corresponds to a basic information table, which is stored in the database; wherein the basic information table includes the algorithm name, algorithm version number, algorithm deployment location, algorithm call path, feature requirements of algorithm input samples, algorithm function interface call input parameter list, output result storage path and file.

[0019] Furthermore, the data processing module processes the data, including data cleaning, format conversion, and outlier detection, to convert the data into a format that meets the requirements.

[0020] Furthermore, the database is a MySQL database structure.

[0021] Furthermore, during the transmission, storage and processing of data in the management system, multi-level encryption strategies and access control strategies are adopted to ensure the confidentiality and integrity of the data.

[0022] Furthermore, the encryption strategy includes encryption in data storage and encryption in data storage; the access control strategy includes identity authentication, authority management, and auditing and monitoring.

[0023] The present invention also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data collection and data and algorithm management system.

[0024] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the data collection and data and algorithm management systems described above.

[0025] In summary, compared with the prior art, the above technical solution has the following beneficial effects:

[0026] (1) This system eliminates the phenomenon of data silos. Through data calling, transmission, and processing between various modules, this system not only promotes the comprehensive utilization of data, but also enhances the collaborative working ability between various systems and improves the efficiency of overall data processing and decision-making;

[0027] (2) This system integrates data collection, storage, processing and algorithm management into an intelligent whole with a high level of intelligence. The system can adaptively adjust and optimize data processing and algorithm application, and intelligently schedule and optimize according to the actual environment and needs to improve the flexibility and efficiency of the system. The data and algorithm linkage module of this system can link data and algorithms, combine the entire data and algorithm, make the process intelligent, and form a closed loop, so as to better cope with complex and changing application scenarios and give full play to the value of data and algorithms. This system has high real-time data collection and processing capabilities, which can meet the usage scenarios of most departments. This system can achieve millisecond-level data collection and processing, ensuring that the system can respond to and handle emergencies in a timely manner, and improve the reliability and stability of the overall system.

[0028] (3) The system provides a flexible algorithm management mechanism, supports algorithm version management, online updates, and dynamic scheduling. The system implements intelligent algorithm scheduling and optimization through the data and algorithm linkage module. The system can dynamically select and adjust the optimal algorithm according to actual needs to improve the application effect of the algorithm. At the same time, the system can ensure accurate version control and seamless deployment of the algorithm to adapt to complex and changing application scenarios.

[0029] (4) This system provides comprehensive data security and privacy protection. During the transmission, storage and processing of data, advanced encryption technology and strict access control mechanisms are used to prevent data from being intercepted, tampered with and leaked, ensuring the confidentiality and integrity of data and enhancing user trust and system security.

[0030] (5) The system adopts a modular design and is easy to expand and integrate. Whether adding new data acquisition equipment or integrating new data processing algorithms, it can be easily expanded and upgraded to meet the ever-changing business needs and technological development trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a system framework diagram of an embodiment of the present invention;

[0032] Figure 2 is a schematic diagram of a data acquisition module according to an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of a data processing module according to an embodiment of the present invention;

[0034] Figure 4 Schematic diagram of the linkage between data and algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The principles and features of the present invention are described below in conjunction with all the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0036] The embodiment of the present invention discloses a data collection and data and algorithm management system.

[0037] Reference Figures 1 to 4 , a data acquisition and data and algorithm management system, including: data acquisition module, data processing module, data storage module, algorithm management module and data and algorithm linkage module. This system adopts modular design and is easy to expand and integrate. Whether it is adding new data acquisition equipment or integrating new data processing algorithms, it can be easily expanded and upgraded to meet the ever-changing business needs and technological development trends. This system integrates data acquisition, storage, processing and algorithm management into an intelligent whole with a high level of intelligence. The system can adaptively adjust and optimize data processing and algorithm application, perform intelligent scheduling and optimization according to actual environment and needs, and improve the flexibility and efficiency of the system. Through this intelligent linkage mechanism, this system can better cope with complex and changing application scenarios and give full play to the value of data and algorithms.

[0038] The data acquisition module collects data from the device at a certain time interval through the Modbus-TCP protocol, that is, it periodically collects the data of the device and transmits the collected data to the data processing module. The data acquisition module sends a data acquisition signal to the device through the Modbus-TCP protocol and obtains the corresponding data through the Modbus point table.

[0039] Specifically, Figure 2As shown, the data acquisition module program establishes a connection with the device through a network cable. The data acquisition module sends a data acquisition signal to the device through the Modbus-TCP protocol, and obtains the corresponding data through the Modbus point table. The data acquisition module periodically obtains device data by periodically sending acquisition signals.

[0040] To address the problem of poor real-time performance, when using the Modbus protocol for data collection, the traditional method is to collect data sequentially one address at a time, and it is not possible to collect data from multiple addresses at the same time point. In addition, sequential collection is slow for data from multiple addresses, and millisecond-level data collection is not possible.

[0041] In this embodiment, multi-threaded parallel data collection is used, and Modbus supports up to 1000 acquisition devices to collect data at the same time. The same acquisition task is divided into 1000 threads to collect data in different Modbus addresses at the same time, so that data from different addresses can be collected at the same time, the acquisition speed can be increased to milliseconds, and the data collection time point can be unified.

[0042] Multiple threads process data simultaneously, and each thread processes the acquired data and finally combines them together, which improves the processing speed. Monitor the processed data, and promptly feedback and upload abnormal data to developers to detect and handle abnormalities in a timely manner.

[0043] Existing technologies are used to implement data cleaning, format conversion, outlier detection, etc. During the data collection process, a data information table will be automatically generated. Each piece of data corresponds to the data information table one by one. The data information table includes: data name, collection location, product and version number, data type, collection accuracy, measurement cycle, etc. Data with the same attributes such as product and version number, data type, collection accuracy, measurement cycle, etc. in the corresponding data information table will be automatically classified into one category, and indexes will be automatically established based on the product and version number to achieve effective data intercommunication between different devices and systems.

[0044] The data processing module receives the data transmitted by the data acquisition module, processes the data, and transmits the processed data to the data storage module for storage. The data processing module processes the data, including data cleaning, format conversion, and outlier detection, converts the data into a format that meets the requirements, and transmits the processed data to the data storage module for storage.

[0045] Specifically, Figure 3As shown, the data acquisition module transmits the processed data to the data processing module, which performs further data cleaning on the data, removes redundant and erroneous data, converts the data into a unified format for subsequent processing and storage, and detects and marks abnormal values ​​in the data for subsequent analysis and processing. The processed data is transmitted to the data storage module for storage, and the data storage module stores the data according to a predetermined structure, and uploads the data separately through the MySQL database structure.

[0046] The data storage module is responsible for storing data in a structured or unstructured manner for subsequent retrieval and analysis. The data storage module uploads the stored data to the database. The data and algorithms stored in the database interact through the data and algorithm linkage module, and the algorithms in the database are matched and selected with the required data information to ensure the normal operation of the algorithm. The database is a MySQL database structure.

[0047] The algorithm management module is responsible for managing various data analysis algorithms and uploading these algorithms to the algorithm part in the database. These algorithms can be machine learning models, deep learning algorithms, statistical analysis methods, optimization algorithms, etc.; the data and algorithms stored in the database interact through the data and algorithm linkage module, and the data information required by the algorithm in the database is matched and selected in the database to ensure the normal operation of the algorithm.

[0048] Specifically, by analyzing the algorithm information, multiple information of the algorithm is recorded in a table, and the data type, format, time interval and other information required by the algorithm are recorded and saved in the MySQL database to facilitate the subsequent linkage between data and algorithm.

[0049] The algorithm management module is used to store algorithms. This module supports version management, online updates, and dynamic scheduling of algorithms to ensure the flexibility and scalability of the algorithms.

[0050] In the process of algorithm management, algorithm updates are based on business needs and data requirements. First, during product operation, certain functions or algorithms need to be implemented and developed according to the requirements. As for the developed algorithms, due to the continuous changes in requirements, the algorithms are also constantly changing, and the same algorithm will produce different versions. For example, algorithm updates may add or delete functions, and some customers prefer to continue using the functions in the old version, that is, some customers will have the need to continue using the old version.

[0051] Each algorithm will have a basic information table, such as: algorithm name, algorithm version number, algorithm deployment location, algorithm call path, algorithm input sample feature requirements, algorithm function interface call input parameter list, output result storage path and file. Each algorithm corresponds to the algorithm basic information table one by one, and the algorithm information table is saved in the database. A program may involve several algorithms, which only need to be processed during initialization.

[0052] Use Docker to encapsulate different algorithms in the server, and use the algorithm name_version as the naming standard. (soxestimate_10) represents sox estimation version 1.0 and corresponds to the algorithm basic information table.

[0053] Dynamically select and adjust the optimal algorithm according to actual needs: In actual applications, first select the type of task to be performed, such as: sox estimation, soh estimation, etc.; according to the selected task type, the system will list the relevant algorithms and versions; select the algorithm and version you want to execute; after selection, output the basic information of the algorithm according to the algorithm information table in the database, and reselect the algorithms that do not meet the requirements (the detailed information of the algorithm will be presented after the algorithm is selected, and the algorithm that does not meet the requirements can be reselected); after determining the algorithm and version, start the corresponding image in Docker to run the algorithm, and the real-time collected data will automatically modify the sample length and the number of input features according to the feature requirements of the algorithm input sample to make it meet the input requirements of the algorithm; the algorithm runs continuously, and the corresponding algorithm running results are found according to the storage path and file of the algorithm output results.

[0054] How to closely integrate algorithm updates with data and application scenarios: First, develop and update algorithms based on demand; second, automatically modify sample length and the number of input features based on real-time collected data and the feature requirements of algorithm input samples to meet the algorithm input requirements.

[0055] During the transmission, storage and processing of data in the management system, multi-level encryption strategies and access control strategies are adopted to ensure the confidentiality and integrity of the data.

[0056] Furthermore, the encryption strategy includes encryption in data transmission and encryption in data storage. Specifically, encryption in transmission includes (1) using the SSL protocol to encrypt data transmission to ensure that data is not intercepted and tampered with during Modbus-TCP network transmission; (2) using a virtual private network (VPN) to establish a secure communication channel to protect the data upload process and remote access.

[0057] Data storage encryption specifically includes data encryption and database encryption. Data encryption: Algorithmic encryption is performed on the collected data to convert it into other data, and the data is restored according to the algorithm when it is used. Encryption refers to converting each character to its ASCII value, adding the secret key to the ASCII value (other mathematical expressions such as multiplication can be used), taking the result modulo 256, and converting the result back to a character. Decryption refers to converting each encrypted character to its ASCII value, subtracting the secret key from the ASCII value, taking the result modulo 256, and converting the result back to a character. Database encryption: The information in the database is encrypted and can only be accessed within the LAN. VPN access is required outside the LAN.

[0058] The access control strategy includes identity authentication, authority management, and auditing and monitoring.

[0059] Specifically, identity verification: multi-factor verification, using multiple verification methods (company email + accurate personal information) to confirm the user's identity.

[0060] Permission management includes role-based access control and attribute-based access control. Role-based access control is to assign access rights based on the user's role and position, ensuring that each user can only access data and functions within his or her scope of responsibility. Attribute-based access control is to dynamically assign access rights based on user attributes, such as department, position, and work location, to enhance flexibility and fine-grained control.

[0061] Auditing and monitoring: Monitor and filter the transmission of sensitive data to prevent unauthorized data leakage; analyze user behavior and detect abnormal activities such as large data downloads or abnormal access patterns.

[0062] By providing comprehensive data security and privacy protection, advanced encryption technology and strict access control mechanisms are used to prevent data from being intercepted, tampered with, and leaked during transmission, storage, and processing, ensuring the confidentiality and integrity of data, and enhancing user trust and system security.

[0063] like Figure 4 As shown, data and algorithms are linked, and the following is explained using the example of calling a C program:

[0064] Take the example of user A and user B completing different requirements. User A needs to complete the estimation of soc (battery state of charge) and uses the soc_estimate.py program. User B needs to complete the estimation of sox (battery state description) and uses the sox.c program. User A selects the soc estimation algorithm, and the system lists various soc estimation algorithms and their version numbers. User A selects the soc estimation algorithm to be run and its version number, such as: soc_estimate_20.py, and selects the soc estimation version 2.0 algorithm. After selecting the algorithm, the system lists the algorithm information. The user can check the algorithm information to confirm whether it is the algorithm he needs. After confirming that it is correct, he will enter yes to confirm; the algorithm is based on the feature requirements of the input sample in its algorithm information, and the format is as follows:

[0065] {

[0066] "featurelist":[{

[0067] "featurename":"workvolt",

[0068] "FV":"y",

[0069] "DT":"int",

[0070] "RV":1,

[0071] "RU":"V",

[0072] "PV":1,

[0073] "PU":"sec"

[0074] },

[0075] {

[0076] "featurename":"current",

[0077] "FV":"y",

[0078] "DT":"int",

[0079] "RV":1,

[0080] "RU":"A",

[0081]

[0082] Among them, featurename is the input feature name, FV: field valid flag, field valid flag, if it is Y, it means that the field has data, if it is N, it means that the field has no data; DT: Data Type, data type; RV: Resolution Value, precision value; RU: Resolution Unit, precision unit, such as the precision unit of voltage can be millivolt (mV), volt (V), etc.; PV: Period Value, collection period value; PU: Period Unit, collection period unit, such as seconds (sec) and minutes (min).

[0083] The system searches for collected data that meets the requirements based on the feature requirements of the input sample. When collecting data, there will be specific fields to save the information of the collected data, including the name of the collected features, DT, RV, RU, PV, PU, ​​and compare the input sample features with the data information. The ones that meet the requirements are: BMMB1, BMM2, BMM3. The user selects the data source for the algorithm to be executed, such as BMMB1 data, which contains workvol, openvol, current, and temperature features. The system automatically selects the workvol and current features that meet the featurelist requirements and inputs them into the algorithm in real time. Each piece of data outputs a soc estimation result.

[0084] User B selects the sox estimation algorithm. The system lists various sox estimation algorithms and their version numbers. User B selects the sox estimation algorithm and its version number to be run, such as sox_estimate_10.c, and selects the sox estimation version 1.0 algorithm. After selecting the algorithm, the system lists the algorithm information. User B believes that this algorithm cannot meet his needs, so he enters no to reselect another algorithm, such as sox_estimate_20.c. After confirming that it is correct, enter yes to confirm; the algorithm is based on the feature requirements of the input sample in its algorithm information, and the format is as follows:

[0085]

[0086]

[0087]

[0088] The system searches for collected data that meets the requirements based on the feature requirements of the input sample, and the one that meets the requirements is: BMMB1. The user selects BMMB1 as the data source for the execution fee algorithm. The system automatically selects features that meet the featurelist requirements and inputs them into the algorithm in real time. Each piece of data outputs a sox estimation result.

[0089] Each algorithm corresponds to a basic information table, which is stored in the database; the basic information table includes the algorithm name, algorithm version number, algorithm deployment location, algorithm call whale, feature requirements of algorithm input samples, algorithm function interface call input parameter list, output result storage path and file.

[0090] An embodiment of the present invention also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data collection and data and algorithm management system.

[0091] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the data collection and data and algorithm management systems described above.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data collection and data and algorithm management system, It is characterized in that include: Data acquisition module, data processing module, data storage module, algorithm management module and data and algorithm linkage module; The data acquisition module periodically collects data from the device and transmits the collected data to the data processing module; The data processing module receives the data transmitted by the data acquisition module, processes the data, and transmits the processed data to the data storage module for storage; The data storage module is used to store data in a structured or unstructured manner for subsequent retrieval and analysis, and the data storage module uploads the stored data to a database; The algorithm management module is responsible for managing various data analysis algorithms and uploading the algorithms to the algorithm part in the database, wherein the database includes a data part and an algorithm part; the data analysis algorithms include but are not limited to machine learning models, deep learning algorithms, statistical analysis algorithms, and optimization algorithms; The data and algorithms stored in the database interact through the data and algorithm linkage module. The algorithm linkage module matches and selects the algorithms in the database with the required data information to ensure the normal operation of the algorithm.

2. A data collection and data and algorithm management system according to claim 1, Features: The data acquisition module sends a data acquisition signal to the device through the Modbus-TCP protocol and obtains the corresponding data through the Modbus point table.

3. A data collection and data and algorithm management system according to claim 1, Features: Each algorithm corresponds to a basic information table, which is stored in the database; the basic information table includes the algorithm name, algorithm version number, algorithm deployment location, algorithm call path, feature requirements of algorithm input samples, algorithm function interface call input parameter list, output result storage path and storage file.

4. A data collection and data and algorithm management system according to claim 1, Features: The data processing module processes the data, including data cleaning, format conversion, and outlier detection, and converts the data into a format that meets the requirements.

5. A data collection and data and algorithm management system according to claim 1, Features: The database is a MySQL database structure.

6. A data collection and data and algorithm management system according to claim 1, Features: The management system adopts multi-level encryption strategy and access control strategy during the transmission, storage and processing of data to ensure the confidentiality and integrity of the data.

7. A data collection and data and algorithm management system according to claim 6, Features: The encryption strategy includes encryption in data transmission and encryption in data storage; the access control strategy includes identity authentication, authority management, auditing and monitoring.

8. An electronic device, Features: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a data collection and data and algorithm management system according to any one of claims 1 to 7.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements a data collection and data and algorithm management system according to any one of claims 1 to 7.

Images