High-performance real-time market record information auditing and inputting system

Abstract

The invention discloses a high-performance real-time market record information auditing and inputting system. The system comprises a data acquisition module; a data preprocessing module; an auditing rule engine module; the data entering module is used for sending the audited data into the data entering module, and the data entering module is responsible for storing the audited data into a database; the system comprises a user interaction interface and a database, the user interaction interface provides an operation platform for a system user and comprises a data display unit, an auditing result feedback unit and a manual adjustment input data unit, and the database is used for storing all audited market record information. Through the highly automatic and intelligent design, the data processing speed and quality are improved, the data security is enhanced, the user experience is improved, and powerful data support is provided for enterprises so as to adapt to constantly changing market environments and business requirements.

Description

Technical Field

[0001] The present invention belongs to the field of information audit and entry, and more specifically, particularly relates to a high-performance real-time market record information audit and entry system. Background Art

[0002] In the existing field of information review and entry, with the continuous growth of data volume and changes in business needs, the existing system may be difficult to flexibly expand to meet larger-scale data processing needs. Many existing information review and entry systems still rely on manual operations to review and enter data, which may lead to slow processing speed, high error rate, and difficulty in coping with large-scale data streams. At the same time, the existing technology has deficiencies in data encryption, access control and prevention of data leakage, and fails to fully protect user data security and privacy. Therefore, we propose a high-performance real-time market record information review and entry system. Summary of the invention

[0003] The purpose of the present invention is to solve the shortcomings existing in the prior art and propose a high-performance real-time market record information review and entry system. This high-performance real-time market record information review and entry system, through its highly automated and intelligent design, not only improves the speed and quality of data processing, but also enhances data security, improves user experience, and provides enterprises with strong data support to adapt to the ever-changing market environment and business needs.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] High-performance real-time market record information review and entry system, including:

[0006] The data collection module is responsible for collecting market data from various data sources in real time, and the data collection module ensures the integrity and timeliness of the data;

[0007] Data preprocessing module: Before the data is reviewed and entered, the preprocessing module will clean and format the collected raw data, including removing invalid or erroneous data points, standardizing data formats, and calculating indicators. The purpose of preprocessing is to ensure data quality and reduce errors in subsequent processing;

[0008] The audit rule engine module verifies data according to preset rules. The rules include data range check, historical data comparison, and outlier detection. The rule engine needs to be flexible and efficient to quickly process real-time data streams and accurately identify data points that require manual review.

[0009] Data entry module: the audited data will be sent to the data entry module, which is responsible for storing the audited data into the database;

[0010] A user interaction interface and a database. The user interaction interface provides an operating platform for system users, including a data display unit, an audit result feedback unit, and a manual adjustment and input data unit. The database is used to store all audited market record information.

[0011] Preferably, the data acquisition module collects specific processes as follows:

[0012] Define the data sources that need to be collected, determine the data format and collection frequency to ensure that subsequent processes can process the data correctly;

[0013] Capture data from various data sources, taking real-time and timeliness into consideration;

[0014] Verify the loaded data and check whether there are any errors in the data transmission and loading process.

[0015] Preferably, the data preprocessing module preprocesses the collected raw data to remove invalid or erroneous data points, standardize the data format, and ensure data consistency and accuracy; wherein, the invalid or erroneous data points are removed using the following calculation formula:

[0016]

[0017] In the formula, x is the original data, μ is the mean, σ is the standard deviation, and Z describes the distance between a data point and the mean of the entire data set;

[0018] The data preprocessing module converts the data from the business format to the storage format of the data warehouse or database according to the requirements of the target system to maintain the integrity and consistency of the data, specifically:

[0019] Scale the data to the range of [0,1] and convert the data to a distribution with a mean of 0 and a standard deviation of 1. The calculation formula is:

[0020]

[0021] In the formula, x is the original data, x norm Normalized data value, x min is the minimum value of the data, x max is the maximum value of the data.

[0022] Preferably, the audit rule engine module verifies the data according to preset rules, and the specific workflow is as follows:

[0023] Receive real-time data streams from the data collection module, which have been pre-processed to conform to the input format of the audit rule engine;

[0024] The system predefines and loads a series of audit rules, including but not limited to data range checking, historical data comparison, and outlier detection;

[0025] Check whether the data item is within a reasonable value range;

[0026] Compare current data with historical data to check for unusual fluctuations;

[0027] Use statistical methods to identify outliers;

[0028] The data that passes the verification will be entered into the database, while the data that fails the verification will be marked and fed back to the relevant personnel by the system for manual review;

[0029] Data that the system cannot automatically determine will be submitted to manual review, and the reviewer will make the final decision based on contextual information and experience;

[0030] The data confirmed to be correct after review will be officially entered into the database to ensure that the information in the database is accurate and reliable.

[0031] Preferably, the audit rule expression is:

[0032] check_range(d, min, max) = min ≤ d ≤ max;

[0033] In the formula, the purpose of the check_range(d,min,max) function is to ensure that the data point d falls between the specified minimum and maximum values. If the data point d meets the condition, that is, min≤d≤max, the function returns true, otherwise it returns false;

[0034] Preferably, the current data is compared with the historical data, and the expression is:

[0035] compare_with_history(c,h,t)=|ch|>t;

[0036] In the formula, the compare_with_history(c,h,t) function is used to check whether the difference between the current data point c and the corresponding historical data point h exceeds the preset threshold t, c represents the value of the current data point, h represents the value of the historical data point, and t is the maximum allowable difference threshold set by the user; the function's operation is based on calculating the absolute difference between the current data point c and the historical data point h, that is, |ch|. If the difference is greater than the threshold t, that is, |ch|>t, then the function returns true, indicating that the current data point is significantly different from the historical data.

[0037] Preferably, the statistical method is used to identify outliers, and the expression is:

[0038] Lower Limit=μ-n×σ

[0039] Upper Limit = μ + n × σ;

[0040] In the formula, μ is the mean, σ is the standard deviation, and n is 2 or 3, representing data points within different standard deviation ranges. Any data point less than the Lower Limit or greater than the Upper Limit is considered an outlier.

[0041] Preferably, before entering the data into the database, the data entry module will perform a final verification on the data to ensure that the data format and content meet the storage requirements of the database;

[0042] Then, according to the data content, dynamically construct the structured query language insertion instruction, interact with the database management system, execute the structured query language insertion instruction, and write the data into the specified data table;

[0043] After data entry is completed, the module will feedback the data processing results to the upstream module or system.

[0044] Preferably, the data display unit of the user-provided operation platform dynamically updates and displays the real-time market record information that has been reviewed and approved, so that users can instantly understand market dynamics, provide historical data retrieval functions, and users can review past market records by setting query conditions, and display data in the form of charts, such as trend charts, bar charts, pie charts, etc., to help users intuitively understand data changes and associations;

[0045] The audit result feedback unit of the user-provided operating platform instantly displays the data audit results, including the status of passed audit, requiring manual review, or failed audit, thereby improving transparency and providing detailed audit feedback information, reasons for failed audit, specific data points of abnormal values, etc., to facilitate user understanding and taking corresponding measures.

[0046] The high-performance real-time market record information review and entry system provided by the present invention has the following advantages compared with the prior art:

[0047] Improve data processing efficiency: Through automated data collection, preprocessing, review and entry processes, the system can quickly process large amounts of real-time market data, reducing the need for manual operations, thereby improving overall processing speed and efficiency;

[0048] Ensure data quality: The system uses multi-level data inspection and cleaning to ensure that only accurate, consistent and complete data is entered into the database; this helps improve data reliability and accuracy, and provides a high-quality data foundation for subsequent analysis and application;

[0049] Reduce human errors: Automated data processing reduces the need for human intervention, thereby reducing data errors or omissions caused by human operational errors;

[0050] Enhanced data security: Through the final verification of the audit rule engine and data entry module, the system can effectively prevent the entry of invalid or malicious data and protect the database from contamination;

[0051] Improve user experience: Dynamically updated data display units and detailed audit result feedback units provide users with an intuitive and transparent operating platform; users can easily obtain real-time market information, understand data change trends, and conduct manual review or adjustments when necessary;

[0052] Support complex analysis and decision-making: Since data quality and integrity are guaranteed, the information entered into the database can support more complex data analysis and decision-making processes, providing strong data support for the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the architecture diagram of the high-performance real-time market record information review and entry system of the present invention.

[0054] Figure 2 This is a flow chart of the audit rule engine module in an embodiment of the present invention verifying data according to preset rules. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0056] The present invention provides a high-performance real-time market record information review and entry system to solve the problem that traditional manual or semi-automatic data processing methods cannot meet such high-speed requirements, resulting in decision delays. The system significantly improves data processing speed and efficiency through automated data collection, preprocessing, review and entry processes, while reducing the direct impact of erroneous or low-quality data on analysis results and decision accuracy. The system ensures data accuracy and consistency through multi-level data inspection and cleaning, thereby ensuring data quality.

[0057] like Figure 1 As shown, the system includes:

[0058] The data collection module is responsible for collecting market data from various data sources in real time, and the data collection module ensures the integrity and timeliness of the data; the specific collection process of the data collection module is as follows:

[0059] Define the data sources that need to be collected, determine the data format and collection frequency to ensure that subsequent processes can process the data correctly;

[0060] Capture data from various data sources, taking real-time and timeliness into consideration;

[0061] Verify the loaded data and check whether there are any errors in the data transmission and loading process.

[0062] Data preprocessing module: Before the data is reviewed and entered, the preprocessing module will clean and format the collected raw data, including removing invalid or erroneous data points, standardizing data formats, and calculating indicators. The purpose of preprocessing is to ensure data quality and reduce errors in subsequent processing; the data preprocessing module preprocesses the collected raw data, removes invalid or erroneous data points, standardizes data formats, and ensures data consistency and accuracy; the following calculation formula is used to remove invalid or erroneous data points:

[0063]

[0064] In the formula, x is the original data, μ is the mean, σ is the standard deviation, and Z describes the distance between a data point and the mean of the entire data set;

[0065] The data preprocessing module converts data from the business format to the storage format of the data warehouse or database according to the requirements of the target system to maintain the integrity and consistency of the data. Specifically:

[0066] Scale the data to the range of [0,1] and convert the data to a distribution with a mean of 0 and a standard deviation of 1. The calculation formula is:

[0067]

[0068] In the formula, x is the original data, x norm Normalized data value, x min is the minimum value of the data, x max is the maximum value of the data.

[0069] The audit rule engine module verifies the data according to preset rules. The rules include data range check, historical data comparison, and outlier detection. The rule engine needs to be flexible and efficient to quickly process real-time data streams and accurately identify data points that require manual review. The audit rule engine module verifies the data according to preset rules, such as Figure 2 , the specific workflow is:

[0070] 1) Receive real-time data streams from the data acquisition module. These data have been pre-processed to meet the input format of the audit rule engine;

[0071] 2) The system predefines and loads a series of audit rules, including but not limited to data range checking, historical data comparison, and outlier detection;

[0072] 3) Check whether the data item is within a reasonable value range;

[0073] 4) Compare current data with historical data to check for abnormal fluctuations;

[0074] 5) Use statistical methods to identify outliers;

[0075] 6) Data that passes the verification will be entered into the database, while data that fails the verification will be marked and fed back to relevant personnel by the system for manual review;

[0076] 7) Data that the system cannot automatically determine will be submitted to manual review, and the reviewer will make the final decision based on contextual information and experience;

[0077] 8) Data that has been verified to be correct will be formally entered into the database to ensure that the information in the database is accurate and reliable.

[0078] Furthermore, the audit rule expression is:

[0079] check_range(d, min, max) = min ≤ d ≤ max;

[0080] In the formula, the purpose of the check_range(d,min,max) function is to ensure that the data point d falls between the specified minimum and maximum values. If the data point d meets the condition, that is, min≤d≤max, the function returns true, otherwise it returns false;

[0081] Compare the current data with the historical data, the expression is:

[0082] compare_with_history(c,h,t)=|ch|>t;

[0083] Wherein, compare_with_history(c,h,t) function is used to check whether the difference between the current data point c and the corresponding historical data point h exceeds the preset threshold t, c represents the value of the current data point, h represents the value of the historical data point, and t is the maximum difference threshold allowed set by the user; the operation of the function is based on calculating the absolute difference between the current data point c and the historical data point h, that is, |ch|. If the difference is greater than the threshold t, that is, |ch|>t, then the function returns true, indicating that the current data point is significantly different from the historical data;

[0084] Using statistical methods to identify outliers, the expression is:

[0085] Lower Limit=μ-n×σ

[0086] Upper Limit = μ + n × σ;

[0087] In the formula, μ is the mean, σ is the standard deviation, and n is 2 or 3, representing data points within different standard deviation ranges. Any data point less than the Lower Limit or greater than the Upper Limit is considered an outlier.

[0088] Data entry module: the audited data will be sent to the data entry module, which is responsible for storing the audited data in the database. Before entering the data into the database, the data entry module will conduct a final verification of the data to ensure that the data format and content meet the storage requirements of the database.

[0089] Then, according to the data content, dynamically construct the structured query language insertion instruction, interact with the database management system, execute the structured query language insertion instruction, and write the data into the specified data table;

[0090] After data entry is completed, the module will feedback the data processing results to the upstream module or system.

[0091] User interaction interface and database. The user interaction interface provides an operation platform for system users, including data display unit, audit result feedback unit, and manual adjustment and input data unit. The database is used to store all audited market record information;

[0092] The data display unit of the user-provided operation platform dynamically updates and displays the real-time market record information that has been reviewed and approved, so that users can instantly understand market dynamics. It also provides historical data retrieval functions, and users can review past market records by setting query conditions. The data is displayed in the form of charts, such as trend charts, bar charts, pie charts, etc., to help users intuitively understand data changes and associations;

[0093] The audit result feedback unit of the user-provided operating platform instantly displays the data audit results, including the status of passed audit, requiring manual review, or failed audit, improves transparency, and provides detailed audit feedback information, reasons for failed audit, specific data points of abnormal values, etc., to facilitate users to understand and take corresponding measures.

[0094] It should be noted that in order to meet the needs of large-scale data processing, the present invention is specifically implemented as follows:

[0095] Distributed data collection: By deploying multiple data collection nodes, each node is responsible for collecting data from a specific data source. These nodes can be geographically dispersed to reduce data transmission delays and improve the efficiency and reliability of data collection.

[0096] Distributed data preprocessing: Data preprocessing tasks can be distributed on multiple computing nodes and executed in parallel. Each node processes a portion of the data and then sends the processed data to the next processing stage. This approach can significantly increase the data processing speed and shorten the overall processing time.

[0097] Distributed audit rule engine: The audit rule engine can run on multiple nodes, with each node responsible for processing a portion of the data flow. This distributed processing can improve the speed and fault tolerance of the audit process, ensuring that the system can continue to operate even when some nodes fail.

[0098] Distributed data storage: A distributed database system is used to store data in multiple nodes, which not only increases the data storage capacity, but also improves the speed of data reading and writing and the scalability of the system.

[0099] Load balancing and failover: By using a load balancer and failover mechanism, the system can automatically distribute requests to different nodes and reroute requests to healthy nodes when a node fails, ensuring high availability and stability of the system.

[0100] Distributed cache: In order to reduce the number of database accesses and response time, a distributed cache layer can be introduced into the system. Common cache solutions include redis and memcached. The cache can store frequently accessed data, thereby speeding up the reading of data.

[0101] Monitoring and log management: In a distributed environment, monitoring and managing logs becomes particularly important. Use a centralized log management system to collect and analyze log information from different nodes to detect and solve problems in a timely manner.

[0102] Through the above design, the real-time market record information review and entry system of the present invention can not only process larger-scale data, but also provide higher performance and stronger fault tolerance, meeting the industry's high requirements for data processing.

[0103] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, a technician in this field can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. High-performance real-time market record information review and entry system, characterized by: include: The data collection module is responsible for collecting market data from various data sources in real time, and the data collection module ensures the integrity and timeliness of the data; Data preprocessing module: Before the data is reviewed and entered, the preprocessing module will clean and format the collected raw data, including removing invalid or erroneous data points, standardizing data formats, and calculating indicators. The purpose of preprocessing is to ensure data quality and reduce errors in subsequent processing; The audit rule engine module verifies data according to preset rules. The rules include data range check, historical data comparison, and outlier detection. The rule engine needs to be flexible and efficient to quickly process real-time data streams and accurately identify data points that require manual review. Data entry module: the audited data will be sent to the data entry module, which is responsible for storing the audited data into the database; A user interaction interface and a database. The user interaction interface provides an operating platform for system users, including a data display unit, an audit result feedback unit, and a manual adjustment and input data unit. The database is used to store all audited market record information.

2. The high-performance real-time market record information review and entry system according to claim 1 is characterized in that: The specific process of data acquisition module is as follows: Define the data sources that need to be collected, determine the data format and collection frequency to ensure that subsequent processes can process the data correctly; Capture data from various data sources, taking real-time and timeliness into consideration; Verify the loaded data and check whether there are any errors in the data transmission and loading process.

3. The high-performance real-time market record information review and entry system according to claim 1 is characterized in that: The data preprocessing module preprocesses the collected raw data, removes invalid or erroneous data points, standardizes the data format, and ensures data consistency and accuracy; wherein, the following calculation formula is used to remove invalid or erroneous data points: In the formula, x is the original data, μ is the mean, σ is the standard deviation, and Z describes the distance between a data point and the mean of the entire data set; The data preprocessing module converts the data from the business format to the storage format of the data warehouse or database according to the requirements of the target system to maintain the integrity and consistency of the data, specifically: Scale the data to the range of [0,1] and convert the data to a distribution with a mean of 0 and a standard deviation of 1. The calculation formula is: In the formula, x is the original data, x norm Normalized data value, x min is the minimum value of the data, x max is the maximum value of the data.

4. The high-performance real-time market record information review and entry system according to claim 1, characterized in that: The audit rule engine module verifies the data according to the preset rules. The specific workflow is as follows: Receive real-time data streams from the data collection module, which have been pre-processed to conform to the input format of the audit rule engine; The system predefines and loads a series of audit rules, including but not limited to data range checking, historical data comparison, and outlier detection; Check whether the data item is within a reasonable value range; Compare current data with historical data to check for unusual fluctuations; Use statistical methods to identify outliers; The data that passes the verification will be entered into the database, while the data that fails the verification will be marked and fed back to the relevant personnel by the system for manual review; Data that the system cannot automatically determine will be submitted to manual review, and the reviewer will make the final decision based on contextual information and experience; The data confirmed to be correct after review will be officially entered into the database to ensure that the information in the database is accurate and reliable.

5. The high-performance real-time market record information review and entry system according to claim 4 is characterized in that: The audit rule expression is: check_range(d, min, max) = min ≤ d ≤ max; Where, the purpose of the check_range(d, min, max) function is to ensure that the data point d falls between the specified minimum and maximum values. If the data point d satisfies the condition, that is, min≤d≤max, the function returns true, otherwise it returns false.

6. The high-performance real-time market record information review and entry system according to claim 5 is characterized in that: The current data is compared with the historical data, and the expression is: compare_with_history(c,h,t)=|ch|>t; Wherein, compare_with_history(c, h, t) function is used to check whether the difference between the current data point c and the corresponding historical data point h exceeds the preset threshold t, c represents the value of the current data point, h represents the value of the historical data point, and t is the maximum allowable difference threshold set by the user; the operation of the function is based on calculating the absolute difference between the current data point c and the historical data point h, that is, |ch|. If the difference is greater than the threshold t, that is, |ch|>t, then the function returns true, indicating that the current data point is significantly different from the historical data.

7. The high-performance real-time market record information review and entry system according to claim 6 is characterized in that: The statistical method is used to identify abnormal values, and the expression is: Lower Limit=μn×σ Upper Limit = μ + n × σ; In the formula, μ is the mean, σ is the standard deviation, and n is 2 or 3, representing data points within different standard deviation ranges. Any data point less than the Lower Limit or greater than the Upper Limit is considered an outlier.

8. The high-performance real-time market record information review and entry system according to claim 1 is characterized in that: Before entering the data into the database, the data entry module will perform a final verification on the data to ensure that the data format and content meet the storage requirements of the database; Then, according to the data content, dynamically construct the structured query language insertion instruction, interact with the database management system, execute the structured query language insertion instruction, and write the data into the specified data table; After data entry is completed, the module will feedback the data processing results to the upstream module or system.

9. The high-performance real-time market record information review and entry system according to claim 1 is characterized in that: The data display unit of the user-provided operation platform dynamically updates and displays the real-time market record information that has been reviewed and approved, so that users can instantly understand market dynamics, provide historical data retrieval functions, and users can review past market records by setting query conditions, and display data in the form of charts, such as trend charts, bar charts, pie charts, etc., to help users intuitively understand data changes and associations; The audit result feedback unit of the user-provided operating platform instantly displays the data audit results, including the status of passed audit, requiring manual review, or failed audit, thereby improving transparency and providing detailed audit feedback information, reasons for failed audit, specific data points of abnormal values, etc., to facilitate user understanding and taking corresponding measures.

Images