On-demand precise real-time data collection and computation method

Abstract

The present application relates to the technical field of data collection and real-time calculation, and in particular to a method for on-demand accurate real-time data collection and calculation. The method comprises the following steps: through multi-dimensional on-demand collection request analysis and dynamic priority scheduling, the present application performs adaptive data collection and incremental transmission optimization on enterprise data, realizes real-time data processing and end-to-end consistency guarantee, and performs multi-dimensional data readiness confirmation and intelligent calculation triggering, so as to reduce data collection cost, improve data scheduling flexibility and calculation efficiency.

Description

Technical Field

[0001] This invention relates to the field of data acquisition and real-time computing technology, specifically to a method for on-demand, precise, and real-time data acquisition and computing. Background Technology

[0002] With the rapid development of internet technology, enterprises are increasingly reliant on external commercial data in their business decision-making processes. However, existing data collection and computing technologies suffer from several pain points: First, data collection often adopts a full-volume collection model without being customized to meet the actual data needs of enterprises, resulting in high data procurement costs and invalid data consuming significant storage and computing resources. Second, the lack of a dynamic priority mechanism in task scheduling leads to competition for resources between batch collection tasks and real-time query tasks, causing delays in business responses to time-sensitive needs. Third, consistency during data transmission and processing is difficult to guarantee, easily resulting in data duplication, loss, or distortion, affecting data availability. Fourth, the lack of a refined confirmation mechanism for data readiness status makes it impossible to accurately determine data integrity, leading to deviations in subsequent calculation results. Fifth, indicator calculations do not incorporate predictions based on data readiness status, resulting in a large amount of invalid calculations and wasted computing resources.

[0003] In existing technologies, data acquisition scheduling often employs fixed priority strategies, making it impossible to dynamically adjust based on task waiting time, business importance, etc.; data consistency assurance relies on complex distributed transactions, leading to system performance degradation; and data readiness confirmation uses a single judgment rule, making it difficult to adapt to data sources with different quality characteristics. These problems make it difficult for existing technologies to meet enterprises' needs for low-cost, accurate, and real-time data services. Therefore, there is an urgent need for an on-demand, accurate, and real-time data acquisition and computation method to overcome these technical bottlenecks. Summary of the Invention

[0004] The purpose of this invention is to provide an on-demand, accurate, real-time data acquisition and calculation method to solve the problems of high data acquisition cost, inflexible scheduling, poor consistency, inaccurate readiness confirmation, and low calculation efficiency in the prior art.

[0005] A method for on-demand, precise, and real-time data collection and computation, comprising the following steps: Step 1: Multi-dimensional on-demand data collection request parsing and dynamic priority scheduling: S11. Receive on-demand data collection requests from business systems, verify the legality of the requests and resource availability, and extract enterprise identifiers, data dimension sets, real-time requirement levels, and calculation accuracy thresholds. S12. Based on business weight, data timeliness, data dimension importance, request waiting time, and business system load status, the request priority is determined through a non-linear priority calculation model. S13. Deliver requests to multi-level scheduling queues according to priority, and dynamically adjust resource allocation ratios based on queue congestion coefficients. Step 2: Adaptive data acquisition and incremental transmission optimization; Step 3: Real-time data processing and end-to-end consistency assurance; Step 4: Confirmation of Multi-Dimensional Data Readiness and Triggering of Intelligent Computation: S41. Dimensional Readiness Assessment: Construct an enterprise dimensional readiness judgment model, and quantify the degree of readiness by combining data integrity, accuracy and timeliness indicators; S42. Dynamic generation of calculation tasks: Based on the enterprise's readiness status and business calculation rules, automatically generate basic indicator calculation tasks and customized complex indicator calculation tasks.

[0006] Furthermore, calculating the importance of data dimensions specifically includes the following process: determining the importance of each dimension of the data based on the analytic hierarchy process (AHP), and summing the importance of each dimension to obtain the data dimension importance.

[0007] Furthermore, in step S12, the formula for the nonlinear priority calculation model is: ; in, Prioritize requests for on-demand data collection. for Normalization function, , , , and The weighting coefficients are satisfied. , This is the business weight, and its value range is... , For data timeliness, For the importance of data dimensions, For Lambert function, For the request waiting time, For the load sensitivity coefficient of the business system, The current load rate of the business system. For load threshold, This represents the highest priority value.

[0008] Furthermore, in step two, the adaptive data acquisition and incremental transmission optimization specifically includes the following processes: S21. Data source access and collection strategy adaptation: Select the corresponding data source interface according to the characteristics of the data dimension, and dynamically adjust the collection concurrency, page size and retry strategy. S22, Incremental Acquisition and Intelligent Deduplication: Based on the data update timestamp, incremental acquisition rules are constructed, and duplicate requests are filtered through a sliding time window deduplication algorithm; S23. Data Acquisition Buffer and Metadata Enhancement: Store the acquired data in a distributed buffer and supplement it with the acquisition batch number, data integrity identifier, and data source confidence metadata.

[0009] Furthermore, in step three, real-time data processing and end-to-end consistency assurance specifically include the following processes: S31. Heterogeneous data standardization processing: Adaptive parsing engine is used to process multi-format data, and data cleaning, format conversion and anomaly repair are completed through custom UDF functions; S32. Streaming optimization based on Flink: The two-phase commit mechanism ensures exactly-Once semantics of data transmission, and the dynamic threshold adjustment of batch processing achieves efficient data writing. S33. Data consistency verification and conflict resolution: Data conflicts are detected based on version vector and timestamp mechanisms, and eventual data consistency is achieved through conflict resolution algorithms.

[0010] Furthermore, in step S22, the deduplication determination formula of the sliding time window deduplication algorithm is: ; ; in, Representing data For duplicate data, Sliding time window Data within, window size ,in, The average data update cycle, To adjust the coefficient, Data similarity is calculated by combining cosine similarity with dimensional feature hashing: ; Let j be the hash function for the j-th dimension of the data. For similarity threshold, For data Collect timestamps, For data Collect timestamps, This represents the time tolerance threshold.

[0011] Furthermore, in step S32, the formula for adjusting the batch dynamic threshold is: ; ; ; in, , and These are the thresholds for the number of data output records, the data output time interval, and the data packet size. , and These are the basic thresholds for the number of data output records, the basic threshold for the data output time interval, and the basic threshold for the data output packet size. This represents the current Kafka queue backlog. This is the normal backlog threshold. The maximum tolerable backlog, Doris database load rate This is the maximum load capacity for the Doris database. , and These are the proportionality coefficients.

[0012] Furthermore, in step S41, the calculation formula for the enterprise-level readiness determination model is as follows: ; in, For readiness level, It is determined to be ready. For integrity weight, This represents the actual number of data entries collected. For the expected number of data entries, For accuracy weighting, For actual data accuracy, The standard accuracy threshold, For the current time, This represents the expected arrival time of the data.

[0013] Compared to existing solutions, the beneficial effects achieved by this invention are: On-demand data collection and dynamic scheduling reduce costs: By receiving on-demand data collection requests from enterprises, and combining a multi-dimensional priority model and a time-weighted scheduling mechanism, accurate data collection and optimized resource allocation are achieved, avoiding the cost waste caused by full data collection, while ensuring the response speed of high-time-efficiency requests; the intelligent deduplication mechanism further reduces duplicate data collection, thereby reducing system resource consumption and data procurement costs.

[0014] Multi-level consistency guarantees improve data quality: Flink's two-phase commit mechanism ensures data transmission consistency, transaction batching combined with the DorisUnique model enables accurate one-time delivery of data output, and differentiated rules for data readiness confirmation ensure data integrity. Multi-dimensional collaboration ensures strong consistency and accuracy of data from collection to output.

[0015] Refined readiness verification and prediction optimizes computational efficiency: A two-tiered readiness verification mechanism is established at the enterprise level and enterprise level, and data availability is accurately identified by combining differentiated judgment rules; computability prediction before indicator calculation avoids invalid calculations, and the hierarchical calculation mode takes into account both indicator uniformity and customization needs, significantly improving computational efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0017] Figure 1 This is a flowchart illustrating the workflow of the first on-demand, precise, real-time data acquisition and calculation method according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the second on-demand, precise, real-time data acquisition and calculation method according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the third on-demand, precise, real-time data acquisition and calculation method according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0020] This embodiment provides a method for on-demand, precise, and real-time data acquisition and calculation. Figure 1 This is a flowchart illustrating the first on-demand, precise, real-time data acquisition and calculation method according to an embodiment of the present invention, as follows: Figure 1 As shown, the method includes the following steps: Step 1: Multi-dimensional on-demand collection request parsing and dynamic priority scheduling; S11. Receive on-demand data collection requests from business systems, verify the legality of the requests and resource availability, and extract enterprise identifiers, data dimension sets, real-time requirement levels, and calculation accuracy thresholds. Specifically, the data collection request verification and parameter extraction process involves receiving on-demand data collection requests initiated by business systems. These requests include core parameters such as the enterprise's unique identifier, the required data dimension set (e.g., business registration information, legal proceedings, operational data), the real-time requirement level (high / medium / low), and the calculation accuracy threshold. The system executes a multi-layered verification process: identity authentication (verifying business system access permissions based on the OAuth 2.0 protocol), enterprise account legitimacy (verifying enterprise registration status and data usage permissions), resource availability verification (checking whether the current system's remaining data collection quota and concurrent task count are within the threshold range), and request format validity verification. Upon successful verification, key parameters are extracted and stored in the request parameter library to provide data support for subsequent priority calculations; if verification fails, a structured error message is returned, clearly indicating the reason for failure.

[0021] S12. Based on business weight, data timeliness, data dimension importance, request waiting time, and business system load status, the request priority is determined through a non-linear priority calculation model. S13. Deliver requests to multi-level scheduling queues according to priority, and dynamically adjust resource allocation ratios based on queue congestion coefficients. Step 2: Adaptive data acquisition and incremental transmission optimization; Step 3: Real-time data processing and end-to-end consistency assurance; Step 4: Confirmation of Multi-Dimensional Data Readiness and Triggering of Intelligent Computation: S41. Dimensional Readiness Assessment: Construct an enterprise dimensional readiness judgment model, and quantify the degree of readiness by combining data integrity, accuracy and timeliness indicators; S42. Dynamic generation of calculation tasks: Based on the enterprise's readiness status and business calculation rules, automatically generate basic indicator calculation tasks and customized complex indicator calculation tasks. S43. Stratified index calculation and result optimization: The index calculation is performed using a micro-batch calculation mode, and the data accuracy is improved through the confidence evaluation of the calculation results and the error correction mechanism.

[0022] In summary, this invention optimizes enterprise data collection and incremental transmission through multi-dimensional on-demand request parsing and dynamic priority scheduling, achieves real-time data processing and end-to-end consistency assurance, and performs multi-dimensional data readiness confirmation and intelligent calculation triggering. This reduces data collection costs, improves data scheduling flexibility, and enhances computational efficiency.

[0023] In some embodiments, step S12, calculating the importance of data dimensions specifically includes the following process: determining the importance of each dimension of the data based on the analytic hierarchy process (AHP), and summing the importance of each dimension to obtain the data dimension importance: Step 1: Construct a hierarchical model: Establish a three-tiered structure of "goal-criteria-solution," clearly defining the core elements of each level to ensure clear analytical logic: Target layer: Quantify the importance of each dimension of the data, and finally form the sum of the importance of the data dimensions, which provides the core basis for subsequent priority calculation; Criteria Layer: Extracts core evaluation dimensions from on-demand data collection requests from business systems; number of dimensions... ≥2, and each dimension corresponds to a specific data requirement item in the request. For example, if the data collection request includes "user basic information, transaction flow data, device operating status, and marketing activity data", then the criteria layer is... (User basic information dimension) (Transaction flow data dimension) (Equipment operating status dimension) (Marketing campaign data dimensions); Scheme layer: Each specific data dimension whose importance is to be evaluated (i.e., each data dimension listed in the criteria layer), with each dimension participating in the importance ranking as an independent scheme.

[0024] Criterion layer extraction rules The system uses natural language processing technology to analyze the data dimension descriptions in the collection request and identify key fields (such as "user ID", "transaction amount", "device temperature" etc.). Perform semantic classification on key fields and merge synonymous or highly related dimensions (such as "user's mobile phone number" and "user's contact information") to avoid redundancy in criteria; By considering the frequency of occurrence of reference data dimensions in requests and their dependence on business scenarios (such as prioritizing core business data dimensions), we can initially identify potential importance trends and provide a reference for the subsequent construction of the judgment matrix.

[0025] Step Two: Constructing the Judgment Matrix Using the 1-9 scaling method, based on the priority of business needs and the value of data application, constructing the criterion layer to the target layer. Order judgment matrix ,matrix Middle elements Indicator level The dimension relative to the first The importance scale has several dimensions; among them... The number of criteria at the criterion level; Scale definition: 1 (both elements are equally important), 3 (the former is slightly more important than the latter), 5 (the former is significantly more important than the latter), 7 (the former is strongly more important than the latter), 9 (the former is extremely more important than the latter), 2, 4, 6, and 8 are the median values ​​of adjacent scales; if the former is not more important than the latter, take the reciprocal of the corresponding scale.

[0026] Comparison criteria: Based entirely on business scenario requirements and data application priorities. If the data collection request explicitly states that "transaction-related data takes precedence over log data", then in the criteria layer, the "transaction flow data dimension" is more important than the "system log data dimension". Combining the requirements for real-time data and computational accuracy, for example, "equipment fault alarm data" with extremely high real-time requirements is more important than "historical data statistical dimensions" which are not real-time. The greater the impact of the reference data on the business weight (B) and the accuracy of the calculation results, the more important the dimension.

[0027] Step 3, Consistency Check: Perform the following checks on the judgment matrix: Calculate the largest eigenvalue of the judgment matrix : ; in, It is a matrix obtained by normalizing each column vector of the judgment matrix. The value is 1, 2... , For matrix The elements are added row by row to obtain a vector, which is then normalized into a matrix. For matrix The elements are added column by column to obtain a vector, which is then normalized into a matrix. According to the formula Calculate the consistency index ,in To determine the order of a matrix; Find the AHP standard average random consistency index and calculate the consistency ratio CR using the formula CR=CI / RI. If CR<0.1, the matrix consistency passes; if CR≥0.1, adjust the scale of the corresponding judgment matrix and repeat steps two and three until the test passes; where the matrix order is... For orders 1, 2, 3, 4, 5, 6, 7, and 8, the values ​​of RI are 0, 0, 0.58, 0.90, 1.12, 1.24, 1.32, and 1.41, respectively.

[0028] Step 4: Weight Calculation and Accumulation of Data Dimension Importance. The importance weight of each data dimension is calculated by summing the weights of the judgment matrix that has passed the consistency check. The importance weights of each data dimension are calculated using the geometric mean method: Step 1: Calculate the judgment matrix The product of elements in each row ; Step 2: Calculate the product of each row. of Root : The importance of each dimension of the data is obtained, and the importance of each dimension is summed to obtain the data dimension importance.

[0029] In some embodiments, in step S12, the formula for the nonlinear priority calculation model is: ; in, Prioritize requests for on-demand data collection. for Normalization function, , , , and The weighting coefficients are satisfied. In the above calculation formula, the business weight, data dimension importance, timeliness requirement coefficient, current load rate of the business system, and request waiting time are all calculated after parameter normalization preprocessing. Specifically: data dimension importance By using a normalization method, Mapped to Interval: The business weights are predefined by the enterprise: core businesses have a business weight of 1.0, important businesses have a business weight of 0.7, and ordinary businesses have a business weight of 0.3; their value range is... , This is the timeliness demand coefficient, and its value range is... The timeliness requirement coefficient for high real-time demand is 1.0, for medium real-time demand it is 0.6, and for low real-time demand it is 0.3. For the importance of data dimensions, For Lambert This function describes the non-linear increase in priority due to waiting time; the longer the waiting time, the greater the increase in priority. The request wait time, where, After normalization using logarithmic scaling: ,in, This is the maximum allowed waiting time by the system, such as 3600 seconds. This is the load sensitivity factor for the business system, with a default value of 2.0. The Sigmoid function takes this parameter as an example. When the value is 2.0, it can achieve both "rapid suppression after the load exceeds the threshold" and avoid "slight load fluctuations causing sudden changes in priority". The current load rate of the business system (the current average load rate of CPU and memory in the system) will be... Mapped to interval, This is the load threshold, with a default value of 0.7. The higher the system load, the smaller the contribution of this factor to priority. This represents the maximum priority value; the default value is 100. It's worth noting that... , , , and These are the weighting coefficients, with default values ​​of , . , , , , .

[0030] Furthermore, in step S13, requests are delivered to multi-level scheduling queues based on priority, and the resource allocation ratio is dynamically adjusted based on the queue congestion coefficient. Hierarchical queue scheduling and resource adaptation: Requests are divided into high-priority queues based on priority. ), medium priority queue ( ) and low-priority queues ( A tiered resource allocation strategy is adopted, with high-priority queues occupying 50% of system collection resources by default, medium-priority queues occupying 30%, and low-priority queues occupying 20%. A queue congestion coefficient is introduced. ( The current number of tasks in the queue. (where the queue's maximum capacity is specified), when a certain queue... The system dynamically preempts 10%-20% of the resources in the low-priority queue; when the high-priority queue is empty, the low-priority queue can occupy its idle resources. Simultaneously, a preemptive scheduling mechanism is employed, allowing newly arriving high-priority tasks to interrupt currently executing low-priority tasks, ensuring real-time response to high-value requests.

[0031] In some embodiments, Figure 2 This is a flowchart illustrating the second on-demand, precise, real-time data acquisition and calculation method according to an embodiment of the present invention. Figure 2 As shown, step two, adaptive data acquisition and incremental transmission optimization, specifically includes the following processes: S21. Data source access and collection strategy adaptation: Select the corresponding data source interface according to the characteristics of the data dimension, and dynamically adjust the collection concurrency, page size and retry strategy. Specifically, select corresponding data source interfaces based on data dimension characteristics (such as data volume, update frequency, and interface performance) and build a data source interface adaptation pool. Dynamically adjust collection strategy parameters and collection concurrency. Calculated using the following formula: ;in, This represents the maximum concurrency level, with a default value of 50. Load rate of the data source interface (calculated in real time through interface response time). To collect the length of the task queue. This is the queue threshold, which defaults to 100. For the first The data volume in each dimension, where m is the total number of data dimensions. To improve the response speed of the data source interface, the page size is dynamically adjusted based on the average record size of the data dimension, with a default of 100-1000 records per page; the retry strategy adopts an exponential backoff mechanism, with a retry interval set to 1 second.

[0032] S22, Incremental Acquisition and Intelligent Deduplication: Based on the data update timestamp, incremental acquisition rules are constructed, and duplicate requests are filtered through a sliding time window deduplication algorithm; Specifically, the deduplication determination formula of the sliding time window deduplication algorithm is as follows: ; ; in, Representing data For duplicate data, Sliding time window Data within, window size ,in, The average data update cycle, To adjust the coefficient, Setting it to 1.5 allows the window size to complement the time tolerance threshold. Data similarity is calculated by combining cosine similarity with dimensional feature hashing: ; Let m be the hash function for the j-th dimension of the data. The hash function can be MD5, and m is the total number of dimensions in the data. This is the similarity threshold, with a default value of 0.9. For data Collect timestamps, For data Collect timestamps, The time tolerance threshold is set to 300 seconds to ensure the timeliness of data response.

[0033] S23. Data Acquisition Buffer and Metadata Enhancement: Store the acquired data in a distributed buffer and supplement it with the acquisition batch number, data integrity identifier, and data source confidence metadata.

[0034] Specifically, successfully collected data is stored in a distributed NoSQL buffer (such as a MongoDB cluster), employing a sharded storage strategy to improve read and write performance. Simultaneously, multi-dimensional metadata is added: collection batch number (formatted as "Enterprise ID-Dimension Set Hash-Timestamp"), data integrity flag (where a value of 1 indicates complete data, 0 indicates partially missing data, and -1 indicates no data); and data source confidence. (Calculated based on historical interface response success rate and data accuracy) , To improve the success rate, This includes information such as data accuracy and collection time. Metadata is stored in association with the raw data, providing support for subsequent data processing and readiness verification.

[0035] In some embodiments, Figure 3 This is a flowchart illustrating the third on-demand, precise, real-time data acquisition and calculation method according to an embodiment of the present invention. Figure 3 As shown, real-time data processing and end-to-end consistency assurance specifically include the following processes: S31. Heterogeneous data standardization processing: Adaptive parsing engine is used to process multi-format data, and data cleaning, format conversion and anomaly repair are completed through custom UDF functions; Specifically, heterogeneous data standardization is achieved through an adaptive parsing engine that supports parsing various data formats such as JSON, XML, and CSV, automatically identifying field types and structures. Data standardization is implemented via a custom UDF function library, including default value filling (filling missing non-critical fields with default values ​​based on business rules), error field correction (validating field value ranges based on a data dictionary and correcting logically erroneous data), format conversion (unifying date and numeric formats from different data sources to a standard format), and complex field parsing (parsing complex structures such as nested JSON and arrays to extract key information). The UDF function library supports dynamic expansion, allowing the addition of custom functions based on new data formats and processing requirements.

[0036] S32. Streaming optimization based on Flink: The two-phase commit mechanism ensures exactly-Once semantics of data transmission, and the dynamic threshold adjustment of batch processing achieves efficient data writing. Specifically, Flink-based stream processing optimizations include: building a stream processing pipeline based on the Flink real-time computing framework and employing CDC (Change Data Capture) technology to capture data change logs in the buffer in real time. Data transmission uses a two-phase commit mechanism to ensure end-to-end exactly-Once semantics: in the first phase, the Flink job writes data to Kafka and pre-commits the transaction; in the second phase, after a successful checkpoint, the transaction is officially committed, ensuring no data duplication or loss. When writing data to the Doris database, dynamic threshold batching is used, with the batching threshold dynamically adjusted using the following formula: ; ; ; in, , and These are the thresholds for the number of data output records, the data output time interval, and the data packet size. , and These are the basic thresholds for the number of data output records, the basic threshold for the data output time interval, and the basic threshold for the data packet size. As an optional embodiment, the basic thresholds can be set to 500 records, 30 seconds, and 20MB, respectively. This represents the current Kafka queue backlog. The normal backlog threshold is set to 10000. The maximum tolerable backlog is set at 50,000. Doris database load rate This represents the maximum load capacity for the Doris database. Considering this load capacity, setting it to 0.8 ensures the system can operate normally. , and These are the proportional coefficients. Considering that the number of data output records has a greater impact on data writing than the data output time interval, and the data output time interval is greater than the data packet size, therefore... , and The values ​​are 0.4, 0.3, and 0.2 respectively.

[0037] S33. Data consistency verification and conflict resolution: Data conflicts are detected based on version vector and timestamp mechanisms, and eventual data consistency is achieved through conflict resolution algorithms.

[0038] Specifically, based on version vectors and timestamp mechanisms, data conflicts are detected. When multiple updated versions exist in the same dimension for the same enterprise, eventual data consistency is achieved through conflict resolution algorithms: the version data corresponding to the latest timestamp is taken as the final updated version data.

[0039] In some embodiments, multi-dimensional data readiness confirmation and intelligent computing triggering: S41. Dimensional Readiness Assessment: Construct an enterprise dimensional readiness judgment model, and quantify the degree of readiness by combining data integrity, accuracy and timeliness indicators; The calculation formula for the enterprise-level readiness determination model is as follows: ; in, For readiness level, It is determined to be ready. As the integrity weight, taking into account the importance of integrity to the readiness determination, Set to 0.5. This represents the actual number of data entries collected. For the expected number of data entries, For accuracy, the weighting is 0.5 for both accuracy and completeness. For actual data accuracy, Setting the standard accuracy threshold to 0.95 can improve data accuracy. For the current time, The value of e is 2.72, representing the expected arrival time of the data.

[0040] S42. Dynamic generation of calculation tasks: Based on the enterprise's readiness status and business calculation rules, automatically generate basic indicator calculation tasks and customized complex indicator calculation tasks. Specifically, based on the enterprise's data readiness status and preset business calculation rules, calculation tasks are automatically generated. These tasks are divided into basic indicator calculation tasks and customized complex indicator calculation tasks: basic indicators are general-purpose indicators (such as annual operating revenue, total tax payment, etc.), with fixed calculation rules; customized complex indicators are personalized indicators configured by the enterprise according to business needs (such as core customer cooperation depth, regional market share, etc.), supporting flexible configuration of calculation logic. The system reads metadata from the "Enterprise Readiness Table" and only generates calculation tasks for enterprises with more than 90% of the total number of ready dimensions, avoiding invalid calculations.

[0041] It is worth noting that the above formulas are all dimensionless calculations, and the preset parameters in the formulas should be set by those skilled in the art according to the actual situation. The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0042] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0043] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0044] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0045] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0046] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for on-demand, precise, and real-time data acquisition and calculation, characterized in that: The method includes the following steps: Step 1: Multi-dimensional on-demand data collection request parsing and dynamic priority scheduling: S11. Receive on-demand data collection requests from business systems, verify the legality of the requests and resource availability, and extract enterprise identifiers, data dimension sets, real-time requirement levels, and calculation accuracy thresholds. S12. Based on business weight, data timeliness, data dimension importance, request waiting time, and business system load status, the request priority is determined through a non-linear priority calculation model. S13. Deliver requests to multi-level scheduling queues according to priority, and dynamically adjust resource allocation ratios based on queue congestion coefficients. Step 2: Adaptive data acquisition and incremental transmission optimization; Step 3: Real-time data processing and end-to-end consistency assurance; Step 4: Confirmation of Multi-Dimensional Data Readiness and Triggering of Intelligent Computation: S41. Dimensional Readiness Assessment: Construct an enterprise dimensional readiness judgment model, and quantify the degree of readiness by combining data integrity, accuracy and timeliness indicators; S42. Dynamic generation of calculation tasks: Based on the enterprise's readiness status and business calculation rules, automatically generate basic indicator calculation tasks and customized complex indicator calculation tasks.

2. The method for on-demand, precise, real-time data acquisition and calculation according to claim 1, characterized in that, In step S12, calculating the importance of data dimensions specifically includes the following process: The importance of each dimension of the data is determined by the analytic hierarchy process (AHP), and the importance of each dimension is summed to obtain the data dimension importance.

3. The method for on-demand, precise, real-time data acquisition and calculation according to claim 1, characterized in that, In step S12, the formula for the nonlinear priority calculation model is: ； in, Prioritize requests for on-demand data collection. for Normalization function, , , , and The weighting coefficients are satisfied. , This is the business weight, and its value range is... , For data timeliness, For the importance of data dimensions, For Lambert function, For the request waiting time, For the load sensitivity coefficient of the business system, The current load rate of the business system. For the load threshold, This represents the highest priority value.

4. The method for on-demand, precise, real-time data acquisition and calculation according to claim 1, characterized in that, In step two, the adaptive data acquisition and incremental transmission optimization are specifically... The process includes the following: S21. Data source access and collection strategy adaptation: Select the corresponding data source interface according to the characteristics of the data dimension, and dynamically adjust the collection concurrency, page size and retry strategy. S22, Incremental Acquisition and Intelligent Deduplication: Based on the data update timestamp, incremental acquisition rules are constructed, and duplicate requests are filtered through a sliding time window deduplication algorithm; S23. Data Acquisition Buffer and Metadata Enhancement: Store the acquired data in a distributed buffer and supplement it with the acquisition batch number, data integrity identifier, and data source confidence metadata.

5. The method for on-demand, precise, real-time data acquisition and calculation according to claim 1, characterized in that, Step three, real-time data processing and end-to-end consistency assurance, specifically includes the following processes: S31. Heterogeneous data standardization processing: Adaptive parsing engine is used to process multi-format data, and data cleaning, format conversion and anomaly repair are completed through custom UDF functions; S32. Streaming optimization based on Flink: The two-phase commit mechanism ensures exactly-Once semantics of data transmission, and the dynamic threshold adjustment of batch processing achieves efficient data writing. S33. Data consistency verification and conflict resolution: Data conflicts are detected based on version vector and timestamp mechanisms, and eventual data consistency is achieved through conflict resolution algorithms.

6. The on-demand, precise, real-time data acquisition and calculation method according to claim 4, characterized in that, In step S22, the deduplication determination formula of the sliding time window deduplication algorithm is: ； ； in, Representing data For duplicate data, Sliding time window Data within, window size ,in, The average data update cycle, To adjust the coefficient, Data similarity is calculated by combining cosine similarity with dimensional feature hashing: ; Let j be the hash function for the j-th dimension of the data. For similarity threshold, For data Collect timestamps, For data Collect timestamps, This represents the time tolerance threshold.

7. The on-demand, precise, real-time data acquisition and calculation method according to claim 5, characterized in that, In step S32, the formula for adjusting the batch dynamic threshold is: ； ； ； in, , and These are the thresholds for the number of data output records, the data output time interval, and the data packet size. , and These are the basic thresholds for the number of data output records, the basic threshold for the data output time interval, and the basic threshold for the data output packet size. This represents the current Kafka queue backlog. This is the normal backlog threshold. The maximum tolerable backlog, Doris database load rate This is the maximum load capacity for the Doris database. , and These are the proportionality coefficients.

8. The method for on-demand, precise, real-time data acquisition and calculation according to claim 2, characterized in that, In step S41, the calculation formula for the enterprise-level readiness determination model is as follows: ； in, For readiness level, It is determined to be ready. For integrity weight, This represents the actual number of data entries collected. For the expected number of data entries, For accuracy weighting, For actual data accuracy, The standard accuracy threshold, For the current time, This represents the expected arrival time of the data.

Images