Enterprise data real-time acquisition and index calculation cooperative processing method and system

Abstract

The application discloses a kind of enterprise data real-time acquisition and index calculation collaborative processing method and system, it is related to data processing technical field, method includes: the batch data item of acquisition and acquisition data packet are classified into continuous equal length statistical time period according to time, respectively statistics batch quantity and data packet byte number;By adjacent statistical time period constitutes overlapping time window, first window is directly summed, subsequent window is recursively updated by removing removed time period value and adding moved-in time period value, while determining window update direction difference and window update total amount.The application solves the problems of repeated processing of shared time period and equal amount of moved-in and moved-out information being canceled by net difference, which not only reuses existing window results, but also preserves non-zero window boundary processing scale.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for real-time data acquisition and indicator calculation collaborative processing of enterprise data. Background Technology

[0002] In enterprise production execution, equipment operation and maintenance, and business data archiving, data acquisition tasks typically run according to a set acquisition scope and execution cycle. Upon completion of a single acquisition task, batch data is generated to record the task completion status and time; the acquired business content is then transmitted to subsequent processing nodes as application-layer binary data packets. To understand the number of acquisition tasks executed and the actual data processing scale, processing nodes need to statistically analyze the number of completed acquisition batches and the number of bytes received in data packets over time. These statistical results can be used to represent the data acquisition load at different time periods and provide a time-dimensional foundation for storage, transmission, and subsequent computation.

[0003] When it's necessary to balance statistical range and update frequency, the statistical range of an indicator is usually larger than its update interval, resulting in overlapping time windows. For example, with a statistical range of 60 minutes and an update interval of 30 minutes, adjacent time windows share a continuous 30-minute common time interval. If each update rereads all batches and data packets within the current time window, the deduplication of identifiers, byte sequence reading, and byte count accumulation already completed within the common time interval will be performed again. Using the indicator from the previous time window for incremental updates can reduce redundant processing, but if only the net change between the preceding and following time windows is saved, the same values ​​shifted out and shifted into the same time interval will cancel each other out.

[0004] In scenarios where data collection tasks are performed according to a fixed cycle, it is normal for the same batch size or similar data scale to appear in different time intervals. For example, a previous time window might contain 35 collection batches and 2.46 × 10⁻⁶ data. 4 When data B is moved into another time interval with the same batch size and data packet byte count within a later time window, the window metrics and their net changes remain unchanged. However, the batch identifiers, data packet identifiers, and byte sequences of the moved-in and moved-out data may differ. Therefore, the core technical problem to be solved is: how to retain the update direction and non-zero processing scale of the moved-out and moved-in data while reusing the existing statistical results of the common time interval of the overlapping time windows, so that identical amounts of moved-out and moved-in data are not missed during the window update process due to the net change canceling each other out. Summary of the Invention

[0005] This invention provides a method and system for collaborative processing of real-time enterprise data acquisition and indicator calculation, which solves the technical problem of how to fully retain the non-zero window boundary processing information corresponding to equal shift-out and equal shift-in while reusing the existing calculation results of the shared statistical period during the recursive update of overlapping time windows.

[0006] A collaborative processing method for real-time enterprise data acquisition and indicator calculation includes: acquiring batch data items and data packets; classifying the batch data items and data packets into consecutive statistical time periods of equal length according to time, obtaining the number of batches and the number of bytes in the data packets for each time period; forming an overlapping time window by combining each statistical time period with the next statistical time period, adding the values ​​of the same type of time period within the first overlapping time window to obtain a window indicator; for adjacent overlapping time windows, removing the corresponding values ​​of outgoing statistical time periods that are present in the previous overlapping time window but not in the next overlapping time window from the window indicator of the previous overlapping time window, and adding the corresponding values ​​of incoming statistical time periods newly added in the next overlapping time window to obtain the window indicator of the next overlapping time window; determining the window update direction difference based on the corresponding values ​​of the incoming statistical time periods relative to the corresponding values ​​of the outgoing statistical time periods, and adding the corresponding values ​​of the incoming and outgoing statistical time periods to obtain the total window update.

[0007] The beneficial effects of this invention are reflected in: This invention categorizes batch data items and data packets into consecutive, equally long statistical time periods based on time, obtaining the number of batches and the number of bytes in data packets within each time period. Each equally long statistical time period is then combined with its successor to form an overlapping time window. The first overlapping time window obtains a window index by adding the values ​​of the included time periods. When forming a subsequent overlapping time window, the statistical period values ​​that were present in the previous window but are no longer present in the subsequent window are removed from the previous window index, and the values ​​of the newly added statistical period values ​​are added. The aggregation result of shared statistical time periods is reused with the previous window index, thus eliminating the need to reread the batch identifiers and data packet byte sequences within the shared statistical time periods each time the window moves. The recursively obtained window index remains consistent with the result obtained by directly accumulating the values ​​of the time periods contained in the subsequent window.

[0008] This invention also uses the same set of values ​​for the outgoing and incoming statistical periods to determine the window update direction difference and the total window update amount, respectively. The window update direction difference indicates the direction of increase or decrease of the incoming value relative to the outgoing value, and the total window update amount indicates the processing scale of the window boundary formed by the non-negative values ​​on both sides. When the values ​​for the incoming and outgoing statistical periods are equal and not simultaneously zero, the window update direction difference is 0, and the total window update amount remains non-zero. Thus, this invention can reduce redundant processing of shared statistical period data, avoid the loss of non-zero processing scale due to net change cancellation in equal outgoing and equal incoming data, and provide verifiable integer data for subsequent segment boundary determination and continuous update segment extraction. Attached Figure Description

[0009] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0010] Figure 1 This is a schematic diagram illustrating the steps of the collaborative processing method for real-time enterprise data collection and indicator calculation in this invention; Figure 2 This is a schematic diagram of some steps in S1 of the collaborative processing method for real-time enterprise data collection and indicator calculation of the present invention; Figure 3 This is a schematic diagram of the data flow sequence in S2 of the collaborative processing method for real-time enterprise data collection and indicator calculation in this invention; Figure 4 This is a schematic diagram of some steps in S2 of the collaborative processing method for real-time enterprise data collection and indicator calculation of the present invention; Figure 5 This is a schematic diagram of the data flow sequence in S3 of the collaborative processing method for real-time enterprise data acquisition and indicator calculation in this invention; Figure 6 This is a schematic diagram of some steps in S3 of the collaborative processing method for real-time enterprise data collection and indicator calculation of the present invention; Figure 7 This is a schematic diagram of the data flow sequence in S4 of the collaborative processing method for real-time enterprise data collection and indicator calculation in this invention; Figure 8 This is a schematic diagram of some steps in S4 of the collaborative processing method for real-time enterprise data collection and indicator calculation of the present invention; Figure 9 This is a schematic diagram of the module structure and data flow of the enterprise data real-time acquisition and indicator calculation collaborative processing system of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0012] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0013] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0014] This invention provides a method for real-time data collection and indicator calculation collaborative processing of enterprises, such as... Figure 1 As shown, in one embodiment, the method includes: S1. Obtain the batch data items and data packets, complete the necessary field checks, process the same identifier data and unify the time base, and classify the valid data into continuous and non-overlapping equal-length statistical time periods according to the collection completion time or the reception completion time, and obtain the number of batches and the number of bytes of data packets in each time period respectively. S2. In chronological order, each equal-length statistical period is combined with the next equal-length statistical period to form an overlapping time window. The window index of the first overlapping time window is directly calculated, and the window index of the subsequent overlapping time windows is recursively calculated based on the outgoing and incoming statistical periods between adjacent overlapping time windows. S3. For each window update position and each window indicator, determine the window update direction difference by comparing the corresponding value of the moving-in statistical period with the corresponding value of the moving-out statistical period, and add the corresponding values ​​of the moving-in statistical period and the moving-out statistical period to obtain the total window update. S4. For each window indicator, determine the segment boundary based on the numerical distribution of the total number of non-zero window updates of the same type during the current analysis period, extract continuous update segments that reach the segment boundary and have continuous window update positions, determine the starting point, peak point and recovery point of change, and encode the generation time sequence of continuous update segments of different data types with common window update positions.

[0015] In this embodiment, the equal-length statistical period refers to a statistical interval that is continuously divided according to a unified time base, has the same duration, and does not overlap. Its function is to carry batch identifier counting and data packet byte accumulation, and it serves as a statistical unit for overall shifting out or in when the overlapping time window moves. An overlapping time window is a statistical interval formed by each equal-length statistical period and its successor; adjacent overlapping time windows share the middle equal-length statistical period and shift one equal-length statistical period along the time direction. A shifted-out statistical period refers to a statistical period included in the previous overlapping time window but no longer included in the subsequent overlapping time window; a shifted-in statistical period refers to a statistical period added by the subsequent overlapping time window but not included in the previous overlapping time window. These terms directly correspond to the shifting out and shifting in processes that occur when the overlapping time window moves.

[0016] A batch data item refers to a data record indicating the completion of a single enterprise data collection batch, and includes at least a data source identifier, a batch identifier, and a collection completion time. A data packet refers to an application-layer binary data record submitted by the enterprise collection node, and includes at least a data source identifier, a data packet identifier, a reception completion time, and a data packet byte sequence. The data packet byte sequence is a contiguous array of bytes actually delivered for processing by this method, excluding link layer, network layer, or transport layer headers not written into this byte array. The data source identifier is a stable identifier assigned to the collection node, collection interface, or collection channel during data source access registration. Both the batch identifier and the data packet identifier are generated by their respective data sources and are not required to be unique across different data sources; therefore, both must be combined with the data source identifier to locate the corresponding business record.

[0017] Batch processing identifier refers to a structured composite identifier composed of a data source identifier and a batch identifier; data packet processing identifier refers to a structured composite identifier composed of a data source identifier and a data packet identifier. The number of batches within a time period refers to the number of different batch processing identifiers within the same equal-length statistical time period; the number of data packets within a time period refers to the sum of the actual number of bytes contained in the data packet byte sequence retained within the same equal-length statistical time period. The batch quantity window index refers to the total number of batches within an overlapping time window; the data packet byte count window index refers to the total number of data packets within an overlapping time window. Batch quantity and data packet byte count have different dimensions and are processed separately during data storage, window updates, boundary determination, and continuous update segment generation; they are not added, averaged, weighted, or compared across dimensions.

[0018] The window update position refers to the position in the sequence of window update records arranged chronologically when adjacent overlapping time windows shift out and shift in, with the sampling time being the start time of the next overlapping time window. The window update direction difference refers to the signed change of the value corresponding to the shifted-in statistical period relative to the value corresponding to the shifted-out statistical period. The total window update volume refers to the non-negative sum of the same values ​​from the shifted-in and shifted-out statistical periods. The segment boundary refers to the classification starting value determined based on the interval between adjacent values ​​of the same non-zero total window update volume within the current analysis period. A continuous update segment refers to a group of window update records that reach the same segment boundary and have consecutive window update positions. The change start point, peak point, and recovery point represent the start position of the continuous update segment, the earliest position corresponding to the maximum total window update volume within the segment, and the position where the segment first falls below the segment boundary, respectively.

[0019] A continuous update segment pair refers to a combination of batch number continuous update segments and data packet byte number continuous update segments that share a common window update position and have all obtained recovery points. Synchronization code, batch number-first code, data packet byte number-first code, and crossover sequence code refer to discrete results generated based on the positional order of the change start point, peak point, and recovery point in the continuous update segment pair. The pending data table stores records that lack valid necessary fields and have not yet entered the current calculation; the correction details table stores records before and after overwrite, and field differences corresponding to the same processing identifier; the pending completion table stores the status of continuous update segments that have formed change start points but have not yet obtained recovery points at the end of the current calculation cycle.

[0020] Data version refers to a snapshot of input data, defined by the data source configuration version, the cutoff receiving order of the calculation batch, the start and end times of the current analysis period, and the correction status. Result version refers to the overall output version based on the same data version, including time-period values, window metrics, window update records, segment boundaries, continuous update segments, and time-sequence encoding. The cutoff receiving order of the calculation batch refers to the maximum receiving order of records that have completed transaction commits at the start of the calculation, used to limit the range of data that can be read in this calculation. Time-period values, window metrics, and subsequent processing results within the same result version all use the same cutoff receiving order of the calculation batch to avoid using different input data for subsequent steps due to newly arriving records during processing.

[0021] In S1, the received records are first converted into data items or data packets with clearly defined fields. Then, necessary field checks, merging of records with the same identifier, and time mapping are performed to ensure that subsequent statistics only use valid records whose data source, business identifier, and time period can be determined. After valid records are grouped into equal-length statistical time periods, the number of time period batches and the number of bytes in time period data packets are generated. In S2, the first overlapping time window directly reads the equal-length statistical time period values ​​it contains. Subsequent overlapping time windows remove the shifted-out statistical time period values ​​from the previous window's indicators and add the shifted-in statistical time period values. Values ​​sharing statistical time periods are retained. In S3 and S4, the window change direction and window boundary processing scale are saved, and the continuous update segments and time sequence encoding are determined based on the actual total number of window updates.

[0022] like Figure 2 As shown, in one specific embodiment, S1 includes: S11. Check the necessary fields of the batch data items and data packets, generate the receiving order and corresponding processing identifier, merge the duplicate data or correction data corresponding to the same processing identifier, and write the data that is missing a valid necessary field into the supplementary data table. S12. Determine the end time of the calculation cycle and the current analysis period. Divide the current analysis period into equal-length statistical periods of the same duration. Map the valid data to the corresponding equal-length statistical periods according to the rule that the left boundary includes the data and the right boundary does not. S13. Count the batch processing identifiers in each equal-length statistical period, and accumulate the actual number of bytes in each data packet byte sequence to obtain the number of batches and the number of data packets in each period.

[0023] In S11, the essential fields for acquiring batch data items are the data source identifier, batch identifier, and acquisition completion time; the essential fields for acquiring data packets are the data source identifier, data packet identifier, reception completion time, and data packet byte sequence. A field is considered missing if it is not present, has an empty value, cannot be converted to a valid time point, or the byte sequence is unreadable. A readable byte sequence of length 0 bytes is considered a valid byte sequence and is not considered a missing field. Records lacking essential fields are written to the supplementary data table; missing fields are not replaced with 0, nor are missing identifiers or missing times inferred from other records. The supplementary data table must at least store the original record identifier, the readable data source identifier, the name of the missing field, the reception time, the reception order, the reason for writing, and the processing status. After the fields are completed, the record is treated as a newly arrived version and S11 is re-executed.

[0024] The data source identifier is determined by verifying the collection node, interface address, service affiliation, and existing identifiers during data source access registration. An unused stable identifier is assigned to each valid registration record, and the identifier's effective time and data source configuration version are saved. The data source identifier is not affected by batch size, packet size, or temporary changes in current load. For example, if "SRC-015" and "SRC-016" already exist in the access registration table, and the interface address and service affiliation of a newly added collection node differ from existing registration records, "SRC-017" can be assigned. Subsequent batch identifiers and packet identifiers submitted by this node will then constitute the processing identifier along with "SRC-017." This identifier continues to be used when the data source migrates but the service identity remains unchanged; when the service identity changes, a new data source identifier is re-registered and assigned.

[0025] The batch processing identifier and data packet processing identifier are generated using database composite keys or structured objects with field length information. Each component field is stored separately, avoiding the use of unseparated string concatenation that could cause ambiguity regarding field boundaries. The first component field of the batch processing identifier is the data source identifier, and the second component field is the batch identifier; the first component field of the data packet processing identifier is the data source identifier, and the second component field is the data packet identifier. For example, field combinations "AB" and "C" and field combinations "A" and "BC" stored separately belong to different processing identifiers and will not be mistakenly identified as the same record due to identical concatenated characters. The processing identifier is only used to locate the same business object and is not used as a numerical input for batch quantity or byte count.

[0026] Each record entering the processing queue is assigned a unique and incremental receiving order. When multiple receiving threads write simultaneously, the receiving order is assigned by a service that generates records in the same order or a database sequence with unique constraints to form a stable total order relationship. At the start of the computation, the maximum receiving order of completed transaction commits is read and determined as the cutoff receiving order for the computation batch; the current data version only reads records whose receiving order is not greater than the cutoff value. For example, if the maximum committed receiving order is 9800 at the start of the computation, then the processing range ends at 9800. Records arriving during the computation with receiving orders from 9801 to 9810 are not added to the current data version but are processed in the next computation cycle, thus preventing changes in the computation input during execution.

[0027] The set of comparison fields for records corresponding to the same processing identifier is determined according to the interface field description during data source access and is stored together with the data source configuration version. The set of comparison fields for collected batch data items includes at least the data source identifier, batch identifier, collection completion time, and business content fields that can change the statistical results; the set of comparison fields for collected data packets includes at least the data source identifier, data packet identifier, reception completion time, and complete data packet byte sequence. Reception time, reception order, internal storage address, and processing status are not included in the content comparison. For example, if a batch interface has a batch identifier, collection completion time, and batch status, then the business fields, along with the data source identifier, are included in the comparison; a change in batch status indicates different content, while a change in reception order does not indicate a change in content.

[0028] For multiple records with the same processing identifier and identical comparison field content, only the record with the last received position is retained for calculation, and the remaining records are marked as duplicate records. For multiple records with the same processing identifier but different comparison field content, the record with the last received position is taken as the currently valid record, and the record before overwriting, the record after overwriting, the field difference, the received order, and the correction time are written to the correction details table. For example, if two byte sequences with the same content and a length of 620 bytes are received successively for the same data packet processing identifier, only the 620 byte sequence is counted; if the second byte sequence is 700 bytes and the content is different, the second record overwrites the first record, and both versions are stored in the correction details table for subsequent location of the affected equal-length statistical period.

[0029] In S12, external time fields are converted to a unified time base before mapping. Time with time zone offsets is converted to the corresponding Coordinated Universal Time (UTC) time point according to its offset. Data without time zone information is only converted if a unique time zone configuration is stored in the data source registration information; otherwise, it is written to the supplementary data table. External time text uses the complete date, time, and time zone offset format, internally storing the converted unified time point while retaining the original time text and original time zone offset. This processing is consistent with the date and time representation principles of the current GB / T7408.1-2023; when exchanging timestamps via an internet interface, the complete time and offset format specified in RFC3339 can be used.

[0030] The duration of the equal-length statistical period is determined before deployment based on a limited set of historical records that can be fully obtained and have consistent field definitions. Specifically, records are sorted in ascending order of collection completion time from the same data source. All positive time intervals between adjacent valid records are extracted. Identical intervals are counted, and the interval with the highest frequency that is divisible by the target window duration is selected as the statistical period duration. If the frequency is the same, the shorter interval is selected. In this example, complete historical data from 60 consecutive days prior to the configuration taking effect was used, resulting in 2879 valid intervals: 2704 were 30 min, 94 were 15 min, and 81 were 60 min, totaling 2879. Therefore, 30 min was selected. This process is re-executed after the collection frequency changes, generating a new configuration version.

[0031] The current analysis period length is determined based on the complete historical retention time available from all participating data sources. Specifically, it is the shortest effective retention time among all data sources, aligned downwards according to complete statistical periods of equal length. This method avoids misinterpreting a data source with no retained records at the beginning of the analysis period as having an actual value of 0. For example, if the three participating data sources can provide complete records for 30 days, 45 days, and 60 days respectively, then the current analysis period is a continuous 30 days, or 720 hours. If the data retention strategy changes, the current analysis period is redefined based on the shared available time of all data sources after the new configuration takes effect, and historical records with different or discontinuous field definitions are not concatenated.

[0032] The end time of the calculation cycle is taken as the right boundary of the most recently completed statistical period of equal length when the calculation starts. Statistical periods that have not yet ended are not included in the current result version. This value does not depend on whether the data collection record has just arrived, but is determined by the boundary of the statistical period under a unified time base. For example, if the duration of a statistical period is 30 minutes, and the calculation starts at 12:47, the right boundary of the most recently completed statistical period is 12:30. Therefore, the end time of the calculation cycle is taken as 12:30. The statistical period from 12:30 to 13:00 is still in progress and is not included in this calculation. The next time the 13:00 boundary is crossed, the statistical period will be included in subsequent data versions to avoid generating multiple incomplete statistical values ​​for the same statistical period.

[0033] The duration of the overlapping time window is determined based on the statistical period of the indicator to be output, and is verified to be exactly equal to the duration of two adjacent equal-length statistical periods. In this embodiment, the enterprise's collected indicators are required to be output in 60-minute intervals, and the duration of the statistical period is 30 minutes. Therefore, each overlapping time window contains two adjacent equal-length statistical periods and shifts 30 minutes along the time direction. The current analysis period of 30 consecutive days contains 1440 equal-length statistical periods. Each statistical period and its subsequent statistical period form a window, resulting in 1439 overlapping time windows. There are 1438 window update positions between adjacent windows. The above quantities are consistent with the time division and window composition relationship, and there is no over- or under-counting.

[0034] The equal-length statistical periods employ a mapping rule where the left boundary includes the data, while the right boundary excludes it. This ensures that boundary time points only enter one statistical period. For example, if the statistical periods are 09:00 to 09:30 and 09:30 to 10:00, records with a time of 09:00 enter the former, and records with a time of 09:30 enter the latter. The right boundary of the current analysis period does not construct overlapping time windows extending into the future. Before dividing the statistical periods, all time intervals are pre-generated. Even if a certain statistical period has no valid records, that interval is retained to maintain the temporal continuity of the window sequence and window update positions.

[0035] In S13, the batch data items collected within the same equal-length statistical time period are counted after deduplication according to the batch processing identifier, and the resulting non-negative integer is the number of batches in the time period. The data packet byte sequence is counted according to the actual length of the underlying byte array. The lengths of the data packet byte sequences retained within the same equal-length statistical time period are added together, and the resulting non-negative integer is the number of data packet bytes in the time period. The number of data packet bytes is measured in bytes (B), and each byte is processed as 8 bits, consistent with the IEC specifications regarding bytes and binary multiples. The data packets are application layer data records and are not required to correspond one-to-one with the underlying network frames. Therefore, the length of a single packet in this embodiment only needs to be consistent with the actual byte array length and is not directly limited by the length of a certain link layer frame.

[0036] For equal-length statistical periods without any valid batch data items, the batch size for that period is set to 0; for equal-length statistical periods without a valid non-zero data packet byte sequence, the data packet byte size for that period is set to 0 bytes. Here, 0 indicates that the valid record set for that statistical period is empty or the valid byte sequence length is 0 bytes; it does not indicate that 0 is used to fill missing fields. The batch size and data packet byte size are stored and calculated separately. One batch can correspond to one or more data packets; therefore, this implementation does not require the batch size to be equal to the data packet size, nor does it substitute the data packet byte size for the data packet size. Only the actual length of the data packet byte sequence participates in the byte count accumulation; communication headers, padding fields, and stored metadata not included in the sequence are not counted.

[0037] The integer storage type is determined based on the maximum number of records in a single equal-length statistical period, the maximum actual number of bytes in a single data packet, and the number of statistical periods contained in each window, as measured before deployment. When selecting values, the maximum possible period value, the maximum window index, and the maximum window update total are first determined based on the complete historical record. Then, an integer type with a value upper limit greater than the above maximum values ​​while retaining expansion space is selected. For example, in this embodiment, the maximum number of bytes in a data packet for a single period is... The maximum window data packet size is The maximum total number of window updates is It can be completely stored using a signed 64-bit integer. A range check is still performed before writing, and if there is a risk of overflow, it is recalculated using an expandable integer type without truncating the high bits.

[0038] For records arriving after the end of the statistical period but whose collection or reception completion time still falls within that period, they are mapped to the original statistical period based on their own business time, not the current statistical period based on their actual arrival time. If the record's reception order is not greater than the cutoff reception order for this batch, it participates in merging in the current data version; if the reception order is greater than the cutoff value, it proceeds to the next data version. For valid records whose business time is earlier than the current analysis period, their original record and reception information are retained, but they are not included in the current analysis period; when backtracking is required, a separate result version is generated based on the specified historical period. Thus, late data, missing data, data outside the analysis period, and normal data each have a clear processing path and do not substitute for each other.

[0039] like Figure 3 and Figure 4 As shown, in one specific embodiment, S2 includes: S21. Construct overlapping time windows according to the time order of equal-length statistical periods, read the adjacent equal-length statistical periods contained in the first overlapping time window, and obtain the first batch quantity window index and the first data packet byte count window index respectively. S22. For adjacent overlapping time windows, remove the corresponding values ​​of the outgoing statistical period from the same type of window index of the previous overlapping time window, and then add the corresponding values ​​of the incoming statistical period to obtain the same type of window index of the next overlapping time window. S23. Perform reverse recovery verification, window difference verification, and non-negativity verification on the recursively generated window indicators; if any verification fails, read the corresponding values ​​of the equal-length statistical periods that constitute the next overlapping time window, and regenerate the same type of window indicator and its window update record.

[0040] In S21, the start time of the overlapping time window coincides with the left boundary of the first equal-length statistical period it contains, and the end time coincides with the right boundary of the next equal-length statistical period it contains. The overlapping time windows are arranged in ascending order of start time and each receives a unique window number within the same result version. The first overlapping time window has no reusable preceding window, so its adjacent statistical periods are read directly. In this embodiment, the batch sizes of the first and second equal-length statistical periods are 12 and 15, respectively, and the packet sizes are 7200 bytes and 9100 bytes, respectively. Adding these values ​​together yields the first batch size window index 27 and the first packet size window index 27. And save the identifier of the statistical period used for subsequent verification.

[0041] In S22, the second overlapping time window is shifted out of the first equal-length statistical period and into the third equal-length statistical period relative to the first overlapping time window. The number of time period batches in the third equal-length statistical period is 21, and the number of data packets per time period is [number missing]. Remove 12 from the batch quantity window index 27 and add 21 to obtain the next batch quantity window index 36; from the packet byte count window index... Remove 7200 B and add The window index of the number of bytes in the next data packet is obtained. The second shared statistical period does not need to be read again.

[0042] In S23, the reverse recovery check removes the corresponding value for the shifted-in statistical period from the same window index of the subsequent overlapping time window, and then adds the corresponding value for the shifted-out statistical period. The resulting reverse recovery value must be exactly the same as the same window index of the previous overlapping time window. The window difference check uses two independent processing paths: one path compares the integer changes of the subsequent window index relative to the previous window index, and the other path compares the integer changes of the shifted-in statistical period value relative to the shifted-out statistical period value. The results of these changes must be exactly the same. Non-negativity check is used to check whether the batch quantity window index and the data packet byte count window index are less than 0. Since all values ​​are integers, no approximation error or floating-point tolerance is set.

[0043] If any of the following occurs: inconsistent reverse recovery values, inconsistent difference between the window indicator change and the corresponding window update direction, or the updated window indicator is less than 0, the recursive result is stopped. Instead, the values ​​of adjacent equal-length statistical periods constituting the next overlapping time window are directly read, re-added, and the corresponding window update direction difference and window update total are regenerated. After recalculation, the aforementioned verification is performed again. If the verification passes, the recalculated result is written to the current result version; if it still fails, the current data version is marked as a calculation failure, and inconsistent results are not published. The failure record saves the data version, window number, input statistical period identifier, recursive result, recalculated result, and failure type to check for mixed data versions, integer range issues, or storage / read anomalies.

[0044] The data version generation process is completed within the same transaction snapshot. At the start of the calculation, the cutoff receiving order for the calculation batch is determined. The generation of time period values, window metric recursion, window update record generation, segment boundary determination, and continuous update segment extraction all use only records whose receiving order is no greater than this cutoff value. Newly arrived records during processing will not be included in the current transaction snapshot. For example, if the current result version uses a receiving order of 9800 as the cutoff value, the correction made by receiving order 9801 for the third statistical time period will only be included in the next data version. There will be no situation where the time period data packet byte count has already adopted the corrected value while adjacent window metrics still use the old value. If any processing stage fails, the current result version will not be published as a whole.

[0045] like Figure 5 and Figure 6 As shown, in one specific embodiment, S3 includes: S31. For the batch size and the number of bytes in the data packet, compare the value of the shift-in statistical period and the value of the shift-out statistical period for each window update position to obtain the difference in the update direction of the corresponding window. S32. For the batch quantity and the number of bytes in the data packet, add the value of the shift-in statistical period and the value of the shift-out statistical period for each window update position to obtain the total update quantity for the corresponding window. S33. Save the window update direction difference and total window update for each window update position in ascending order of sampling time, forming batch quantity update sequence and data packet byte number update sequence respectively, and retain window update records with direction difference of 0 and total window update greater than 0.

[0046] In S31 and S32, a window update direction difference greater than 0 indicates that the value in the shift-in statistical period is greater than the value in the shift-out statistical period; equal to 0 indicates that the two values ​​are equal; and less than 0 indicates that the value in the shift-in statistical period is less than the value in the shift-out statistical period. The total window update is obtained by adding the values ​​of the two non-negative periods, without any positive or negative cancellation. The batch number window update direction difference and total window update only use the batch number for calculation, and the data packet byte number window update direction difference and total window update only use the data packet byte number for calculation. For example, both the shift-out statistical period and the shift-in statistical period contain 35 batches and both have... When the data packet size is 70 bytes, the batch size difference is 0 in the direction, and the total size is 70. (B) The data packet size difference is 0 in the direction, and the total size is... .

[0047] The total window update only represents the processing scale of the window boundary formed by the outgoing and incoming statistical periods, and does not independently prove that the record identifiers or byte contents on both sides are different. When the difference in window update direction is 0 and the total window update is greater than 0, it can be determined that there are non-zero value outgoing and non-zero value incoming; only after further checking the batch processing identifier, data packet processing identifier, or data packet byte sequence can it be determined whether it belongs to equal replacement with different content. In the example, the batch number of the 5th and 7th equal-length statistical periods is 35, and the number of data packet bytes is also 35. However, the processing identifiers and byte length distributions are different, so it belongs to the replacement of equal content, rather than the repeated statistics of the same record.

[0048] In S33, each window update record stores at least the data type, data version, previous window number, next window number, window update position, sampling time, value removed from statistical periods, value moved into statistical periods, window update direction difference, and total window update amount. The window update position is generated by first arranging the windows in ascending order according to the start time of the overlapping time windows, and then generating one window update record for each pair of adjacent overlapping time windows, with the sampling time taken as the start time of the next overlapping time window. When the sampling times of adjacent window update records differ by one statistical period duration, the corresponding positions are temporally continuous. When both the removed and moved-in values ​​are 0, the update record is still retained to determine whether the continuous update segment is interrupted, but its total amount is not included in the segment boundary calculation.

[0049] The aforementioned seven equal-length statistical periods form six overlapping time windows and five window update positions. The differences in the update direction of the batch quantity windows are 9, 3, and 14, respectively. 4 and 0 correspond to total window update amounts of 33, 33, 56, 32, and 70 respectively. The differences in packet byte count window update direction are 7000 bytes, 2700 bytes, and 10400 bytes respectively. 3200 B and 0 B correspond to the total window update amounts respectively. , , , and The above results are consistent with the values ​​for each statistical period.

[0050] like Figure 7 and Figure 8 As shown, in one specific embodiment, S4 includes: S41. Extract the total non-zero window update amount from the batch quantity update sequence and the data packet byte count update sequence respectively, remove duplicate values ​​of the same type and sort them in ascending order, and determine the corresponding segment boundary according to the interval between adjacent values. S42. Form continuous update segments by combining window update records that reach the same segment boundary and whose window update positions are continuous. Determine the change start point, peak point and recovery point from each continuous update segment, and process continuous update segments that have not yet obtained a recovery point. S43. Form a continuous update segment pair by combining the batch number continuous update segment and the data packet byte number continuous update segment that have a common window update position and have all obtained recovery points, and generate time sequence encoding according to the order of the corresponding key positions.

[0051] In S41, the method for extracting the total non-zero window update is to traverse the same update sequence, copy the total window update for values ​​greater than 0 for boundary calculation, and retain the original update record and its window update position. Deduplication only deletes duplicate values ​​in the boundary calculation copy, not duplicate update records in the original time series. When there is no non-zero window update, no segment boundary or continuous update segment is generated for this data type; when there is only one non-zero value after deduplication, this value is directly used as the segment boundary; when there are at least two different non-zero values, the interval value between each pair of adjacent values ​​and the sorted last endpoint value are obtained in sequence, and then sorted in ascending order according to the interval value and the last endpoint value, and the last endpoint value in the sort is taken as the segment boundary.

[0052] The sorting method for adjacent numerical intervals also specifies the handling rules when multiple maximum intervals are the same. When the interval values ​​are different, the record with the larger interval value is placed later; when the interval values ​​are the same, the record with the larger endpoint value is placed later, thus the last record in the sorting can be uniquely determined. Reaching the segment boundary means that the total window update amount is equal to or greater than the segment boundary. For example, if the batch quantity is non-zero, the total window update amount after deduplication is 24, 26, 27, 28, 29, 30, 32, 33, 56, 61, 63, and 70. The largest adjacent interval is between 33 and 56, with an interval value of 23. Therefore, the segment boundary for the batch quantity is the endpoint value of 56.

[0053] The number of bytes in the data packet is non-zero. The total number of bytes updated in the window after deduplication is: , , , , , , , , , , , , , and The maximum interval is located at and Between, therefore the segment boundary is .

[0054] Segment boundaries do not use pre-written fixed values; their calculation range is determined jointly by the current data version and the current analysis period. After each complete calculation or correction affecting the total window update, the same non-zero window update total is re-extracted, and deduplication, sorting, adjacent interval record generation, and last record selection are re-executed. For example, data corrections may change the original window update total... Become When correcting the data, the corrected values ​​should be placed back into the complete dataset and reordered before determining the new maximum interval and endpoint values. The old segment boundaries should not be used directly. Batch quantity segment boundaries and data packet byte count segment boundaries should always be calculated separately.

[0055] In S42, the original update sequence is traversed starting from the earliest window update position. A continuous update segment is established when the total window update volume first reaches the segment boundary; if subsequent sampling times are exactly increased by one equal-length statistical period duration and the total window update volume still reaches the segment boundary, the update record is added to the same continuous update segment; if a position is discontinuous or the total window update volume is lower than the segment boundary, the current continuous update segment ends. The starting point of change is the earliest position within the segment; for peak points, the maximum total window update volume within the segment is first determined, and then the earliest sampling time is selected from the positions with that maximum value; the recovery point is the position where the first window update volume after the segment is lower than the segment boundary used by that segment.

[0056] For example, the total number of batch updates at window update positions 701 to 710 are 24, 26, 27, 28, 28, 30, 56, 61, 63, and 29, respectively. When the segment boundary is 56, positions 707 to 709 reach the boundary and are continuous, forming a continuous update segment for batch numbers. Position 707 is the starting point of change, and position 709 has the maximum value of 63 within the segment, thus it is the peak point. The value at position 710 is 29, which is below the segment boundary, therefore it is the recovery point. If multiple positions within a segment have the same maximum value, only the position with the earliest sampling time is taken as the peak point to ensure the uniqueness of the processing result.

[0057] The number of data packets and the total number of window updates at the same window update position are as follows: , , , , , , , , and At the segment boundary is At that time, positions 708 to 709 form a continuous update segment for the number of bytes in the data packet. Position 708 is the starting point of the change, position 709 is the peak point, and position 710 is the recovery point.

[0058] When a continuous update segment extends to the end of the current calculation cycle and has not yet reached a recovery point, the continuous update segment is written to the pending completion table and is not output as a completed continuous update segment. The pending completion table at least stores the data type, result version, segment boundary used, change start point, current peak point, last sampling time, and the identifier of the window update record already included. In the next calculation cycle, the pending completion record is read first, and the segment boundary corresponding to the beginning of the segment is used to check for newly added window update records until the total number of window updates below the boundary is first found, at which point the recovery point is determined. For example, if the continuous update segment still reaches the boundary at position 1438, it is written to the pending completion table in this cycle; if the newly generated position in the next cycle is below the boundary, that position is used as the recovery point.

[0059] The segment boundaries corresponding to the beginning of the continuous update segment are retained to avoid changes in the judgment criteria due to changes during the rolling analysis. If only new subsequent records are added, the saved segment boundaries are used to determine the recovery point; if the corrected data changes the window update records included in the continuous update segment, or changes the total set of window updates that generated the segment boundaries, the original record to be completed is marked as invalid, and S41 and S42 are re-executed according to the corrected data version. Completed and output continuous update segments retain their historical state through the result version. The new result version does not directly overwrite and delete the original result, so as to verify the processing before and after the segment boundary change.

[0060] In S43, a common window update position refers to the same position contained in both the batch quantity continuous update segment and the data packet byte count continuous update segment. The method for generating continuous update segment pairs is to first limit the candidate range according to the data source identifier and result version, and then, for each batch quantity continuous update segment that has achieved a recovery point, check all data packet byte count continuous update segments that have achieved a recovery point one by one. A continuous update segment pair is formed when the number of common positions is greater than 0. Random selection is not used, and there is no restriction that a continuous update segment can only participate in one segment pair. The continuous update segment pair identifier is composed of the result version, the batch quantity continuous update segment identifier, and the data packet byte count continuous update segment identifier to avoid duplicate output of the same combination.

[0061] The change start interval is calculated using the signed position difference between the change start position of the continuous update segment for data packet byte count and the change start position of the continuous update segment for batch quantity. The peak point interval and recovery point interval follow the same order. A synchronization code is generated when all three position intervals are 0; a batch quantity-first code is generated when all are not less than 0 and at least one is greater than 0; a data packet byte count-first code is generated when all are not greater than 0 and at least one is less than 0; and an interleaved sequence code is generated when both positive and negative intervals exist. The time sequence encoding only indicates the relative order of key positions and does not represent causal relationships, business responsibilities, or economic implications.

[0062] The starting point, peak point, and recovery point of the aforementioned batch quantity continuous update segment are the 707th, 709th, and 710th window update positions, respectively, and the corresponding positions for the data packet byte count continuous update segment are the 708th, 709th, and 710th positions, respectively. The interval for the starting point is 1, and the intervals for the peak point and recovery point are both 0, thus generating batch quantity-priority codes. The duration of the equal-length statistical period is 30 minutes, so position interval 1 corresponds to 30 minutes. If both positive and negative values ​​appear in the corresponding position interval, cross-order codes are generated, without forcibly specifying a uniform prior data type; if the two types of continuous update segments do not have a common position, no continuous update segment pair is generated.

[0063] In one specific implementation, for correction data that is received last in the order, the records before and after the overlay are first located based on the batch processing identifier or data packet processing identifier. Then, the affected equal-length statistical period is determined based on the collection completion time or reception completion time before and after the overlay. When the data items from the collection batches before and after the overlay are included in the same equal-length statistical period, the number of batches in that period is recalculated, and only the overlapping time windows containing that period and the window update positions using that period as the move-out or move-in statistical period are updated. When the data items before and after the overlay are included in different statistical periods, the statistical periods are recalculated separately, and the affected windows and update positions are merged. When there are overlapping windows or update positions, they are updated only once after deduplication based on the unique identifier.

[0064] The same positioning rules are used for local correction of the acquired data packets. When the acquired data packets before and after the coverage are grouped into the same statistical time period of equal length, the number of bytes of the data packets in that time period is recalculated, and the overlapping time window containing that time period and the window update position with that time period as the outgoing or incoming statistical time period are updated. When the data packets before and after the coverage are grouped into different statistical time periods, the statistical time periods are recalculated separately. A statistical time period can only belong to the adjacent window with it as the preceding or following segment, and can only act as the outgoing or incoming statistical time period during the movement of the corresponding window. Therefore, the local update range is directly determined by the temporal adjacency relationship between the statistical time period and the window, without the need to traverse all windows.

[0065] For example, if the byte sequence corresponding to a packet processing identifier is corrected from 640 bytes to 700 bytes, and the reception completion time before and after the overwrite both fall within the third equal-length statistical period, then the number of packets in that statistical period is […]. Updated to The second data packet byte count window indicator is determined by... Updated to The third window indicator is... Updated to The correction increment is 60 B, consistent with the increase in the affected window index.

[0066] In the above correction example, the difference in the number of data packets in the window update direction at the first window update position was updated from 7000 bytes to 7060 bytes, and the total window update amount was changed from... Updated to The direction difference for updating the position of the third window is determined by... Updated to The total number of window updates is from Updated to The former is determined by the updated third statistical period relative to the first statistical period, while the latter is determined by the fifth statistical period relative to the updated third statistical period. The calculation direction and values ​​are consistent.

[0067] After a partial update is completed, the total update volume of all non-zero windows within the current analysis period is re-extracted for the affected data types, the segment boundaries are redefined, and continuous update segments, pending records, and continuous update segment pairs are regenerated. Period values ​​and window indicators for unaffected data types can be reused, but continuous update segment pairs involving different data types must be re-evaluated based on the new results. After a new result version is generated, the old result version is marked as replaced, while the old version, corresponding data version, correction details, and generation time are retained for auditing purposes. External queries return the latest valid result version by default; when specifying a historical result version, the results before correction can still be read, and historical results are not physically deleted.

[0068] This invention also provides a collaborative processing system for real-time enterprise data acquisition and indicator calculation, such as... Figure 9 As shown, the system includes a processor, memory, a data mapping module, a window update processing module, and a time-coordinated output module. The data mapping module acquires batch data items and data packets, checks for valid and necessary fields, generates batch processing identifiers and data packet processing identifiers, processes duplicate and corrected data, assigns valid records to equal-length statistical time periods, and outputs the number of batches and data packets per time period. The window update processing module constructs overlapping time windows, calculates the first window index, recursively generates subsequent window indices, performs reverse recovery verification, window difference verification, and non-negativity verification, and generates the window update direction difference, total window update amount, segment boundaries, and continuous update segments. The time-coordinated output module generates continuous update segment pairs and their time sequence encoding.

[0069] The memory is used to store received records, data to be supplemented, correction details, time period values, window indicators, window update records, segment boundaries, records to be completed, continuously updated segments, continuously updated segment pairs, data versions, and result versions. When the processor executes the computer program in the memory, it enables each module to complete the aforementioned methods. When multiple processors execute in parallel, statistical time periods of equal length can be divided according to the data source identifier and time range, but the same result version must use the same calculation batch cutoff receiving order; after each processing partition is completed, they are merged according to the window sequence number and sampling time, and no incomplete result version is published if any partition fails. Each module can be implemented by different program segments in the same processor, or by multiple processors accessing the same transaction snapshot.

[0070] A company's data acquisition nodes continuously receive batch information on production equipment operation and application-layer binary data packets. Before deployment, 60 consecutive days of historical data with consistent field definitions were obtained, yielding 2879 valid data collection intervals: 2704 for 30 minutes, 94 for 15 minutes, and 81 for 60 minutes. The sum of these intervals is 2879; therefore, in S1, the duration of the equal-length statistical period is determined to be 30 minutes. The data sources involved in the calculation can provide complete historical data for 30, 45, and 60 days respectively; the commonly available 30-day period is used in the current analysis. The statistical period for the output indicator is 60 minutes; therefore, each overlapping time window contains two adjacent equal-length statistical periods.

[0071] In S1, the batch identifier and data packet identifier of the data source "SRC-017" form a structured processing identifier together with the data source identifier. If two identical byte sequences, each 620 bytes in length, are received sequentially under the same data packet processing identifier, only the record at the end of the received sequence is retained and counted within the 620 bytes. One record lacks a data packet identifier; this record is written to the supplementary data table, not included in the equal-length statistical period, and its identifier is not inferred from other records. The maximum submitted received sequence at the start of the calculation is 9800; therefore, the current data version only processes records with a received sequence no greater than 9800. Records arriving during processing with a received sequence after 9801 are included in the next data version to prevent changes to the input set during processing.

[0072] The batch sizes for the seven consecutive statistical periods of equal length are 12, 15, 21, 18, 35, 14, and 35, respectively. The first statistical period contains 12 data packets of 600 bytes each, totaling 7200 bytes; the second statistical period contains 10 data packets of 600 bytes each and 5 data packets of 620 bytes each, totaling 9100 bytes; the third statistical period contains 20 data packets of 650 bytes each and 1 data packet of 1200 bytes each, totaling 1.42 × 10⁻⁶ bytes. 4 B; The fourth statistical period contains 12 650 B, 4 700 B, and 2 600 B data packets, totaling 1.18 × 10⁻⁶. 4 B.

[0073] The fifth statistical period includes 20 700B, 10 720B, and 5 680B data packets, totaling 2.46 × 10⁻⁶. 4 B; The 6th statistical period contains 10 packets of 600 bytes and 4 packets of 650 bytes, totaling 8600 bytes; The 7th statistical period contains 15 packets of 720 bytes, 10 packets of 700 bytes, and 10 packets of 680 bytes, totaling 2.46 × 10⁻⁶ bytes. 4 B. The above single packet lengths fall within the common application layer data block range, and the product of each group matches the total value. The values ​​for the 5th and 7th statistical periods are the same, but the processing identifiers and byte length distributions are different.

[0074] In S2, the seven equal-length statistical periods form six overlapping time windows. The batch quantity window indices are 27, 36, 39, 53, 49, and 49, respectively, and the data packet byte count window indices are 1.63 × 10⁻⁶, respectively. 4 B, 2.33×10 4 B, 2.6×10 4 B, 3.64×10 4 B, 3.32×10 4 B and 3.32×10 4B. The second window removes 12 from the first batch quantity window indicator of 27 and adds 21, resulting in 36; it removes 7200 B from the first packet byte count window indicator and adds 1.42 × 10⁻⁶. 4 B, yielding 2.33 × 10 4 B.

[0075] In S3, the batch size and window update direction differences for the 1st to 5th window update positions are 9, 3, and 14 respectively. The total window update amounts for 4 and 0 are 33, 33, 56, 32, and 70 respectively. The differences in packet byte count and window update direction are 7000 bytes, 2700 bytes, and 1.04 × 10⁻⁶ bytes respectively. 4 B. For 3200 B and 0 B, the total window update amount is 2.14 × 10⁻⁶ respectively. 4 B, 2.09×10 4 B, 3.88×10 4 B, 2.04×10 4 B and 4.92×10 4 B. Although the directional difference at position 5 is 0, it still retains a non-zero window update total.

[0076] In S4, after deduplication and ascending sorting, the maximum adjacent interval for the total non-zero batch number update during the complete current analysis period is 23, corresponding to a subsequent endpoint value of 56. Therefore, the batch number segment boundary is 56; the maximum adjacent interval for data packet bytes is 1.74 × 10^6 bytes. 4 B, corresponding to a trailing endpoint value of 3.88 × 10 4 B, therefore the data packet byte segmentation boundary is 3.88 × 10 4 B. Positions 707 to 709 form a continuous update segment for batch quantity, and positions 708 to 709 form a continuous update segment for data packet byte count. The two have the same position; the change start interval is 1, and the peak point interval and recovery point interval are both 0. Therefore, batch quantity advance code is generated.

[0077] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0078] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0079] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for real-time data collection and indicator calculation collaborative processing of enterprises, characterized in that, The methods include: Acquire batch data items and data packets, and group the batch data items and data packets into consecutive statistical time periods of equal duration according to time, thereby obtaining the number of batches and the number of bytes of data packets in each time period. Each equal-length statistical period is combined with the next equal-length statistical period to form an overlapping time window. The values ​​of the same period within the first overlapping time window are added together to obtain the window index. For adjacent overlapping time windows, remove the corresponding values ​​of the outgoing statistical periods that the previous overlapping time window has but the subsequent overlapping time window does not have from the window index of the previous overlapping time window, and add the corresponding values ​​of the incoming statistical periods newly added by the subsequent overlapping time window to obtain the window index of the subsequent overlapping time window. The window update direction difference is determined by comparing the corresponding value of the moving-in statistical period with the corresponding value of the moving-out statistical period, and the corresponding values ​​of the moving-in statistical period and the moving-out statistical period are added together to obtain the total window update.

2. The method for real-time enterprise data acquisition and indicator calculation collaborative processing according to claim 1, characterized in that, The batch data items include a data source identifier, a batch identifier, and a collection completion time; the data packets include a data source identifier, a data packet identifier, a reception completion time, and a data packet byte sequence. A batch processing identifier is composed of the data source identifier and the batch identifier, and a data packet processing identifier is composed of the data source identifier and the data packet identifier. According to the collection completion time, the collected batch data items are assigned to the corresponding equal-length statistical time period, and the number of different batch processing identifiers within the same equal-length statistical time period is used as the number of batches in the time period. According to the received completion time, the collected data packets are assigned to the corresponding equal-length statistical time period. The actual number of bytes in the byte sequence of each data packet within the same equal-length statistical time period is added together to obtain the number of bytes of data packets in the time period.

3. The method for real-time enterprise data acquisition and indicator calculation collaborative processing according to claim 1, characterized in that, The boundary of a completed equal-length statistical period is taken as the end time of the calculation cycle, and a continuous time range forward from the end time of the calculation cycle is selected as the current analysis period. The current analysis period is divided into continuous and non-overlapping statistical periods of equal length according to the rule that the left boundary includes the right boundary and the right boundary does not. Each equal-length statistical period is combined with the next equal-length statistical period in chronological order to form an overlapping time window, and the start time of the next overlapping time window is used as the sampling time for the corresponding window update position.

4. The method for real-time enterprise data acquisition and indicator calculation collaborative processing according to claim 2, characterized in that, Add any batch data items or data packets that lack valid and necessary fields to the supplementary data table and exclude them from the current calculation cycle; For multiple records with the same identifier and identical content, only the record that is received last in the order will be retained; For multiple records with the same identifier but different content, the record that is last in the receiving order is taken as the current valid record, and the record before overwriting, the record after overwriting, and their field differences are written to the correction details table.

5. The method for real-time enterprise data acquisition and indicator calculation collaborative processing according to claim 1, characterized in that, Remove the corresponding values ​​for the period being moved into the statistical period from the same window index of the next overlapping time window, and add the corresponding values ​​for the period being moved out of the statistical period to obtain the reverse recovery values. Compare the changes of the same window index in the later overlapping time window with those in the previous overlapping time window, and compare them with the difference in the corresponding window update direction. When the reverse recovery value is different from the same window index of the previous overlapping time window, the difference between the change of the window index and the window update direction is different, or the updated window index is negative, the equal-length statistical period value constituting the next overlapping time window is reread, and the corresponding window index, window update direction difference and window update total are regenerated.

6. The method for real-time enterprise data acquisition and indicator calculation collaborative processing according to claim 1, characterized in that, Generate a batch quantity window index and a data packet byte count window index respectively, and generate the window update direction difference and the total window update amount corresponding to the batch quantity window index and the data packet byte count window index respectively; The batch size for each time period and the number of bytes in the data packet for each time period are stored and calculated respectively; When the same value in the statistical period being moved out and the statistical period being moved in are equal and not both zero at the same time, the difference in window update direction is set to zero, and the two are added together to obtain the non-zero total window update amount.

7. The method for real-time enterprise data acquisition and indicator calculation collaborative processing according to claim 1, characterized in that, For each window metric, the total update volume of the same non-zero window is deduplicated and then sorted in ascending order; When there are different adjacent values, obtain the interval value between the adjacent values ​​and the last endpoint value in the sorted order, sort them in ascending order according to the interval value and the last endpoint value, and use the last endpoint value in the sorted order as the segment boundary. The total number of window updates that reach the segment boundary and whose window update positions are continuous are formed into a continuous update segment, and the change start point, peak point and recovery point are determined from the continuous update segment; If a recovery point is not obtained in the current calculation cycle, the continuously updated segment is written into the pending completion table and output after a recovery point is obtained in a subsequent calculation cycle.

8. The method for real-time enterprise data acquisition and indicator calculation collaborative processing according to claim 7, characterized in that, The batch number continuous update segment and the data packet byte number continuous update segment, which have a common window update position and have all obtained recovery points, are combined into a continuous update segment pair; Compare the positional order of the starting point, peak point, and recovery point of the continuously updated segment; Synchronization code is generated when all corresponding positions are the same. Batch quantity-first code is generated when none of the corresponding positions in the continuous update segment of data packet byte count are earlier than the continuous update segment of batch quantity and at least one position is later. Data packet byte count-first code is generated when none of the corresponding positions in the continuous update segment of data packet byte count are later than the continuous update segment of batch quantity and at least one position is earlier. In other cases, cross-order code is generated.

9. The method for real-time enterprise data acquisition and indicator calculation collaborative processing according to claim 4, characterized in that, The affected equal-length statistical period is determined based on the time fields of the records before and after the overwrite, and the number of time period batches or the number of time period data packets in the affected equal-length statistical period is recalculated. Only update the overlapping time windows that include the affected equal-length statistical periods and the window update position that uses the affected equal-length statistical periods as the period to be moved out of or into the statistical period; The segment boundaries are redefined based on the updated total number of the same type of window, and the corresponding continuous update segments and continuous update segment pairs are regenerated.

10. A collaborative processing system for real-time enterprise data acquisition and indicator calculation, characterized in that, The system is used to implement the enterprise data real-time acquisition and indicator calculation collaborative processing method as described in any one of claims 1 to 9, including a processor, a memory, a data mapping module, a window update processing module, and a time collaborative output module; The data mapping module is used to acquire batch data items and data packets, and to group valid records into continuous and equally long statistical periods according to time, thereby obtaining the number of batches and the number of bytes of data packets in each period, and to form an overlapping time window with each equal-length statistical period and the next equal-length statistical period. The window update processing module is used to calculate the window index of the first overlapping time window, and for adjacent overlapping time windows, remove the corresponding value of the outgoing statistical period from the window index of the previous overlapping time window, and then add the corresponding value of the incoming statistical period to obtain the window index of the next overlapping time window. The window update processing module is also used to determine the window update direction difference and the total window update, determine the segmentation boundary based on the non-zero total window update and generate continuous update segments; The time-coordinated output module is used to output window indicators, window update direction differences, total window update volume, continuous update segments, and time sequence codes.

Images