Correlation data updating method, system, device and readable storage medium

Abstract

This disclosure provides a method, system, device, and readable storage medium for updating correlated data. The method includes: configuring a status table and a materialized view, the materialized view storing the correlation results of written stable data and unstable data, and the status table recording completion status identifiers corresponding to write operations on unstable data; the completion status identifier includes at least one of a data partition identifier, a data block identifier, a business date, or a key-value range; in response to a change in the completion status identifier recorded in the status table, parsing the changed completion status identifier to determine the refresh range of the materialized view it indicates, and determining whether the changed completion status identifier is a valid completion status; if so, refreshing the originally stored correlation results within the refresh range to the updated correlation results. Compared to existing technologies, this disclosure can eliminate invalid calculations and reduce the overhead of data updates.

Description

Technical Field

[0001] This disclosure belongs to the field of data processing technology, and specifically relates to a method, system, device, and readable storage medium for updating correlated data. Background Technology

[0002] In hybrid data processing systems, such as RPA-fetched commodity prices and API vendor market information, the writing time of unstable data is random and may be frequently corrected. Traditional processing methods usually adopt timed scheduling or full refresh strategies. Summary of the Invention

[0003] Triggering a complete ETL process or materialized view reconstruction at fixed intervals fails to capture the true completion status of the data source, leading to a large amount of invalid computation. For example, a full refresh is performed even when no new data is being written, or downstream computations are triggered prematurely before the data is fully written. When external data sources need to be re-collected due to errors or omissions, manual intervention or timed waiting is often required, making it impossible to achieve precise linkage between data writing completion and downstream refresh. Furthermore, a full reconstruction of the entire materialized view incurs high computational overhead and response latency, making it unsuitable for scenarios with high-frequency, small-batch external data updates.

[0004] The purpose of this disclosure is to provide a method, system, device, and readable storage medium for updating correlated data, which solves the problems of invalid calculations and high data update overhead in updating correlated data.

[0005] To achieve the above objectives, the technical solution provided in this disclosure is as follows:

[0006] Firstly, this disclosure provides a method for updating related data, which includes:

[0007] Configure a status table and a materialized view. The materialized view stores the association results of written stable data and unstable data. The status table is used to record the completion status identifier corresponding to the write operation of the unstable data. The completion status identifier includes at least one of the following: data partition identifier, data block identifier, business date, or key value range.

[0008] In response to a change in the completion status identifier recorded in the status table, the changed completion status identifier is parsed to determine the refresh range of the materialized view it indicates, and to determine whether the changed completion status identifier is a valid completion status.

[0009] If the changed completion status identifier is a valid completion status, the original stored association results within the refresh range will be refreshed to the updated association results, wherein the updated association results are determined based on the unstable data corresponding to the changed completion status identifier.

[0010] In one or more embodiments of this disclosure, the method further includes:

[0011] Obtain the batch number and / or version number of the written unstable data, and determine whether there is a corresponding completion status identifier in the status table;

[0012] If the changed completion status identifier is a valid completion status, then the corresponding completion status identifier entry in the status table is updated with the written unstable data;

[0013] If the changed completion status identifier is a valid completion status, then a new status entry is added to the status table, and the new status entry in the status table is initialized with the written unstable data.

[0014] In one or more embodiments of this disclosure, the method further includes:

[0015] If the completion status identifier of the change is an invalid completion status, and / or the completion status identifier corresponding to the written unstable data is not received, then the refresh of the materialized view is prevented.

[0016] In one or more embodiments of this disclosure, the method further includes:

[0017] Obtain the batch number and / or version number of the completion status identifier of the change, and determine whether there is a corresponding refreshed record;

[0018] If so, retain the original stored association results within the refresh range;

[0019] If not, refresh the previously stored association results within the refresh range to the updated association results.

[0020] In one or more embodiments of this disclosure, the method further includes:

[0021] Delete each refreshed record after a preset time interval following its creation; and / or,

[0022] Each refreshed record is deleted after a preset number of global refresh operations following its creation, wherein the global refresh is a refresh of the entire range of the materialized view.

[0023] In one or more embodiments of this disclosure, the method further includes:

[0024] Obtain the number of records of unstable data corresponding to the changed completion status identifier; wherein, the number of records of the corresponding unstable data is carried by the status table, or determined based on the status table indexed by the changed completion status identifier;

[0025] If the number of records is less than the preset minimum record count threshold, the current refresh operation is suspended and an alarm message is generated.

[0026] If the number of records is greater than or equal to the preset minimum record number threshold, the original stored association results within the refresh range will be refreshed to the updated association results.

[0027] In one or more embodiments of this disclosure, the method further includes:

[0028] In response to simultaneous changes in multiple completion status identifiers recorded in the status table, the priority identifier carried by each completion status identifier is obtained to determine the refresh order of the corresponding refresh range of each completion status identifier in the materialized view; and / or

[0029] The unstable data is external data, and the stable data is internal data.

[0030] Secondly, this disclosure provides a correlation data update system, which includes:

[0031] The configuration module is used to configure the status table and materialized view. The materialized view stores the association results of written stable data and unstable data. The status table is used to record the completion status identifier corresponding to the write operation of the unstable data. The completion status identifier includes at least one of the following: data partition identifier, data block identifier, business date, or key value range.

[0032] The parsing module is used to respond to a change in the completion status identifier recorded in the status table, parse the changed completion status identifier to determine the refresh range of the materialized view it indicates, and determine whether the changed completion status identifier is a valid completion status.

[0033] The refresh module is used to refresh the previously stored association results within the refresh range to the updated association results, wherein the updated association results are determined based on the unstable data corresponding to the changed completion status identifier.

[0034] Thirdly, this disclosure provides an electronic device comprising:

[0035] At least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the associative data update method.

[0036] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the aforementioned correlation data update method.

[0037] Compared with existing technologies, the correlation data update method provided in this disclosure avoids invalid calculations when there is no data or the data is not ready by using the change of the completion status identifier as the condition for triggering the refresh. At the same time, it precisely limits the scope of the refresh to the affected local area and only calculates the changed part, which significantly shortens the execution time of each refresh, ensures that the materialized view is updated only when the data is complete and valid, and achieves near real-time refresh response, thus achieving an optimal balance between data consistency and timeliness. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram illustrating a use case of the correlation data update method in one embodiment of this disclosure;

[0040] Figure 2 This is a flowchart of a correlation data update method in one embodiment of the present disclosure;

[0041] Figure 3 This is a structural block diagram of a correlation data update system according to another embodiment of this disclosure;

[0042] Figure 4 This is a structural block diagram of an electronic device according to one embodiment of the present disclosure;

[0043] Figure 5 This is a schematic diagram of the database architecture in one embodiment of the present disclosure. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0045] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0046] In existing technical solutions, materialized view updates for related data often employ mechanisms based on scheduled tasks or full refresh triggers. Specifically, the system performs a full reconstruction of the materialized view at fixed time intervals, or performs a full data scan of the base table on which the materialized view depends to determine if changes exist, and performs a comprehensive update when changes are detected.

[0047] Because the externally unstable data write time is completely random and may be duplicated due to resampling, supplementary recording, etc., the scheduled data cannot perceive the actual completion status of data writing. This inevitably leads to calculations being performed even when there is no new data or the data has not been completely written, resulting in a large amount of invalid calculations and resource idleness. When data needs to be corrected after resampling, the next scheduled cycle must be waited for the update to be triggered, resulting in poor timeliness. Furthermore, the full refresh method cannot distinguish the affected local data range, and the entire materialized view needs to be rebuilt for each update. The computational overhead increases linearly with the amount of data, severely limiting the scalability of the system.

[0048] The inventors of this disclosure discovered that the root cause of the aforementioned shortcomings lies in the mismatch between the time-driven mechanism and the data-driven mechanism, namely, the update mechanism cannot perceive the real-time state changes of the data source. Based on this, the inventors propose a novel technical approach that decouples data writing operations from downstream refresh operations. This is achieved by introducing an independent status table to record the completion status, batch information, and affected data range of each write operation, and by changing the materialized view's refresh action from timed polling to signal triggering.

[0049] This approach fundamentally solves the problems of ineffective computation and latency caused by information asymmetry, achieving the ideal effect of write-aware and precise partial refresh. Because existing technologies lack guidance or motivation to associate completion status markers with partial refresh of materialized views, and because this solution introduces additional state tables and active listening mechanisms at the architectural level, those skilled in the art cannot directly derive an incremental refresh scheme driven by completion status markers from traditional timed scheduling or full-scan methods without creative effort.

[0050] Please refer to Figure 1 The diagram illustrates an application scenario of the correlation data update method provided in this disclosure under one embodiment. This scenario specifically includes: a first database for processing stable data and a second database for processing unstable data. Continuing with... Figure 5 As shown, the second database includes at least a storage layer (CC), a module layer (MK), and a bottom table layer (DB).

[0051] In this scenario, the storage layer can receive and store unstable data from external platforms using a unique key model, and append a write status identifier and version number to each data record. In actual business applications, such unstable data can originate from... Figure 5 The storage layer includes external or semi-structured inputs such as "market competition" (e.g., competitor pricing), "platform traffic" (e.g., real-time clickstream), and "manual marking" (e.g., abnormal orders marked by operations personnel). It also maintains a status table to record the completion status of each data write operation, which includes at least a task completion identifier, a timestamp, a data partition identifier, and a batch number.

[0052] The module layer contains at least one materialized view, which predefines the logic for performing association calculations between unstable data in the storage layer and stable data provided by a first database, and physically stores the association results. Figure 5 For example, the stable data in the first database may include "order system" data, "financial revenue / cost" data, etc.; the materialized views in the module layer can be oriented towards specific analysis targets such as "market group analysis," "market performance," and "financial management," and predefined association rules can be used. For example, "platform traffic" can be associated with "order system" for attribution transformation, or "market competition" prices can be associated with "financial costs" to calculate real-time gross profit, etc. This disclosed embodiment does not limit this.

[0053] The bottom layer contains logical views, which are used to further merge and encapsulate the associated results from the module layers, providing a unified query interface. For example, the results of "Market Group Analysis" and "Market Performance" can be merged into a comprehensive competitive dashboard for use by application layers such as "Planned Orders".

[0054] In this scenario, when externally unstable data, such as newly collected competitor prices, is written to the storage layer, the corresponding completion status flag in the status table is updated to "valid completion." The module layer listens for changes in this completion status flag, parses the refresh range carried in the signal, and performs a partial refresh only on the affected portion of the materialized view. This allows the newly written unstable data to be re-associated with the stable data in the first database, resulting in the latest association result.

[0055] Meanwhile, the system performs idempotency checks based on the batch number of the completion status identifier, and avoids repeated refreshing of the same batch of data by checking whether a corresponding refreshed record already exists.

[0056] Through the above architecture, this disclosure achieves efficient and reliable correlation and update between external unstable data and internal stable data, significantly reduces the computational overhead caused by full refresh, and ensures near real-time linkage between data writing completion and downstream view update.

[0057] Please refer to Figure 2 The diagram shown is a flowchart illustrating a method for updating correlated data according to an embodiment of this disclosure. This method specifically includes the following steps:

[0058] S201: Configure a status table and a materialized view. The materialized view stores the association results of written stable data and unstable data. The status table is used to record the completion status identifier corresponding to the writing operation of the unstable data.

[0059] It should be noted that, in this disclosure, stable data refers to data with predictable update patterns that are not prone to duplication or missing data; unstable data refers to data with random update times, large fluctuations in data quality, and the possibility of duplicate data collection or missing fields.

[0060] In one implementation, the data type of the input data can be determined based on the source characteristics of the input data. Specifically, data sources can be classified into high-stability sources and low-stability sources based on physical indicators such as the controllability of the data source, the predictability of the update pattern, and the quality of historical data; then, input data from the corresponding source can be assigned a corresponding data type according to the stability category of the data source.

[0061] For example, taking a real-time data warehouse of an e-commerce platform as an example, the system needs to associate product order data with user behavior log data to generate a materialized view of order summary attribution for user behavior.

[0062] Of the two types of input data, product order data originates from the platform's core transaction database and is written to the data warehouse in fixed batch processing cycles via a reliable data synchronization link. This link is highly controllable; therefore, the transaction database is considered a high-stability source, and the order data written to it is classified as "stable data." User behavior log data originates from a front-end tracking system deployed on multiple content delivery network nodes and is collected in real-time via a message queue over the public internet. Affected by network fluctuations and client limitations, user behavior data often experiences delays in arrival, missing fields, or minor duplication, and its writing frequency varies drastically over time. Therefore, the front-end tracking system is considered a low-stability source, and the user behavior data written to it is classified as "unstable data."

[0063] Based on the above determination results, a status table can be configured during step S201 to record the completion status identifier corresponding to each write operation of the user behavior log. This status table can contain fields such as: log date partition (data partition identifier), event type (data block identifier), business date, batch number, etc.

[0064] In the same configuration step, a materialized view is constructed, which performs association calculations between order data and user behavior logs based on preset matching conditions, and persists the association results. Here, association calculation refers to matching, combining, and aggregating stable and unstable data according to predetermined association keys and join conditions, ultimately forming a materialized view that can be directly used for querying or analysis. In attribution scenarios, association conditions can be defined as follows: the order identifier in the order table is equal to the attribution click identifier in the behavior log, the behavior occurred within a preset window before the order was created, and the order status is "completed," etc. During initial construction, the system can use all historical data to complete a baseline calculation, generating an initial snapshot of the materialized view. Subsequent maintenance relies on the partial refresh in step S203.

[0065] To tightly link completion status identifiers with the physical storage of materialized views, in one implementation, the materialized views employ multi-level partitioning. For example, the first level partitions by business date, the second level partitions by data block identifier, and the third level buckets by key value range. Simultaneously, during the creation of the materialized view, the system records the combination of completion status identifier dimensions corresponding to each physical partition; this mapping relationship can be stored in the system directory. When a completion status identifier changes subsequently, this mapping can directly locate one or more specific physical partitions in the materialized view, thus providing a precise basis for the refresh range resolution in step S202, eliminating the need to re-traverse the associated conditions or scan the entire table with each refresh. This design reduces the computational complexity of the refresh range from being related to the amount of data to being related to the size of the partition metadata, greatly improving decision-making efficiency.

[0066] In another implementation, stability quantification metrics can be used to more precisely measure the degree of data instability. These metrics include, but are not limited to, the jitter rate between actual and expected data update times, the gap rate of data records, the field missing rate, and the duplicate record rate. At the data access layer, the system continuously collects these metrics to form a stability profile for each data stream. After collecting all stability quantification metrics, the corresponding unstable data can be further classified into different stability subcategories, such as mild instability and severe instability, based on the number of metrics exceeding a preset effective range. Alternatively, each stability quantification metric can be normalized and weighted to obtain a stability score for the input data, and the stability of the input data can be evaluated accordingly. When the stability score is below a certain preset threshold, the system can automatically mark it as unstable data and trigger the maintenance logic of the status table. This quantitative judgment method makes the identification of unstable data more objective and adaptive, and can reflect changes in the quality of the data source itself in a timely manner.

[0067] Regarding the creation of materialized views, it should also be noted that the specific logic for relational calculations can be flexibly defined according to business needs. In addition to simple equivalence joins, more complex operations such as time window joins, multi-condition matching, deduplication aggregation, and multi-table joins can also be introduced.

[0068] In one implementation, the definition of a materialized view can be represented as a declarative materialized query statement. After parsing the query statement, the system decomposes it into the stable and unstable data tables it depends on, and automatically registers the corresponding completion status flags. Simultaneously, the materialized view can be configured with various refresh strategies, such as partial incremental refresh and global full refresh. Partial incremental refresh is driven by the completion status flag, while global full refresh can be executed at fixed intervals to correct potential data drift. Thus, S201 configures not only a static storage structure, but also a management framework that integrates data characteristics, driving signals, and view maintenance.

[0069] S202: In response to a change in the completion status identifier recorded in the status table, the changed completion status identifier is parsed to determine the refresh range of the materialized view it indicates, and to determine whether the changed completion status identifier is a valid completion status.

[0070] It should be noted that when a batch of unstable data is written to the system, it will be organized into specific logical storage units according to its inherent business attributes. These business attributes include the business date to which the data belongs, the event type and time period, and the range of values ​​for the associated keys covered by the data. The completion status identifier does not save the actual data content when recording this write operation, but only registers the above-mentioned business attributes as metadata.

[0071] On the other hand, materialized views, as the result storage entity formed after performing correlation calculations on multiple types of data, are also physically partitioned according to business dates. Each row of result data within a materialized view contains association key information such as order creation time and user identifier. Therefore, the content recorded in the completion status identifier can correspond to a specific partition of the materialized view. The data block identifier in the completion status identifier can be used to deduce the time range of affected orders within that partition through association logic, and the key value range in the completion status identifier can be directly used as a filtering condition to select data rows in the materialized view whose associated key values ​​fall within that range. Therefore, by parsing the changed completion status identifier, the scope of the materialized view that needs to be refreshed this time can be determined.

[0072] Furthermore, changes to the completion status identifier in this disclosure can include adding or replacing the completion status identifier. Before responding to a write operation of unstable data and writing the corresponding completion status identifier to the status table, the system can first obtain the batch number or version number carried by the data being written, and use this identifier to search in the status table to determine whether a completion status identifier record corresponding to the same write operation already exists.

[0073] In one implementation, if the status table already contains an entry corresponding to the batch number or version number, it means that there has been a previous record of writing data of the same batch. In this case, the system will not add a new record, but will update the existing completion status identifier entry with the unstable data written this time, such as updating its record number, data block identifier range or status field.

[0074] In another implementation, if there is no corresponding record in the status table, a new status table entry is added to the status table, and the new entry is initialized with the unstable data written this time, and metadata such as batch number or version number, business date, data block identifier, and key value range are completely filled in.

[0075] Based on the two complementary implementation methods described above, even if unstable data in the same batch is written multiple times due to the system retry mechanism or repeated pushes from the application side, the status table always maintains only one completion status identifier record that precisely corresponds to it, without generating redundant status entries. This eliminates the bloat of completion status identifiers and data management chaos caused by repeated writing. On the other hand, it also ensures that subsequent materialized view refreshes triggered based on completion status identifiers will not be executed repeatedly, thereby avoiding the waste of computing resources and the inconsistency problem of materialized view results.

[0076] It should also be noted that after the status table is established, a completion status identifier is inserted or overwritten for each data write operation, and its initial status is marked as writing or other intermediate states. Once the corresponding data write operation is fully completed and passes integrity and quality checks, the data write engine or scheduling system atomically updates the status field of the corresponding record in the status table to a valid completion status. Simultaneously, auxiliary fields such as the record count and timestamp may also be updated. Therefore, in this disclosure, whether the completion status identifier is in a valid completion state indicates whether the corresponding data write operation has been fully completed and passed the corresponding checks.

[0077] Furthermore, the system can capture completion status indicator change events in various ways. For example, a database trigger can be set to actively push a change notification when an update operation occurs in the status table; alternatively, a timed polling component can periodically read the status table, filter out records whose status fields have changed but have not yet been refreshed, and add them to the pending queue. This embodiment of the present disclosure does not limit this approach. Regardless of the mechanism used, the input received in step S202 is one or more completed status indicators that have been changed.

[0078] S203: If the completion status identifier of the change is a valid completion status, then the original stored association results within the refresh range are refreshed to the updated association results, wherein the updated association results are determined based on the unstable data corresponding to the changed completion status identifier.

[0079] It should be noted that this step is used to safely and efficiently refresh the materialized view when a valid completion status is detected. The basic process involves parsing the data partition identifier, data block identifier, business date, or key-value range from the changed completion status identifier. Based on this, the refresh range of the materialized view is defined. The previously stored association results within this range are cleared, and the association calculation is re-executed using the unstable data from the corresponding batch and the current stable data. The newly generated association results are then written to the refresh range, completing the update. This basic mechanism avoids a full reconstruction of the materialized view, reacting only to the affected local area, significantly reducing computational and input / output overhead.

[0080] In one implementation, if the system determines that the completion status indicator of the change is not a valid completion status, for example, the completion status indicator is still in an intermediate state of writing, aborted, or rolled back; or if no completion status indicator corresponding to the write operation is received within a preset time, then step S203 will not be triggered at all, and the materialized view refresh operation will be completely blocked. This design prevents the materialized view from being contaminated by uncommitted or abnormal data, ensuring that the view always reflects a consistent data snapshot.

[0081] For example, in a data warehouse system, stable data consists of internally maintained product dimension tables, while unstable data consists of order detail streams written in near real-time via message queues. A status table records the progress markers for each order stream write task, including data partition identifiers and write status. The system scans the status table at fixed intervals. When a completion status marker changes from "in progress" to "rollback," step S203 parses the signal and determines it to be an invalid completion status, then executes a blocking operation, without generating any refresh task, and the materialized view remains unchanged. In another scenario, if the upstream system fails to insert the corresponding completion status marker for this write into the status table due to a fault, step S203, within a preset detection window, does not find any signal matching this write operation, similarly determining it to be invalid and prohibiting refresh triggering.

[0082] This mechanism completely eliminates materialized view pollution caused by uncommitted or anomalous data, ensuring that the materialized view only reflects a consistent snapshot of successfully committed data.

[0083] Furthermore, step S203 of this disclosure can also enhance robustness by incorporating multiple protection measures.

[0084] In one implementation, to prevent the same batch of unstable data from being refreshed multiple times due to upstream replay, supplementary recording, or duplicate message delivery, step S203 extracts the batch number or version number carried by the completion status identifier before determining the refresh and searches for it in the refreshed record set. When a refreshed record matching the batch number or version number is found, it indicates that the corresponding data has already completed the association result refresh, and the system skips this operation directly, retaining the original stored association result as is. Only when no matching refreshed record is found is the replacement writing of the association result actually performed, and the current batch number or version number is recorded in the refreshed record. This anti-duplicate logic can prevent duplicate calculations while maintaining the idempotency of the refresh operation.

[0085] To achieve efficient deduplication under high concurrency, the storage of refreshed records can leverage a distributed database or key-value store system that supports atomic insertion and unique constraints. Queries and insertions can be combined into a single atomic operation. For example, by utilizing the semantics of inserting if the record does not exist, a refresh is performed when the insertion is successful, and skipped when a conflict is returned, thus avoiding race windows between queries and insertions.

[0086] Furthermore, the lifecycle of refreshed records also needs to be managed; otherwise, the record set will grow indefinitely and reduce the efficiency of each query match. Step S203 may include at least one of the following cleanup strategies.

[0087] First, each refreshed record is automatically deleted after a preset period of time from its creation. This strategy cleans up records caused by repeated write requests in the short term, keeping the table size within a reasonable range in the near term. Second, each refreshed record is also deleted if it undergoes a preset number of global refresh operations after its creation. Here, a global refresh refers to a full refresh of the materialized view. This counting baseline mechanism eliminates useless historical records due to a lack of repeated writes for a long time, while ensuring that records are promptly cleared even with frequent global refreshes.

[0088] In another implementation, step S203 performs a reasonableness check on the amount of data written upstream before deciding to execute the refresh. It obtains the number of unstable data records corresponding to the change completion status identifier and compares it with a preset minimum record count threshold. If the number of records is less than the threshold, it indicates a possible upstream data extraction anomaly, missing files, or logical errors. In this case, the refresh is not executed, the current refresh operation is suspended, and an alarm message is output to notify relevant personnel to intervene and investigate. The refresh is only executed when the number of records reaches or exceeds the minimum record count threshold. This gating mechanism effectively avoids erroneous correlation results or data collapse due to insufficient data, protecting the accuracy of downstream business operations.

[0089] It should be noted that when multiple completion status identifiers change simultaneously within the same time window in the status table, step S203 also needs to coordinate the execution order of these refresh tasks. Specifically, the priority identifier carried by each completion status identifier can be read. The priority identifier is appended by the upstream data production system according to the business importance when submitting the write operation.

[0090] For example, real-time transaction data can be identified as high priority, while offline supplementary data is identified as normal priority. Based on the priority identification, step S203 constructs an ordered refresh task queue, with higher priority refresh ranges placed at the head of the queue and scheduled for execution first. Under the same priority, the sorting can be further determined according to the order of batch numbers or the size of the data.

[0091] In scenarios such as content distribution or recommendation systems, this ensures that the behavioral data of high-value users can be reflected in the materialized view earlier, thereby influencing the downstream recommendation model updates. If a higher-priority signal is inserted during the refresh process, the system can also support limited preemption or dynamic insertion at the head of the queue to further improve timeliness.

[0092] Please refer to Figure 3 As shown, based on the same inventive concept as the aforementioned correlation data update method, one embodiment of this disclosure provides a correlation data update system 300, including: a configuration module 301, a parsing module 302, and a refresh module 303.

[0093] Specifically, the configuration module 301 is used to configure the status table and the materialized view. The materialized view stores the association results of written stable data and unstable data. The status table is used to record the completion status identifier corresponding to the writing operation of the unstable data. The completion status identifier includes at least one of the following: data partition identifier, data block identifier, business date, or key value range.

[0094] The parsing module 302 is used to respond to a change in the completion status identifier recorded in the status table, parse the changed completion status identifier to determine the refresh range of the materialized view it indicates, and determine whether the changed completion status identifier is a valid completion status.

[0095] The refresh module 303 is used to refresh the original stored association results within the refresh range to the updated association results, wherein the updated association results are determined based on the unstable data corresponding to the changed completion status identifier.

[0096] Please refer to Figure 4 As shown, this disclosure also provides an electronic device 400, which includes at least one processor 401, a memory 402 (e.g., non-volatile memory), a main memory 403, and a communication interface 404, and the at least one processor 401, memory 402, main memory 403, and communication interface 404 are connected together via an internal bus 405. The at least one processor 401 is configured to invoke at least one program instruction stored or encoded in the memory 402 to cause the at least one processor 401 to perform various operations and functions of the correlated data update method described in the various embodiments of this specification.

[0097] In the embodiments of this specification, electronic device 400 may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile electronic device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable electronic device, consumer electronic device, etc.

[0098] This disclosure also provides a computer-readable medium carrying computer-executable instructions, which, when executed by a processor, can be used to implement various operations and functions of the correlation data update methods described in the various embodiments of this specification.

[0099] The computer-readable medium in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0100] In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0101] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, systems, and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable associative data update device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable associative data update device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] The foregoing description of specific exemplary embodiments of this disclosure is for illustrative and explanatory purposes. These descriptions are not intended to limit this disclosure to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of this disclosure and their practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of this disclosure, as well as various different choices and variations. The scope of this disclosure is intended to be defined by the claims and their equivalents.

[0104] It will be apparent to those skilled in the art that this disclosure is not limited to the details of the exemplary embodiments described above, and that this disclosure can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of this disclosure is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this disclosure. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0105] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for updating correlated data, characterized in that, include: Configure a status table and a materialized view. The materialized view stores the association results of written stable data and unstable data. The status table is used to record the completion status identifier corresponding to the write operation of the unstable data. The completion status identifier includes at least one of the following: data partition identifier, data block identifier, business date, or key value range. In response to a change in the completion status identifier recorded in the status table, the changed completion status identifier is parsed to determine the refresh range of the materialized view it indicates, and to determine whether the changed completion status identifier is a valid completion status. If the changed completion status identifier is a valid completion status, the original stored association results within the refresh range will be refreshed to the updated association results, wherein the updated association results are determined based on the unstable data corresponding to the changed completion status identifier.

2. The correlation data update method according to claim 1, characterized in that, The method further includes: Obtain the batch number and / or version number of the written unstable data, and determine whether there is a corresponding completion status identifier in the status table; If the changed completion status identifier is a valid completion status, then the corresponding completion status identifier entry in the status table is updated with the written unstable data; If the changed completion status identifier is a valid completion status, then a new status entry is added to the status table, and the new status entry in the status table is initialized with the written unstable data.

3. The method for updating correlated data according to claim 1, characterized in that, The method further includes: If the completion status identifier of the change is an invalid completion status, and / or the completion status identifier corresponding to the written unstable data is not received, then the refresh of the materialized view is prevented.

4. The correlation data update method according to claim 1, characterized in that, The method further includes: Obtain the batch number and / or version number of the completion status identifier of the change, and determine whether there is a corresponding refreshed record; If so, retain the original stored association results within the refresh range; If not, refresh the previously stored association results within the refresh range to the updated association results.

5. The correlation data update method according to claim 4, characterized in that, The method further includes: Delete each refreshed record after a preset time interval following its creation; and / or, Each refreshed record is deleted after a preset number of global refresh operations following its creation, wherein the global refresh is a refresh of the entire range of the materialized view.

6. The method for updating correlated data according to claim 1, characterized in that, The method further includes: Obtain the number of records of unstable data corresponding to the changed completion status identifier; wherein, the number of records of the corresponding unstable data is carried by the status table, or determined based on the status table indexed by the changed completion status identifier; If the number of records is less than the preset minimum record count threshold, the current refresh operation is suspended and an alarm message is generated. If the number of records is greater than or equal to the preset minimum record number threshold, the original stored association results within the refresh range will be refreshed to the updated association results.

7. The method for updating correlated data according to claim 1, characterized in that, The method further includes: In response to simultaneous changes in multiple completion status identifiers recorded in the status table, the priority identifier carried by each completion status identifier is obtained to determine the refresh order of the corresponding refresh range of each completion status identifier in the materialized view; and / or The unstable data is external data, and the stable data is internal data.

8. A correlation data update system, characterized in that, include: The configuration module is used to configure the status table and materialized view. The materialized view stores the association results of written stable data and unstable data. The status table is used to record the completion status identifier corresponding to the write operation of the unstable data. The completion status identifier includes at least one of the following: data partition identifier, data block identifier, business date, or key value range. The parsing module is used to respond to a change in the completion status identifier recorded in the status table, parse the changed completion status identifier to determine the refresh range of the materialized view it indicates, and determine whether the changed completion status identifier is a valid completion status. The refresh module is used to refresh the previously stored association results within the refresh range to the updated association results, wherein the updated association results are determined based on the unstable data corresponding to the changed completion status identifier.

9. An electronic device, characterized in that, include: At least one processor; And a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the associative data update method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the correlation data update method according to any one of claims 1-7.

Images