Data processing method and device, storage medium and computer program product

Abstract

The invention discloses a data processing method and device, a storage medium and a computer program product. The method comprises the following steps: acquiring a data query request; a target materialized view of a target relation corresponding to the data query request is determined, a segment-level fragmentation abstract corresponding to the target materialized view is obtained, the target materialized view comprises at least one data identifier under the target relation, the at least one data identifier is stored in the target materialized view in a segmented mode, and a segment-level fragmentation abstract corresponding to the segment-level fragmentation abstract is obtained; each segment corresponds to one segment-level fragmentation abstract; determining a candidate fragment set corresponding to the data query request according to the segment-level fragment abstract; and querying each fragment in the candidate fragment set to obtain a data content query result corresponding to the data query request. According to the method and the device, the technical problem of high data query delay caused by undifferentiated access of a query strategy to fragments after a bottom-layer main table is horizontally fragmented in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data processing method, apparatus, storage medium, and computer program product. Background Technology

[0002] With the development of large-scale interactive communities, social platforms, and content distribution systems, a common data access pattern involves querying a series of IDs through materialized views of relationships, and then using these IDs to access records in the main table in batches. For example, a user might request to browse the recent activities of a specific circle or topic. This requires first obtaining a list of activity IDs related to that circle or topic, and then reading the content of these activities in batches. To improve query efficiency, relationship mappings (such as circle / topic / user lists, etc.) are usually maintained as materialized views or cached, while the main table is horizontally partitioned or sharded according to certain rules to enhance write scalability.

[0003] However, when the main table is horizontally sharded, the list of IDs returned by the relational materialized view is often scattered across multiple shards. Query methods in related technologies, especially without any optimization, typically require traversing all possible shards to find the required ID, which not only increases query latency but can also lead to high I / O consumption.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a data processing method, apparatus, storage medium, and computer program product to at least solve the technical problem in the related art where, after the underlying master table is horizontally sharded, the query strategy accesses the shards indiscriminately, resulting in high data query latency.

[0006] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: obtaining a data query request, wherein the data query request is used to request viewing data content in data corresponding to a target relationship that meets target query conditions, the target relationship indicating that the data belongs to the same user group or a topic; determining a target materialized view of the target relationship corresponding to the data query request, and obtaining a segment-level fragment summary corresponding to the target materialized view, wherein the target materialized view contains at least one data identifier under the target relationship, different data identifiers are used to identify different data content, the at least one data identifier is stored in segments in the target materialized view, each segment corresponds to a segment-level fragment summary, the segment-level fragment summary is used to characterize the fragments in which the data content corresponding to the data identifier in the segment is distributed; determining a candidate fragment set corresponding to the data query request based on the segment-level fragment summary, wherein the candidate fragment set is a set of fragments storing data content that meets the target query conditions; querying each fragment in the candidate fragment set to obtain a data content query result corresponding to the data query request.

[0007] Optionally, determining the candidate shard set corresponding to the data query request based on the segment-level shard summary includes: filtering out segments in the target materialized view that meet the target query conditions; for each filtered segment, determining the shard containing the data content corresponding to the data identifier in the segment based on the segment-level shard summary corresponding to the segment, and adding the shard to the candidate shard set; obtaining the shard mapping index corresponding to the target relationship, and verifying each shard in the candidate shard set based on the shard mapping index to determine whether the shard actually contains the data content corresponding to the data identifier in the segment, wherein the shard mapping index is used to characterize the correspondence between the data identifier and the shard; if it is determined that the shard does not contain the data content corresponding to the data identifier in the segment, the shard is removed from the candidate shard set to obtain the final candidate shard set.

[0008] Optionally, the method further includes: when new data content is detected being written to a shard under the target relation, recording the data identifier corresponding to the data content, the shard identifier of the shard, and the timestamp of the data writing; and updating the correspondence between the data identifier and the shard identifier, as well as the timestamp, to the shard mapping index corresponding to the target relation.

[0009] Optionally, the method further includes: obtaining the original materialized view of the target relationship, and segmenting the data identifiers in the original materialized view to obtain multiple segments, wherein different segments correspond to different time windows and / or data scores, wherein the data scores are used to characterize the importance of the data; determining the segment attribute information corresponding to each segment to obtain the target materialized view, wherein the segment attribute information includes: segment-level fragment summary, segment identifier, segment version number, and compressed sequence of data identifiers.

[0010] Optionally, segmenting the data identifiers in the original materialized view includes: segmenting the data identifiers in the original materialized view according to the time window corresponding to the data identifier, wherein the time window is used to characterize the period length range of the data existence; determining the data score corresponding to the data identifier based on the data validity information of the data content, wherein the data validity information is used to characterize the popularity and timeliness of the data content; and sorting or further segmenting the data identifiers according to the data score within each time window.

[0011] Optionally, the method further includes: identifying the abnormal segments when the data content query result is missing and / or the segment version number of the segment is less than the version number of the shard mapping index; determining the shards corresponding to each data identifier in the abnormal segment based on the shard mapping index to obtain a minimum shard set; performing a batch scan on the shards in the minimum shard set to obtain the repaired segments; replacing the abnormal segments with the repaired segments; and updating the segment-level shard summary and segment version number of the repaired segments.

[0012] Optionally, the method further includes: pre-storing a preset number of snapshots of data content with the highest data scores in the segments, wherein the data scores are used to characterize the importance of the data; when a data query request is obtained, aggregating snapshots in the segments that meet the target query conditions in the target materialized view, and querying the data content corresponding to the data query request in all the aggregated snapshots to obtain initial query results; if the initial query results meet the query requirements of the data query request, directly returning the initial query results to the user; and if the initial query results do not meet the query requirements, further querying is performed in the segments.

[0013] According to another aspect of the embodiments of this application, a data processing apparatus is also provided, comprising: a query request acquisition module, configured to acquire a data query request, wherein the data query request is used to request to view data content in data corresponding to a target relationship that meets the target query conditions, the target relationship indicating that the data belongs to the same user group or a topic; a fragment summary determination module, configured to determine a target materialized view of the target relationship corresponding to the data query request, and acquire a segment-level fragment summary corresponding to the target materialized view, wherein the target materialized view contains at least one data identifier under the target relationship, different data identifiers are used to identify different data content, at least one data identifier is stored in segments in the target materialized view, each segment corresponds to a segment-level fragment summary, the segment-level fragment summary is used to characterize the fragments in which the data content corresponding to the data identifier in the segment is distributed; a candidate fragment filtering module, configured to determine a candidate fragment set corresponding to the data query request based on the segment-level fragment summary, wherein the candidate fragment set is a set of fragments storing data content that meets the target query conditions; and a fragment access query module, configured to query each fragment in the candidate fragment set to obtain a data content query result corresponding to the data query request.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes a data processing method by running the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of a data processing method.

[0016] In this embodiment, a data query request is used to request and view data content that meets the target query conditions within the data corresponding to the target relationship. The target relationship indicates that the data belongs to the same user group or topic. A target materialized view of the target relationship corresponding to the data query request is determined, and a segment-level fragment summary corresponding to the target materialized view is obtained. The target materialized view contains at least one data identifier under the target relationship. Different data identifiers are used to identify different data content. At least one data identifier is stored in segments within the target materialized view, and each segment corresponds to a segment-level fragment summary. The segment-level fragment summary is used... This approach identifies the shards where the data content corresponding to the data identifier in a segment is distributed. Based on the segment-level shard summary, a candidate shard set corresponding to the data query request is determined. The candidate shard set is a set of shards that store data content that meets the target query conditions. The query results corresponding to the data query request are obtained by querying each shard in the candidate shard set. By introducing the segment-level shard summary, the number of shards accessed during the query is effectively reduced. This solves the technical problem in related technologies where the query strategy accesses shards indiscriminately after the underlying main table is horizontally sharded, resulting in high data query latency. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a data processing method according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of a data processing method flow according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the architecture of a table-aware ID list materialized view maintenance and minimum fragmentation access method provided in an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of a data processing device provided according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] On the one hand, when the main table is horizontally sharded, the list of IDs returned by the relational materialized view is often scattered across multiple shards. The query methods in related technologies, especially without any optimization measures, typically require traversing all possible shards to find the required ID, which not only increases query latency but may also lead to high I / O consumption.

[0025] On the other hand, due to dynamic data changes, inconsistencies may arise between relational materialized views and underlying shards. When inconsistencies or data misses are detected, a full refill of the materialized view or a scan of numerous shards is required to correct them. This approach is not only time-consuming but also significantly impacts system performance in high-concurrency scenarios, leading to a degraded user experience. Furthermore, the lack of effective version control and consistency checking mechanisms makes maintaining data consistency and handling anomalies in large-scale systems complex and uncontrollable.

[0026] To address the aforementioned issues, this application provides relevant solutions that minimize the actual number of accessed shards during queries, significantly reducing average latency and tail latency; asynchronously backfilling with minimal repair costs and maintaining system availability when inconsistencies or misses are detected; and balance memory costs, scalability, and engineering operability, supporting auditing and playback. These solutions are described in detail below.

[0027] According to an embodiment of this application, a method embodiment for data processing is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware structure block diagram of a computer terminal (or electronic device) for implementing a data processing method is shown. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or electronic device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0031] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0032] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or electronic device).

[0033] Under the above operating environment, this application provides a data processing method. Figure 2 This is a schematic diagram of a data processing method flow provided according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0034] Step S202: Obtain a data query request, wherein the data query request is used to request to view the data content that meets the target query conditions in the data corresponding to the target relationship. The target relationship means that the data belongs to the same user group or the same topic.

[0035] Step S204: Determine the target materialized view of the target relationship corresponding to the data query request, and obtain the segment-level fragment summary corresponding to the target materialized view. The target materialized view contains at least one data identifier under the target relationship. Different data identifiers are used to identify different data contents. At least one data identifier is stored in segments in the target materialized view. Each segment corresponds to a segment-level fragment summary. The segment-level fragment summary is used to characterize the fragments to which the data contents corresponding to the data identifiers in the segment are distributed.

[0036] Step S206: Based on the segment-level shard summary, determine the candidate shard set corresponding to the data query request, wherein the candidate shard set is a set of shards that store data content that meets the target query conditions;

[0037] Step S208: Query each shard in the candidate shard set to obtain the data content query results corresponding to the data query request.

[0038] Through the above steps and efficient data access strategies, the number of accesses to the underlying database is reduced, the database load is lowered, and the overall resource utilization efficiency is improved. This solves the technical problem in related technologies where, when the underlying main table is horizontally sharded, the query strategy accesses all shards indiscriminately, resulting in high data query latency.

[0039] The data processing method in steps S202 to S208 of the embodiments of this application will be further described below.

[0040] The overall architecture of the data query process is as follows: Figure 3 As shown, after the client initiates a data query request, the request is routed to the materialized view service; the materialized view service then sends a query request to the sharded storage cluster; the sharded storage cluster returns the query results. This embodiment introduces an innovative indexing and segmentation mechanism into the above process, combined with version control and asynchronous repair tasks, which can effectively solve the data access optimization problem under a horizontally partitioned architecture in a distributed database. It not only significantly improves query performance but also considers cost, scalability, and engineering operability, making it suitable for large-scale online service scenarios. A detailed introduction follows.

[0041] In this embodiment of the application, the materialized view is maintained according to the target relationship (such as circle or topic). The list of data identifiers (IDs) under a target relationship is stored in segments in the target materialized view (Segmented-ID-list), as follows.

[0042] In some embodiments of this application, the method further includes the following steps: obtaining an original materialized view of the target relationship, and segmenting the data identifiers in the original materialized view to obtain multiple segments, wherein different segments correspond to different time windows and / or data scores, wherein the data scores are used to characterize the importance of the data; determining the segment attribute information corresponding to each segment to obtain the target materialized view, wherein the segment attribute information includes: segment-level fragment summary, segment identifier, segment version number, and compressed sequence of data identifiers.

[0043] Specifically, a data identifier (ID) under a target relation can be divided into several segments according to a time window and / or data score. The segment attribute information corresponding to each segment can include the following fields: segment_id (segment identifier), segment_version (segment version number), segment_shard_summary (segment-level shard summary), id_list_compressed (compressed sequence of data identifiers), and top_k_snapshot (snapshot of the best k data contents) (optional). Among them, id_list_compressed is the compressed sequence of data identifiers (IDs) (which can be varint or binary encoding); segment_shard_summary (segment-level shard summary) is used to indicate the probability that the data identifiers (IDs) of this segment are distributed to different shards. For example, in a circle of "basketball enthusiasts", the segment attribute information corresponding to a certain segment after the data identifier (ID) is stored in segments may be as follows: Segment 1: {seg_id:seg_20251113_10_11h,segment_version:v1,shard_summary:[shard_1,shard_2],id_list:[12345,12346]}.

[0044] In this embodiment, the segment_shard_summary can be implemented using compact data structures, such as Bloom filters, bitmaps, or inverted substructures, to efficiently represent the probability of IDs within a segment being distributed across various shards. This allows for quick filtering of candidate shards during queries, reducing unnecessary network transmission and data loading, and lowering access latency.

[0045] Specifically, 1) If a Bloom filter is used to implement segment-level shard digests: For n = 10000 IDs within a segment, with an expected false positive rate p = 0.01, the number of bits m in the Bloom filter and the number of hash functions k can be calculated using the formulas: m = -n ln p / (ln2)^2, k = (m / n)ln2. Example estimation: m ≈ 95,800 bits, k ≈ 7; 2) If a bitmap is used: When the number of shards N is small (e.g., N ≤ 1024), an N-bit bitmap can be maintained for each segment, with acceptable memory usage and no false positives; 3) If an inverted substructure is used: In some scenarios, an inverted substructure (shard_id -> id_list offset) can be maintained for popular segments to speed up access, but its maintenance cost is high.

[0046] The strategy of segmenting data identifiers (IDs) based on time windows and / or data scores is described below.

[0047] In some embodiments of this application, segmenting data identifiers in the original materialized view includes the following steps: segmenting the data identifiers in the original materialized view according to the time window corresponding to the data identifier, wherein the time window is used to characterize the period length range of the data existence; determining the data score corresponding to the data identifier based on the data validity information of the data content corresponding to the data identifier, wherein the data validity information is used to characterize the popularity and timeliness of the data content; and sorting or further segmenting the data identifiers according to the data score within each time window.

[0048] Specifically, in this embodiment, data identifiers (IDs) can first be coarsely segmented by time window (to ensure temporal relevance and facilitate the rapid elimination of old data); then, within each time period, fine-grained organization can be performed by score (to facilitate the rapid return of high-value candidates). That is, segment by time first, and then sort / bucket within each segment by score.

[0049] For example, when segmenting using time windows (which can be either fixed or sliding), the time windows can include: real-time window (0–1 hour, supporting strong real-time scenarios), short-term window (1 hour–24 hours), mid-term window (1 day–7 days), and long-term window (>7 days, or archived to cold storage), etc. Each circle / topic maintains several segments, and each segment corresponds to a time window (or time slice). The segment_id contains the time interval metadata (e.g., seg_20250918_00_01h).

[0050] Furthermore, within each time period, data identifiers can be sorted by predefined data scores (e.g., scores based on popularity, user interaction frequency, etc.), and further categorized by score into buckets (e.g., high, medium, and low tiers), or by percentile (top 1%, 1–5%, 5–20%, and the rest). The formula for calculating the data score is: score(t) = Σw_i * action_i * exp(-(now - t_action) / τ), where w_i is the weighting coefficient, action_i represents user behavior (e.g., likes, comments), and τ` is a time decay constant (e.g., τ = 24h or can be set according to different behavior types).

[0051] In subsequent data queries, the system can start from the latest time window, prioritizing the retrieval of high-score candidates within that window. If insufficient, it can then explore short-term / medium-term windows, thus balancing real-time performance with long-term value. Through this segmented processing based on time and data scores, the system can achieve efficient access and management of data, reduce unnecessary data retrieval, improve query performance and system resource utilization. Furthermore, this segmentation strategy supports dynamic data updates and maintenance, ensuring the accuracy and timeliness of materialized views, thereby enhancing the user experience of online services and the overall stability of the system.

[0052] On the other hand, this application embodiment also introduces a shard-index to maintain the mapping between the entity primary key ID (object_id) and the shard ID (shard_id) (and version / timestamp). With the help of the shard-index, the shard that holds the data content corresponding to the ID can be located during the query. Furthermore, the shard-index can be incrementally maintained and persisted. The maintenance process of the shard-index is as follows.

[0053] In some embodiments of this application, the method further includes the following steps: when new data content is detected to be written to a shard under the target relationship, the data identifier corresponding to the data content, the shard identifier of the shard, and the timestamp of the data writing are recorded; the correspondence between the data identifier and the shard identifier, as well as the timestamp, are updated to the shard mapping index corresponding to the target relationship.

[0054] Specifically, when new data content is detected being written to a shard under a target relationship, the system records the data identifier corresponding to the data content, the shard identifier of the shard, and the timestamp of the data write. For example, the write shard layer (the logic for writing to the dynamic master table) sends an event message to the event bus after each write operation: {object_id (i.e., data identifier), shard_id (i.e., shard identifier), logical_ts (i.e., timestamp)}. The shard-index service consumes this event and incrementally updates the local shard mapping index (which can exist in a RocksDB or MySQL small table with an LRU cache) to maintain the consistency between the index and the data distribution.

[0055] By maintaining sharding information and version control in the index in real time, the system can effectively cope with changes in data distribution and ensure the accuracy and timeliness of query results.

[0056] Based on the aforementioned shard-index and segment_shard_summary, this embodiment of the application can calculate the actual set of candidate shards S to be accessed during the data query process, and initiate parallel batch point queries only on S, thereby achieving minimum shard access query, as detailed below.

[0057] In some embodiments of this application, determining the candidate shard set corresponding to a data query request based on the segment-level shard summary includes the following steps: filtering out segments in the target materialized view that meet the target query conditions; for each filtered segment, determining the shard containing the data content corresponding to the data identifier in the segment based on the segment-level shard summary corresponding to the segment, and adding the shard to the candidate shard set; obtaining the shard mapping index corresponding to the target relationship, and verifying each shard in the candidate shard set based on the shard mapping index to determine whether the shard actually contains the data content corresponding to the data identifier in the segment, wherein the shard mapping index is used to characterize the correspondence between the data identifier and the shard; if it is determined that the shard does not contain the data content corresponding to the data identifier in the segment, removing the shard from the candidate shard set to obtain the final candidate shard set.

[0058] Specifically, the actual storage of each segment can be initially determined using segment-level fragment digests, thereby constructing a candidate fragment set. Subsequently, by obtaining and verifying the fragment mapping index corresponding to the target relationship, it can be ensured that each fragment in the candidate fragment set does indeed contain the data content corresponding to the required data identifier.

[0059] For example, in a social network application, when a data query request is received to obtain the first 50 user dynamics followed by a specific user, the segments in the materialized view of the user's follow relationship that match the time range can be filtered out first; for each selected segment, segment-level shard summaries, such as Bloom filters or bitmaps, are used to determine the shards that may store the IDs of the required dynamics, and these shards are added to the candidate shard set (there may be false positives); then, the shard mapping index corresponding to the user dynamics is obtained, and based on this index, the shards in the candidate shard set are verified one by one to check whether they exactly contain the actual dynamic content corresponding to the dynamic ID in the required segment. If it is found that a certain shard does not actually contain the required dynamic content, the shard is immediately removed from the candidate shard set. After this process, a smallest candidate shard set that truly needs to be accessed is finally determined to process the original data query request.

[0060] After obtaining the final candidate shard set S (usually |S| << N), the materialized view service can initiate a batch query (select id in (...) or mget) in parallel to each shard in S, aggregate the results, sort them by score, and return them.

[0061] In the above query process, the candidate shard set that may contain the required data is quickly filtered out through segment-level shard summaries, avoiding the situation of blindly scanning all shards and significantly improving the query speed. Secondly, the use of shard-index enables the system to accurately locate the specific shard where each data identifier is located, avoiding excessive network communication and data transmission, and further reducing the query latency.

[0062] In addition, in the paging / top-k scenario, a bounded top-k approximate merge strategy can also be adopted to avoid accessing all shards, as follows.

[0063] In some embodiments of the present application, the method further includes the following steps: pre-saving snapshots of a preset number of data contents with the highest data scores in the segment, where the data score is used to represent the importance of the data; when a data query request is obtained, aggregating the snapshots in the segments that meet the target query conditions in the target materialized view, and querying the data content corresponding to the data query request in all the aggregated snapshots to obtain an initial query result; when the initial query result meets the query requirements of the data query request, directly returning the initial query result to the user, and when the initial query result does not meet the query requirements, further querying in the shards.

[0064] Specifically, to avoid accessing all shards, a top-M snapshot (e.g., top-200) can be pre-stored for each segment. This means pre-storing a preset number of snapshots of data content with the highest data scores. During a query, the top-M results from each segment are aggregated first, and the top-K results (initial query results) are returned. If the initial query results can fully meet the query requirements, the system will directly return this data to the user. Conversely, if the initial query results are insufficient to meet the query conditions (insufficient top-K or the completeness flag is false), the system will initiate a further query process, performing a detailed search of the shards to supplement or correct the initial results, ensuring the completeness and accuracy of the returned data.

[0065] In this embodiment of the application, the above-mentioned top-M snapshot is a snapshot of the most valuable M data contents of each segment (for fast bounded merging). Its update strategy can adopt a hybrid triggering strategy (event-driven + periodic reconstruction), and the expiration strategy can adopt a TTL based on time window and version number + hot-driven refresh, and atomic replacement to ensure consistency, as follows.

[0066] The update strategy for top-M snapshots includes at least one of the following:

[0067] 1) Event-Driven Updates (Instant Insertion / Replacement): When there is a new dynamic write or the score of a dynamic data item increases beyond a threshold (e.g., new score > current_min_topM), a fast comparison is performed. If score_new > min(topM), the item is inserted into the top-M, the lowest-scoring item is removed, and the top-M is replaced with an atomic operation (Redis Lua or CAS). This strategy is suitable for scenarios with sudden hot topics (new trending articles, sudden increases in rankings), ensuring that the top-M promptly includes new hot topics. Simultaneously, to avoid high write amplification caused by a large number of updates in a short period, event-driven updates can protect system stability through rate limiters (e.g., maximum number of updates per second) or merge windows (e.g., merging multiple insertions within 1 second).

[0068] 2) Periodic Reconstruction (Batch Refresh): This employs timed batch reconstruction (the cycle depends on the segment's popularity), which can include: Hot segments (frequent access): reconstruct a complete top-M every 1 minute or 5 minutes (ensuring accuracy); Medium segments: reconstruct every 10-60 minutes; Cold segments (low access): reconstruct hourly or daily, or even lazy reconstruction (request-triggered). The advantage of periodic reconstruction is that it corrects potential errors caused by event-driven processes and cleans up the top-M data "corroded" by long-tail behavior.

[0069] The above event-driven strategy can ensure extreme real-time performance, and the periodic reconstruction strategy can ensure statistical correctness and long-term consistency. The combination of the two can achieve a balance between low latency and high correctness.

[0070] The expiration and invalidation strategies for top-M snapshots include at least one of the following:

[0071] 1) Expiration based on time window (default TTL): The top-M of each segment inherits the time window lifecycle of that segment. For example, the top-M TTL of the real-time segment is set to be short (e.g., 1h), and the TTL of the medium-term segment is longer. When the TTL is exceeded, it is automatically deleted or moved to the cold table. This avoids meaningless long-term memory occupation and is consistent with the time segmentation strategy.

[0072] 2) Consistent expiration based on version number: The top-M and the segment have version numbers (segment_version); when the underlying write or shard-index change causes the segment_version to increase, the old top-M is marked as stale and triggers reconstruction. Specifically, when querying, if it is found that topM.version < shard_index_version, asynchronous reconstruction can be immediately triggered and "may be incomplete" can be marked when returning to the client.

[0073] 3) Adaptive freshness preservation based on popularity: If the popularity of a segment (request frequency / hit rate) exceeds the threshold, the reconstruction frequency and TTL of the top-M of that segment are extended (refreshed more frequently); if the popularity decreases, the refresh frequency is reduced until lazy cleaning, so as to ensure that resources are preferentially used for hot segments.

[0074] For the above write / replacement operations of top-M snapshots, atomic operations can be used. For example, in Redis, Lua scripts can be used for compare-and-swap; in the database, transactions or compare-and-swap can be used. To avoid write conflicts, periodic reconstruction or event writing can both write to the audit / version log, compare versions before replacement and use CAS, retry / rollback when unsuccessful and record the audit.

[0075] In addition, during the above query process, the embodiments of the present application can also use a version management mechanism controlled by logical timestamps to timely discover and repair consistency problems caused by data updates. For example, if the returned result is missing or there is a version conflict between segment_version and the version in shard-index, it is marked and a repair task is generated to maintain high query availability and data consistency, as follows.

[0076] In some embodiments of this application, the method further includes the following steps: when the content of the data content query result is missing and / or the segment version number of the segment is less than the version number of the shard mapping index, identify the abnormal segment; based on the shard mapping index, determine the shards corresponding to each data identifier in the abnormal segment to obtain a minimum shard set; perform batch scanning on the shards in the minimum shard set to obtain the repaired segment; replace the abnormal segment with the repaired segment, and update the segment-level shard summary and segment version number of the repaired segment.

[0077] Specifically, during the query process, if the number of misses for a segment (i.e. the number of missing data identifiers in the query results) exceeds a preset threshold, it is considered that the segment may have a problem (such as data loss or inconsistency); or, if the version number of the segment (segment_version) is less than the version number (shard_index_version) recorded in the Shard-index, it is considered that the segment may have expired or be partially missing.

[0078] In the above situation, the system can automatically identify and determine the segment that has an anomaly. The materialized view service generates a repair task and writes it to the message bus. The task includes fields such as: task_id (task identifier), affected_segment_id (affected segment ID, i.e., the segment identifier of the segment that has an anomaly), affected_id_list or id range summary (a list of affected data identifiers, or a summary of the range of data identifiers), origin_version (the original version number of the segment), and priority (task priority).

[0079] Then, the backend worker pulls repair tasks from the task queue and determines the repair order according to task priority and segment popularity; the worker uses shard-index to locate the minimum set of shards S_rep that only need to be accessed, and initiates batch pull or scanning of these shards (there is a rate limit during scanning); aggregates the list of repaired segment IDs (i.e., the repaired segments), and constructs a new segment_shard_summary and segment_version.

[0080] Then, atomic replacement (RedisLua / compare-and-swap / DB transaction) can be used to perform an atomic write-back, replacing the original abnormal segment and recording an audit log for replay, rollback, and investigation. Specifically, the Worker can use a Redis Lua script or database transaction to perform the atomic replacement operation, replacing the old segment with the new one and updating the version; at the same time, an audit log is written to record the details of this repair (including the affected ID, the list of accessed shards, and the timestamp). This audit log supports subsequent replay or rollback operations, facilitating problem localization and compliance auditing.

[0081] The above operations minimize maintenance costs without affecting normal system operation, ensure the consistency and completeness of query results, and significantly reduce latency and IO peaks, thereby improving system stability and response speed. Furthermore, by recording the repair process through audit logs, the system also possesses the ability to locate problems and conduct compliance audits, enhancing the project's operability.

[0082] This application proposes a collaborative mechanism between shard-index and segment-level shard summaries in its engineering approach, and designs causal version control and asynchronous minimum repair task scheduling to avoid full scans. Compared with related technologies that replicate materialized views to each shard, this solution achieves similar high availability for queries while significantly reducing storage and update costs through lightweight indexes and summaries. Furthermore, this solution focuses on the access pattern of "relationship → ID list → batch master table read," proposing bounded merging and integrity flags for pagination / top-k, as well as an audit replay mechanism, enhancing the engineering adaptability of online services. In actual business scenarios, this solution effectively improves service response speed, reduces the number of accesses to the underlying MySQL sharded data, and achieves the goals of reducing system response latency and minimizing peak traffic.

[0083] According to an embodiment of this application, an embodiment of a data processing apparatus is also provided. Figure 4 This is a schematic diagram of the structure of a data processing device according to an embodiment of this application. Figure 4 As shown, the device includes:

[0084] The query request acquisition module 40 is used to acquire data query requests. The data query request is used to request to view the data content that meets the target query conditions in the data corresponding to the target relationship. The target relationship means that the data belongs to the same user group or the same topic.

[0085] The shard summary determination module 42 is used to determine the target materialized view of the target relationship corresponding to the data query request and obtain the segment-level shard summary corresponding to the target materialized view. The target materialized view contains at least one data identifier under the target relationship. Different data identifiers are used to identify different data contents. At least one data identifier is stored in segments in the target materialized view. Each segment corresponds to a segment-level shard summary. The segment-level shard summary is used to characterize the shards to which the data contents corresponding to the data identifiers in the segments are distributed.

[0086] The candidate shard filtering module 44 is used to determine the candidate shard set corresponding to the data query request based on the segment-level shard summary, wherein the candidate shard set is a set of shards that store data content that meets the target query conditions;

[0087] The shard access query module 46 is used to query each shard in the candidate shard set to obtain the data content query results corresponding to the data query request.

[0088] Optionally, determining the candidate shard set corresponding to the data query request based on the segment-level shard summary includes: filtering out segments in the target materialized view that meet the target query conditions; for each filtered segment, determining the shard containing the data content corresponding to the data identifier in the segment based on the segment-level shard summary corresponding to the segment, and adding the shard to the candidate shard set; obtaining the shard mapping index corresponding to the target relationship, and verifying each shard in the candidate shard set based on the shard mapping index to determine whether the shard actually contains the data content corresponding to the data identifier in the segment, wherein the shard mapping index is used to characterize the correspondence between the data identifier and the shard; if it is determined that the shard does not contain the data content corresponding to the data identifier in the segment, the shard is removed from the candidate shard set to obtain the final candidate shard set.

[0089] Optionally, the data processing device is further configured to: when new data content is detected being written to a shard under the target relation, record the data identifier corresponding to the data content, the shard identifier of the shard, and the timestamp of the data writing; and update the correspondence between the data identifier and the shard identifier, as well as the timestamp, to the shard mapping index corresponding to the target relation.

[0090] Optionally, the data processing device is further configured to: obtain an original materialized view of the target relationship, and segment the data identifiers in the original materialized view to obtain multiple segments, wherein different segments correspond to different time windows and / or data scores, wherein the data scores are used to characterize the importance of the data; determine the segment attribute information corresponding to each segment to obtain the target materialized view, wherein the segment attribute information includes: segment-level fragment summary, segment identifier, segment version number, and compressed sequence of data identifiers.

[0091] Optionally, segmenting the data identifiers in the original materialized view includes: segmenting the data identifiers in the original materialized view according to the time window corresponding to the data identifier, wherein the time window is used to characterize the period length range of the data existence; determining the data score corresponding to the data identifier based on the data validity information of the data content, wherein the data validity information is used to characterize the popularity and timeliness of the data content; and sorting or further segmenting the data identifiers according to the data score within each time window.

[0092] Optionally, the data processing device is further configured to: identify the abnormal segment when the content of the data content query result is missing and / or the segment version number of the segment is less than the version number of the shard mapping index; determine the shards corresponding to each data identifier in the abnormal segment according to the shard mapping index to obtain a minimum shard set; perform batch scanning on the shards in the minimum shard set to obtain the repaired segments; replace the abnormal segment with the repaired segments; and update the segment-level shard summary and segment version number of the repaired segments.

[0093] Optionally, the data processing device is further configured to: pre-store a preset number of snapshots of data content with the highest data scores in the segments, wherein the data scores are used to characterize the importance of the data; when a data query request is obtained, aggregate the snapshots in the segments that meet the target query conditions in the target materialized view, and query the data content corresponding to the data query request in all aggregated snapshots to obtain initial query results; if the initial query results meet the query requirements of the data query request, directly return the initial query results to the user; and if the initial query results do not meet the query requirements, further query in the segments.

[0094] It should be noted that each module in the above data processing device can be a program module (for example, a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0095] It should be noted that the data processing device provided in this embodiment can be used to perform... Figure 2 The data processing method shown above is also applicable to the embodiments of this application, and will not be repeated here.

[0096] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following data processing method by running the computer program: obtaining a data query request, wherein the data query request is used to request viewing data content that meets the target query conditions in data corresponding to a target relationship, where the target relationship indicates that the data belongs to the same user group or a topic; determining a target materialized view of the target relationship corresponding to the data query request, and obtaining a segment-level fragment summary corresponding to the target materialized view, wherein the target materialized view contains at least one data identifier under the target relationship, different data identifiers are used to identify different data content, at least one data identifier is stored in segments in the target materialized view, each segment corresponds to a segment-level fragment summary, the segment-level fragment summary is used to characterize the fragments where the data content corresponding to the data identifier in the segment is distributed; determining a candidate fragment set corresponding to the data query request based on the segment-level fragment summary, wherein the candidate fragment set is a set of fragments storing data content that meets the target query conditions; querying each fragment in the candidate fragment set to obtain the data content query result corresponding to the data query request.

[0097] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method described in various embodiments of this application: obtaining a data query request, wherein the data query request is used to request to view data content that meets the target query conditions in the data corresponding to the target relationship, and the target relationship indicates that the data belongs to the same user group or the same topic; determining the target materialized view of the target relationship corresponding to the data query request, and obtaining the segment-level fragment summary corresponding to the target materialized view, wherein the target materialized view contains at least one data identifier under the target relationship, different data identifiers are used to identify different data content, at least one data identifier is stored in segments in the target materialized view, each segment corresponds to a segment-level fragment summary, and the segment-level fragment summary is used to characterize the fragments in which the data content corresponding to the data identifier in the segment is distributed; determining a candidate fragment set corresponding to the data query request based on the segment-level fragment summary, wherein the candidate fragment set is a set of fragments that store data content that meets the target query conditions; querying each fragment in the candidate fragment set to obtain the data content query result corresponding to the data query request.

[0098] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0099] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0104] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized by, include: Obtain a data query request, wherein the data query request is used to request to view the data content that meets the target query conditions in the data corresponding to the target relationship, and the target relationship means that the data belongs to the same user group or the same topic; Determine the target materialized view of the target relationship corresponding to the data query request, and obtain the segment-level fragment summary corresponding to the target materialized view. The target materialized view contains at least one data identifier under the target relationship. Different data identifiers are used to identify different data contents. The at least one data identifier is stored in segments in the target materialized view. Each segment corresponds to a segment-level fragment summary. The segment-level fragment summary is used to characterize the fragments in which the data contents corresponding to the data identifier in the segment are distributed. Based on the segment-level shard summary, a candidate shard set corresponding to the data query request is determined, wherein the candidate shard set is a set of shards that store data content that meets the target query conditions; Each fragment in the candidate fragment set is queried to obtain the data content query result corresponding to the data query request.

2. The data processing method according to claim 1, characterized in that, Based on the segment-level shard summary, the candidate shard set corresponding to the data query request is determined to include: Filter out the segments in the target materialized view that meet the target query conditions; For each of the selected segments, based on the segment-level fragment digest corresponding to the segment, a fragment containing the data content corresponding to the data identifier in the segment is determined, and the fragment is added to the candidate fragment set; Obtain the shard mapping index corresponding to the target relationship, and verify each shard of the candidate shard set based on the shard mapping index to determine whether the shard actually contains the data content corresponding to the data identifier in the segment. The shard mapping index is used to characterize the correspondence between the data identifier and the shard. If it is determined that the fragment does not contain the data content corresponding to the data identifier in the segment, the fragment is removed from the candidate fragment set to obtain the final candidate fragment set.

3. The data processing method according to claim 2, characterized in that, The method further includes: If new data content is detected to be written to a shard under the target relationship, the data identifier corresponding to the data content, the shard identifier of the shard, and the timestamp of the data writing are recorded. The correspondence between the data identifier and the shard identifier, as well as the timestamp, are updated in the shard mapping index corresponding to the target relationship.

4. The data processing method according to claim 2, characterized in that, The method further includes: Obtain the original materialized view of the target relationship, and segment the data identifiers in the original materialized view to obtain multiple segments, wherein different segments correspond to different time windows and / or data scores, wherein the data scores are used to characterize the importance of the data; The segment attribute information corresponding to each segment is determined to obtain the target materialized view, wherein the segment attribute information includes: segment-level fragment summary, segment identifier, segment version number, and data identifier compressed sequence.

5. The data processing method according to claim 4, characterized in that, The segmentation process for data identifiers in the original materialized view includes: Based on the time window corresponding to the data identifier, the data identifier in the original materialized view is segmented, wherein the time window is used to characterize the period length range of the data existence; Based on the data validity information that determines the data content corresponding to the data identifier, the data score corresponding to the data identifier is determined, wherein the data validity information is used to characterize the popularity and timeliness of the data content; Within each time window, the data identifiers are sorted or further segmented based on the data scores.

6. The data processing method according to claim 2, characterized in that, The method further includes: If the data content query result is missing and / or the segment version number of the segment is less than the version number of the shard mapping index, the segment that is abnormal is identified. Based on the sharding mapping index, determine the shards corresponding to each data identifier in the segment where the anomaly occurred, and obtain the minimum shard set; A batch scan is performed on the fragments in the minimum fragment set to obtain the repaired segments. The abnormal segments are replaced with the repaired segments, and the segment-level fragment summary and segment version number of the repaired segments are updated.

7. The data processing method according to claim 1, characterized in that, The method further includes: In the segment, a preset number of snapshots of data content with the highest data scores are pre-stored, wherein the data scores are used to characterize the importance of the data; Upon receiving the data query request, snapshots from the segments in the target materialized view that meet the target query conditions are aggregated, and the data content corresponding to the data query request is queried from all the aggregated snapshots to obtain the initial query result; If the initial query result satisfies the query requirements of the data query request, the initial query result is directly returned to the user; if the initial query result does not satisfy the query requirements, further querying is performed in the segment.

8. A data processing apparatus, characterized in that, include: The query request acquisition module is used to acquire data query requests, wherein the data query request is used to request to view the data content that meets the target query conditions in the data corresponding to the target relationship, and the target relationship means that the data belongs to the same user group or the same topic; The shard summary determination module is used to determine the target materialized view of the target relationship corresponding to the data query request, and to obtain the segment-level shard summary corresponding to the target materialized view. The target materialized view contains at least one data identifier under the target relationship. Different data identifiers are used to identify different data content. The at least one data identifier is stored in segments in the target materialized view. Each segment corresponds to a segment-level shard summary. The segment-level shard summary is used to characterize the shards in which the data content corresponding to the data identifier in the segment is distributed. The candidate shard filtering module is used to determine a set of candidate shards corresponding to the data query request based on the segment-level shard summary, wherein the set of candidate shards is a set of shards that store data content that meets the target query conditions; The shard access query module is used to query each shard in the candidate shard set to obtain the data content query result corresponding to the data query request.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the data processing method according to any one of claims 1 to 7 by running the computer program.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method according to any one of claims 1 to 7.

Images