Data processing method and device, equipment, storage medium and program product

Abstract

The application discloses a data processing method, device, equipment, storage medium and program product. The method comprises the following steps: obtaining interaction data of an account, wherein the interaction data is used for indicating an interaction operation of the account on multimedia content in a first application program; performing real-time deduplication processing on the interaction data to obtain deduplicated interaction data; performing real-time aggregation processing on the deduplicated interaction data according to an aggregation dimension to obtain aggregation data, wherein the aggregation dimension comprises an account identifier and a subject identifier, and the subject identifier is used for indicating an associated subject corresponding to the multimedia content interacted by the account; and performing bitmap compression on the aggregation data to obtain storage data, wherein the storage data is used for providing real-time query service for the account.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the rapid development of the multimedia streaming industry, the multimedia content playback behavior of accounts in applications has generated massive amounts of data. Real-time statistics and efficient querying of this data have become key requirements to support core businesses such as personalized recommendations and account behavior analysis.

[0003] In related technologies, the processing of such playback data often adopts a batch processing architecture combined with a relational database. That is, the batch processing framework processes the account playback data offline at fixed intervals, and then stores each playback record into the relational database as a single record, optimizing the query speed of some hot data only through a simple caching mechanism.

[0004] However, batch processing architecture results in high data processing latency, which cannot meet the needs of real-time statistics. Single-storage method makes the storage cost of massive data high, and queries need to traverse a large number of scattered records, resulting in slow response speed and low data processing efficiency. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, storage medium, and program product. The technical solution is as follows.

[0006] On the one hand, a data processing method is provided, the method comprising: Obtain the interaction data corresponding to the account, the interaction data being used to instruct the account on interactive operations of multimedia content in the first application; The interaction data is processed in real time to obtain the deduplicated interaction data; The deduplicated interactive data is processed in real time according to the aggregation dimension to obtain aggregated data. The aggregation dimension includes account identifier and subject identifier. The subject identifier is used to indicate the associated subject corresponding to the multimedia content of the account interaction. Bitmap compression is performed on the aggregated data to obtain stored data, which is used to provide real-time query services to the account.

[0007] On the other hand, a data processing apparatus is provided, the apparatus comprising: The acquisition module is used to acquire the interaction data corresponding to the account, and the interaction data is used to indicate the account's interactive operations on multimedia content in the first application; The deduplication module is used to perform real-time deduplication processing on the interactive data to obtain deduplicated interactive data. The aggregation module is used to perform real-time aggregation processing on the deduplicated interactive data according to the aggregation dimension to obtain aggregated data. The aggregation dimension includes account identifier and subject identifier. The subject identifier is used to indicate the associated subject corresponding to the multimedia content of the account interaction. The storage module is used to perform bitmap compression on the aggregated data to obtain stored data, which is used to provide real-time query services to the account.

[0008] In some embodiments, the deduplication module is further configured to: Divide the data window, which is used to indicate the interactive data generated by the account for the multimedia content corresponding to the associated subject within the window duration; Sort the interactive data within the data window to obtain the sorting result; Based on the sorting results, valid interactive data is filtered to obtain the deduplicated interactive data.

[0009] In some embodiments, the deduplication module is further configured to: The interactive data within the data window is sorted according to the time of data generation to obtain the sorting result; The step of filtering valid interaction data based on the sorting results to obtain the deduplicated interaction data includes: The interactive data that is ranked first from the last to the first in the sorting results is selected as the valid data, and the deduplicated interactive data is obtained.

[0010] In some embodiments, the aggregation module is further configured to update statistical indicators according to the aggregation dimension, summarize the interaction-related information of the deduplicated interaction data, and obtain the aggregated data. The aggregated data includes summary data aggregated according to the statistical indicators and an association set. The statistical indicators include at least one of interaction count, interaction duration, first interaction time, and most recent interaction time. The association set includes a set of content identifiers corresponding to the multimedia content that the account has interacted with under the aggregation dimension.

[0011] In some embodiments, the storage module is further configured to convert the set of content identifiers in the aggregated data into a binary sequence to obtain the stored data, wherein the binary bits in the binary sequence are used to indicate whether the multimedia content indicated by the corresponding content identifier has been interacted with by the account.

[0012] In some embodiments, the acquisition module is further configured to: Receive the event tracking data reported by the first application based on interactive events; The interaction data is obtained by parsing the data points. The interaction data includes at least one of the following: the account identifier and the content identifier corresponding to the multimedia content, the subject identifier, the interaction duration, and the time when the data was generated.

[0013] In some embodiments, the apparatus further includes a query module for: Receive a query request initiated by the account, the query request including query dimensions; In response to the query request, the stored data is queried in real time according to the query dimension to obtain the query results.

[0014] In some embodiments, the query module is further configured to: Receive the query request initiated by the account targeting the first entity identifier; In response to the query request, the stored data is queried according to the combination dimension of the account identifier and the first subject identifier to obtain the query result.

[0015] In some embodiments, the query module is further configured to: Receive the query request initiated by the account for the content set; In response to the query request, extract the set of content identifiers corresponding to the account; Perform an intersection operation between the binary sequences corresponding to the content identifier set and the content set in the query request to obtain the operation result; The result of the operation is decoded into a list of content identifiers to obtain the query result.

[0016] In some embodiments, the storage module is further configured to: A data storage preparation request is sent to the storage layer, the data storage preparation request including the storage data. The storage layer is used to verify the storage status based on the data storage preparation request. If the storage status is normal, the storage data is written to a temporary storage area and the corresponding storage resource is locked. The storage layer includes at least two storage nodes participating in storage. If at least two storage nodes return a full readiness response, a data confirmation storage request is sent to the storage layer. The data confirmation storage request is used to instruct the storage layer to migrate the stored data in the temporary storage area to the final storage area and release the storage resources. If, in the case where a first storage node among the at least two storage nodes does not return a preparation failure response, a data discard request is sent to the storage layer. The data discard request is used to instruct the storage layer to delete the stored data in the temporary storage area and release the storage resources.

[0017] In some embodiments, the apparatus further includes a normalization module and a splitting module; The acquisition module is also used to acquire behavioral data reported by multiple applications. The behavioral data includes at least one of playback data, reading data, and viewing data. The behavioral data corresponds to a global account identifier, a content identifier, a subject identifier, and a content type identifier. The global account identifier is used to associate the same account in different applications. The normalization module is used to perform normalization processing on the behavior data based on the global account identifier to obtain standardized behavior data, wherein the standardized behavior data has the same format as the interaction data. The aggregation module is also used to perform real-time aggregation processing on the standardized behavioral data according to the global aggregation dimension to obtain cross-type aggregated data. The global aggregation dimension includes a combination dimension of the global account identifier, the subject identifier, and the content type identifier. The splitting module is used to split the set of content identifiers in the cross-type aggregated data according to the content type identifier, and obtain at least two splitting results; The storage module is further configured to perform bitmap compression on the at least two splitting results respectively to obtain type-isolated storage data, which is associated with the global account identifier.

[0018] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the data processing method as described in any of the embodiments of this application above.

[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the data processing method as described in any of the embodiments of this application above.

[0020] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the data processing methods described in the above embodiments.

[0021] The beneficial effects of the technical solutions provided in this application include at least the following: By constructing a full-link streaming processing architecture that integrates real-time deduplication, real-time aggregation, bitmap compression, and real-time query services, the deduplication and aggregation stages of account interaction data are brought forward with real-time processing logic. This eliminates the need to wait for fixed offline processing cycles, reducing data latency from hours to seconds and meeting real-time statistical requirements. Aggregation processing combines multiple scattered playback records into a single aggregated data entry containing a set of statistical indicators and content identifiers. Bitmap compression technology then converts the content identifier set into a binary sequence, replacing the complete identifier with the smallest storage unit, thus reducing storage costs. Aggregation processing enables the pre-calculation of statistical indicators required for queries, and bitmap compression supports fast bit operations, compressing storage response time and improving data processing efficiency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a computer system provided in an exemplary embodiment of this application; Figure 2 This is a flowchart of a data processing method provided in an exemplary embodiment of this application; Figure 3 This is a flowchart of a two-stage data storage method provided in an exemplary embodiment of this application; Figure 4 This is a flowchart of a cross-application type data processing method provided in an exemplary embodiment of this application; Figure 5 This is a structural block diagram of a data processing apparatus provided in an exemplary embodiment of this application; Figure 6 This is a structural block diagram of a data processing apparatus module provided in an exemplary embodiment of this application; Figure 7 This is a structural block diagram of a terminal provided in an exemplary embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0025] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0026] In related technologies, the processing of such playback data often employs a batch processing architecture combined with a relational database. This involves using a batch processing framework to process account playback data offline at fixed intervals, and then storing each playback record individually in a relational database. Only a simple caching mechanism is used to optimize the query speed for some frequently accessed data. However, the batch processing architecture results in high data processing latency, failing to meet real-time statistical needs. The individual record storage method leads to high storage costs for massive amounts of data, and queries require traversing a large number of scattered records, resulting in slow response times and low data processing efficiency.

[0027] The data processing method provided in this application embodiment constructs a full-link streaming processing architecture that includes real-time deduplication, real-time aggregation, bitmap compression, and real-time query services. By using real-time processing logic to bring forward the deduplication and aggregation stages of account interaction data after it is generated, it eliminates the need to wait for fixed offline processing cycles, reducing data latency from hours to seconds and meeting real-time statistical requirements. Aggregation processing summarizes multiple scattered playback records into a single aggregated data entry containing statistical indicators and content identifier sets. Bitmap compression technology then converts the content identifier set into a binary sequence, replacing the complete identifier with the smallest storage unit, reducing storage costs. Aggregation processing enables pre-calculation of the statistical indicators required for querying, and bitmap compression supports fast bitwise operations, compressing storage response time and improving data processing efficiency.

[0028] First, the computer system described in this application will be introduced. Please refer to... Figure 1 The illustration shows a schematic diagram of a computer system provided in an exemplary embodiment of this application, which includes: terminal 10, server 20, and terminal 30.

[0029] Terminal 10 and Terminal 30 are clients of the first application, and Server 20 is the backend server of the first application. Terminal 10 and Terminal 30 are connected to Server 20 via a network.

[0030] Optionally, the first application may include, but is not limited to, any application capable of providing multimedia content for account interaction, such as music playback software, video playback platform, e-reading software, social media program, virtual game, online shopping platform, map navigation software, online teaching platform, virtual reality program, augmented reality program, mixed reality program, etc. This application embodiment does not limit this.

[0031] Server 20 is used to provide data processing services for the first application, and to collect and store relevant data in real time based on the interaction events in the client for account query.

[0032] Schematic illustration: Server 20 acquires interaction data reported by the client, which indicates the account's interactive operations on multimedia content in the first application. It performs real-time deduplication on the interaction data to obtain deduplicated data. Then, it performs real-time aggregation on the deduplicated data according to aggregation dimensions, obtaining aggregated data. The aggregation dimensions include account identifier and subject identifier, where the subject identifier indicates the associated subject corresponding to the multimedia content interacted with by the account. Finally, it performs bitmap compression on the aggregated data to obtain stored data, which is used to provide real-time query services to the account. For example, if an account requests a query for statistical data related to a first subject identifier through terminal 10, the server retrieves the relevant data corresponding to the account identifier and the first subject identifier from the stored data based on the query request and returns it to terminal 10.

[0033] In some embodiments, server 20 can also provide a unified data processing service for multiple applications, supporting cross-application data queries for multiple applications.

[0034] It is worth noting that the data interaction between the server and the terminal described above is merely an exemplary example, and the embodiments of this application do not limit it.

[0035] The aforementioned terminal is optional and can be a desktop computer, laptop computer, mobile phone, tablet computer, e-book reader, Moving Picture Experts Group Audio Layer III (MP3) player, Moving Picture Experts Group Audio Layer IV (MP4) player, smart TV, smart vehicle, and other types of terminal devices. This application embodiment does not limit the specific terminal device to these types.

[0036] It is worth noting that the aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud security, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0037] Cloud technology refers to a managed technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0038] In some embodiments, the server described above can also be implemented as a node in a blockchain system.

[0039] It should be noted that all information (including but not limited to account device information, account personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the account holder or by all parties in full, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions. For example, the operational data and account information involved in this application were obtained with full authorization.

[0040] To further clarify, this application can display prompt interfaces, pop-ups, or output voice prompts before and during the collection of account-related data (e.g., account information, historical operation data, and real-time operation data mentioned in this application). These prompt interfaces, pop-ups, or voice prompts are used to inform the account that its relevant data is currently being collected. This ensures that the application only begins the steps for acquiring account-related data after receiving confirmation from the account regarding the prompt interface or pop-up; otherwise (i.e., without receiving confirmation from the account), the steps for acquiring account-related data end, meaning no account-related data is acquired. In other words, all account data collected in this application is collected with the account's consent and authorization, and the collection, use, and processing of related account data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0041] This is illustrative; please refer to it. Figure 2 This document illustrates a flowchart of a data processing method provided in an exemplary embodiment of this application. This method can be executed by a terminal, a server, or both simultaneously. This embodiment uses the execution of the method by a server as an example for illustration. Figure 2As shown, the method includes the following steps: Step 210: Obtain the interaction data corresponding to the account.

[0042] Interactive data is used to indicate the account's interactive actions on multimedia content in the first application.

[0043] The goal of acquiring interaction data is to collect standardized, full-volume raw data on account interaction behavior to provide reliable material for subsequent real-time processing.

[0044] Optionally, the first application may include, but is not limited to, any application capable of providing multimedia content for account interaction, such as music playback software, video playback platform, e-reading software, social media program, virtual game, online shopping platform, map navigation software, online teaching platform, virtual reality program, augmented reality program, mixed reality program, etc. This application embodiment does not limit this.

[0045] Multimedia content refers to interactive content provided in the first application, including but not limited to audio content (such as songs in music software), video content (such as short videos in video platforms), text content (such as novels in e-reading software), and other media content in any form. This application embodiment does not limit this.

[0046] Interactive operations on multimedia content include, but are not limited to, operations corresponding to any interactive event such as play, pause, switch, read, favorite, and watch. Interactive data is data used to record interactive events, including but not limited to the interaction time, number of interactions, duration of interaction, and multimedia content identifier of the interaction.

[0047] In some embodiments, the system receives event tracking data reported by a first application based on interaction events; and parses the event tracking data to obtain interaction data.

[0048] Interaction data includes at least one of the following: account identifier and content identifier corresponding to multimedia content, subject identifier, interaction duration, and data generation time.

[0049] Regarding the source and triggering method of interactive data, when an account performs operations such as play, pause, and end playback in the first application, the application's built-in Software Development Kit (SDK) will automatically trigger data reporting without requiring manual operation by the account. The reported data is received and cached through a message queue (such as a distributed message queue (Kafka) cluster) to avoid data loss or blocking in high-concurrency scenarios, and adapts to high-concurrency write requirements of 50,000 / second.

[0050] Playback data must include key information to support subsequent processing, including at least account identifiers (such as account identifiers, used to uniquely identify accounts), multimedia content identifiers (such as song identifiers and video identifiers, used to indicate specific content), subject identifiers (such as singer identifiers and video blogger identifiers, corresponding to the core associated subject of multimedia content), playback duration (such as 30 seconds of song playback or 2 minutes of video viewing), and data generation time (such as the timestamp of the playback behavior, accurate to the millisecond level), ensuring that the data has the characteristics of being identifiable, calculable, and traceable.

[0051] After receiving data, the message queue performs preliminary verification and standardization of the data format, such as unifying the timestamp format and filling in missing required fields, to avoid subsequent processing anomalies due to inconsistent data formats and ensure the smoothness of the entire chain of processing. This step is seamlessly connected with the distributed computing cluster to achieve low-latency data transmission.

[0052] The method provided in this application embodiment receives the event tracking data reported by the application based on interactive events, and parses out core fields such as account identifier, content identifier, subject identifier, interaction duration, and data generation time. This achieves standardized access to interactive data, thereby ensuring the smoothness of subsequent deduplication, aggregation, and other end-to-end processing, avoiding processing anomalies caused by inconsistent data formats or missing fields, and ensuring the stability of system processing.

[0053] Step 220: Perform real-time deduplication on the interactive data to obtain the deduplicated interactive data.

[0054] By streamlining and deduplicating data in real time, we ensure that the data entering the aggregation stage is valid data that reflects the actual playback behavior of the accounts.

[0055] For the deduplication triggering mechanism, a streaming real-time processing mode is adopted. After the playback data flows into the computing cluster (Flink) from the message queue, the deduplication process is triggered immediately without waiting for the accumulation of batch data, ensuring the real-time performance of data processing and controlling data latency to the second level.

[0056] For the core deduplication logic, data windows are divided based on a three-dimensional grouping rule of "account identifier + subject identifier + time window". The time window is a short fixed duration (e.g., 1 minute, which can be configured as needed). Only interactive data generated by the same account for multimedia content of the same associated subject within the window is deduplicated to avoid misjudgment of duplicate data across accounts, subjects or long time spans. Then, the playback data in the window is sorted from newest to oldest according to the time of data generation by the window function ROW_NUMBER(). The latest data at the top of the sort is selected as valid data, and redundant data such as duplicate reports (e.g., secondary reports caused by network fluctuations) and segmented reports (e.g., multiple records caused by restarting after pausing in the middle of playback) are removed.

[0057] Leveraging Flink's stateful stream processing capabilities, the processing status of each data window is maintained in real time, such as received data, sorting results, and filtered valid data, ensuring that the deduplication process is delayed and without omissions, ultimately achieving a statistical data accuracy rate of 99.9%.

[0058] In some embodiments, the real-time deduplication process includes: dividing a data window, which indicates the interactive data generated by an account for multimedia content corresponding to an associated subject within the window duration; sorting the interactive data within the data window to obtain a sorting result; and filtering valid interactive data based on the sorting result to obtain deduplicated interactive data.

[0059] The data window is used to define the deduplication range and avoid misjudgments across scenarios. It employs a three-dimensional grouping strategy of "account identifier + subject identifier + time window." The account identifier uniquely identifies the operating subject, the subject identifier (e.g., singer identifier) ​​corresponds to the core related subject of the multimedia content (e.g., singer, author, video blogger), and the time window is a configurable short-term fixed duration (e.g., 1 minute, adapting to real-time processing needs in high-concurrency scenarios). A data window is the set of interactive data generated by the same account for the same related subject within a specified time period. For example, all interactive data of account A playing singer B's song between 10:00 and 10:01 on May 20, 2024.

[0060] By dividing the data into windows, we can avoid data misjudgment and duplication issues across accounts (such as account A and account C playing songs by the same artist), across subjects (such as account A playing songs by singers B and D), and over long time spans (such as account A playing songs by the same artist today and yesterday). This lays the foundation for accurate deduplication in the future and is suitable for business scenarios such as "singer memory coordinates" that are statistically analyzed by subject dimension.

[0061] Prioritize the interactive data within the window by sorting, identify which data best reflects the account's true behavior, extract valid data from the window using clear rules, and eliminate all redundant data.

[0062] The illustrative process of filtering effective interactive data includes: sorting the interactive data in the data window according to the time of data generation to obtain a sorting result; filtering the interactive data that is ranked first from the last to the first in the sorting result as effective data, thus obtaining the deduplicated interactive data.

[0063] The data is sorted based on its generation time, which is a millisecond-level timestamp of the account's playback operation, such as the precise time of playback start, end, or playback status update. This is automatically reported by the front-end event tracking SDK to ensure the accuracy and uniqueness of the timestamps. For example, if an account plays a song and the same playback action is reported twice due to network fluctuations, the corresponding timestamps are 2024-06-01 10:00:03.123 and 2024-06-01 10:00:05.456, respectively. The latter is the more complete data generated later.

[0064] During the sorting process, all interactive data within the data window is sorted from newest to oldest, meaning the most recently timestamped interactive data is placed first, and earlier generated data is placed later. Since duplicate or redundant data (such as secondary reports due to network latency or segmented reports from pauses and restarts during playback) inevitably have temporal differences, and the most recently generated interactive data usually contains the most complete behavioral information (such as the complete playback duration and final playback status), sorting based on the time of data generation better reflects the account's true intent.

[0065] The screening process uses clear rules to extract valid data, completely eliminates redundant data, and ensures that only one core valid record is retained in the same data window.

[0066] The filtering rules include retaining only the interaction data with a ranking of "1" as valid data from the sorted results, and marking all other data with a ranking of 2 or higher as redundant data and discarding them directly. For example, if a data window contains 3 duplicate interaction data with timestamps of 10:00:01, 10:00:03, and 10:00:05, after sorting, only the first data corresponding to 10:00:05 is retained, and the other 2 redundant records are deleted, achieving a precise deduplication effect of retaining only one data for each valid pregnancy behavior.

[0067] The filtering logic is implemented based on Flink's ROW_NUMBER() window function. The system assigns a unique consecutive sequence number to each sorted data in each data window, with the first sequence number being 1 and increasing sequentially. The structured query language (SQL) logic of "filtering data with sequence number = 1" ensures the accuracy and unambiguity of the filtering process and avoids data omissions or accidental deletions.

[0068] The final deduplicated interactive data contains core fields that support subsequent aggregation processing, including account identifier, subject identifier, content identifier, effective playback duration, and data generation time.

[0069] The method provided in this application sorts the playback data in the window from newest to oldest according to the time of data generation, and selects the latest data at the top of the sort as valid data. This accurately locks in the complete data reflecting the account's real playback behavior, thereby effectively solving problems such as repeated reporting of network fluctuations and segmented reporting of playback. It ensures that only one piece of data is retained for each valid playback behavior, thus improving the accuracy of statistical data.

[0070] The method provided in this application, by dividing the window duration data corresponding to "account-associated subject", sorting the playback data in the window and filtering the effective data, clarifies the scope boundary and core operation logic of real-time deduplication, thereby avoiding misjudgment and duplication of data across accounts, subjects or long time spans, providing a clean and non-redundant data source for subsequent aggregation processing, and improving the basic accuracy of statistical data.

[0071] Step 230: Perform real-time aggregation processing on the deduplicated interactive data according to the aggregation dimension to obtain aggregated data.

[0072] The aggregation dimension includes account identifier and subject identifier. The subject identifier is used to indicate the associated subject corresponding to the multimedia content played by the account.

[0073] A related subject refers to a topic that is related to multimedia content. This relationship includes, but is not limited to, attribution, creation, and interpretation.

[0074] Indicative, related entities include, but are not limited to, singers of songs, authors of novels, bloggers / directors / actors of videos, production teams of programs, and live streamers.

[0075] The subject identifier is an identifier used to indicate the associated subject. Taking the music scene as an example, the subject identifier includes the singer identifier corresponding to the song, such as the singer's name.

[0076] This application embodiment transforms scattered, valid playback data into structured, statistically valuable aggregated data through real-time pre-aggregation.

[0077] The fixed aggregation dimension is "account identifier + subject identifier". The subject identifier can be flexibly adapted to different multimedia scenarios, such as singer identifier (ID) for music scenarios, author ID for novel scenarios, and blogger ID for video scenarios, to ensure the universality of the solution. This dimension focuses on the behavioral binding between the account and the core related subject of the content, which fits the needs of the account to collect the preference statistics of the specific subject in the business scenario, such as the account's playback preference for a certain singer or viewing preference for a certain blogger.

[0078] In some embodiments, statistical indicators are updated according to the aggregation dimension, and the interaction-related information of the deduplicated interaction data is summarized to obtain aggregated data.

[0079] Aggregated data includes summary data aggregated according to statistical indicators and related sets. Statistical indicators include at least one of the following: number of interactions, interaction duration, first interaction time, and most recent interaction time. Related sets include the set of content identifiers corresponding to the multimedia content that the account has interacted with under the aggregation dimension.

[0080] For the aggregation processing logic, based on the real-time computing capabilities of the distributed computing cluster (Flink), the deduplicated playback data is dynamically and cumulatively updated according to the aggregation dimension. On the one hand, it summarizes statistical indicators, including cumulative playback count (the total number of valid playback behaviors under the same account-subject dimension), cumulative playback duration (the sum of the durations of all valid playback behaviors), first playback time (the timestamp of the first playback by the account under this dimension), and most recent playback time (the timestamp of the latest playback by the account under this dimension). On the other hand, it integrates related sets, that is, it collects the content identifiers (such as song ID sets, video ID sets) corresponding to all multimedia content played by the account under this dimension, forming complete behavioral trajectory data.

[0081] In some embodiments, a state maintenance mechanism is adopted, which uses Flink's stateful storage capability to maintain the historical aggregated state of each "account identifier + subject identifier" dimension in real time. When new valid playback data is generated, it is only necessary to update the statistical indicators and supplement the content identifier based on the historical state, without having to recalculate the full amount of data, which greatly improves processing efficiency and reduces computing resource consumption.

[0082] The method provided in this application updates statistical indicators such as playback count, playback duration, and first / recent playback time according to the dimension of "account identifier + subject identifier". At the same time, it summarizes the set of content identifiers played by the account, realizing real-time pre-calculation and structured integration of data. This avoids the need to traverse massive single data records for repeated calculations during queries, shortens query response time, and provides a standardized data foundation for subsequent bitmap compression, supporting efficient storage.

[0083] Step 240: Perform bitmap compression on the aggregated data to obtain the stored data.

[0084] The stored data is used to provide real-time query services to accounts.

[0085] Bitmap compression technology enables efficient storage of aggregated data while supporting high-concurrency, low-latency real-time queries.

[0086] The compressed objects include the associated sets in the aggregated data, namely the content identifier sets, such as the song ID set. Statistical metrics (such as play counts and duration) are directly bound to the compressed associated sets for storage due to their small data volume, and do not require additional compression.

[0087] In some embodiments, the set of content identifiers in the aggregated data is converted into a binary sequence to obtain the stored data.

[0088] The binary bits in the binary sequence are used to indicate whether the multimedia content indicated by the corresponding content identifier has been interacted with by the account.

[0089] First, a globally unified "content identifier - binary bit" mapping rule is established, assigning a unique binary bit sequence number to each multimedia content identifier (such as song ID), e.g., song ID=1 corresponds to the 1st bit, ID=2 corresponds to the 2nd bit; then, the content identifiers in the associated set are converted into binary sequences according to the mapping rule, where a binary bit of "1" indicates that the account has played the content, and "0" indicates that it has not been played, e.g., the content identifier set {1,3,5} corresponds to the binary sequence "10101"; finally, through the aggregation model of the storage cluster (such as Apache Doris), historical data of the same "account identifier + subject identifier" dimension are automatically merged to avoid duplicate storage and achieve ultimate optimization of storage costs.

[0090] The method provided in this application converts the set of content identifiers in aggregated data into a binary sequence, and uses the binary bit status to indicate whether the corresponding content has been played by the account. It replaces the complete content identifier with the smallest storage unit, thereby significantly reducing the storage space occupied by the content set and reducing storage costs. Moreover, the compressed binary sequence supports fast bit operations, laying an efficient processing foundation for subsequent intersection queries and other scenarios.

[0091] The compressed storage data (including statistical indicators and binary sequences) is stored in a distributed storage cluster (Apache Doris). This cluster adopts a high-availability deployment architecture across three availability zones, supporting data sharding and parallel querying. The query service is provided through an SQL interface, enabling efficient response for different query scenarios.

[0092] In some embodiments, based on the stored data obtained after data processing, a real-time query service can be provided to an account. This process includes receiving a query request initiated by a first account, the query request including query dimensions; and responding to the query request by querying the stored data in real time according to the query dimensions to obtain query results.

[0093] Taking a query targeting the subject dimension as an example, the system receives a query request initiated by an account targeting the first subject identifier; in response to the query request, it queries the stored data according to the combination dimension of the account identifier and the first subject identifier, and obtains the query results.

[0094] As an example, the query request indicates "query the playback statistics of account A for singer B", which directly retrieves the statistical indicators and binary sequence of the "account ID + singer ID" dimension, decodes them, and returns results such as the number of plays, duration, and list of played songs.

[0095] The method provided in this application embodiment receives query requests from accounts for specific entity identifiers and queries stored data according to the combined dimension of "account identifier + specific entity identifier". This accurately matches the query needs of core business scenarios such as "singer memory coordinates", thereby realizing the rapid retrieval of statistical data related to accounts and specific entities, shortening the query response time, and enhancing the user experience of core functions.

[0096] Taking a query on the intersection of content sets as an example, the system receives a query request from an account for a content set; in response to the query request, it extracts the content identifier set corresponding to the account; it performs an intersection operation on the binary sequences corresponding to the content identifier set and the content set in the query request to obtain the operation result; and it decodes the operation result into a list of content identifiers to obtain the query result.

[0097] As an illustration, the query request indicates "the overlap between the songs played by account A and the recommended playlist". The binary sequence of the account association set and the binary sequence of the recommended playlist are subjected to bitwise operations to quickly filter out the intersection results, avoiding the inefficiency of traversal comparison in traditional solutions and supporting high-concurrency query load.

[0098] The method provided in this application receives a query request from an account for a content set, extracts the binary sequence corresponding to the account's content identifier set, performs an intersection operation with the binary sequence of the query content set, and then decodes and returns the result. By utilizing the efficient characteristics of bitmap bit operations, it avoids the inefficiency of traversal comparison in traditional solutions, quickly filters out the overlapping parts of the account's played content and the specified set, and supports the efficient implementation of scenarios such as recommendation deduplication and content matching.

[0099] The method provided in this application embodiment receives a query request from an account containing query dimensions, queries the stored data after bitmap compression in real time according to the corresponding dimensions and returns the results. It clarifies the core execution link of real-time query, thereby meeting the account's need for immediate access to personal playback statistics, breaking the hour-level latency limitation of traditional batch processing solutions, and improving the account user experience.

[0100] In summary, the method provided in this application constructs a full-link streaming processing architecture that includes real-time deduplication, real-time aggregation, bitmap compression, and real-time query services. By using real-time processing logic to bring forward the deduplication and aggregation stages of account interaction data after generation, it eliminates the need to wait for fixed offline processing cycles, reducing data latency from hours to seconds and meeting real-time statistical requirements. Aggregation processing combines multiple scattered playback records into a single aggregated data entry containing statistical indicators and content identifier sets. Bitmap compression technology then converts the content identifier set into a binary sequence, replacing the complete identifier with the smallest storage unit, reducing storage costs. Aggregation processing enables pre-calculation of the statistical indicators required for querying, and bitmap compression supports fast bitwise operations, compressing storage response time and improving data processing efficiency.

[0101] In some embodiments, during data storage, this application can employ a two-phase commit mechanism to achieve exactly-once semantics, improving data storage accuracy. Please refer to [reference needed]. Figure 3 , Figure 3 This is a flowchart of a two-stage data storage method provided in an exemplary embodiment of this application. This method can be executed by a terminal, a server, or both simultaneously. This embodiment illustrates the method as being executed by a server. Figure 3 As shown, the method consists of the following steps.

[0102] Step 310: Send a data storage preparation request to the storage layer.

[0103] The data storage preparation request includes storage data. The storage layer is used to verify the storage status based on the data storage preparation request. If the storage status is normal, the storage data is written to the temporary storage area and the corresponding storage resources are locked. The storage layer includes at least two storage nodes participating in the storage.

[0104] This application embodiment verifies storage feasibility in advance and locks in dedicated resources to avoid data storage anomalies caused by insufficient resources or node failures during the subsequent confirmation stage.

[0105] For the request initiator and data content of the data storage preparation request, the Flink real-time computing layer acts as the request initiator. After completing real-time data deduplication, aggregation and Bitmap compression, it generates storage data, namely structured aggregated data, which includes statistical indicator summary data of account identifier + subject identifier, and a set of content identifiers after Bitmap compression. It is then encapsulated into a data storage preparation request and sent to the storage layer (Apache Doris cluster).

[0106] Regarding the composition and node distribution of the storage layer, the storage layer is an Apache Doris distributed cluster, which includes a multi-node FE (Frontend) cluster (responsible for request parsing and node scheduling) and a multi-node BE (Backend) cluster (responsible for actual data storage). All storage nodes are deployed across three availability zones (e.g., at least one backend (BE) node is deployed in each of availability zones A, B, and C), satisfying the requirement that at least two storage nodes participate in storage, thus avoiding the risk of single point of failure from the physical architecture perspective.

[0107] Regarding the storage status verification logic, after receiving a request, the Frontend (FE) node allocates the storage task to at least two BE nodes of the corresponding shard according to the data sharding rules (based on account identifier + subject identifier hash sharding). (For example, the shards correspond to BE1, BE2, and BE3, which are distributed in different availability zones.) After receiving the task, each BE node synchronously performs multiple checks: whether the disk space is sufficient to accommodate the stored data; whether the node load is within the threshold, such as the central processing unit (CPU) or memory usage rate being <0%, to avoid high load causing write failures; and whether the network connection is normal, such as the communication links with the FE node and other BE nodes being unobstructed.

[0108] For temporary storage and resource locking operations, if the BE node verification status is normal, the stored data is immediately written to the local temporary storage area. The temporary storage area is a cache area independent of the final storage directory, which supports fast writing and deletion. At the same time, the corresponding storage resources are locked through a distributed lock mechanism, including the storage directory corresponding to the data shard, the storage inode node of the Bitmap compressed file, and the index slot of the hash shard, to prevent other concurrent storage requests from occupying the same resource and avoid data overwriting or conflict.

[0109] After each BE node completes temporary writing and resource locking, it returns a ready-to-complete response to the FE node, carrying the node identifier, temporary storage path, and resource lock identifier. If the verification fails, such as due to insufficient disk space or network interruption, it returns a ready-to-fail response, carrying the reason for the failure. The FE node summarizes the response results of all participating storage nodes to provide a basis for decision-making in subsequent steps.

[0110] Step 320: If at least two storage nodes return a full readiness response, send a data confirmation storage request to the storage layer.

[0111] The data confirmation storage request is used to instruct the storage layer to migrate the stored data in the temporary storage area to the final storage area and release the locked storage resources.

[0112] The data confirmation storage request will only be sent when the FE node receives a readiness completion response from all participating BE nodes. For example, if BE1, BE2, and BE3 participating in the storage all report readiness completion and the nodes across the three availability zones are all in normal status, this will prevent data inconsistency caused by some nodes being ready.

[0113] In response to the data confirmation storage request, after receiving the full preparation completion notification forwarded by the FE node, the Flink compute layer sends a data confirmation storage request to the storage layer. This request includes at least the following: a migration instruction, requiring each BE node to atomically migrate the stored data in the temporary storage area to the final storage area (Doris's core data directory, associated with the aggregation model, and supporting SQL queries); a merge instruction, based on Doris's aggregation model, automatically merging the newly migrated aggregated data with the historical aggregated data of the "account identifier + subject identifier" dimension (such as cumulative playback count overlay, Bitmap content identifier set supplementation); and a release instruction, after the migration and merging are completed, releasing the storage resources (distributed lock, inode node, etc.) locked in step 310, allowing other storage requests to reuse the resources.

[0114] When each BE node performs migration operations, it adopts atomic operations of write-after-renaming. The temporary data is first written completely to the final storage directory, and then the temporary data is deleted. This ensures that during the migration process, the data either exists completely in the final storage area or is still in the temporary storage area, and there is no intermediate state of partial migration. At the same time, Doris's aggregation model uses primary key unique constraints (the primary key is the account identifier + subject identifier) ​​to ensure that new data is not duplicated or omitted when merging with historical data, thus maintaining the accuracy of statistical indicators.

[0115] After all BE nodes have completed migration, merging, and resource release, they send a confirmation response to the FE nodes. The FE nodes update their metadata (recording data storage path, sharding information, and Bitmap compression version) and synchronize the successful storage result to the Flink computing layer. Flink updates its state storage (marking that the data has been successfully stored), thus completing the entire storage process loop.

[0116] Step 330: If, among at least two storage nodes, the first storage node does not return a preparation failure response, a data discard request is sent to the storage layer.

[0117] Data discard requests are used to instruct the storage layer to delete stored data in temporary storage areas and release locked storage resources.

[0118] This application's embodiments employ a rollback mechanism to ensure data consistency. This mechanism is triggered when any participating storage node fails to prepare, thoroughly cleaning up temporary data and locking resources to prevent invalid data from occupying storage or causing conflicts.

[0119] A data scrapping request is triggered if any of the following conditions are met: at least one BE node returns a preparation failure response, such as BE2 failing verification due to disk failure; some BE nodes do not return a response after timeout, such as network interruption causing FE nodes not to receive a response from BE3, with the timeout threshold set to 5 seconds; or a single availability zone fails in a cross-availability zone deployment, such as availability zone C losing power and BE3 being unable to communicate, ensuring that rollback can be triggered under any storage anomaly scenario.

[0120] After receiving the partial preparation failure notification forwarded by the FE node, the Flink compute layer sends a data discard request to the storage layer, instructing all BE nodes participating in storage to perform two operations: delete all storage data in the temporary storage area (including data that has been successfully written to BE1 and BE2 in the temporary area) to avoid invalid data residue occupying storage resources; and forcibly release all storage resources locked in step 310 (regardless of whether the resources are in use, they are forcibly unlocked through the distributed lock timeout mechanism) to prevent resource leakage.

[0121] After the abandonment operation is completed, the FE node returns the storage failure result to Flink. Flink triggers the retry logic, adds the stored data back to the processing queue, waits for a preset time (e.g., 10 seconds), and then re-executes the entire process of steps 310-330. If the retry count still fails after reaching the threshold, the data is written to the exception queue (Kafka dead letter queue) for operations personnel to investigate the fault (e.g., node failure, network problem) to ensure that the data is not lost.

[0122] In summary, the method provided in this application sends a preparation request containing storage data to the storage layer, and then instructs the migration to the final storage area after all storage nodes return a preparation completion response. If any node fails to prepare, the temporary data is discarded. This constructs a step-by-step submission mechanism of preparation, confirmation / rollback, thereby realizing exactly-Once semantics, avoiding duplicate storage or omission of data, and ensuring the consistency of stored data.

[0123] In some embodiments, the data processing method provided in this application can also implement cross-application type data processing; that is, the first application may include multiple applications. Please refer to [reference needed]. Figure 4 , Figure 4 This is a flowchart of a cross-application type data processing method provided in an exemplary embodiment of this application. This method can be executed by a terminal, a server, or both simultaneously. This embodiment illustrates the method executed by a server as an example. Figure 4 As shown, the method includes the following steps.

[0124] Step 410: Obtain behavioral data reported by multiple applications.

[0125] Behavioral data includes at least one of playback data, reading data, and viewing data. Behavioral data corresponds to a global account identifier, content identifier, subject identifier, and content type identifier. The global account identifier is used to associate the same account across different applications.

[0126] Behavioral data originates from multiple multimedia applications within the same ecosystem, covering content consumption behavior across all scenarios, including but not limited to song playback data from music software, chapter reading data from novel software, drama viewing data from video platforms, and listening data from audio program software. All applications deploy a unified specification event tracking SDK to ensure the standardization of behavioral data reporting and avoid data loss or format confusion caused by differences in reporting logic.

[0127] A global account identifier is a globally unique account identifier, such as a mobile phone number hash value or a unified account system identifier. It is used to associate the same account in different applications. For example, the local account identifier of an account in a music software is "music_001" and the local account identifier of a novel software is "novel_001". By unifying the association through the global account identifier "global_001", the problem of fragmented account identity across applications is solved, and a unified identity foundation is provided for the construction of full-scenario profiles.

[0128] Each content identifier corresponds to a unique code for the specific content (such as song ID, novel chapter ID, video ID), ensuring that each piece of content can be accurately located.

[0129] The subject identifier is the core associated subject code of the content (such as singer ID, novel author ID, video blogger ID), ensuring that account preferences can still be statistically analyzed by subject in cross-type scenarios.

[0130] Content type identifiers are standardized tags used to distinguish content types, such as "music", "novel", and "video". They are used for subsequent type isolation, aggregation, and storage to avoid confusion between different types of content data, such as separating the statistics of "song playback" and "novel reading" behavior data.

[0131] Behavioral data from each application is transmitted in real time to a unified message queue (Kafka) cluster via a dedicated line. Kafka splits independent topics according to content type identifiers to achieve parallel reception and caching of multiple types of data. Relying on Kafka's high throughput characteristics, data loss or transmission blockage in high-concurrency scenarios is avoided. At the same time, the message persistence mechanism ensures the reliability of data access and provides a stable data source for subsequent real-time computing.

[0132] Step 420: Perform normalization processing on the behavioral data based on the global account identifier to obtain standardized behavioral data.

[0133] Standardized behavioral data and interaction data are in the same format.

[0134] By normalizing the data, we can achieve unified account identity (the behavioral data of the same account is aggregated under the same global identifier) ​​and unified data format (multi-source data is formatted according to the original scheme standard), so that cross-application data has the characteristics of homogeneity and computability, and can be processed directly without reconstructing the original computing chain.

[0135] To ensure unified account identity, a pre-defined global account identifier mapping table is used to uniformly replace the local account identifiers of each application with the global account identifier. The mapping table is maintained by the unified account system and supports real-time updates. When a new application is added to the account, it is automatically associated, ensuring that all behavioral data of the same account (regardless of which application it comes from) is aggregated under the same global identifier, thus avoiding the fragmentation of account preference profiles.

[0136] To standardize field formats, the format of all behavioral data is unified according to the field specifications of the above-mentioned interactive data. For example, the format of timestamps, the unit of interaction duration, and the encoding rules of identifier fields are unified. At the same time, missing required fields are automatically filled in. For example, if some applications do not report the main identifier, the singer / author ID is filled in by associating the content identifier with the metadata dictionary.

[0137] By uniformly mapping different behavior types such as playing, reading, watching, and listening to the above-mentioned interaction behavior types, it is ensured that subsequent aggregation logic (such as interaction count and interaction duration statistics) can directly reuse the calculation rules of the original solution without additional development.

[0138] Normalization is implemented using Flink's Table API. It automatically parses multi-source data through a predefined standardized schema, performs format conversion and field completion. The processing runs in parallel with Kafka data consumption, and the normalization time for a single data record is controlled in milliseconds, which does not affect the real-time performance of the overall data processing and ensures that the processing latency of cross-application data remains in the second range.

[0139] Step 430: Perform real-time aggregation processing on the standardized behavioral data according to the global aggregation dimension to obtain cross-type aggregated data.

[0140] The global aggregation dimension includes a combination of global account identifier, subject identifier, and content type identifier.

[0141] Based on the two-dimensional aggregation dimension of account identifier + subject identifier in the data processing of a single application, a new content type identifier is added to form a three-dimensional global aggregation dimension, which enables accurate collection of cross-application and cross-type behavioral data.

[0142] The global aggregation dimension is a combination of global account identifier, subject identifier, and content type identifier, such as "global_001+author_102+novel" and "global_001+singer_203+music". It continues the original logic while distinguishing new scenarios. It retains the core aggregation dimension of "account-subject" (fitting the business needs of account preference analysis) and achieves isolated statistics for different types of content through content type identifiers, avoiding confusion of different types of behavior under the same account-subject. For example, the reading of novels and listening to audiobooks by the same author by the account are counted separately.

[0143] Reuse Flink's stateful stream processing capabilities from single-application data processing to maintain independent historical aggregation states for each global aggregation dimension, including cumulative interaction counts, total interaction duration, first interaction time, most recent interaction time, and content identifier set. When new standardized behavioral data flows in, only incremental updates are needed based on the historical state, such as incrementing the interaction count by 1, accumulating the duration, and adding new content identifiers. There is no need to recalculate the entire dataset, ensuring that processing latency remains within the second level.

[0144] Cross-type aggregated data continues the structure of statistical indicator summary data and associated sets in single-application data processing. Statistical indicators are cumulative data under this three-dimensional dimension, such as an account reading a novel by a certain author 5 times, with a total duration of 120 minutes and the first reading time being 2024-01-10. The associated set is the set of all content identifiers that the account has interacted with under this dimension, such as the set of novel chapter IDs of a certain author and the set of song IDs of a certain singer.

[0145] By using three-dimensional global aggregation, it achieves precise collection of behavioral data from the same account, the same subject, and different types of content, breaking the limitations of single application and single type, providing a data foundation for building full-scenario account preference profiles, and supporting innovative businesses such as cross-application recommendations.

[0146] Step 440: Split the set of content identifiers in the cross-type aggregated data according to the content type identifier to obtain at least two splitting results.

[0147] Type splitting ensures that the sets of identifiers for different types of content are independent, which not only meets the requirement of Bitmap compression for orderly arrangement of similar identifiers, but also avoids interference between different types of content during querying.

[0148] After cross-type aggregated data is generated, the system automatically triggers a splitting operation based on the content type identifier. For example, the content identifier set corresponding to a certain global aggregation dimension (global_001+author_102+novel) is the novel chapter ID, and the content identifier set corresponding to another dimension (global_001+author_102+audio) is the audiobook ID. The system splits the data according to the two identifiers "novel" and "audio" to obtain two independent content identifier sets, each containing only a single type of content identifier.

[0149] Based on the uniqueness of content type identifiers, different types of content identifier sets under the same "global account identifier + subject identifier" are completely separated to avoid Bitmap mapping conflicts caused by mixed storage of different types of content identifiers (such as novel ID and music ID), such as duplicate content IDs of different types, which may lead to misjudgment of binary bit status. After splitting, each split result is bound to a three-dimensional index of "global account identifier + subject identifier + content type identifier" to ensure that the target data can be quickly located through the index during subsequent queries.

[0150] The split content identifier set has the characteristics of single type and centralized identifier. The single type ensures that Bitmap mapping rules can be designed independently according to type, such as novel ID mapping table and music ID mapping table, to avoid conflicts. The centralized identifier improves the Bitmap compression ratio. IDs of the same type of content are usually consecutive, reducing invalid "0" bits in the binary sequence and laying the foundation for subsequent low-cost storage.

[0151] Step 450: Perform bitmap compression on at least two split results to obtain type-isolated storage data.

[0152] Type-isolated stored data is associated with a global account identifier.

[0153] It reuses Bitmap compression technology from single-application data processing, but adapts to cross-type scenarios through type-isolated storage, maintaining storage cost advantages while supporting efficient queries across types and single types.

[0154] Bitmap compression logic works in conjunction with single-application data processing. For each split content identifier set, the same Bitmap compression process as single-application data processing is executed. First, the corresponding global mapping rules are loaded according to the content type identifier. For example, the novel ID mapping table assigns a unique binary bit sequence number to each novel ID. Then, the content identifier set is converted into a binary sequence. A binary bit of "1" indicates that the account has interacted with the content, and a binary bit of "0" indicates that it has not been interacted with.

[0155] The compressed binary sequence (i.e., type-isolated stored data) is stored in the Apache Doris distributed storage cluster, using a storage strategy that combines type partitioning with a three-dimensional index.

[0156] Doris divides content into independent physical partitions based on content type identifiers, such as music partitions, novel partitions, and video partitions. Different types of stored data are physically isolated, avoiding storage directory chaos. At the same time, queries only need to scan the corresponding type partition, without traversing the entire data, thus improving query efficiency.

[0157] Each type of isolated stored data is bound to a three-dimensional index consisting of a "global account identifier + subject identifier + content type identifier," ensuring that single-type queries can be quickly located. For example, querying "the list of novels read by author B by account A" directly matches the three-dimensional index. At the same time, cross-partition associations are established through the global account identifier. For example, all types of stored data of account A are associated through "global_001," supporting cross-type aggregate queries, such as querying "all interaction behaviors of account A related to subject B."

[0158] Type-isolated storage data and single-application storage data share the same Doris cluster, providing unified support for query services. The query layer receives requests through an SQL interface and automatically matches the corresponding partitions and indexes based on the query dimensions. Single-type queries only scan the corresponding partition, while cross-type queries associate data from multiple partitions through a global account identifier.

[0159] In summary, the method provided in this application, by acquiring behavioral data such as playback, reading, and viewing from multiple applications, and after normalizing it based on global account identifiers, aggregates and performs bitmap compression by category according to the dimensions of global account identifier + subject identifier + content type identifier. This breaks down the data isolation barriers of single applications and single content types, thereby forming a complete profile of account content preferences, expanding the applicable business scenarios, and maintaining the low-cost storage advantages of aggregation model and bitmap compression without the need for additional large amounts of storage resources.

[0160] In conjunction with the above embodiments, the application of the data processing method provided in this application to a personal statistics system for account behavior on a music streaming platform will be used as an example for explanation.

[0161] Application scenarios include, but are not limited to, the singer recall coordinate function: accounts can view their playback history statistics for specific singers; personal music profiles: providing accounts with statistical analysis and review of personal playback behavior; account playback behavior tracking: recording and statistically analyzing the account's listening behavior data in real time.

[0162] Features include real-time playback statistics: the system immediately updates playback statistics after each song is played; artist-based aggregation: playback behavior of an account is statistically analyzed by artist, including playback duration, number of plays, and first playback time; song set query: supports quick query of the list of songs played by an account, as well as intersection operations with a specified song set.

[0163] The system automatically records playback behavior when an account plays songs in a music app; accounts can view personal playback statistics through the "Artist Memory Coordinates" page; and the system provides playback data query and statistics functions by artist.

[0164] For the hardware environment, the computing cluster adopts a distributed cluster based on the Flink streaming computing engine; the storage cluster uses the Apache Doris distributed database, which includes a 5-node FE cluster and a multi-node BE cluster; the message queue uses a Kafka cluster for data streaming transmission; and the network environment is deployed across 3 availability zones and connected via dedicated lines.

[0165] Regarding the front-end implementation logic, when the music app client plays a song, it reports playback events through the event tracking SDK; the playback event includes information such as account ID, song ID, artist ID, and playback duration.

[0166] The backend processing flow includes the following steps: 1. Data access layer: Kafka receives playback event streams from clients.

[0167] 2. Real-time computing layer: Flink consumes the Kafka data stream and performs the following processing: • Data deduplication is performed based on the ROW_NUMBER() window function; • Aggregate data by (userid, singerid) dimension; • Use stateful streaming to maintain account playback status.

[0168] 3. Storage layer: Processed data is written to the Doris database. • Automatically merge playback records of the same account-singer using an aggregation model; • Compress and store the song ID set using Bitmap technology.

[0169] 4. Query Layer: Provides real-time data query services through SQL interface.

[0170] The data processing flow includes account playback behavior → data tracking and reporting → Kafka queue → Flink real-time computing → data deduplication and aggregation → Doris storage → real-time query service. Through this data processing flow, accurate deduplication is achieved based on ROW_NUMBER(), space utilization is optimized based on Bitmap storage technology, data consistency is guaranteed based on two-phase commit, and real-time pre-aggregation is achieved based on the aggregation model.

[0171] In summary, the method provided in this application proposes a real-time processing system for large-scale account music playback behavior, capable of handling trillions of account-singer-song relationship data. The system achieves high-concurrency write processing of 50,000 records per second through the Flink streaming engine, employs a deduplication algorithm based on ROW_NUMBER() window sorting to ensure data uniqueness, and combines a two-phase commit mechanism to guarantee exactly-Once semantics. At the storage level, an aggregation model combined with Bitmap storage technology is used to achieve efficient compressed storage and fast querying of the song collection played by each account. The system adopts a high-availability deployment architecture across three availability zones, supporting a query load of 3000+ queries per second (QPS), providing real-time and accurate data support for music recommendation and account profile construction.

[0172] The method provided in this application improves performance, including shortening query response time from seconds in traditional solutions to milliseconds (average response time <100ms); enhancing concurrency processing capabilities, supporting 50,000 transactions per second (TPS) and 3000+ query QPS; ensuring data real-time performance, reducing playback behavior data latency to seconds to meet real-time recommendation needs; optimizing costs, including reduced storage costs (over 80% reduction through aggregation models and bitmap compression); optimized computing resources (pre-aggregation reduces computation during queries, lowering CPU utilization by 60%); reduced operational costs (high-availability architecture reduces system failure rate, lowering operational costs by 50%); and enhanced business value, including optimized account experience (second-level response for personal music statistics, improving account satisfaction); and guaranteed data quality (precise deduplication mechanism ensures statistical data accuracy reaches 99.9%).

[0173] Figure 5 This is a structural block diagram of a data processing apparatus provided in an exemplary embodiment of this application, such as... Figure 5 As shown, the device includes the following parts: The acquisition module 510 is used to acquire the interaction data corresponding to the account, and the interaction data is used to indicate the account's interactive operation on multimedia content in the first application. The deduplication module 520 is used to perform real-time deduplication processing on the interactive data to obtain deduplicated interactive data. The aggregation module 530 is used to perform real-time aggregation processing on the deduplicated interactive data according to the aggregation dimension to obtain aggregated data. The aggregation dimension includes an account identifier and a subject identifier. The subject identifier is used to indicate the associated subject corresponding to the multimedia content of the account interaction. Storage module 540 is used to perform bitmap compression on the aggregated data to obtain stored data, which is used to provide real-time query services to the account.

[0174] In some embodiments, the deduplication module 520 is further configured to: Divide the data window, which is used to indicate the interactive data generated by the account for the multimedia content corresponding to the associated subject within the window duration; Sort the interactive data within the data window to obtain the sorting result; Based on the sorting results, valid interactive data is filtered to obtain the deduplicated interactive data.

[0175] In some embodiments, the deduplication module 520 is further configured to: The interactive data within the data window is sorted according to the time of data generation to obtain the sorting result; The step of filtering valid interaction data based on the sorting results to obtain the deduplicated interaction data includes: The interactive data that is ranked first from the last to the first in the sorting results is selected as the valid data, and the deduplicated interactive data is obtained.

[0176] In some embodiments, the aggregation module 530 is further configured to update statistical indicators according to the aggregation dimension, summarize the interaction-related information of the deduplicated interaction data, and obtain the aggregated data. The aggregated data includes summary data aggregated according to the statistical indicators and an association set. The statistical indicators include at least one of interaction count, interaction duration, first interaction time, and most recent interaction time. The association set includes a set of content identifiers corresponding to the multimedia content that the account has interacted with under the aggregation dimension.

[0177] In some embodiments, the storage module 540 is further configured to convert the set of content identifiers in the aggregated data into a binary sequence to obtain the stored data, wherein the binary bits in the binary sequence are used to indicate whether the multimedia content indicated by the corresponding content identifier has been interacted with by the account.

[0178] In some embodiments, the acquisition module 510 is further configured to: Receive the event tracking data reported by the first application based on interactive events; The interaction data is obtained by parsing the data points. The interaction data includes at least one of the following: the account identifier and the content identifier corresponding to the multimedia content, the subject identifier, the interaction duration, and the time when the data was generated.

[0179] Please refer to Figure 6 , Figure 6 This is a structural block diagram of a data processing apparatus module provided in an exemplary embodiment of this application, such as... Figure 6 As shown, in some embodiments, the device further includes a query module 550, used for: Receive a query request initiated by the account, the query request including query dimensions; In response to the query request, the stored data is queried in real time according to the query dimension to obtain the query results.

[0180] In some embodiments, the query module 550 is further configured to: Receive the query request initiated by the account targeting the first entity identifier; In response to the query request, the stored data is queried according to the combination dimension of the account identifier and the first subject identifier to obtain the query result.

[0181] In some embodiments, the query module 550 is further configured to: Receive the query request initiated by the account for the content set; In response to the query request, extract the set of content identifiers corresponding to the account; Perform an intersection operation between the binary sequences corresponding to the content identifier set and the content set in the query request to obtain the operation result; The result of the operation is decoded into a list of content identifiers to obtain the query result.

[0182] In some embodiments, the storage module 540 is further configured to: A data storage preparation request is sent to the storage layer, the data storage preparation request including the storage data. The storage layer is used to verify the storage status based on the data storage preparation request. If the storage status is normal, the storage data is written to a temporary storage area and the corresponding storage resource is locked. The storage layer includes at least two storage nodes participating in storage. If at least two storage nodes return a full readiness response, a data confirmation storage request is sent to the storage layer. The data confirmation storage request is used to instruct the storage layer to migrate the stored data in the temporary storage area to the final storage area and release the storage resources. If, in the case where a first storage node among the at least two storage nodes does not return a preparation failure response, a data discard request is sent to the storage layer. The data discard request is used to instruct the storage layer to delete the stored data in the temporary storage area and release the storage resources.

[0183] In some embodiments, the apparatus further includes a normalization module 560 and a splitting module 570; The acquisition module 510 is also used to acquire behavioral data reported by multiple applications. The behavioral data includes at least one of playback data, reading data, and viewing data. The behavioral data corresponds to a global account identifier, a content identifier, a subject identifier, and a content type identifier. The global account identifier is used to associate the same account in different applications. The normalization module 560 is used to perform normalization processing on the behavior data based on the global account identifier to obtain standardized behavior data, wherein the standardized behavior data has the same format as the interaction data. The aggregation module 530 is further configured to perform real-time aggregation processing on the standardized behavioral data according to the global aggregation dimension to obtain cross-type aggregated data. The global aggregation dimension includes a combination dimension of the global account identifier, the subject identifier, and the content type identifier. The splitting module 570 is used to split the set of content identifiers in the cross-type aggregated data according to the content type identifier, and obtain at least two splitting results; The storage module 540 is further configured to perform bitmap compression on the at least two splitting results respectively to obtain type-isolated storage data, which is associated with the global account identifier.

[0184] In summary, the apparatus provided in this application, by constructing a full-link streaming processing architecture that includes real-time deduplication, real-time aggregation, bitmap compression, and real-time query services, pre-processes the deduplication and aggregation stages of account interaction data after it is generated using real-time processing logic. This eliminates the need to wait for fixed offline processing cycles, reducing data latency from hours to seconds, thus meeting real-time statistical requirements. Aggregation processing summarizes multiple scattered playback records into a single aggregated data record containing a set of statistical indicators and content identifiers. Bitmap compression technology then converts the content identifier set into a binary sequence, replacing the complete identifier with the smallest storage unit, reducing storage costs. Aggregation processing enables the pre-calculation of statistical indicators required for querying, and bitmap compression supports fast bit operations, compressing storage response time and improving data processing efficiency.

[0185] It should be noted that the data processing device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0186] Figure 7 The diagram illustrates a structural block diagram of a terminal 700 provided in an exemplary embodiment of this application. The terminal 700 may be a smartphone, tablet computer, MP3 player, MP4 player, laptop computer, or desktop computer. The terminal 700 may also be referred to as an account device, portable terminal, laptop terminal, desktop terminal, or other names.

[0187] Typically, terminal 700 includes a processor 701 and a memory 702.

[0188] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0189] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 are used to store at least one instruction, which is executed by the processor 701 to implement the data processing method provided in the method embodiments of this application.

[0190] In some embodiments, the terminal 700 also includes other components 703, as those skilled in the art will understand. Figure 7 The structure shown does not constitute a limitation on terminal 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0191] Embodiments of this application also provide a computer device that can be implemented as follows: Figure 1 The terminal or server shown. The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, at least one program, code set, or instruction set being loaded and executed by the processor to implement the data processing methods provided in the above-described method embodiments.

[0192] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the data processing method provided in the above-described method embodiments.

[0193] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing methods provided in the above-described method embodiments.

[0194] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0195] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0196] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the interaction data corresponding to the account, the interaction data being used to instruct the account on interactive operations of multimedia content in the first application; The interaction data is processed in real time to obtain the deduplicated interaction data; The deduplicated interactive data is processed in real time according to the aggregation dimension to obtain aggregated data. The aggregation dimension includes account identifier and subject identifier. The subject identifier is used to indicate the associated subject corresponding to the multimedia content of the account interaction. Bitmap compression is performed on the aggregated data to obtain stored data, which is used to provide real-time query services to the account.

2. The method according to claim 1, characterized in that, The real-time deduplication process for the interaction data to obtain deduplicated interaction data includes: Divide the data window, which is used to indicate the interactive data generated by the account for the multimedia content corresponding to the associated subject within the window duration; Sort the interactive data within the data window to obtain the sorting result; Based on the sorting results, valid interactive data is filtered to obtain the deduplicated interactive data.

3. The method according to claim 2, characterized in that, The process of sorting the interactive data within the data window to obtain the sorting result includes: The interactive data within the data window is sorted according to the time of data generation to obtain the sorting result; The step of filtering valid interaction data based on the sorting results to obtain the deduplicated interaction data includes: The interactive data that is ranked first from the last to the first in the sorting results is selected as the valid data, and the deduplicated interactive data is obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The real-time aggregation processing of the deduplicated interactive data according to the aggregation dimension yields aggregated data, including: The statistical indicators are updated according to the aggregation dimension, and the interaction-related information of the deduplicated interaction data is summarized to obtain the aggregated data. The aggregated data includes the summary data aggregated according to the statistical indicators and the association set. The statistical indicators include at least one of the number of interactions, interaction duration, first interaction time, and most recent interaction time. The association set includes the content identifier set corresponding to the multimedia content that the account has interacted with under the aggregation dimension.

5. The method according to claim 4, characterized in that, The step of performing bitmap compression on the aggregated data to obtain stored data includes: The set of content identifiers in the aggregated data is converted into a binary sequence to obtain the stored data. The binary bits in the binary sequence are used to indicate whether the multimedia content indicated by the corresponding content identifier has been interacted with by the account.

6. The method according to any one of claims 1 to 3, characterized in that, The acquisition of interaction data corresponding to the account includes: Receive the event tracking data reported by the first application based on interactive events; The interaction data is obtained by parsing the data points. The interaction data includes at least one of the following: the account identifier and the content identifier corresponding to the multimedia content, the subject identifier, the interaction duration, and the time when the data was generated.

7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Receive a query request initiated by the account, the query request including query dimensions; In response to the query request, the stored data is queried in real time according to the query dimension to obtain the query results.

8. The method according to claim 7, characterized in that, Receiving the query request initiated by the account includes: Receive the query request initiated by the account targeting the first entity identifier; In response to the query request, the stored data is queried in real time according to the query dimension to obtain query results, including: In response to the query request, the stored data is queried according to the combination dimension of the account identifier and the first subject identifier to obtain the query result.

9. The method according to claim 7, characterized in that, Receiving the query request initiated by the account includes: Receive the query request initiated by the account for the content set; In response to the query request, the stored data is queried in real time according to the query dimension to obtain query results, including: In response to the query request, extract the set of content identifiers corresponding to the account; Perform an intersection operation between the binary sequences corresponding to the content identifier set and the content set in the query request to obtain the operation result; The result of the operation is decoded into a list of content identifiers to obtain the query result.

10. The method according to any one of claims 1 to 3, characterized in that, The method further includes: A data storage preparation request is sent to the storage layer, the data storage preparation request including the storage data. The storage layer is used to verify the storage status based on the data storage preparation request. If the storage status is normal, the storage data is written to a temporary storage area and the corresponding storage resource is locked. The storage layer includes at least two storage nodes participating in storage. If at least two storage nodes return a full readiness response, a data confirmation storage request is sent to the storage layer. The data confirmation storage request is used to instruct the storage layer to migrate the stored data in the temporary storage area to the final storage area and release the storage resources. If a first storage node among the at least two storage nodes does not return a preparation failure response, a data discard request is sent to the storage layer. The data discard request is used to instruct the storage layer to delete the stored data in the temporary storage area and release the storage resources.

11. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain behavioral data reported by multiple applications. The behavioral data includes at least one of playback data, reading data, and viewing data. The behavioral data corresponds to a global account identifier, a content identifier, a subject identifier, and a content type identifier. The global account identifier is used to associate the same account with different applications. The behavior data is normalized based on the global account identifier to obtain standardized behavior data, which has the same format as the interaction data. The standardized behavioral data is aggregated in real time according to the global aggregation dimension to obtain cross-type aggregated data. The global aggregation dimension includes a combination of the global account identifier, subject identifier, and content type identifier. The content identifier set in the cross-type aggregated data is split according to the content type identifier to obtain at least two splitting results; Bitmap compression is performed on the at least two splitting results to obtain type-isolated storage data, which is associated with the global account identifier.

12. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the interaction data corresponding to the account, and the interaction data is used to indicate the account's interactive operations on multimedia content in the first application; The deduplication module is used to perform real-time deduplication processing on the interactive data to obtain deduplicated interactive data. The aggregation module is used to perform real-time aggregation processing on the deduplicated interactive data according to the aggregation dimension to obtain aggregated data. The aggregation dimension includes account identifier and subject identifier. The subject identifier is used to indicate the associated subject corresponding to the multimedia content of the account interaction. The storage module is used to perform bitmap compression on the aggregated data to obtain stored data, which is used to provide real-time query services to the account.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the data processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer program, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1 to 11.

Images