Data publishing method and device, storage medium and computer device

Abstract

Embodiments of the present application disclose a data publishing method and device, a storage medium and a computer device. The method comprises: generating flow data corresponding to change data generated by a data producer according to the change data, wherein the flow data is used for recording data change log information of the change data, each piece of flow data has a spatially unique flow number; preprocessing the flow data, the data volume of the preprocessed flow data being less than that of the flow data before preprocessing; caching the change data and the preprocessed flow data to a data cache layer; monitoring a flow number service, and if it is determined that the flow number of the flow space is changed, reading target flow data and target change data recorded by the target flow data from the data cache layer according to a target flow number that is changed, and publishing the target change data to a data user, so that the data publishing capability can be improved and the data publishing can be quickly and efficiently realized.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data publishing method, apparatus, storage medium, and computer equipment. Background Technology

[0002] Currently, large-scale, complex internet advertising systems can generate tens of millions or even hundreds of millions of data entries per minute. Advertising data has certain logical object relationships, and the common advertising model is a relational entity data model. However, a single relational data model has significant limitations for large-scale, complex advertising systems. How to quickly and effectively distribute data generated by the data producer to the data user has become a major challenge for advertising systems. Summary of the Invention

[0003] This application provides a data publishing method, apparatus, storage medium, and computer equipment, which can improve data publishing capabilities and enable data publishing quickly and efficiently.

[0004] Firstly, a data publishing method is provided, the method comprising: generating serial data corresponding to the changed data generated by the data producer, wherein the serial data is used to record data change log information of the changed data, each serial data has a spatially unique serial number, which is maintained by a serial number service; preprocessing the serial data, wherein the data volume of the preprocessed serial data is less than the data volume of the serial data before preprocessing; caching the changed data and the preprocessed serial data in a data cache layer; monitoring the serial number service to determine whether the serial number in the serial space has changed, wherein the serial space is a data set recording the serial data and the serial number; if it is determined that the serial number in the serial space has changed, then reading the target serial data and obtaining the target changed data recorded in the target serial data from the data cache layer according to the target changed serial number, and publishing the target changed data to the data user.

[0005] Secondly, a data publishing device is provided, comprising: a generation unit, configured to generate serial data corresponding to the changed data generated by a data producer, wherein the serial data is used to record data change log information of the changed data, and each serial data has a spatially unique serial number, which is maintained by a serial number service; a preprocessing unit, configured to preprocess the serial data, wherein the data volume of the preprocessed serial data is less than the data volume of the serial data before preprocessing; a caching unit, configured to cache the changed data and the preprocessed serial data in a data caching layer; a monitoring unit, configured to monitor the serial number service to determine whether the serial number in the serial space has changed, wherein the serial space is a data set recording the serial data and the serial number; and a publishing unit, configured to, if it is determined that the serial number in the serial space has changed, read the target serial data and obtain the target changed data recorded in the target serial data from the data caching layer according to the target changed serial number, and publish the target changed data to the data user.

[0006] Thirdly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program adapted for loading by a processor to perform the steps in the data publishing method as described in any of the above embodiments.

[0007] Fourthly, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the processor executing steps in the data publishing method as described in any of the above embodiments by calling the computer program stored in the memory.

[0008] This application embodiment generates sequential data corresponding to the changed data based on the changed data generated by the data producer. The sequential data records the data change log information of the changed data. Each sequential data entry has a spatially unique sequential number, maintained by a sequential number service. The sequential data is preprocessed, and the amount of data in the preprocessed sequential data is less than the amount of data in the original sequential data. The changed data and the preprocessed sequential data are cached in a data cache layer. The sequential number service is monitored to determine whether the sequential number in the sequential space has changed. The sequential space is a data set that records sequential data and the sequential numbers. If it is determined that the sequential number in the sequential space has changed, the target sequential data and the target changed data recorded in the target sequential data are read from the data cache layer based on the changed target sequential number, and the target changed data is published to the data user. This application embodiment optimizes data by preprocessing the transaction data based on the transaction data corresponding to the changed data, reduces the latency of data ownership and alleviates the pressure of data reading on the business data system by pre-caching the changed data and transaction data, and accelerates the data release rate by monitoring the transaction number service to detect changes in the transaction number. When the transaction number changes, the target transaction data and the target changed data corresponding to the target transaction data are read from the data cache layer according to the changed target transaction number, thereby improving the data release capability and enabling fast and efficient data release. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic diagram of the framework of the advertising system provided in the application embodiment.

[0011] Figure 2 This is a schematic diagram of the entity relationship model provided in the embodiments of this application.

[0012] Figure 3 This is a schematic diagram of a key-value pair data model provided in an embodiment of this application.

[0013] Figure 4 This is a flowchart illustrating the data publishing method provided in an embodiment of this application.

[0014] Figure 5 A schematic diagram of a first application scenario for the data publishing method provided in this application embodiment.

[0015] Figure 6 This is a schematic diagram of a second application scenario for the data publishing method provided in the embodiments of this application.

[0016] Figure 7 A schematic diagram of a third application scenario of the data publishing method provided in the embodiments of this application.

[0017] Figure 8 This is a schematic diagram of a fourth application scenario of the data publishing method provided in the embodiments of this application.

[0018] Figure 9 This is a schematic diagram of the structure of the data publishing device provided in the embodiments of this application.

[0019] Figure 10 Another schematic diagram of the data publishing device provided in the embodiments of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This application provides a data publishing method, apparatus, computer device, and storage medium. Specifically, the data publishing method of this application can be executed by a computer device, which can be a terminal or a server, etc.

[0022] First, some of the nouns or terms that appear in the description of the embodiments of this application are explained as follows:

[0023] Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data within a tolerable timeframe. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the internet, and scalable storage systems.

[0024] A database, simply put, can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, capable of being shared by multiple users, with minimal redundancy, and independent of application programs.

[0025] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0026] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Backend services of cloud computing systems require substantial computing and storage resources, such as those used by video websites, image websites, and many portal websites; therefore, cloud technology relies on cloud computing as its foundation.

[0027] A data entity represents a table in a relational database or a non-relational database.

[0028] A field represents an attribute of a data storage entity.

[0029] A rule represents a configurable read / write strategy or method for a data entity.

[0030] Events include four types: add, delete, modify, and query.

[0031] A view is a virtual entity in a database system. It is a set of query statements that combine to form a queryable virtual data entity object.

[0032] A trigger is a database object associated with an event that affects a data entity. Its execution is triggered by an event. Triggers can query other tables and can contain complex Structured Query Language (SQL) statements. They can also be used to enforce referential integrity, preserving the relationships defined between multiple tables when rows are added, updated, or deleted.

[0033] A stored procedure is a set of SQL statements in a database system designed to perform a specific function. It is stored in the database and remains valid permanently after being compiled once. Stored procedures are an important object in databases; when dealing with extremely large amounts of data, they can significantly improve efficiency.

[0034] A function is a piece of code that can be directly referenced by another piece of code.

[0035] A data service domain typically refers to the owner of a data entity, usually the creator of the data entity or data object, or the producer of the data entity, such as an advertising delivery system for a certain traffic.

[0036] The data circulation domain typically refers to the entity that uses the data, which could be an advertising system of a "traffic" or "traffic alliance".

[0037] Data infrastructure (DI) is a data intermediary infrastructure between data users and data storage systems.

[0038] Internet advertising systems include data producers of advertising-related data, encompassing various business domains such as customer management, ad placement, traffic management, review, feedback logs, and billing. Different business domains generate different advertising-related data. This advertising data needs to be reviewed, retrieved, analyzed, calculated, and played by data users. Ad launches generate user logs, billing data, and other related information. Therefore, advertising data produced by different business domains may be used by various downstream data users. From data producers to data users, data needs to be published rapidly. Complex internet advertising systems produce tens of millions or even hundreds of millions of data entries per minute. Data producers in different business domains may use different models and corresponding transmission or processing objects when producing data. Similarly, data users may also use different models and objects for data utilization.

[0039] Because data producers and users may have different data storage models, transmission objects, and processing objects, how to quickly and efficiently publish data generated by data producers to data users has become a major problem for advertising systems. At the same time, data changes inevitably require consistency between data producers and users. A single data change involves data reassembly, object conversion, object transmission, and the re-retrieval, analysis, and calculation of data. The parallel changes and publication of billions of advertising data points further increase the system's complexity.

[0040] This application proposes a data publishing method between data producers and users of advertising data, enabling rapid and efficient data publishing. This method is applicable to scenarios where data producers and users employ different data models (e.g., object models, storage models, processing models, etc.). The method is based on pipeline data corresponding to table-level changes generated by a middleware layer above the database. Through pipeline data preprocessing—compressing, filtering, and merging the pipeline data—and pre-caching of changes, the method accelerates the data publishing process, achieving fast and efficient data publishing. Simultaneously, during the data publishing process, various data processing attachments (plugins) are embedded, including those for data resource authorization and data object transformation (e.g., rule systems, complex strategy trees), allowing advertising data to be published from the data producer to multiple data users, achieving diverse data publishing capabilities.

[0041] Please see Figure 1 , Figure 1 This is a schematic diagram of the framework of an advertising system provided in an embodiment of this application. The advertising system 100 includes a data publishing platform 110, a data producer 120, a data user 130, and a serial number service 140. The data publishing method is applied to the data publishing platform 110, which can run on a computer device, such as a terminal or a server. The data publishing platform 110 may include a data caching layer 111 (for pre-caching of changed data), a preprocessing layer 112 (for compressing, filtering, and merging data change serial numbers), a data resource authorization layer 113 (for data resource authorization), a data strategy calculation layer 114 (for data strategy calculation), a rule system 115, a data object conversion layer 116 (for data object conversion), and a data publishing layer 117 (for data publishing).

[0042] Among them, the rule system 115 is a data read and write access strategy and rule based on distributed storage. It is a data model abstraction of distributed data storage objects for different data business domains. Using the rule system 115, data can be presented or represented in various ways for the same or several data entities according to different specific business needs. The rule system 115 provides various mapping capabilities for data objects in distributed scenarios for different data business domains or data circulation domains.

[0043] For example, Figure 1 The data flow diagram for data publishing is shown, and its main process is as follows:

[0044] (1) Business data is changed or modified by data producer 120 in the business system;

[0045] (2) Changes in business data trigger the generation of data flow to obtain flow data. Each flow data has a spatially unique flow number; the flow number service maintains the monotonically increasing flow number;

[0046] (3) The data cache layer 111 of the data publishing platform 110 can monitor the change of the serial number and actively pull the changed data to cache. Specifically, it can obtain the serial data according to the serial number to obtain the changed business data recorded in the serial data and cache it. There is another way: when the business data changes and the serial data is generated, the data cache layer 111 is notified synchronously that there is a change in the business data.

[0047] (4) The data publishing layer 117 of the data publishing platform 110 can also monitor the changes in serial number, and then pull the changed business data, and synchronize or send the changed business data to the data user 130.

[0048] (5) When the data publishing layer 117 publishes data to the data user 130, it may involve data object conversion through data object conversion (such as using the rules of the rule system, data strategy calculation layer, etc. to perform conversion). This is because different data users may have different data models for data use and storage, and the same data object generated by the data producer may correspond to different data objects in different data users.

[0049] (6) Data user 130 may also mainly pull data from data publishing platform 110 by sending access requests. Data publishing platform 110 needs to authorize data user 130 to use data (data resource authorization).

[0050] (7) The preprocessing of the pipeline data may involve intelligent optimization of the pipeline processing: such as pipeline compression; for example, if the same data is modified multiple times in a short period of time, the data publishing platform 110 can perform pipeline compression through the preprocessing layer 112 when performing pipeline data preprocessing, and the amount of pipeline data after preprocessing is less than the amount of pipeline data before preprocessing.

[0051] The production of advertising data mainly includes customer management, ad placement, review, and traffic management (such as ad placement management). Advertisers' ad placements generate a large amount of advertising-related data. The advertising system model is typically a three- or four-layer entity-relationship (ER) model.

[0052] For example, advertisers can match account information, funds, etc.

[0053] For example, a promotion plan may include information such as promotion methods, bidding models, and purchasing methods.

[0054] For example, an ad group can include cost-related information such as budget, bid, billing, and deductions, as well as information such as ad targeting, products, and industry categories, and information such as site collections, ad placements, and traffic packages.

[0055] For example, advertising creative ideas can include information such as creative form, creative specifications, materials, and elements.

[0056] For example, such as Figure 2 The Entity Relationship (ER) model shown includes an advertiser that may have multiple campaigns, a campaign that may correspond to multiple ad groups, an ad group that may have multiple ad creatives, and an ad creative that may also correspond to multiple ad groups.

[0057] For example, the use of advertising data includes ad retrieval, analysis, calculation, and playback. After advertising data is generated, it is processed by the users: for example, by retrieving, analyzing, calculating, and playing (launching) ads. After the ads are launched, billing for the ads and further analysis of the generated user logs may be required. Ad retrieval, analysis, calculation, and playback use completely different data models from ad delivery. For example, in ad retrieval or playback, a key-value pair (key, value) structure with ads as the dimension is more convenient for ad retrieval, analysis, or calculation. This may involve a rapid mapping or object conversion from an entity relationship model to a key-value pair data model.

[0058] For example, such as Figure 3 The key-value pair data model shown has the keyword "ad ID" as the key, and the stored values ​​include advertiser information, campaign information, ad group information, targeting information, ad creative information, product information, traffic packages, ad placements, and other information.

[0059] Among them, for the data producers of advertising data, after the advertising data is changed or generated, it needs to be quickly published to the data users through the data publishing platform.

[0060] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0061] The embodiments of this application provide a data publishing method, which can be executed by a terminal or a server, or by both a terminal and a server. The embodiments of this application illustrate the data publishing method executed by a server as an example.

[0062] Please see Figures 4 to 8 , Figure 4 This is a flowchart illustrating the data publishing method provided in an embodiment of this application. Figures 5 to 8 This is a schematic diagram illustrating an application scenario of the data publishing method provided in an embodiment of this application. The method includes:

[0063] Step 401: Based on the change data generated by the data producer, generate the corresponding transaction data. The transaction data is used to record the data change log information of the change data. Each transaction data has a spatially unique transaction number, which is maintained by the transaction number service.

[0064] In this system, data producers are the various business domains within the advertising system, while data users are the data circulation domains, requiring timely access to the latest data change information. Database systems typically employ write-ahead logging (such as the binary log in MySQL), with the granularity of this logging at the instance level. For example, the transaction logs used to record data change log information can be represented using transaction logs. In this application, the transaction logs are table-level data change log information above the database. Granularity refers to the level of detail in data statistics within the same dimension; table granularity indicates the table level. A database instance may have many tables, and transaction logs can be configured precisely at the table level. Transaction logs are the middleware layer's log records of data changes. For example, a transaction log contains snapshot information about data changes, such as change time, source, change request body, transaction type, transaction number, and transaction body. The transaction body may include details of the change transaction, such as the entity / table of the changed data, the primary key / unique index of the changed data, the data partition key, and the old and new values ​​of the changed fields.

[0065] The transaction log can be a global transaction space, or it can be formed by combining different entity tables based on different business domains or across business domains. For example, one transaction log corresponds to one data change transaction, meaning that one transaction log may include changes to one or more data records. For example, one transaction log has a unique transaction number within a transaction space.

[0066] The serial number service maintains the serial numbers. For example, the serial number service maintains a monotonically increasing serial number, incrementing the serial number in the serial space whenever new data changes. This serial number service could be a sequence number service or a sequence service. The sequence number service is used to implement globally unique and monotonically increasing sequence numbers. For example, the sequence number service could be a Redis-based service, based on the Redis INCR command. Redis's "INCR AND GET" is an atomic operation, and Redis's single-process, single-threaded architecture prevents duplicate serial numbers from being generated by multiple INCR commands. Therefore, using the Redis INCR command to generate sequence numbers can produce globally unique and monotonically increasing sequence numbers. For example, an atomic operation is an operation that cannot be interrupted by the thread scheduling mechanism; an atomic operation is indivisible and will not be interrupted by any other task or event before completion; when multiple processes (threads) access shared resources, it ensures that all other processes (threads) do not access the same resource at the same time. Among them, Sequence is a special table in the data that stores arithmetic sequences. This table is controlled by the database system. At any time, the database system can obtain the next record in the table based on the current number of records plus the step size. This table is often used as a primary key.

[0067] For example, a serial number can be represented by a sequence number. The serial number can be an integer value, and it will increment continuously as data changes. For example, if there are concurrent business changes, the serial number service can ensure atomic operations.

[0068] Step 402: Preprocess the streaming data, wherein the amount of data in the preprocessed streaming data is less than the amount of data in the streaming data before preprocessing.

[0069] Optionally, the preprocessing of the pipeline data includes: performing merging, compression, and filtering processing on the pipeline data within each data processing window of the same pipeline space to obtain the preprocessed pipeline data.

[0070] Optionally, the transaction data includes a transaction log, which contains the primary key and partition key of the target change data. The process of merging, compressing, and filtering the transaction data includes:

[0071] Merge and compress the log entries corresponding to the same row of data in the aforementioned log data, wherein the same row of data contains data whose primary key and partition key are consistent; and / or

[0072] The streaming data is filtered by discarding streaming logs that are not of interest to the data user or that are meaningless.

[0073] Optionally, merging and compressing the logs corresponding to the same row of data in the log data includes:

[0074] When multiple transaction logs exist, the transaction logs belonging to the same primary key are identified from the multiple transaction logs, and the data corresponding to the partition key that is consistent with the primary key is obtained from the transaction space according to the primary key to obtain the same row of data;

[0075] If the same data packet contains multiple changed log entries, then determine whether the changed objects of the multiple changed log entries are the same;

[0076] If the changed objects in the multiple changed logs are the same, then when merging and compressing the same row of data, the log with the last change in the multiple changed logs will be determined as the preprocessed log data; or

[0077] If the changed objects in the multiple changed logs are different, then when merging and compressing the same row of data, the different changed objects in the multiple changed logs are merged, and the same non-changed objects in the multiple changed logs are compressed into a single non-changed object, so that the data containing the single non-changed object and the multiple different changed objects is determined as the preprocessed log data.

[0078] For example, the same row of data may be modified multiple times within a period of time, and some of these modifications may be meaningless or irrelevant to the data user. Therefore, it is essential to preprocess the streaming data to reduce or eliminate irrelevant or meaningless data, thereby optimizing the streaming data. The consistency between the data produced and the data used by the data user is an eventual consistency over a period of time. Complex internet advertising systems generate tens of millions or even hundreds of millions of data entries per minute. Therefore, preprocessing the streaming data is an important means for data publishing platforms to accelerate data distribution. For example, streaming data preprocessing can include stream merging, compression, and filtering.

[0079] This involves preprocessing the pipeline data, which means performing merging, compression, and filtering of the pipeline data within each data processing window (time window) for the same pipeline space. For example, when data is concurrent, there may be multiple data processing windows. The pipeline data within multiple data processing windows can be preprocessed in parallel, or the pipeline data within multiple data processing windows can be preprocessed sequentially.

[0080] For example, the merge and compression process includes merging and compressing change data logs (transaction logs) for the same row of data (with consistent primary key and partition key). While a primary key uniquely identifies data, it doesn't necessarily pinpoint the data's storage location. The transaction log records both the primary key and partition key of the changed data. Therefore, the changed data can be located first through the primary key; multiple transaction logs belonging to the same primary key correspond to changes in the same business data. The partition key can then be used to locate the storage location of the transaction log (or the corresponding changed data). Finally, the consistency between the primary key and partition key confirms that the data belongs to the same row. For example, during pipeline merging and compression, if the same row of data corresponding to a certain business is modified multiple times within a period of time, there will be multiple pipeline logs for that row of data. The first pipeline log records the modified detail A (the object of change) and the unmodified content in the first changed data, and the second pipeline log records the modified detail B (the object of change) and the unmodified content in the second changed data. During merging and compression, the first and second pipeline logs are compared. If detail A and detail B belong to the same object of change, then the pipeline log of the last change is determined as the preprocessed pipeline data according to the order of change time. For example, if detail A and detail B do not belong to the same object of change, then after merging and compression, a preprocessed pipeline data containing detail A, detail B, and the unmodified content common to the first and second pipeline logs is generated to reduce the data volume of the original pipeline data. For example, the object of change can be an entity / table, primary key / unique index, data partition key, field, etc. within the business data.

[0081] For example, filtering includes discarding data change logs (transaction logs) that are not relevant to or meaningless to the data user. Understandably, different filtering can be performed based on different data users. For example, for the same data table, data user A may need it and therefore cannot be filtered, while data user B may not be interested in it and therefore needs to be filtered.

[0082] For example, such as Figure 5 As shown, pipeline data preprocessing is performed on the original pipeline spatial sequence. Specifically, pipeline data within a time window T is merged, compressed, and filtered to obtain a compressed pipeline spatial sequence. The compressed pipeline data queue within the compressed time window T' is smaller than the pipeline data queue within the uncompressed time window T. Here, the pipeline data queue represents the chronological order in which the pipeline data is generated, with the pipeline numbers arranged in ascending order. Typically, the time window is fixed, i.e., T equals T', for example, a time window of 100 milliseconds. The preprocessed pipeline data will definitely be less than the uncompressed pipeline data, thereby improving the system's processing capacity and accelerating data release time.

[0083] Step 403: Cache the changed data and the preprocessed stream data to the data cache layer.

[0084] Optionally, caching the changed data and the preprocessed serial data in the data cache layer includes: controlling the data cache layer to monitor changes in the serial number; when a change in the serial number is detected, controlling the data cache layer to obtain the corresponding preprocessed serial data based on the serial number and cache it; and actively pulling the changed data recorded in the preprocessed serial data and caching it; or

[0085] When the transaction data corresponding to the changed data is generated, the change notification is sent to the data cache layer, and the data cache layer is controlled to obtain the preprocessed transaction data and the change data recorded in the preprocessed transaction data according to the change notification and cache them.

[0086] The change logs corresponding to the changed data are data change logs (stream logs) built on top of the database in the middleware layer. Therefore, the data publishing platform can detect data changes through the stream log mechanism, thus providing conditions for pre-caching of upcoming changed data releases (such as data queries and reads). Pre-caching changed data rows in a caching system (such as a data caching layer) reduces data read latency, alleviates the pressure on the business data system from large-scale data reads, and accelerates data publishing.

[0087] In this system, data producers are the various business domains, while circulation domains, acting as data users, subscribe to these business domains and, upon data changes (the data flow), retrieve the modified data based on the change messages. Data change messages in the database system are primarily recorded in log files (e.g., binlog), which are used to monitor these messages. Data users build a database replication system to monitor data changes. A data change flow is used to communicate changes between data producers and users. Specifically, when business data is modified by a business domain, the Data Infrastructure (DI) generates a data flow record of these changes.

[0088] There are two main methods for updating the data cache on the data publishing platform: The first method involves the data cache layer monitoring changes in the serial number and proactively retrieving and caching the changed data. Specifically, it retrieves the serial data based on the serial number to obtain the changed business data recorded in the serial data and caches it. The second method involves synchronously notifying the data cache layer of the business data change when serial data is generated.

[0089] For example, such as Figure 6 The image shows the details of the second update method:

[0090] a) The business domain initiates a data change request (data write request) to the data system;

[0091] b) Data writing: The data system first modifies the business data;

[0092] c) Replication Notification: Notifies the data cache layer of changes to business data; the data cache layer retrieves the changed data and the corresponding transaction logs based on the notification content;

[0093] d) Generate a log of the changed data.

[0094] Step 404: Monitor the serial number service to determine whether the serial number in the serial space has changed. The serial space is a data set that records the serial data and the serial number.

[0095] For example, a serial number service has multiple serial nodes, each corresponding to a serial space. Serial numbers are managed through the serial number service and can be represented by a sequence number. The serial space is a data set that records the serial data and the serial numbers. For instance, entity tables with similar usage can be grouped into a data set, and changes to these entity tables share a single serial node from the serial number service. For example, entity tables related to advertising operations can be combined into a data set to form a corresponding serial space, which stores the advertising-related serial numbers and changed data storage tables. If multiple data changes occur in the advertising operation, and each changed data corresponds to a serial log with a unique serial number, the serial numbers of the multiple changed data corresponding to the serial logs for that advertising operation are stored in the data set (i.e., the serial space) corresponding to that advertising operation. A data set can store the serial numbers corresponding to multiple serial logs, and each serial log has a unique serial number within a serial space.

[0096] Optionally, monitoring the serial number service to determine whether the serial number in the serial space has changed includes: querying the serial number service every preset time period, comparing the value of the currently queried serial number with the value of the previously recorded serial number; if the value of the currently queried serial number is greater than the value of the previously recorded serial number, then it is determined that the currently queried serial number has changed.

[0097] In the process of publishing changes to advertising data, the serial number service is monitored through the data publishing platform. If a change in the serial number of the relevant serial space is detected, the serial log is read and the changed data record is obtained. Then, the business domain data is read and the data is published to the data user.

[0098] Specifically, when monitoring the serial number service to detect changes in the serial numbers of the relevant serial space, the serial number service is queried every preset time period (e.g., 100 milliseconds) to determine whether the serial number of the serial space has changed (increased). For example, the value of the currently queried serial number is compared with the value of the previously recorded serial number to determine the change of the serial number. For example, how much the serial number has increased indicates how much business data has changed.

[0099] Step 405: If it is determined that the serial number of the serial space has changed, then according to the changed target serial number, read the target serial data from the data cache layer and obtain the target change data of the target serial data record, and publish the target change data to the data user.

[0100] During the process of publishing changes to advertising data, the data publishing platform monitors the serial number service. If a change in the serial number of a relevant serial space is detected, the platform reads the serial log, retrieves the changed data record, reads the business domain data, and publishes the data to the data user. For example, when reading business domain data, it typically reads the changed data.

[0101] Optionally, the step of reading target flow data from the data cache layer and obtaining target change data of the target flow data record based on the changed target flow number includes: determining the flow log corresponding to the target flow data based on the target flow number; addressing the data row of the target change data based on the primary key and partition key of the flow log corresponding to the target flow data, so as to obtain the target change data of the target flow data record from the data cache layer.

[0102] For example, a transaction log contains snapshot information about data changes, including change time, source, change request body, transaction type, transaction number, and transaction body. The transaction body may include details of the transaction, such as the entity / table of the changed data, the primary key / unique index of the changed data, the data partition key, and the old and new values ​​of the changed fields. If a transaction number in the transaction space has changed, the corresponding transaction log for the target transaction data can be determined based on the target transaction number. In other words, the transaction number can be used to address the data change transaction and related snapshot information in the transaction log. The target transaction number is the transaction number that has changed. The transaction log contains the primary key and partition key of the changed data, which can be used to address the data row.

[0103] Optionally, before publishing the target change data to the data user, the method further includes: converting the first data object of the target change data generated by the business domain where the data producer is located into a second data object corresponding to the data user according to the different needs of the data user.

[0104] To accelerate data publishing, the data publishing platform employs technologies such as preprocessing and data pre-caching. Simultaneously, because data users may have different data models than their business domains, and different data users may use different data models, the data publishing platform may perform corresponding data object transformations based on different needs. For example, it may pre-set corresponding rule systems and strategy calculations for data corresponding to the business domains of business producers to form a set of rules to be matched.

[0105] In this context, the business domain, as the owner of the data entity or data object, is typically the creator of the data entity or data object and the producer of the entity data. Complex business systems, such as internet advertising systems, often involve massive amounts of data for a single data entity, and the storage of these data entities may be distributed and partitioned. Because data usage and circulation are diverse, the same data entity may require different mappings or data objects in different data circulation domains of different data users.

[0106] The data publishing platform needs to have the capability of data object transformation, that is, to transform the first data object of the target change data generated by the business domain where the data producer is located into a second data object corresponding to the data user according to the different needs of the data user, and then publish the transformed second data object to the data user. The rule system in this application integrates database system views, stored procedures, triggers, user-defined functions, complex strategy tree calculations, and other possible object transformations. Among them, different data rows of the same data entity table can have various data mappings or object transformations through different rule configurations; the same data entity table can have various data mappings or projections through different rule configurations; multiple data entity tables can have different data mappings and strategy calculation results through different rule or complex strategy tree configurations.

[0107] Optionally, converting the first data object of the target changed data generated by the business domain where the data producer is located into a second data object corresponding to the data user according to the different needs of the data user includes: receiving a first access request sent by the data user for the first data object of the target changed data; determining a target rule from a set of matching rules that correspond to the first data object according to the first access request; and converting the first data object into a second data object corresponding to the data user according to the target rule.

[0108] The data object transformation involves the data publishing platform using a set of matching rules (such as rules, strategy tree calculations, and other object transformations) to convert a first data object representing the target changed data generated by the business domain of the data producer into a second data object corresponding to the data user, based on the different needs of the data user. Specifically, if a first access request for the first data object representing the target changed data is received from a data user, including a first request statement and a data user identifier (indicating the target data circulation domain), a set of matching rules is determined based on the first data object and the target data circulation domain. This set of matching rules corresponds to the first data object and includes at least one matching rule. Each matching rule includes a statement rule hit condition and a correspondence between the statement rule and the operation rule. If the first request statement satisfies the statement rule hit condition included in the target rule, the first data object is converted into a second data object corresponding to the data user based on the first request statement and the operation rule included in the target rule. The target rule belongs to the set of matching rules. For example, the first request statement carries the fields corresponding to the first data object, which is a table. The data user can be identified based on the data user identifier, thus determining the target data flow domain corresponding to that data user. After parsing the first access request, the first request statement is obtained. Based on the content of the first request statement (such as data entities, conditions, and requested data content), the rules that the access request might match are analyzed, resulting in a set of rules to be matched. This set includes at least one rule to be matched. A rule to be matched may have a definite result (e.g., hit or miss) or an indefinite result (e.g., pending). For indefinite results, it may be necessary to determine whether a match has occurred during subsequent execution stages or during the data result fusion stage after execution. Therefore, a single access request rule may have one or more result sets. If the first request statement satisfies the hit conditions of the statement rules included in the target rule, then according to the first request statement and the operation rules included in the target rule, the first data object is converted into a second data object corresponding to the data user.

[0109] For example, such as Figure 7As shown, the first data object takes the data entity table as an example. After parsing the first access request, the first request statement is obtained. For example, the first request statement includes specific events, such as query, insert, update, and delete. Based on the first data object and the corresponding target data flow domain, the set of rules to be matched for the event corresponding to the first request statement is obtained. Each set of rules to be matched includes the statement rule hit condition and its corresponding operation rule. The statement rule hit condition includes the event and the rule hit condition. For example, if the first request statement is the "delete" event, then all the rules to be matched in the set of rules to be matched belong to the rules under the "delete" event. Therefore, it is then determined whether the first request statement hits the "rule hit condition" in the "statement rule hit condition". For example, if a statement rule for an event hits condition 1 and the corresponding operation rule is a rule, the first data object can be converted into a second data object corresponding to the data user through the rule corresponding to condition 1. Similarly, if a statement rule for an event hits condition 2 and the corresponding operation rule is another object conversion method, the first data object can be converted into a second data object corresponding to the data user through the other object conversion method corresponding to condition 2. Finally, if a statement rule for an event hits condition n and the corresponding operation rule is strategy tree 1, the first data object can be converted into a second data object corresponding to the data user through strategy tree 1 corresponding to condition n.

[0110] Optionally, after receiving the first access request for the first data object of the target changed data sent by the data user, the method further includes: matching a first authorization policy tree that matches the first authorization feature field through a predefined first authorization feature field carried in the first access request, and authorizing data resources for the first access request according to the first authorization policy tree.

[0111] For example, data is also a resource, and the use of data requires necessary authorization. Data publishing platforms also need to authorize data resources.

[0112] For example, for data resources that require access authorization, data resource authorization (authentication) must be performed first, and then data object transformation (rule system, data strategy calculation) may be involved during data access.

[0113] In this context, the business domain, as the owner of data entities or data objects, has a certain say in the circulation and use of data. Data, as a resource, needs protection, and access to data resources requires authorization. Resource authorization can be a policy-tree-based authorization strategy. For access requests requiring authorization, predefined authorization feature fields carried in the access request are matched against the corresponding policy tree. The policy engine then authorizes the data resource based on the policy tree matching the authorization feature fields. For example, the authorization feature fields can also include authentication or authorization fields. These authorization feature fields can be used to retrieve the corresponding authorization policy tree from a pre-defined rules database to authorize data resources based on the policy tree matching the authorization feature fields.

[0114] like Figure 8 As shown, for data resources in a data entity, if an access request carries predefined authorization feature fields including AAA field 1 and AAA field 2, the corresponding authorization policy tree is policy tree 1; if an access request carries predefined authorization feature fields including AAA field 1 and AAA field 3, the corresponding authorization policy tree is policy tree 2.

[0115] Optionally, before publishing the target change data to the data user, the method further includes: establishing a dedicated publishing channel for publishing the target change data to the data user through the dedicated publishing channel.

[0116] Optionally, a dedicated release channel is established, including: pre-establishing multiple dedicated release channels, with different dedicated release channels matching different types of data producers; when the data producer of the target changed data to be released belongs to the data producer matched by the multiple dedicated release channels, a dedicated release channel corresponding to the data producer of the target changed data is selected from the multiple dedicated release channels.

[0117] Optionally, establishing a dedicated publishing channel includes: establishing a dedicated publishing channel when preset conditions are met, wherein the preset conditions include any of the following:

[0118] The amount of the target change data to be released is greater than the preset amount of data;

[0119] The data producer corresponding to the target change data to be released has a data release volume greater than the preset release volume within the historical period.

[0120] The activity level of the data producer corresponding to the target change data to be released is greater than the preset activity level.

[0121] The data publishing process is parallel and multi-channel. Based on the specific characteristics of the advertising system and the different characteristics of advertisers, the data publishing platform can establish dedicated publishing channels for specific advertisers, such as those with large advertising volumes or high activity levels. For example, multiple dedicated publishing channels can be established in advance, and different dedicated publishing channels can be matched with different types of data producers. Data producers who need to use dedicated channels can publish data through the corresponding type of dedicated publishing channel. Alternatively, dedicated publishing channels can be established only when preset conditions are met, and data is published through these established dedicated publishing channels.

[0122] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0123] This application embodiment generates sequential data corresponding to the changed data based on the changed data generated by the data producer. The sequential data records the data change log information of the changed data, and each sequential data entry has a unique sequential number, maintained by a sequential number service. The sequential data is preprocessed, with the preprocessed sequential data having a smaller data volume than the original sequential data. The changed data and the preprocessed sequential data are cached in a data cache layer. The sequential number service is monitored to determine if the sequential number in the sequential space has changed. The sequential space is a data set recording sequential data and sequential numbers. If a change in the sequential number in the sequential space is determined, the target sequential data and the target changed data recorded in the target sequential data are read from the data cache layer based on the changed target sequential number, and the target changed data is published to the data user. This application embodiment optimizes data by preprocessing the transaction data corresponding to the changed data, and reduces the latency of data ownership and alleviates the pressure on the business data system by pre-caching the changed data and transaction data. When a transaction number change is detected by the transaction number monitoring service, the target transaction data and the corresponding changed data are read from the data cache layer based on the changed target transaction number, thereby accelerating the data release rate and improving data release capabilities. This allows for fast and efficient data release. Simultaneously, during the data release process, data resource authorization, data object conversion, and other data processing can be performed to release advertising data from the data producer to multiple data users, achieving diverse data release methods.

[0124] To facilitate better implementation of the data publishing method of this application, this application also provides a data publishing apparatus. Please refer to... Figure 9 , Figure 9 A schematic diagram of the structure of a data publishing device provided in an embodiment of this application. The data publishing device 900 may include:

[0125] The generation unit 901 is used to generate serial data corresponding to the change data generated by the data producer. The serial data is used to record the data change log information of the change data. Each serial data has a spatially unique serial number, which is maintained by the serial number service.

[0126] Preprocessing unit 902 is used to preprocess the streaming data, wherein the amount of data in the preprocessed streaming data is less than the amount of data in the streaming data before preprocessing.

[0127] Cache unit 903 is used to cache the changed data and the preprocessed pipeline data to the data cache layer;

[0128] The monitoring unit 904 is used to monitor the serial number service to determine whether the serial number in the serial space has changed, wherein the serial space is a data set that records the serial data and the serial number;

[0129] The publishing unit 905 is used to, if it is determined that the serial number of the serial space has changed, read the target serial data from the data cache layer according to the changed target serial number and obtain the target change data of the target serial data record, and publish the target change data to the data user.

[0130] Optionally, the preprocessing unit 902 can be used to perform merging, compression, and filtering processing on the pipeline data within each data processing window of the same pipeline space to obtain the preprocessed pipeline data.

[0131] Optionally, the transaction data includes transaction logs, which contain the primary key and partition key of the target changed data. The preprocessing unit 902 is specifically used to: merge and compress the transaction logs corresponding to the same row of data in the transaction data, wherein the same row of data contains data with the same primary key and partition key; and / or discard transaction logs that are not of interest to the data user, or discard meaningless transaction logs, in order to filter the transaction data.

[0132] Optionally, when merging and compressing the logs corresponding to the same row of data in the log data, the preprocessing unit 902 can be used to: when there are multiple logs, determine the logs belonging to the same primary key from the multiple logs, and obtain the data corresponding to the partition key consistent with the primary key from the log space according to the primary key to obtain the same row of data; if the same row of data contains logs with multiple changes, determine whether the changed objects of the multiple changed logs are the same; if the changed objects of the multiple changed logs are the same, then when merging and compressing the same row of data... When merging and compressing row data, the last changed log in the multiple changed logs is identified as the preprocessed log data; or if the changed objects in the multiple changed logs are different, when merging and compressing the same row data, the different changed objects in the multiple changed logs are merged, and the same non-changed objects in the multiple changed logs are compressed into a single non-changed object, so that the data containing the single non-changed object and the multiple different changed objects is identified as the preprocessed log data.

[0133] Optionally, when the publishing unit 905 reads target serial data from the data cache layer and obtains target change data of the target serial data record based on the changed target serial number, it can be used to: determine the serial log corresponding to the target serial data based on the target serial number; address the data row of the target change data based on the primary key and partition key of the serial log corresponding to the target serial data, so as to obtain the target change data of the target serial data record from the data cache layer.

[0134] Optionally, the caching unit 903 can be used to: control the data caching layer to monitor changes in the serial number; when a change in the serial number is detected, control the data caching layer to obtain the corresponding preprocessed serial data based on the serial number and cache it; and actively retrieve the changed data of the preprocessed serial data record and cache it; or when the serial data corresponding to the changed data is generated, send a change notification to the data caching layer, and control the data caching layer to obtain the preprocessed serial data and the changed data of the preprocessed serial data record based on the change notification and cache them.

[0135] Optionally, the monitoring unit 904 can be used to: query the serial number service at preset time intervals, compare the value of the currently queried serial number with the value of the previously recorded serial number; if the value of the currently queried serial number is greater than the value of the previously recorded serial number, then determine that the currently queried serial number has changed.

[0136] Optionally, before publishing the target change data to the data user, the publishing unit 905 can also be used to: convert the first data object of the target change data generated by the business domain where the data producer is located into a second data object corresponding to the data user according to the different needs of the data user.

[0137] Optionally, when the publishing unit 905 converts the first data object of the target changed data generated by the business domain where the data producer is located into a second data object corresponding to the data user according to the different needs of the data user, it can be used to: receive a first access request sent by the data user for the first data object of the target changed data; determine a target rule from the set of matching rules that correspond to the first data object according to the first access request; and convert the first data object into a second data object corresponding to the data user according to the target rule.

[0138] Optionally, after receiving the first access request for the first data object of the target changed data sent by the data user, the publishing unit 905 can also be used to: match a first authorization policy tree that matches the first authorization feature field through the predefined first authorization feature field carried in the first access request, and authorize the first access request for data resources according to the first authorization policy tree.

[0139] Optionally, before publishing the target change data to the data user, the publishing unit 905 may also be used to: establish a dedicated publishing channel so that the target change data can be published to the data user through the dedicated publishing channel.

[0140] Optionally, when establishing a dedicated publishing channel, the publishing unit 905 can be used to: pre-establish multiple dedicated publishing channels, with different dedicated publishing channels matching different types of data producers; when the data producer of the target changed data to be published belongs to the data producer matched by the multiple dedicated publishing channels, select the dedicated publishing channel corresponding to the data producer of the target changed data from the multiple dedicated publishing channels.

[0141] Optionally, when establishing a dedicated publishing channel, the publishing unit 905 can be used to: establish a dedicated publishing channel when preset conditions are met, wherein the preset conditions include any of the following:

[0142] The amount of the target change data to be released is greater than the preset amount of data;

[0143] The data producer corresponding to the target change data to be released has a data release volume greater than the preset release volume within the historical period.

[0144] The activity level of the data producer corresponding to the target change data to be released is greater than the preset activity level.

[0145] It should be noted that the functions of each module in the data publishing device 900 in this application embodiment can be referred to the specific implementation of any embodiment in the above method embodiments, and will not be repeated here.

[0146] Each unit in the aforementioned data publishing device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can invoke and execute the operations corresponding to each unit.

[0147] The data publishing device 900 can be integrated, for example, into a terminal or server with storage and a processor, or the data publishing device 900 can be the terminal or server itself. The terminal can be a smartphone, tablet, laptop, smart TV, smart speaker, wearable smart device, personal computer (PC), etc. The terminal can also include a client, which can be a video client, browser client, or instant messaging client, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0148] Figure 10 Another schematic structural diagram of the data publishing device provided in the embodiments of this application, such as Figure 10 As shown, the data publishing device 1000 may include: a communication interface 1001, a memory 1002, a processor 1003, and a communication bus 1004. The communication interface 1001, memory 1002, and processor 1003 communicate with each other via the communication bus 1004. The communication interface 1001 is used for data communication between the device 1000 and external devices. The memory 1002 can be used to store software programs and modules, and the processor 1003 runs the software programs and modules stored in the memory 1002, such as the software programs for corresponding operations in the aforementioned method embodiments.

[0149] In some embodiments, the processor 1003 may invoke software programs and modules stored in the memory 1002 to perform the following operations:

[0150] Based on the change data generated by the data producer, corresponding transaction data is generated. This transaction data records the data change log information of the change data. Each transaction data entry has a unique transaction number, maintained by a transaction number service. The transaction data is preprocessed, with the preprocessed data size being smaller than the original data size. The change data and the preprocessed transaction data are cached in a data cache layer. The transaction number service is monitored to determine if the transaction number in the transaction space has changed. The transaction space is a data set recording the transaction data and the transaction number. If a change in the transaction number is detected, the target transaction data and the target change data recorded in the target transaction data are read from the data cache layer based on the changed target transaction number, and the target change data is published to the data user.

[0151] In some embodiments, the data publishing device 1000 may be integrated into a terminal or server that has storage and a processor and thus computing power, or the data publishing device 1000 may be the terminal or server. The terminal may be a smartphone, tablet, laptop, smart TV, smart speaker, wearable smart device, personal computer, or other similar device. The server may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0152] In some embodiments, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0153] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the data publishing method of this application embodiment; for brevity, further details are omitted here.

[0154] This application also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding process in the data publishing method of this application embodiment. For simplicity, further details are omitted here.

[0155] This application also provides a computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding process in the data publishing method of this application embodiment. For simplicity, further details are omitted here.

[0156] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0157] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0158] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0159] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0160] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0164] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0165] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data publishing method, characterized in that, The method includes: Based on the change data generated by the data producer, corresponding transaction data is generated. The transaction data is used to record the data change log information of the change data. Each transaction data has a spatially unique transaction number, which is maintained by a transaction number service. The transaction data includes a transaction log, which contains the primary key and partition key of the target change data. The pipeline data is preprocessed, and the amount of preprocessed pipeline data is less than the amount of pipeline data before preprocessing. The preprocessing includes: within each data processing window of the same pipeline space, performing merging and compression processing and filtering processing on the pipeline data; the merging and compression processing includes merging and compressing the pipeline logs corresponding to the same row of data in the pipeline data, wherein the same row of data contains data with the same primary key and partition key; the filtering processing includes discarding pipeline logs that are not of interest to the data user or discarding meaningless pipeline logs from the pipeline data. The changed data and the preprocessed stream data are cached in the data cache layer; Monitor the serial number service to determine whether the serial number in the serial space has changed, wherein the serial space is a data set that records the serial data and the serial number; If it is determined that the serial number of the pipeline space has changed, then according to the changed target serial number, the target pipeline data and the target change data of the target pipeline data record are read from the data cache layer. Before publishing the target change data to the data user, the first data object of the target change data generated by the business domain where the data producer is located is converted into a second data object corresponding to the data user according to the different needs of the data user. After receiving a first access request for a first data object for the target changed data sent by the data user, the system matches a first authorization policy tree that matches the first authorization feature field carried in the first access request, and authorizes data resources for the first access request according to the first authorization policy tree. The target change data, after being authorized by the data resources and transformed by the data objects, will be published to the data users.

2. The data publishing method as described in claim 1, characterized in that, The step of reading target serial number data from the data cache layer and obtaining target change data of the target serial number data record based on the changed target serial number includes: Determine the flow log corresponding to the target flow data based on the target flow number; The target change data row is addressed based on the primary key and partition key of the log corresponding to the target flow data, so as to obtain the target change data of the target flow data record from the data cache layer.

3. The data publishing method as described in claim 1, characterized in that, The step of caching the changed data and the preprocessed pipeline data to the data cache layer includes: The data caching layer is controlled to monitor changes in the serial number. When a change in the serial number is detected, the data caching layer is controlled to retrieve the corresponding preprocessed serial data based on the serial number and cache it, and to actively retrieve and cache the changed data of the preprocessed serial data records; or When the transaction data corresponding to the changed data is generated, the change notification is sent to the data cache layer, and the data cache layer is controlled to obtain the preprocessed transaction data and the change data recorded in the preprocessed transaction data according to the change notification and cache them.

4. The data publishing method as described in claim 1, characterized in that, The monitoring of the serial number service to determine whether the serial number in the serial space has changed includes: The serial number service is queried at preset time intervals, and the value of the currently queried serial number is compared with the value of the previously recorded serial number. If the value of the currently retrieved serial number is greater than the value of the previously recorded serial number, then it is determined that the currently retrieved serial number has changed.

5. The data publishing method as described in claim 1, characterized in that, The step of converting the first data object of the target change data generated by the business domain where the data producer is located into a second data object corresponding to the data user according to the different needs of the data user includes: Receive a first access request for a first data object targeting the changed data sent by the data user; Based on the first access request, a target rule is determined from the set of matching rules that correspond to the first data object, and the first data object is converted into a second data object corresponding to the data user according to the target rule.

6. The data publishing method according to any one of claims 1-5, characterized in that, Before publishing the target change data to the data user, the method further includes: Establish a dedicated publishing channel so that the target change data can be published to the data user through the dedicated publishing channel.

7. The data publishing method as described in claim 6, characterized in that, The establishment of a dedicated publishing channel includes: Multiple dedicated publishing channels are pre-established, and different dedicated publishing channels are matched with different types of data producers; When the data producer of the target change data to be published belongs to the data producer matched by the plurality of dedicated publishing channels, a dedicated publishing channel corresponding to the data producer of the target change data is selected from the plurality of dedicated publishing channels.

8. The data publishing method as described in claim 7, characterized in that, The establishment of a dedicated publishing channel includes: A dedicated publishing channel will be established when preset conditions are met, including any one of the following: The amount of the target change data to be released is greater than the preset amount of data; The data producer corresponding to the target change data to be released has a data release volume greater than the preset release volume within the historical period. The activity level of the data producer corresponding to the target change data to be released is greater than the preset activity level.

9. A data publishing device, characterized in that, The device includes: The generation unit is used to generate transaction data corresponding to the changed data generated by the data producer. The transaction data is used to record the data change log information of the changed data. Each transaction data has a spatially unique transaction number, which is maintained by a transaction number service. The transaction data includes a transaction log, which contains the primary key and partition key of the target changed data. A preprocessing unit is used to preprocess the pipeline data, wherein the amount of preprocessed pipeline data is less than the amount of pipeline data before preprocessing. The preprocessing includes: performing merging and compression processing and filtering processing on the pipeline data within each data processing window of the same pipeline space; the merging and compression processing includes merging and compressing the pipeline logs corresponding to the same row of data in the pipeline data, wherein the same row of data contains data whose primary key and partition key are consistent; the filtering processing includes discarding pipeline logs that are not of interest to the data user or discarding meaningless pipeline logs from the pipeline data. A caching unit is used to cache the changed data and the preprocessed pipeline data to the data caching layer; The monitoring unit is used to monitor the serial number service to determine whether the serial number in the serial space has changed, wherein the serial space is a data set that records the serial data and the serial number; The publishing unit is configured to, if a change is determined in the serial number of the serial space, read the target serial data and obtain the target change data of the target serial data record from the data cache layer according to the changed target serial number; before publishing the target change data to the data user, convert the first data object of the target change data generated by the business domain where the data producer is located into a second data object corresponding to the data user according to the different needs of the data user; after receiving a first access request for the first data object of the target change data sent by the data user, match a first authorization policy tree that matches the first authorization feature field through the predefined first authorization feature field carried in the first access request, and authorize the data resources for the first access request according to the first authorization policy tree; and publish the target change data after the data resource authorization and the data object conversion to the data user.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the steps of the data publishing method as described in any one of claims 1-8.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executes the steps of the data publishing method according to any one of claims 1-8 by calling the computer program stored in the memory.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the data publishing method according to any one of claims 1-8.

Images