Data processing method and device, computer readable storage medium and equipment

Abstract

The invention is suitable for the field of data processing, and provides a data processing method and device, a computer readable storage medium and computer device.The data processing method comprises the steps that according to a preset data collection strategy, original data are collected from multiple advertisement channels through a public data interface; storing the original data into a temporary database on the basis of a mapping relationship between the self-defined quantitative information of each advertisement channel and preset standard quantitative information; detecting the data integrity of the original data stored in the temporary database; when the data integrity of the original data of the temporary database meets the preset condition, format conversion is conducted on the original data meeting the preset condition, and the data obtained after format conversion is loaded into the target database, and according to the scheme, the efficiency and timeliness of data collection can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data processing, and particularly relates to a data processing method and device, a computer readable storage medium and a computer device. BACKGROUND

[0002] With the development of the Internet, online advertising has gradually become a means for advertisers in the Internet industry to expand new pages and activate. Therefore, the analysis of advertising effectiveness is also a crucial matter for advertisers. At present, the collection and integration of analysis data still relies on manual collection, and then key indicators are extracted and tabulated by manual extraction to complete.

[0003] However, with the increase in data volume, the labor cost increases significantly, and due to the lag of manual processing, it is difficult to obtain timely data, that is, the current data processing scheme has the problems of poor efficiency and timeliness. SUMMARY

[0004] The purpose of the present application is to provide a data processing method, device, computer readable storage medium and computer device, which can improve the efficiency and timeliness of data collection.

[0005] In a first aspect, the present application provides a data processing method, comprising: According to a preset data collection strategy, collecting original data from a plurality of advertising channels through a public data interface; Based on the mapping relationship between the self-defined quantitative information of each advertising channel and the preset standard quantitative information, the original data is stored in a temporary database; Detecting the data integrity of the original data stored in the temporary database; When the data integrity of the original data in the temporary database meets the preset condition, the original data that meets the preset condition is format-converted, and the format-converted data is loaded into a target database.

[0006] In a second aspect, the present application provides a data processing device, comprising: The collection module is configured to collect original data from a plurality of advertising channels through a public data interface according to a preset data collection strategy; The storage module is configured to store the original data in a temporary database based on the mapping relationship between the self-defined quantitative information of each advertising channel and the preset standard quantitative information; The detection module is configured to detect the data integrity of the original data stored in the temporary database; The loading module is configured to, when the data integrity of the original data of the temporary database meets a preset condition, perform format conversion on the original data meeting the preset condition, and load the data after the format conversion into the target database.

[0007] Optionally, in some embodiments of the present application, the collection module is specifically configured to: determine a collection frequency corresponding to the preset data collection strategy; collect original data from a plurality of advertising channels through a public data interface according to the task scheduling component and the collection frequency.

[0008] Optionally, in some embodiments of the present application, the collection module is specifically further configured to: configure a calling interface corresponding to a software module of each advertising channel; collect original data from the calling interface corresponding to the software module of each advertising channel according to the task scheduling component and the collection frequency.

[0009] Optionally, in some embodiments of the present application, the storage module is specifically configured to: determine a version number corresponding to storage of the original data into the temporary database based on a mapping relationship between the custom quantification information of each advertising channel and the preset standard quantification information; associate the version number with the original data, and store the version number and the original data into the temporary database.

[0010] Optionally, in some embodiments of the present application, the storage module is specifically further configured to: when it is detected that the mapping relationship between the custom quantification information of the advertising channel and the preset standard quantification information is changed, change the version number corresponding to storage of the original data into the temporary database.

[0011] Optionally, in some embodiments of the present application, the storage module is specifically further configured to: map the custom quantification information of the advertising channel to the standard quantification information, and / or; convert the custom quantification information of the advertising channel to the standard quantification information through a preset conversion strategy.

[0012] Optionally, in some embodiments of the present application, the loading module is specifically configured to: when the data integrity of the original data of the temporary database meets a preset condition, convert the custom quantification information of the original data meeting the preset condition to the standard quantification information; based on a target database structure, batch load the data after the format conversion into the target database.

[0013] In a third aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of any of the above methods.

[0014] In a fourth aspect, the present application also provides a computer device, which comprises one or more processors, a memory, and one or more computer programs, wherein the processor and the memory are connected through a bus, the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and the processor executes the programs to implement the steps of any of the above methods.

[0015] The embodiments of the present application provide a data processing method, device, storage medium and computer device. After collecting original data from multiple advertising channels through a public data interface according to a preset data collection strategy, the original data is stored in a temporary database based on the mapping relationship between the custom quantification information of each advertising channel and the preset standard quantification information. Then, the data integrity of the original data stored in the temporary database is detected. When the data integrity of the original data in the temporary database meets a preset condition, the original data meeting the preset condition is format-converted, and the format-converted data is loaded into a target database. It can be seen that, in the data processing scheme of the present application, the original data is collected according to the preset data collection strategy, avoiding the collection delay or omission caused by manual intervention, ensuring the efficiency and timeliness of data collection. Meanwhile, the original data is collected from multiple advertising channels, ensuring the data transmission protocol specification and data format specification, avoiding the unstable data problem caused by anti-crawling mechanism, webpage structure change and the like, thereby ensuring the stability and legality of data collection. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 FIG. 1 is a flowchart of a data processing method according to an embodiment of the present application.

[0017] Figure 2 FIG. 2 is another flowchart of a data processing method according to an embodiment of the present application.

[0018] Figure 3 FIG. 3 is still another flowchart of a data processing method according to an embodiment of the present application.

[0019] Figure 4 FIG. 4 is a function module block diagram of a data processing device according to an embodiment of the present application.

[0020] Figure 5 FIG. 5 is a specific structure block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purposes, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application and not to limit the present application.

[0022] In order to illustrate the technical solutions described in the present application, the following will be described by specific embodiments.

[0023] Please refer to Figure 1 is a flowchart of the data processing method provided by an embodiment of the present application. The embodiment mainly takes the application of the data processing method to a computer device as an example to illustrate the data processing method provided by an embodiment of the present application, which includes the following steps: S101, according to a preset data collection strategy, collecting original data from multiple advertising channels through a public data interface.

[0024] The data collection strategy refers to a strategy of collecting original data of advertising effect indicators from multiple advertising channels in an automatic and timed manner. The public data interface refers to a type of interface that allows callers to obtain data through a public data network interface, which is different from the crawler collection. The data obtained through the public interface has stability, accuracy and legality, wherein the following conditions need to be met when the public data interface is called: Identity verification: input the identity token of the caller (authentication information assigned by the channel).

[0025] Parameter transmission: input parameters as required by the data provider, such as query time: specify the time range of the data (such as “2025-06-01”); indicator type: specify the indicators to be collected (such as exposure, click rate).

[0026] The advertising channel refers to different platforms or media for advertising, such as search engine advertising platforms, social media advertising platforms, etc.; the original data refers to the original indicator data directly obtained from the advertising channel through the public data interface without processing.

[0027] For example, data can be collected in a timed and fixed manner according to the time and day dimension (such as fixed time every day), a task scheduling component is used to configure a timed task (for example, every day at 2 a.m.), and data collection is triggered, that is, in some embodiments of the present application, step S101 can specifically include: determining the collection frequency corresponding to the preset data collection strategy; According to the task scheduling component and the collection frequency, collecting original data from multiple advertising channels through a public data interface.

[0028] The data collection strategy corresponds to a collection frequency, which can be a flexible configuration of a timing and fixed-point collection cycle by time and day dimensions. Since the data update frequency of different advertising channels is different (such as daily update, hourly update), in some embodiments of the present application, the collection frequency can be synchronized with the data update frequency of different advertising channels. For example: if the channel updates the data of the previous day at dawn every day, the collection frequency is set to 1 time per day (such as 2 o'clock in the morning); if the channel supports a real-time data interface, the collection frequency can be set to 1 time per hour or 1 time per minute.

[0029] The task scheduling component is the core module for realizing automatic collection, and is used for triggering a collection task at a preset time and day dimension (such as daily, hourly, or minute-by-minute). The timing task (for example, 2 o'clock in the morning every day) can be set through the task scheduling component, so that the public data interface of each advertising channel can be called without manual intervention, and the original data can be obtained and stored persistently.

[0030] For example, if a channel updates data at dawn every day, the collection frequency can be set to 1 time per day (such as 2 o'clock in the morning); if the channel supports real-time updating, it can be set to 1 time per hour or higher frequency. High-frequency analysis scenarios (such as real-time optimization of placement) use high-frequency collection (such as 1 time per 30 minutes). Low-frequency analysis (such as daily reports) uses low-frequency collection (such as 1 time per day).

[0031] In order to improve the low coupling, stability and scalability of the system, in some embodiments of the present application, an independent interface is configured for each software module of the advertising channel, the interface defines standardized methods such as “collection data” and “conversion data”, so that each module can be independently developed, tested and maintained, and when the collection logic or interface of a channel is changed, only the corresponding module needs to be modified, without affecting other channels, that is, in some embodiments of the present application, the following can also be included: The calling interface corresponding to the software module of each advertising channel is configured; The step of “collecting original data from multiple advertising channels through a public data interface according to the task scheduling component and the collection frequency” specifically includes: collecting original data from the calling interface corresponding to the software module of each advertising channel according to the collection frequency through the task scheduling component.

[0032] In which, a separate channel SDK (software module) is designed for each advertising channel, which inherits the public SDK and implements a unified abstract interface (such as "collection interface"), which defines standardized methods such as "collection data" and "conversion data". Configure the authentication parameters (such as identity tokens) and index mapping rules (establish the mapping relationship between channel indicators and public indicators through the configuration management system) in the channel SDK to ensure that the interface call can legally access the channel public data interface and process the original data according to the public indicator format. In this way, each channel SDK interacts with the main system in a low-coupling manner, and changes to a single channel interface only need to modify the corresponding SDK, without affecting other modules, achieving flexibility and maintainability of the system.

[0033] Specifically, configure the timing rules of time and day dimensions (such as the cron expression of every day at 2 a.m. or every hour) in the task scheduling component to trigger the collection task according to the preset collection frequency; when the specified time arrives, the task scheduling component calls the channel SDK (software module) independently developed by each advertising channel through a unified abstract interface (such as "collection interface") to trigger its "collection data" method; after receiving the call instruction, each channel SDK automatically completes the identity token verification and dynamically generates parameters such as query time and indicator type according to the collection frequency (such as passing the time range of "previous day date" or "current hour - 1"), and calls the public data interface of the corresponding channel through the HTTP / HTTPS protocol; after obtaining the original data (JSON / XML format), the channel SDK stores the data in the temporary database for subsequent data integrity detection and format conversion. In this process, the task scheduling component is only responsible for scheduling, and each channel SDK independently encapsulates the interface call logic to ensure that changes to the collection logic of a single channel do not affect other channels, achieving low-coupling automatic collection.

[0034] S102, based on the mapping relationship between the custom quantification information of each advertising channel and the preset standard quantification information, store the original data in the temporary database.

[0035] The custom quantification information refers to the original definition and calculation method of the index of each advertising channel, and there may be differences between different channels. The preset standard quantification information is a set of unified public index system defined in advance, which is used to eliminate the index differences between channels, facilitate data integration and analysis. The mapping relationship refers to the corresponding rule between the custom quantification information and the standard quantification information of each advertising channel in the configuration management system. Specifically, a set of public indexes (standard quantification information) can be defined in advance through the configuration management system, and the mapping relationship between the custom quantification information (channel original index) of each advertising channel and the public index can be established. For example, the “click rate” of Google Ads (the calculation method is “clicks / impressions”) is mapped to the public index “click rate”, and the “link clicks” of Facebook Ads is mapped to the public index “clicks”.

[0036] When the original data is obtained through the channel SDK, the SDK will associate the native index field returned by the channel with the public index according to the mapping relationship of the configuration management system, but not immediately perform data conversion. For example, when “clicks” (clicks) and “impressions” (impressions) are obtained from Google Ads, the SDK records the mapping rule of the two fields corresponding to the public indexes “clicks” and “impressions”.

[0037] The original data is stored in the temporary database in the following manner: Table design: create a temporary table (such as raw_data_temp), and the fields include channel_id (channel ID), raw_data (original data JSON field), mapping_rule_id (mapping rule ID), collection time, etc.

[0038] Storage content: the original data returned by the channel is stored in the raw_data field, and the corresponding mapping rule is recorded to ensure that the mapping rule can be called for format conversion during subsequent data cleaning.

[0039] Optionally, in some embodiments of the present application, step S102 can specifically include: Based on the mapping relationship between the custom quantification information of each advertising channel and the preset standard quantification information, determine the version number corresponding to the storage of the original data in the temporary database; Associate the version number with the original data, and store the version number and the original data in the temporary database.

[0040] Specifically, when collecting the original data, the current valid mapping rule version identifier is requested from the configuration management system synchronously. For example, if the “click” mapping rule of Google Ads in the current configuration is the third updated version, the version number is “V3”. After the original data collection is completed, the system associates the version number with the original data, ensuring that each original data record corresponds to the mapping rule version used when it was collected.

[0041] It should be noted that the temporary database adds a version number field (such as “mapping_version”) when storing the original data, which is used to record the mapping rule version at the time of collection. For example, the storage structure includes “channel identifier”, “original data content”, “collection time”, “version number”, and the like.

[0042] When the original data (such as channel indicators in JSON format) is stored in the temporary database, the version number is stored as metadata together with the original data. For example, the “link click” original data collected from Facebook Ads is associated with the current mapping rule version number “20250604V1”, ensuring that the data can be cleaned and converted in the future, and the rules of this version can be called to map it to the public indicator “click”.

[0043] Optionally, in some embodiments of the present application, the configuration management system can define a public indicator system to establish a mapping relationship between the self-defined indicators of each channel and the public indicators, such as mapping the “click rate” (click number / display number) of Google Ads and the “link click number / display number” of Facebook Ads to the public indicator “click rate”; and / or, in the configuration management system, for different channel indicator definitions (such as the “click rate” calculation method of Google Ads is “click number / display number”, and the “link click number / display number” of Facebook Ads), configure the mapping rule with the public indicator “click rate”.

[0044] That is, the data processing method of the present application can further include: mapping the self-defined quantification information of the advertising channel to standard quantification information, and / or; converting the self-defined quantification information of the advertising channel to standard quantification information through a preset conversion strategy.

[0045] In addition, in some embodiments of the present application, when the mapping relationship is changed, the newly collected original data will be automatically associated with the latest version number when stored in the temporary database, that is, optionally, it can further include: When the mapping relationship between the custom quantification information of the advertising channel and the preset standard quantification information is changed, the version number of the original data stored in the temporary database is changed.

[0046] For example, if the mapping rule of “click rate” of Facebook Ads is changed from V1.0 (the original calculation method is “link click number / display number”) to V2.0 (the new calculation method is “total click number / display number”), the version number of the collected data after the change is V2.0.

[0047] S103, detecting the data integrity of the original data stored in the temporary database.

[0048] In this application, the content of the integrity check can be as follows: Format verification: verifying whether the original data meets the preset format requirements (such as whether the JSON / XML structure is complete), to ensure that the data can be correctly parsed by the system. For example, check whether there are invalid characters, and whether the fields match the specifications returned by the channel API.

[0049] Field integrity check: according to the index mapping rules defined in the configuration management system, check whether the original data contains all the required fields. For example, if “click number” and “display number” of Google Ads are required fields, the system needs to verify whether these two fields exist and are not empty.

[0050] Logical verification: verify the data rationality through the preset business logic rules (such as numerical range, time validity). For example, check whether the “click rate” is between 0 and 1, to avoid abnormal values such as negative numbers or exceeding 1.

[0051] If the check finds that the data is incomplete (such as missing key fields or format errors), the system can trigger an exception handling process, such as recording error logs, marking abnormal data batches, and re-collecting or requesting the channel to re-push data, to ensure that the data finally entered into the data warehouse is complete and reliable.

[0052] If the data integrity of the original data meets the preset conditions, step S104 is performed.

[0053] S104, when the data integrity of the original data in the temporary database meets the preset conditions, the original data that meets the preset conditions is format-converted, and the format-converted data is loaded into the target database.

[0054] The preset condition refers to a completeness check standard for the original data in the temporary database. For example, the preset condition can include that the original data contains all mandatory fields defined in the configuration management system, and that the original data meets the business logic rules (such as reasonable numerical range, valid timestamp, and correct index calculation logic). For example, the click rate should be between 0 and 1.

[0055] Specifically, the mapping rules and conversion rules corresponding to each channel index (such as index definition, calculation logic, etc.) are obtained from the configuration management system, and the rules are applied to the original data for standardized processing. For example, the “clicks” and “impressions” of Google Ads are converted into the public indexes “clicks” and “impressions” according to the rules, or the calculation method is adjusted according to the channel difference (such as the “link clicks” of Facebook Ads is converted into “clicks”). This process is completed in the “data cleaning and conversion” module to ensure that the data format is uniform and conforms to the public index system specification. The converted data is stored in the target database (unified data warehouse or database) in a unified format through unified data collection and export. Before loading, the accuracy of the conversion result (such as numerical range, field completeness) needs to be verified. After verification, the data is organized according to the specification (such as uniform field name, unit, time format), and finally provided to the analysis tool or report system.

[0056] Optionally, in some embodiments of the present application, step S104 can specifically include: When the data completeness of the original data of the temporary database meets the preset condition, the custom quantitative information of the original data meeting the preset condition is converted into standard quantitative information; Based on the structure of the target database, the format-converted data is batch-loaded into the target database.

[0057] Specifically, when the raw data passes the format validation (e.g. JSON / XML structure integrity), field integrity check (no missing mandatory fields), and logical validity check (e.g. numerical range is reasonable), the system triggers the format conversion process. Then, the mapping rules and conversion rules of channel metrics and public metrics are obtained through the configuration management system. For example, the “clicks” and “impressions” of Google Ads are converted to the public metrics “clicks” and “impressions” according to the rules, or the “click rate” is calculated according to the “link clicks” of Facebook Ads. Next, the same-named metrics (e.g. “click rate”) of different channels are converted according to the unified calculation logic (e.g. “clicks / impressions”), unified field naming (e.g. “clicks” is unified to “click_count”), and data format (e.g. timestamp standardization). Then, the converted data is organized according to the structure of the target database (unified data warehouse or database) through the unified data collection and export module, for example, the data of different channels is integrated into a unified table structure, including public metric fields, timestamps, channel identifiers, etc. Finally, the standardized data is batch imported into the target database using the standard interface (e.g. API or SQL query) provided by the data service layer. The loading process supports batch processing to ensure data consistency, for example, batch loading of data verified the day before at midnight.

[0058] In order to further understand the data processing scheme of the present application, the following will be further described, please refer to Figure 2 The present application provides a data acquisition system, which includes an automated acquisition module, a configuration management system, a low-coupling acquisition module, a data cleaning and conversion module, and a unified data output module. The automated acquisition module periodically acquires data from various advertising channels, the configuration management system stores and manages metric definitions and conversion rules, the low-coupling acquisition module independently processes data of each channel, the data cleaning and conversion module converts data into a unified format, and finally the unified data output module provides a standardized data interface.

[0059] Further, please refer to Figure 3, the automation collection module: using Job component, periodically calling the API provided by each advertising channel to obtain the original data. The timing task can be set, such as data collection at 2 o'clock in the morning every day. Configuration management system: a set of mapping relationship scheme is designed, a set of common indicators is defined, and the mapping relationship between the indicator definition of each channel and the common indicator is configured. For example, the indicator called exposure on my side, for Google Ads, the calculation method of "click rate" is defined as "click number / display number", and for Facebook Ads, it may be defined as "link click number / display number". Low coupling collection module: develop independent channel SDK for each channel, which inherits the common SDK. These SDLs all implement multiple common interfaces, such as "collection interface". The interface defines "collection data", "conversion data" and other methods to ensure that each module can be developed and tested independently. Data cleaning and conversion: through a set of unified data import and export, the collected data is converted into a unified format, and then the data is cleaned and converted, and then loaded into the data warehouse. Unified data output: develop a data service layer to provide standard API or SQL query interface for analysis tools or report system.

[0060] The above is the data processing process provided in the present application.

[0061] The present application provides a data processing method, after collecting original data from multiple advertising channels through a public data interface according to a preset data collection strategy, storing the original data into a temporary database based on the mapping relationship between the self-defined quantification information of each advertising channel and the preset standard quantification information, then detecting the data integrity of the original data stored in the temporary database, when the data integrity of the original data in the temporary database meets the preset condition, then performing format conversion on the original data that meets the preset condition, and loading the format converted data into the target database. It can be seen that in the data processing scheme of the present application, the original data is collected according to the preset data collection strategy, which avoids the collection delay or omission caused by manual intervention, ensures the efficiency and timeliness of data collection, at the same time, the original data is collected from multiple advertising channels, which ensures the stability and legality of data collection.

[0062] Please refer to Figure 4An embodiment of the present application provides a data processing apparatus which can be a computer program or a program code running in a computer device, for example, an application software. The data processing apparatus can be used to execute corresponding steps in the data processing method provided by the embodiments of the present application. The data processing apparatus provided by an embodiment of the present application comprises a collection module 201, a storage module 202, a detection module 203 and a loading module 204, and specifically comprises the following: The collection module 201 is configured to collect original data from a plurality of advertising channels through a public data interface according to a preset data collection strategy. The storage module 202 is configured to store the original data into a temporary database based on a mapping relationship between custom quantification information of each advertising channel and preset standard quantification information. The detection module 203 is configured to detect data integrity of the original data stored in the temporary database. The loading module 204 is configured to perform format conversion on the original data satisfying the preset condition, and load the data after format conversion into a target database when the data integrity of the original data in the temporary database satisfies a preset condition.

[0063] Optionally, in some embodiments of the present application, the collection module 201 is specifically configured to: determine a collection frequency corresponding to the preset data collection strategy; collect original data from a plurality of advertising channels through a public data interface according to a task scheduling component and the collection frequency.

[0064] Optionally, in some embodiments of the present application, the collection module 201 is specifically further configured to: configure a calling interface corresponding to a software module of each advertising channel; collect original data from the calling interface corresponding to the software module of each advertising channel according to a task scheduling component and a collection frequency.

[0065] Optionally, in some embodiments of the present application, the storage module 202 is specifically configured to: determine a version number corresponding to storage of the original data into a temporary database based on a mapping relationship between custom quantification information of each advertising channel and preset standard quantification information; associate the version number with the original data, and store the version number and the original data into the temporary database.

[0066] Optionally, in some embodiments of the present application, the storage module 202 is specifically further configured to: when it is detected that the mapping relationship between custom quantification information of an advertising channel and preset standard quantification information is changed, change the version number corresponding to storage of the original data into the temporary database.

[0067] Optionally, in some embodiments of the present application, the storage module 202 is specifically further used for: mapping the custom quantitative information of the advertising channel to the standard quantitative information, and / or; converting the custom quantitative information of the advertising channel to the standard quantitative information through a preset conversion strategy.

[0068] Optionally, in some embodiments of the present application, the loading module 204 is specifically used for: when the data integrity of the original data of the temporary database meets a preset condition, converting the custom quantitative information of the original data meeting the preset condition to the standard quantitative information; based on the target database structure, loading the format-converted data to the target database in batches.

[0069] The data processing device provided by an embodiment of the present application and the data processing method provided by an embodiment of the present application belong to the same concept, and the specific implementation process is described in detail in the full text of the specification, which will not be repeated here.

[0070] The present application provides a data processing device, after the acquisition module 201 collects original data from multiple advertising channels through a public data interface according to a preset data acquisition strategy, the storage module 202 stores the original data in a temporary database based on the mapping relationship between the custom quantitative information of each advertising channel and the preset standard quantitative information, then the detection module 203 detects the data integrity of the original data stored in the temporary database, and the loading module 204 converts the original data meeting the preset condition when the data integrity of the original data of the temporary database meets the preset condition, and loads the format-converted data to the target database. It can be seen that in the data processing scheme of the present application, the original data is collected according to the preset data acquisition strategy, which avoids the collection delay or omission caused by manual intervention, ensures the efficiency and timeliness of data collection, at the same time, the original data is collected from multiple advertising channels, which ensures the data transmission protocol specification, data format clear, avoids the data instability problem caused by anti-crawling mechanism, webpage structure change and the like, thereby ensures the stability and legality of data collection.

[0071] An embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the data processing method provided by an embodiment of the present application.

[0072] Figure 5A specific structure block diagram of a computer device provided by an embodiment of the present application is shown, the computer device 100 includes one or more processors 101, a memory 102, and one or more computer programs, wherein the processor 101 and the memory 102 are connected through a bus, the one or more computer programs are stored in the memory 102 and are configured to be executed by the one or more processors 101, and the processor 101 implements the steps of the data processing method provided by the embodiment of the present application when executing the computer program. The computer device includes servers and terminals, etc. The computer device can be a desktop computer, a mobile terminal or a vehicle-mounted device, and the mobile terminal includes at least one of a mobile phone, a tablet computer, a personal digital assistant or a wearable device, etc.

[0073] It should be understood that the steps in the embodiments of the present application are not necessarily executed in the order indicated by the step labels. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least a part of the steps in the embodiments can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or other steps or stages.

[0074] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a non-volatile computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0075] Any combination of the technical features in the above embodiments can be made, and for the sake of brevity, not all possible combinations are described above, however, as long as the combination of the technical features does not exist in contradiction, it shall be considered within the scope of the present disclosure.

[0076] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present application. It shall be pointed out that, for ordinary skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application patent shall be subject to the appended claims.

Claims

1. A data processing method, characterized by, The method comprises the following steps: According to the preset data collection strategy, the original data is collected from multiple advertising channels through the public data interface; Based on the mapping relationship between the custom quantification information of each advertising channel and the preset standard quantification information, the original data is stored in the temporary database; Detect the data integrity of the original data stored in the temporary database; When the data integrity of the original data in the temporary database meets the preset condition, the original data that meets the preset condition is format-converted, and the format-converted data is loaded into the target database.

2. The data processing method according to claim 1, characterized in that, According to the preset data collection strategy, the original data is collected from multiple advertising channels through the public data interface, which comprises the following steps: Determine the collection frequency corresponding to the preset data collection strategy; According to the task scheduling component and the collection frequency, the original data is collected from multiple advertising channels through the public data interface.

3. The data processing method according to claim 2, characterized in that, Further comprising: Configure the calling interface corresponding to the software module of each advertising channel; According to the task scheduling component and the collection frequency, the original data is collected from multiple advertising channels through the public data interface, which comprises the following steps: through the task scheduling component, the original data is collected from the calling interface corresponding to the software module of each advertising channel according to the collection frequency.

4. The data processing method of claim 1, wherein, Based on the mapping relationship between the custom quantification information of each advertising channel and the preset standard quantification information, the original data is stored in the temporary database, which comprises the following steps: Based on the mapping relationship between the custom quantification information of each advertising channel and the preset standard quantification information, determine the version number corresponding to the storage of the original data in the temporary database; Associate the version number with the original data, and store the version number and the original data in the temporary database.

5. The data processing method according to claim 4, characterized in that, Further comprising: When it is detected that the mapping relationship between the custom quantification information of the advertising channel and the preset standard quantification information is changed, the version number corresponding to the storage of the original data in the temporary database is changed.

6. The data processing method according to claim 4, characterized in that, Further comprising: Map the custom quantification information of the advertising channel to the standard quantification information, and / or; Convert the custom quantification information of the advertising channel to the standard quantification information through the preset conversion strategy.

7. The data processing method according to any one of claims 1 to 6, characterized in that, When the data integrity of the original data in the temporary database meets the preset condition, the original data that meets the preset condition is format-converted, and the format-converted data is loaded into the target database, which comprises the following steps: When the data integrity of the original data in the temporary database meets the preset condition, the custom quantification information of the original data that meets the preset condition is converted to the standard quantification information; Based on the structure of the target database, the format-converted data is batch-loaded into the target database.

8. A data processing apparatus, characterized by, The method comprises the following steps: The collection module is used for collecting original data from multiple advertising channels through the public data interface according to the preset data collection strategy; The storage module is used for storing the original data in the temporary database based on the mapping relationship between the custom quantification information of each advertising channel and the preset standard quantification information; The detection module is used for detecting the data integrity of the original data stored in the temporary database; The detection module is used for detecting the data integrity of the original data stored in the temporary database; The loading module is configured to, when the data integrity of the original data of the temporary database meets a preset condition, perform format conversion on the original data meeting the preset condition, and load the format-converted data into the target database.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 7.

10. A computer device comprising: one or more processors; a memory; and one or more computer programs, the processor and the memory being connected by a bus, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and wherein the processor implements the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program.

Images