Data processing method, system, device, medium and program product

Abstract

The present application relates to the field of computer data processing, and particularly relates to a data processing method, system, device, medium and program product. The present application is suitable for electronic products, and characterized in that: a data acquisition step, acquiring original data from at least one data platform; a data retention step, storing the original data into a table in an ODS library; a data association step, associating the original data in the table in the ODS library with data of a table of a predetermined format to generate an associated table; and a data updating step, updating the data of the table of the predetermined format based on the data of the associated table. The present application can simultaneously support data synchronization of multiple advertising platforms, and is convenient for association analysis with self-owned data, can meet the demand for overall analysis of advertising attribution, cost and backtracking data, is a technology that can update and manage aggregated data in real time, and effectively solves the difficult problem of application of advertising platform aggregated data.

Description

Technical Field

[0001] This invention relates to the field of computer data processing, and in particular to a data processing method, system, device, medium, and program product. Background Technology

[0002] In real-world business scenarios, with the increasing demand for aggregated data analysis on advertising platforms, real-time updating and governance of aggregated data from multiple platforms has become a pressing issue for the industry. Among these, the technology for synchronizing and governing data from multiple platform data sources is the core technology of the entire advertising platform aggregated data analysis process. Currently, advertising platform data synchronization only provides some interfaces for data acquisition; however, the data format cannot be customized, and it cannot be linked to existing data. For data application scenarios, a technology is needed that can update and govern aggregated data in real time to ensure data quality, availability, integrability, security, and ease of use, addressing the pain points and difficulties in applying aggregated data from advertising platforms. This invention is a multi-advertising platform data synchronization, update, and governance technology that supports multiple data sources and guarantees the real-time nature, reusability, and scalability of data synchronization. Summary of the Invention

[0003] The purpose of this invention is to provide a data processing method, system, device, medium, and program product. This invention solves the difficult problems encountered in the real-time updating and governance analysis of aggregated data from multiple platforms through methods such as original data retention, data parsing, data association, and partitioned data rewriting. It supports multiple data sources, customized data formats, and association with one's own data, effectively ensuring the quality, availability, reusability, security, and scalability of data.

[0004] An embodiment of the present invention discloses a data processing method, the method comprising:

[0005] The data acquisition step involves acquiring raw data from at least one data platform.

[0006] The data retention step involves storing the original data into a table in the ODS database;

[0007] The data association step involves associating the original data in the tables of the ODS database with the data in a table of a predetermined format to generate an association table.

[0008] The data update step involves updating the data in the table with the predetermined format based on the data in the associated table.

[0009] Optionally, the data acquisition step includes converting the raw data into JSON format.

[0010] Optionally, the data retention step includes storing the original data of the same data type and format into the same table of the ODS database using a partitioned storage method, wherein the number of tables in the ODS database is at least one.

[0011] Optionally, the data association step includes associating the original data in the tables of the ODS database with data in tables of a predetermined format based on the user system.

[0012] Optionally, the data association step further includes converting the associated data based on the field types of the table in the predetermined format, and generating an association table based on the associated data.

[0013] Optionally, the data update step includes updating the data by overwriting the old data with a partition.

[0014] An embodiment of the present invention discloses a data processing system, the system comprising:

[0015] The data acquisition module acquires raw data from at least one data platform;

[0016] The data retention module stores the original data into a table in the ODS database;

[0017] The data association module associates the original data in the tables of the ODS database with the data in the tables of a predetermined format to generate an association table;

[0018] The data update module updates the data in the table with the predetermined format based on the data in the associated table.

[0019] An embodiment of the present invention discloses an electronic device, characterized in that the device includes a memory storing computer-executable instructions and a processor, the processor being configured to execute the instructions to implement the data processing method.

[0020] The present invention discloses a computer storage medium using computer program encoding, characterized in that the computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the data processing method.

[0021] This invention provides a highly efficient real-time update and governance analysis technology for aggregated data. It solves the difficult problems encountered in the real-time update and governance analysis of aggregated data from multiple platforms by means of original data retention, data parsing, data association, and partitioned data rewriting. It supports multiple data sources, customized data formats, and association with its own data, effectively ensuring the quality, availability, reusability, security, and scalability of the data.

[0022] Compared with the prior art, the main differences and effects of the embodiments of the present invention are as follows: it supports data sources of different formats and types from multiple data platforms, can perform standardized formatted data processing on aggregated data, associates with its own data, and solves the difficult problems of real-time updating and governance analysis of aggregated data.

[0023] In existing technologies, advertising platforms only provide data interfaces, but the data format cannot be customized and cannot be linked to their own data. For data governance, especially data updates, this has always been a pain point that is difficult to solve in aggregated data analysis scenarios.

[0024] Compared to existing technologies, the distinguishing technical feature of this invention is that it establishes a table with a predetermined format and associates it with the data in the ODS database to generate an associated table. Through the data in the associated table, the data in the table with the predetermined format is updated simply and efficiently.

[0025] The technical advantage of this invention lies in the fact that the multi-advertising platform data processing method of this invention can simultaneously support data synchronization of multiple platforms by integrating data governance and updates, and can facilitate correlation analysis with proprietary data, effectively solving the problem of comprehensive analysis of data such as advertising attribution, cost, and retrospective analysis. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating a scenario in which a data processing method according to an embodiment of the present invention is applied;

[0027] Figure 2 This is a flowchart of a data processing method according to an embodiment of the present invention;

[0028] Figure 3 This is a structural block diagram of a data processing method for acquiring data according to an embodiment of the present invention;

[0029] Figure 4 This is a structural block diagram of updated data in a data processing method according to an embodiment of the present invention;

[0030] Figure 5 This is a structural block diagram of a data processing system according to an embodiment of the present invention;

[0031] Figure 6 This is a hardware structure block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0032] The present invention will be further described below with reference to specific embodiments and accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, for ease of description, the accompanying drawings show only the parts relevant to the invention, and not all of the structures or processes. It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings.

[0033] It should be understood that although the terms "first," "second," etc., may be used herein to describe various features, these features should not be limited by these terms. The use of these terms is merely for distinction and should not be construed as indicating or implying relative importance. For example, without departing from the scope of the exemplary embodiments, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0035] Figure 1 This is a schematic diagram illustrating a scenario in which a data processing method according to an embodiment of the present invention is applied.

[0036] like Figure 1 As shown, the system includes a server 100 and a data platform 110. In the data processing method application of this embodiment, the server 100 primarily comprises an ODS library 120, a pre-formatted table 130, and a related table 140. The server 100 and the data platform 110 communicate via a wired / wireless network connection, allowing users to access the system. In one example, the data platform 110 may be an advertising data platform. In practical applications, data analysis for advertising data platforms mainly includes three phases: before advertising, during advertising, and after advertising. Real-time data updates and governance analysis are crucial throughout the entire process. Synchronizing and governing data from multiple advertising platform data sources is the core technology for the entire advertising platform's aggregated data analysis process. Data analysis allows for strategy formulation before advertising, estimation of actual budgets and conversion costs, real-time adjustment and optimization of advertising based on actual performance during advertising, and post-advertising summary and review to formulate future advertising plans. In the example above, the data objects of the advertising platform that the server-side data processing 100 includes time, dimensions, and traffic conversion data. Specifically, time can include advertising placement periods, cycles, year-on-year and month-on-month data, etc. Dimensions can include indicators such as ad placements, user audiences, and creative materials. Traffic conversion data can include data such as ad consumption, impressions, clicks, click-through rate, conversions, conversion rate, and conversion cost.

[0037] Figure 1 The diagram illustrates a data platform 110. It is worth noting that those skilled in the art will understand that the number of data platforms 110 is not limited to one; there may be one data platform 110 or several data platforms 110. The number of data platforms 110 should not be considered a limitation of the present invention. Furthermore, the data platform 110 can be any client or server that provides an interface to a data source. The data processing method of the present invention can be applied to electronic devices that require real-time updates and governance analysis of data.

[0038] As one example, the data platform can be a multi-advertising data platform. The server 100 performs real-time data processing on three data indicators of multiple advertising data. Based on the aggregated data of the multi-advertising data platform, the total number of data updates, the number of deduplications of the IP (Internet Protocol)-UA (User Agent)-device type merged field, and the number of deduplications of IDFA (Identifier For Advertising) are obtained in real time.

[0039] As another embodiment, the data platform can be a multi-advertising data platform, where server 100 processes the aggregated data from the multi-advertising data platform to obtain exposure data, click data, and spending data for governance analysis based on the aggregated data from the multi-advertising data platform.

[0040] The server-side 100 can store data and provide functions for standard formatting, updating, and normalizing data.

[0041] Currently, real-time updating and governance analysis of aggregated data is a major challenge in multi-data source aggregated data applications. Among these challenges, the technology for synchronizing and governing aggregated data from multiple data sources is the core technology of the entire aggregated data analysis process. Existing data processing technologies, especially in the synchronization and governance of data from multiple advertising platforms, cannot support multiple data sources and cannot effectively guarantee the real-time performance, reusability, and scalability of data synchronization.

[0042] To address the above problems, the present invention provides a data processing method. The method will be described in detail below with reference to the accompanying drawings.

[0043] Figure 2 This is a flowchart of a data processing method according to an embodiment of the present invention.

[0044] like Figure 2As shown, an embodiment of the present invention provides a data processing method that requires the cooperation of raw data 111, ODS database 120, table 121 in the ODS database, table 130 with a predetermined format, and associated table 140 in steps S1-S4. The method includes:

[0045] Step S1: Obtain raw data 111 from at least one data platform 110.

[0046] Big data collection can be achieved through database acquisition, system log acquisition, network data acquisition, and sensor device data acquisition. Traditional relational databases like MySQL and Oracle can store data, while in the big data era, NoSQL databases such as Redis, MongoDB, and HBase are also commonly used for data acquisition. This involves deploying a large number of databases at the acquisition end and implementing load balancing and sharding among them to complete the big data acquisition task. System log acquisition primarily collects the large amounts of log data generated daily by the data platform for use by offline and online big data analysis systems. High availability, high reliability, and scalability are fundamental characteristics of log collection systems. System log collection tools all employ a distributed architecture, capable of meeting the requirements for collecting and transmitting hundreds of megabytes of log data per second. Network data acquisition refers to the process of obtaining data information from websites through web crawlers or publicly available website APIs. Web crawlers start with one or more initial webpage URLs, obtain content from each webpage, and continuously extract new URLs from the current page and add them to a queue until a set stopping condition is met. This allows unstructured and semi-structured data to be extracted from webpages and stored in a local storage system. Data acquisition by sensing devices refers to the automatic collection of signals, images, or videos through sensors, cameras, and other intelligent terminals to obtain data. Big data intelligent sensing systems need to achieve intelligent identification, location, tracking, access, transmission, signal conversion, monitoring, preliminary processing, and management of massive amounts of structured, semi-structured, and unstructured data.

[0047] The various aggregated data collected above can be used as the raw data 111 obtained by the server 100 of the data processing method provided by this invention from the data platform 110.

[0048] Step S2: Store the original data 111 into table 121 in the ODS database.

[0049] ODS (Operational Data Store) is operational data, and an ODS database is an operational database. The data structure of an ODS database generally maintains consistency with the data source, which helps reduce the complexity of ETL (Extract-Transform-Load, data warehousing technology). Furthermore, the data lifecycle of an ODS database is typically short. An ODS database stores the current data status, providing users with the current state and meeting their needs for immediate, operational, and integrated information. As a transitional form from database to data warehouse, ODS offers high-performance response times. ODS design employs a hybrid approach: the data in an ODS is "real-time value," while the data in a data warehouse is "historical value." Generally, data stored in an ODS does not exceed one month. The biggest difference between tables in an ODS database and permanent tables in other databases is that the data in an ODS table is not permanent. Data in an ODS table is temporary; when a session or transaction ends, the data in the ODS table is automatically cleared by the ODS database without the user needing to delete it manually.

[0050] In one embodiment, based on the characteristics of the ODS library 120, the original data 111 is stored in table 121 in the ODS library. When an event or process ends, the corresponding original data 111 stored in table 121 of the ODS library is automatically cleared by the ODS library 120.

[0051] Storing the raw data obtained from data platform 110 into the ODS database has the following advantages:

[0052] An isolation layer is formed between business systems and the data warehouse. Typical data warehouse application systems have very complex data sources. For aggregated data from multiple data platforms, this data is stored in different geographical locations, different databases, and different applications. Extracting data from these business systems is not an easy task. Therefore, the ODS (Optical Data Store) is used to store data directly extracted from business systems. This data maintains a high degree of consistency with the business systems in terms of data structure and logical relationships, thus greatly reducing the complexity of data transformation during the extraction process. The focus is now primarily on issues such as the data extraction interface, data volume, and extraction methods.

[0053] Before the data warehouse was established, a significant portion of the detailed query functionality from the business systems was transferred. This was crucial because, in the process of generating complex reports, the business systems directly handled a large volume of reports and analyses, placing considerable pressure on their operation. Since the ODS database maintains consistency with the business systems in terms of granularity and organization, queries for reports and detailed data previously generated by the business systems can now be performed from the ODS database, thus reducing the query burden on the business systems.

[0054] To accomplish functions that a data warehouse cannot, the ODS layer typically stores aggregated data and operational metrics in its architecture, rather than detailed data for each transaction. However, in certain applications, it may be necessary to query detailed transaction data. In such cases, this detailed data query functionality needs to be transferred to the ODS. Furthermore, the ODS's data model, stored in a subject-oriented manner, easily supports multidimensional analysis and other query functions. In short, the data warehouse meets the enterprise's decision support requirements from a macro perspective, while the ODS layer reflects detailed transaction data or low-granularity data query requirements from a micro perspective.

[0055] Step S3 associates the original data 111 in table 121 of the ODS database with the data in table 130 of the predetermined format to generate the association table 140.

[0056] When a pre-formatted table 130 is created, unless otherwise specified, the pre-formatted table 130 is a permanent relational data table. The data in the pre-formatted table 130 will always exist unless a deletion instruction is explicitly requested.

[0057] In one example, a pre-formatted table 130 is created to design a complete user system. Tag instances are created based on the tags required by the business. Through the execution data in the tag instances, the user tag system includes natural attributes, product attributes, consumption attributes, resource attributes, etc. The execution data is associated with the original data 111 obtained from table 121 in the ODS library to generate an association table 140.

[0058] It should be noted that those skilled in the art will understand that the user system is one aspect of aggregated data analysis, and the analysis and governance of aggregated data includes many aspects. The user system on which the predefined format Table 130 is based is merely a specific example and is not intended to limit the data processing method of this invention.

[0059] Step S4: Update the data in table 130 with a predefined format based on the data in the associated table 140.

[0060] Association table 140 is a table composed of ordered pairs, used to express basic data types that are related.

[0061] In one example, the data that needs to be updated is temporarily stored in a related table 140, which is linked to a pre-formatted table 130. An update statement is then used to update the pre-formatted table 130.

[0062] As one implementation method, the original data 111 in table 121 of the ODS database is associated with the data in table 130 of a predetermined format based on the user system to form an association table 140. The association table has order number (order_id), order issuer (operator), order date (oper_date), and memo, etc. The predetermined format table 130 has order number (order_id), serial number (id), product code (code), product name (name), and remak, etc. The data that needs to be updated is temporarily stored in the association table 140, and the update statement is used to update the data in the association table 140 to the predetermined format table 130.

[0063] Next, combined Figure 3 and Figure 4 The acquisition and updating of data according to the data processing method of the present invention will be described.

[0064] Figure 3 This is a structural block diagram of a data processing method for acquiring data according to an embodiment of the present invention.

[0065] Please refer to Figure 3 The raw data 111 is provided by multiple data platforms 110. It is unavoidable that the data types and formats of different data interfaces are different, namely raw data 111A and raw data 111B. When the server 100 obtains the raw data 111 from the data platform 110, in order to customize a unified data format, the obtained raw data 111 is converted into JSON format. For the convenience of data governance, the JSON format raw data 111A and raw data 111B are stored in the corresponding areas of table 121 of the ODS database, namely table 121A and table 121B of the ODS database, respectively.

[0066] In one example, a partitioned storage method is used to store raw data 111 of the same data type and format into the same table in the ODS database 120, and the number of tables 121 in the ODS database is at least one.

[0067] In one implementation, the raw data 111 includes raw data 111A and raw data 111B of different data types and formats. The tables of the ODS database 120, ODS database table 121A and ODS database table 121B respectively store data of the same data type and format. Raw data 111A is stored in ODS database table 121A and raw data 111B is stored in ODS database table 121B.

[0068] It should be noted that those skilled in the art will understand that the number of tables 121 in the ODS library is based on the actual data processing operations performed by those skilled in the art. There is no fixed standard for the number of tables 121 in the ODS library. The number is set according to actual needs, based on the actual data processing situation, to ensure that the original data 111 can be fully stored.

[0069] Figure 4 This is a structural block diagram of updated data according to a data processing method based on an embodiment of the present invention.

[0070] In one example, based on the data in the associated table 140, the data in the predefined format table 130 is updated by means of partitioning and overwriting the old data.

[0071] Please refer to Figure 4 To facilitate data synchronization, based on the same field type, the pre-formatted table 130 is updated by partitioning and overwriting old data. As one implementation method, the associated table 140A and the pre-formatted table 130A have the same data field type, and the associated table 140B and the pre-formatted table 130B have the same data field type. Based on the data of the associated table 140A, the data of the pre-formatted table 130A is updated, and based on the data of the associated table 140B, the data of the pre-formatted table 130B is updated.

[0072] It should be noted that those skilled in the art will understand that the number of tables 130 and associated tables 140 in the predetermined format is based on the actual operation of data processing by those skilled in the art. There is no fixed standard for the number of tables 130 and associated tables 140 in the predetermined format. The number is based on the actual data processing situation and can fully store the original data 111, and is set according to actual needs.

[0073] Thus, one data processing step according to an embodiment of the present invention is completed. Through the above... Figures 1-4 As will be understood by those skilled in the art, the data processing method according to the present invention can effectively support multiple data sources, customize data formats, associate its own data, and effectively ensure the quality, availability, reusability, security and scalability of the data.

[0074] Figure 5 This is a structural block diagram of a data processing system according to an embodiment of the present invention.

[0075] like Figure 5 As shown, system 500 includes a data acquisition module 501, a data retention module 502, a data association module 503, and a data update module 504;

[0076] The data acquisition module 501 acquires raw data 111 from at least one data platform 110;

[0077] The data retention module 502 stores the original data 111 into table 121 in the ODS database;

[0078] The associated data module 503 associates the original data 111 in table 121 of the ODS database with the data in table 130 of a predetermined format to generate an associated table 140.

[0079] Update data module 504, based on the data in associated table 140, update the data in table 130 in a predefined format.

[0080] This embodiment is a corresponding method embodiment to the aforementioned embodiments, and can be implemented in conjunction with the aforementioned embodiments. The relevant technical details mentioned in the aforementioned embodiments remain valid in this embodiment, and will not be repeated here to avoid repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the aforementioned embodiments.

[0081] According to some embodiments of the present invention, an electronic device is disclosed, the device including a memory storing computer-executable instructions and a processor configured to execute the instructions to implement a data processing method.

[0082] Figure 6 This is a hardware structure block diagram of an electronic device implementing an embodiment of the present invention.

[0083] like Figure 6 As shown, the electronic device 600 may include one or more processors 602, a system motherboard 608 connected to at least one of the processors 602, system memory 605 connected to the system motherboard 608, non-volatile memory (NVM) 606 connected to the system motherboard 608, and a network interface 610 connected to the system motherboard 608.

[0084] Processor 602 may include one or more single-core or multi-core processors. Processor 602 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments of the invention, processor 602 may be configured to perform operations according to... Figure 2 The method shown.

[0085] In some embodiments, system motherboard 608 may include any suitable interface controller to provide any suitable interface to at least one of processors 602 and / or any suitable device or component communicating with system motherboard 608.

[0086] In some embodiments, system motherboard 608 may include one or more memory controllers to provide an interface to system memory 605. System memory 605 may be used to load and store data and / or instructions. In some embodiments, system memory 605 of electronic device 600 may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM).

[0087] The NVM 606 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the NVM 606 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of an HDD (Hard Disk Drive), a CD (Compact Disc) drive, or a DVD (Digital Versatile Disc) drive.

[0088] The NVM 606 may include a portion of the storage resources on a device installed on electronic device 600, or it may be accessible by the device, but is not necessarily part of the device. For example, the NVM 606 may be accessed over a network via network interface 610.

[0089] Specifically, system memory 605 and NVM 606 may each include a temporary copy and a permanent copy of instruction 620, respectively. Instruction 620 may include, when executed by at least one of processors 602, causing electronic device 600 to perform, as Figure 2 The instructions for the method shown. In some embodiments, the instructions 620, hardware, firmware and / or their software components may additionally / alternatively be located in the system motherboard 608, network interface 610 and / or processor 602.

[0090] Network interface 610 may include a transceiver for providing a radio interface to electronic device 600, thereby enabling communication with any other suitable device (e.g., front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 610 may be integrated into other components of electronic device 600. For example, network interface 610 may be integrated into at least one of processor 602, system memory 605, NVM 606, and firmware device (not shown) with instructions, wherein electronic device 600 implements [the desired functionality] when at least one of processor 602 executes the instructions. Figure 2 One or more embodiments of the various embodiments shown.

[0091] The network interface 610 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 610 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0092] In one embodiment, at least one of the processors 602 may be packaged together with one or more controllers for the system motherboard 608 to form a system-in-package (SiP). In another embodiment, at least one of the processors 602 may be integrated on the same die with one or more controllers for the system motherboard 608 to form a system-on-a-chip (SoC).

[0093] The electronic device 600 may further include an input / output (I / O) device 612 connected to the system motherboard 608. The I / O device 612 may include a user interface enabling a user to interact with the electronic device 600; the peripheral component interface is designed to allow peripheral components to also interact with the electronic device 600. In some embodiments, the electronic device 600 may also include sensors for determining at least one type of environmental condition and location information related to the electronic device 600.

[0094] In some embodiments, I / O device 612 may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash) and a keyboard.

[0095] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0096] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 600. In other embodiments of this application, the electronic device 600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0097] Program code can be applied to input instructions to perform the functions described in this invention and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a system for processing instructions including processor 602 includes any system having a processor such as a digital signal processor (DSP), microcontroller, application-specific integrated circuit (ASIC), or microprocessor.

[0098] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this invention are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0099] According to one embodiment of the present invention, a computer-readable storage medium is also provided, wherein at least one computer instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the aforementioned method.

[0100] According to one embodiment of the present invention, a computer program product is also provided, the computer program product comprising computer instructions, which, when executed, implement the aforementioned method.

[0101] The illustrative embodiments of the present invention include, but are not limited to, a data processing method, system, device, medium, and program product.

[0102] Various aspects of the illustrative embodiments will be described using terminology commonly employed by those skilled in the art to convey the essence of their work to others skilled in the art. However, it will be apparent to those skilled in the art that some alternative embodiments will be practiced using some of the features described herein. Specific figures and configurations are set forth for purposes of explanation in order to provide a more thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that alternative embodiments may be practiced without specific details. In some other instances, well-known features have been omitted or simplified herein to avoid obscuring the illustrative embodiments of the invention.

[0103] Furthermore, the various operations will be described as multiple separate operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order of description, and many of these operations may be performed in parallel, concurrently, or simultaneously. Moreover, the order of the operations may also be rearranged. The process may be terminated when the described operations are completed, but may also include additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0104] References to phrases such as "an example," "in an example," "an embodiment," and "an implementation" in the specification indicate that the described embodiment may include specific features, structures, or properties; however, each embodiment may or may not necessarily include specific features, structures, or properties. Furthermore, these phrases are not necessarily directed at the same embodiment. Additionally, when specific features are described in conjunction with specific embodiments, the knowledge of those skilled in the art can influence the combination of these features with other embodiments, whether or not those embodiments are explicitly described.

[0105] Unless the context otherwise specifies, the terms “comprising,” “having,” and “including” are synonyms. The phrase “A and / or B” means “(A), (B), or (A and B).”

[0106] As used herein, the term "module" may refer to, as part of, or include: a memory (shared, dedicated, or grouped), an application-specific integrated circuit (ASIC), electronic circuitry and / or a processor (shared, dedicated, or grouped), combinational logic circuitry, and / or other suitable components that provide the said functionality for running one or more software or firmware programs.

[0107] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not necessary. Rather, in some embodiments, these features may be illustrated in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular drawing does not mean that all embodiments need to include such features; in some embodiments, these features may be omitted or may be combined with other features.

[0108] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0109] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in fewer than all features of the single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0110] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature serving the same, equivalent, or similar purpose.

[0111] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

Claims

1. A data processing method for electronic devices, characterized in that, include: The data acquisition step involves acquiring raw data from at least one data platform. The data retention step involves storing the original data into a table in the ODS database; The data association step involves associating the original data in the tables of the ODS database with the data in a table of a predetermined format to generate an association table. The data update step involves updating the data in the predefined format table based on the data in the associated table. The data association step includes associating the raw data in the tables of the ODS database with data in tables of a predetermined format based on the user system. The data update step includes updating the data by overwriting the old data with partitions.

2. The data processing method according to claim 1, characterized in that, The data acquisition step includes converting the raw data into JSON format.

3. The data processing method according to claim 1, characterized in that, The data retention step includes storing the original data of the same data type and format into the same table of the ODS database using a partitioned storage method, wherein the number of tables in the ODS database is at least one.

4. The data processing method according to claim 1, characterized in that, The data association step further includes converting the associated data based on the field types of the table in the predetermined format, and generating an association table based on the associated data.

5. A data processing system, characterized in that, include: The data acquisition module acquires raw data from at least one data platform; The data retention module stores the original data into a table in the ODS database; The data association module associates the original data in the tables of the ODS database with the data in the tables of a predetermined format to generate an association table; The data update module updates the data in the predefined format table based on the data in the associated table. The associated data module includes associating the raw data in the tables of the ODS database with data in tables of a predetermined format based on the user system. The updated data module includes an update method that involves overwriting old data by partitioning.

6. An electronic device, characterized in that, The device includes a memory storing computer-executable instructions and a processor configured to execute the instructions to implement the data processing method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1-4.

Images