Data processing and storage method, device, equipment, medium and product

Abstract

The invention relates to a data processing and storage method and device, equipment, a medium and a product, and the method comprises the steps: obtaining step: obtaining incremental data from an offline database; in the processing step, the obtained incremental data is processed into processing data, and the processing data comprises data needing to be updated, data needing to be newly added and data needing to be deleted; in the monitoring step, data processing is monitored, and if data processing fails, an alarm is given; and a storage step: storing the processing data in an online database using one cluster. According to the data processing method and device, the technical problems that in a traditional mode, the data processing performance is poor, and storage resources are wasted are solved, the data processing performance is improved, and the storage space is saved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device, equipment, medium and product for data processing and storage. Background Art

[0002] In recent years, with the development of the Internet, various large models have become more and more complex. Coupled with the recent popularity of large models, the amount of data required for the model is increasing, the complexity is increasing, and the feature distinguishability is also increasing. As a result, the stability and real-time use of offline features are required to be relatively high. Previously, data was usually stored in an offline database, and now there is an increasing need to synchronize the offline database with the online database.

[0003] The existing synchronization method usually uses the Hbase database as the online database, which has the following defects:

[0004] 1. The existing synchronization method uses a lot of Python code, which is poorly readable, has a complex and messy engineering path, and is a huge project;

[0005] 2. Since there is no monitoring, if the system fails or reports an error, there will be no alarm or notification. The operation and maintenance personnel will only check the log on the machine when they find that no data has been generated.

[0006] 3. Poor data processing performance. All data is stored in the Hbase database each time, resulting in a very long storage period.

[0007] 4. Since the Hbase database uses two clusters, it wastes resources

[0008] 5. Feature management is rather chaotic and has no blood relationship, so it is impossible to find out how a certain feature is generated.

[0009] It can be seen that whether it is possible to provide an improved data processing and storage technology based on the deficiencies in the existing technology has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the invention

[0010] Problems to be solved by the invention

[0011] The purpose of the present invention is to overcome the defects of the prior art and provide an improved data processing and storage method, device, equipment and medium. The improved data processing and storage method, device, equipment and medium provided by the present invention solve the technical problems of poor data processing performance and waste of storage resources in the traditional method, improve the performance of data processing and save storage space.

[0012] Methods for solving problems

[0013] A first aspect of the present invention relates to a data processing and storage method, comprising the following steps:

[0014] Acquisition step: acquiring incremental data from the offline database;

[0015] Processing step: processing the acquired incremental data into processed data, including data to be updated, data to be added, and data to be deleted;

[0016] Monitoring step: monitoring data processing and issuing an alarm if data processing fails;

[0017] The storage step stores the processed data in an online database using a cluster.

[0018] Preferably, the offline database is a dataphin database.

[0019] Preferably, the calculation of the diff wide table is performed on a calculation engine in the dataphin database to obtain the incremental data.

[0020] Preferably, spark tasks are executed on a computing engine in a dataphin database to perform data processing.

[0021] Preferably, the computing engine is maxcompute.

[0022] Preferably, the online database is a TIDB database.

[0023] A second aspect of the present invention relates to a case screening device, comprising:

[0024] The acquisition module is used to obtain incremental data from the offline database;

[0025] The processing module is used to process the acquired incremental data into processed data, and the processed data includes: data that needs to be updated, data that needs to be added, and data that needs to be deleted;

[0026] The monitoring module is used to monitor data processing and issue an alarm if data processing fails;

[0027] The storage module is used to store the processed data in an online database using a cluster.

[0028] The third aspect of the present invention relates to a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing and storage method of the first aspect when executing the computer program.

[0029] A fourth aspect of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data processing and storage method of the first aspect.

[0030] A fifth aspect of the present invention relates to a computer program product, comprising a computer program, which implements the steps of the data processing and storage method of the first aspect when executed by a processor.

[0031] Effects of the Invention

[0032] The improved data processing and storage method provided by the present invention solves the technical problems of poor data processing performance and waste of storage resources in the traditional method, improves the data processing performance and saves storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the data processing and storage method according to the first embodiment of the present invention.

[0034] Figure 2 This is a structural diagram of a computer device according to a third embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of the comparison task of specific data processing and storage in Example 1. DETAILED DESCRIPTION

[0036] Hereinafter, the data processing and storage method involved in the present invention will be described in detail.

[0037] Figure 1 FIG. 1 is a flow chart of a data processing and storage method according to a first embodiment of the present invention. Figure 1 As shown, the specific process of the data processing and storage method is as follows: first, an acquisition step (step S100) is performed to obtain incremental data from an offline database. Then, a processing step (step S101) is performed to process the acquired incremental data into processed data, and the processed data includes: data that needs to be updated, data that needs to be added, and data that needs to be deleted. Then, a monitoring step (step S102) is performed to monitor the data processing, and an alarm is issued if the data processing fails. Finally, a reply step (step S103) is performed to store the processed data in an online database using a cluster.

[0038] Step S100 is described. Specifically, from an offline database, preferably from a dataphin database, using a computing engine, the application system featurestorejob will submit a scheduling configuration that needs to generate a diff wide table, and specifically will generate the following Figure 3The multiple comparison tasks shown determine which data is incremental data by comparing the structural differences of wide tables containing multiple fields, so that the data that needs to be synchronized is incremental, the data volume is small, and the synchronization speed is fast. The calculation engine of the dataphin database is preferably maxcompute.

[0039] Step S101 is described below. Specifically, a spark task is run on the computing engine to process data, and the processed data includes: data that needs to be updated, data that needs to be added, and data that needs to be deleted.

[0040] Step S102 is explained. Specifically, during the data processing, since the task status is known in real time, the data processing is monitored, and an alarm notification will be issued if an error is reported.

[0041] Step S103 is explained. Specifically, after the data is processed, it will be imported into an online database using a cluster. The online database is preferably a TIDB database. The TIDB database itself is a distributed database. Its TIDB Server supports a relational data query language similar to MySQL. The underlying layer is TiKv's K, V storage. For this purpose, it can use its relational query and convert all offline feature table data columns into rows and store them in a table in the TIDB database. This can ensure the speed of point query and batch query using SQL statements.

[0042] It can be seen that the data processing and storage method according to the first embodiment of the present invention solves the technical problems of poor data processing performance and waste of storage resources in the traditional method, improves the performance of data processing and saves storage space. Specifically, the data processing and storage method according to the first embodiment has the following advantages:

[0043] High performance in data processing: Importing full offline data into the Hbase database instead of incremental data into the TIDB database has made the processing time faster. The following is a comparison table.

[0044] Classification 2 o'clock 4 o'clock 6 o'clock 8 o'clock 10 o'clock 12 o'clock Hbase 144 378 690 913 1070 1112 TIDB 206 445 738 964 1199 1205

[0045] Data processing logic can be monitored: the task status can be known in real time during the execution of spark tasks, and an alarm will be notified if an error is reported;

[0046] Offline feature online storage is more economical: the Hbase database used in the existing technology uses two clusters for storage, while the TIDB database uses only one cluster for storage, saving about 150,000 yuan in costs per month;

[0047] More diversified queries: When query efficiency is similar, the TIDB database can be used to perform diversified queries using relational SQL statements, which is more scalable.

[0048] The data processing and storage device of the second embodiment of the present invention includes: an acquisition module for acquiring incremental data from an offline database; a processing module for processing the acquired incremental data into processed data, and the processed data includes: data that needs to be updated, data that needs to be added, and data that needs to be deleted; a monitoring module for monitoring data processing and issuing an alarm if data processing fails; and a storage module for storing the processed data in an online database using a cluster. The data processing and storage device corresponds to the data processing and storage method of the first embodiment, so the various variations in the first embodiment are also applicable to the second embodiment, which will not be described in detail here.

[0049] As described above, the data processing and storage device according to the second embodiment of the present invention solves the technical problems of low efficiency, easy errors and high resource consumption of traditional manual text messages, reduces operating costs and improves customer experience.

[0050] A third embodiment of the present invention provides a computer device, the internal structure diagram of which can be shown as follows: Figure 2 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to connect to an external device to process and store data with the external device. When the computer program is executed by the processor, the data processing and storage method involved in the first embodiment of the present invention is implemented.

[0051] Those skilled in the art will understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0052] The fourth embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When a processor executes the computer program, the data processing and storage method involved in the first embodiment of the present invention is implemented.

[0053] The fifth embodiment of the present invention provides a computer program product, including a computer program. When a processor executes the computer program, the data processing and storage method involved in the first embodiment of the present invention is implemented.

[0054] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0055] Industrial Applicability

[0056] The data processing and storage method, device, computer equipment, storage medium and computer program product involved in the present invention solve the technical problems of poor data processing performance and waste of storage resources in traditional methods, improve data processing performance and save storage space.

[0057] While the present invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

Claims

1. A data processing and storage method, characterized in that: The following steps are involved: Acquisition step: acquiring incremental data from the offline database; Processing step: processing the acquired incremental data into processed data, including data to be updated, data to be added, and data to be deleted; Monitoring step: monitoring data processing and issuing an alarm if data processing fails; The storage step stores the processed data in an online database using a cluster.

2. The data processing and storage method according to claim 1, characterized in that: The offline database is the dataphin database.

3. The data processing and storage method according to claim 2, characterized in that: The calculation of the diff wide table is performed on the calculation engine in the dataphin database to obtain incremental data.

4. The data processing and storage method according to claim 2, characterized in that: Execute spark tasks on the computing engine in the dataphin database to process data.

5. The data processing and storage method according to claim 3 or 4, characterized in that: The computing engine is maxcompute.

6. The data processing and storage method according to claim 1, characterized in that: The online database is the TIDB database.

7. A data processing and storage device, characterized in that: include: The acquisition module is used to obtain incremental data from the offline database; The processing module is used to process the acquired incremental data into processed data, and the processed data includes: data that needs to be updated, data that needs to be added, and data that needs to be deleted; The monitoring module is used to monitor data processing and issue an alarm if data processing fails; The storage module is used to store the processed data in an online database using a cluster.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data processing and storage method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing and storage method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing and storage method according to any one of claims 1 to 6 are implemented.

Images