Data storage method, device, equipment and medium

Abstract

The present application relates to the field of data storage technology, and in particular to a data storage method, apparatus, equipment and medium. In the implementation of the present application, each buried point data can be stored in a distributed database corresponding to the first target identification information of the distributed database in which each buried point data is stored. On the one hand, compared with a relational database, a distributed database is more suitable for storing large amounts of data, and there will be no problem of affecting the running speed of the business system after the buried point data reaches a certain scale. Moreover, each buried point data is stored in a separate database, which can effectively reduce the storage pressure of each distributed database. On the other hand, the method does not require developers to develop programs, which reduces the workload of developers and reduces the requirements for disk performance and capacity of the business system.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a data storage method, apparatus, device, and medium. Background Art

[0002] With the vigorous development of various businesses, business data has also accumulated. Business system administrators, superior regulatory departments, etc. have put forward the need to collect and analyze various indicators such as system activity, transaction volume, user behavior analysis, etc. for different business systems. For example, statistics show how many times Zhang San has conducted transaction A in the business system in the past three years. Specifically, various indicators can be collected and analyzed through buried data.

[0003] Related technologies can store the tracking data to be analyzed in a relational database within the business system. The business system then analyzes the stored tracking data to obtain various indicators. Because relational databases are not suitable for storing large amounts of data, when the amount of tracking data stored in a relational database reaches a certain scale, it will affect the operation speed of the business system and even cause other functions of the business system to malfunction.

[0004] In related technologies, the buried data that needs to be analyzed can also be written into the log, and then the buried data in the log is loaded into the big data computing environment, and the data analysis is performed in the big data computing environment to obtain various indicators of the completed analysis. Since the process of writing the buried data into the log requires developers to perform program development operations, the workload of developers is increased, and this method has very high requirements on the disk performance and capacity of the business system. Summary of the Invention

[0005] The present application provides a data storage method, apparatus, equipment and medium to solve the problem in the prior art that storing large-scale tracking data based on relational databases will affect the operating speed of the business system, and writing tracking data into logs will increase the workload of developers.

[0006] The present application provides a data storage method, the method comprising:

[0007] Read the configuration file, obtain the burying point data to be buried carried in the configuration file and the first target identification information of the distributed database corresponding to the burying point data;

[0008] The buried point data is stored in a distributed database of the first target identification information.

[0009] Furthermore, the configuration file also carries second target identification information of the business system to be embedded; and storing the embedded data in the distributed database of the first target identification information includes:

[0010] According to the correspondence between the second target identification information of the business system to be embedded, the pre-saved identification information of the business system and the identification information of the topic in Kafka, determine the third target identification information of the topic in Kafka corresponding to the second identification information, and write the embedded data to the topic of the third target identification information in Kafka;

[0011] Identify the embedded point data written in the subject of the third target identification information, and store the identified embedded point data in the distributed database of the first target identification information.

[0012] Furthermore, the method further comprises:

[0013] If the buried point data in the distributed database of the first target identification information is lost, then after the identification conditions are met, the buried point data written in the subject of the third target identification information is identified, and the identified buried point data is stored in the distributed database of the first target identification information.

[0014] Furthermore, the identifying the tracking data written in the subject of the third target identification information includes:

[0015] Identify the embedded data written in the topic of the third target identification information based on Flink; or

[0016] Based on Flink SQL, the embedded data written in the topic of the third target identification information is identified.

[0017] Furthermore, the configuration file also carries fourth target identification information of the data table;

[0018] The storing the buried point data in the distributed database of the first target identification information includes:

[0019] According to the fourth target identification information, the burial point data is stored in the target data table of the fourth target identification information in the distributed database of the first target identification information.

[0020] Furthermore, the method further comprises:

[0021] Read the configuration file to determine whether the configuration file carries information for processing the buried data;

[0022] If so, data analysis is performed on the buried data.

[0023] Furthermore, the data analysis of the buried data includes:

[0024] Perform data analysis on the embedded data based on the Flink filter operator; or

[0025] Based on Flink SQL, the tracking data is analyzed.

[0026] The present application also provides a data storage device, comprising:

[0027] An acquisition module is configured to read a configuration file, obtain the buried point data to be buried carried in the configuration file, and obtain the first target identification information of the distributed database corresponding to the buried point data;

[0028] A storage module is used to store the buried point data in a distributed database of the first target identification information.

[0029] Furthermore, the storage module is specifically used to, if the configuration file also carries the second target identification information of the business system to be embedded, determine the third target identification information of the topic in Kafka corresponding to the second identification information according to the correspondence between the second target identification information of the business system to be embedded, the pre-saved identification information of the business system and the identification information of the topic in Kafka, and write the embedding data into the topic of the third target identification information in Kafka; identify the embedding data written in the topic of the third target identification information, and store the identified embedding data in the distributed database of the first target identification information.

[0030] Furthermore, the storage module is also used to identify the buried point data written in the subject of the third target identification information after the identification conditions are met if the buried point data in the distributed database of the first target identification information is lost, and store the identified buried point data in the distributed database of the first target identification information.

[0031] Furthermore, the storage module is specifically used to identify the embedded data written in the topic of the third target identification information based on Flink; or to identify the embedded data written in the topic of the third target identification information based on Flink SQL.

[0032] Furthermore, the storage module is specifically used to store the burial point data in the target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information if the configuration file also carries the fourth target identification information of the data table.

[0033] Furthermore, the device further comprises:

[0034] A determination module, configured to read the configuration file and determine whether the configuration file carries information for processing the buried data;

[0035] The analysis module is used to perform data analysis on the buried data if it carries information for processing the buried data.

[0036] Furthermore, the analysis module is specifically used to perform data analysis on the buried data based on the Flink filter operator; or to perform data analysis on the buried data based on Flink SQL.

[0037] The present application also provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any of the above-mentioned data storage methods when executing a computer program stored in the memory.

[0038] The present application also provides a computer-readable storage medium storing a computer program, which implements the steps of any of the above-mentioned data storage methods when executed by a processor.

[0039] The present application also provides a computer program product, which includes: computer program code, which enables the computer to execute any step of the above-mentioned data storage method when the computer program code is run on a computer.

[0040] In an embodiment of the present application, the configuration file is read to obtain the buried point data to be buried in the configuration file and the first target identification information of the distributed database in which the buried point data is stored, and the buried point data is stored in the distributed database of the first target identification information. In the implementation of the present application, each buried point data can be stored in the distributed database of the first target identification information according to the first target identification information of the distributed database in which each buried point data is stored. On the one hand, compared with the relational database, the distributed database is more suitable for storing large amounts of data, and there will be no problem of affecting the running speed of the business system after the buried point data reaches a certain scale. Moreover, each buried point data is stored in a separate database, which can effectively reduce the storage pressure of each distributed database. On the other hand, this method does not require developers to develop programs, which reduces the workload of developers and also reduces the requirements for disk performance and capacity of the business system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 A schematic diagram of a data storage process provided in some embodiments of the present application;

[0043] Figure 2 A schematic diagram of a data tracking process provided in some embodiments of the present application;

[0044] Figure 3 A detailed process diagram of data burial provided in some embodiments of the present application;

[0045] Figure 4 A schematic structural diagram of a data storage device provided in some embodiments of the present application;

[0046] Figure 5 A schematic structural diagram of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0048] When storing buried point data in the embodiment of the present application, the configuration file is read to obtain the buried point data to be buried in the configuration file and the first target identification information of the distributed database where the buried point data is stored; the buried point data is stored in the distributed database of the first target identification information.

[0049] In order to reduce the workload of developers and avoid the problem that storing large-scale tracking data based on relational databases will affect the operating speed of the business system, the embodiments of the present application provide a data storage method, device, equipment and medium.

[0050] Example 1:

[0051] Figure 1 A schematic diagram of a data storage process provided in some embodiments of the present application includes the following steps:

[0052] S101: Read a configuration file, obtain the burying point data to be buried carried in the configuration file and the first target identification information of the distributed database where the burying point data is stored.

[0053] The data storage method provided in the embodiments of the present application is applied to an electronic device, which may be a smart terminal, a PC, or a server.

[0054] In an embodiment of the present application, the electronic device obtains a configuration file in advance, wherein the configuration file carries the buried point data to be buried and the first target identification information of the distributed database where the buried point data is stored. In order to determine which buried point data is stored and where, the electronic device reads the configuration file, obtains the buried point data and the first target identification information of the distributed database where it is stored. The first target identification information of the distributed database where different buried point data are stored may be the same or different. The distributed database may be Hive, HBase, ES, etc., specifically, there is no limitation.

[0055] Among them, the configuration file may include a first sub-file for configuring the buried point data, and may also include a second sub-file for configuring the identification information of the distributed database where the buried point data is stored. In order to determine which buried point data is stored in which distributed database, the electronic device reads the first sub-file to determine the buried point data to be buried, and reads the second sub-file to determine the first target identification information of the distributed database where the buried point data is stored. Specifically, since the first sub-file may contain multiple buried point data to be buried, in order to determine where each buried point data is stored, the correspondence between the buried point data and its identification information may be saved in the first sub-file, and the correspondence between the identification information of the buried point data and the identification information of the distributed database may be saved in the second sub-file. For each buried point data, based on the correspondence between the buried point data and its identification information saved in the first sub-file, the target identification information corresponding to the buried point data is determined, and in the second sub-file, based on the correspondence between the identification information of the buried point data saved in advance and the identification information of the distributed database, the first target identification information of the distributed database where the buried point data of the target identification information is stored is determined.

[0056] S102: Storing the buried point data in a distributed database of the first target identification information.

[0057] After determining the buried point data and the first target identification information stored in the corresponding distributed database, the buried point data is stored in the distributed database of the first target identification information.

[0058] Because in the implementation of this application, each burial point data can be stored in the distributed database corresponding to the first target identification information of the distributed database where each burial point data is stored. On the one hand, compared with the relational database, the distributed database is more suitable for storing large amounts of data, and there will be no problem of affecting the running speed of the business system after the burial point data reaches a certain scale. Moreover, each burial point data is stored in a separate database, which can effectively reduce the storage pressure of each distributed database. On the other hand, there is no need to write the burial point data into the log to synchronize the burial point data in the big data computing environment, which reduces the workload of developers and reduces the requirements for disk performance and capacity of the business system.

[0059] Example 2:

[0060] In order to store the identified embedding data in the distributed database of the first target identification information, based on the above embodiment, in the embodiment of the present application, the configuration file also carries the second target identification information of the business system to be embedded;

[0061] The storing the buried point data in the distributed database of the first target identification information includes:

[0062] According to the correspondence between the second target identification information of the business system to be embedded, the pre-saved identification information of the business system and the identification information of the topic in Kafka, determine the third target identification information of the topic in Kafka corresponding to the second identification information, and write the embedded data to the topic of the third target identification information in Kafka;

[0063] Identify the embedded point data written in the subject of the third target identification information, and store the identified embedded point data in the distributed database of the first target identification information.

[0064] In an embodiment of the present application, in order to achieve decoupling of the electronic device and the distributed database, the electronic device can also write the buried data to Kafka. The subsequent electronic device identifies the buried data in Kafka and stores the identified buried data in the distributed database of the first target identification information to ensure that when the buried data is lost in the distributed database of the first target identification information, it can directly identify the buried data written in Kafka and re-store the buried data in the distributed database of the first target identification information without re-identifying the buried data in the configuration file and the first identification information of the distributed database where the buried data is stored.

[0065] Among them, Kafka is a high-throughput distributed publish-subscribe messaging system. It is an open source stream processing platform developed by the Apache Software Foundation. It provides fast message persistence capabilities and can maintain long-term stable performance even if data is stored in the form of Terabytes (TB). At the same time, it has high throughput characteristics. Even with very ordinary hardware, Kafka can support millions of messages per second.

[0066] In order to collect and analyze various indicators such as the activity, transaction volume, user behavior analysis, etc. of each business system, the distributed database where the tracking data is stored can be the database in the device deployed by the business system.

[0067] To facilitate subsequent identification, the embedded data corresponding to different business systems can also be written into different topics of Kafka for different business systems. Specifically, the configuration file can also carry the second target identification information of the business system to be embedded, and based on the second target identification information of the business system to be embedded, and the correspondence between the pre-saved identification information of the business system and the topic in Kafka, the third target identification information of the topic in Kafka corresponding to the second target identification information is determined, and the embedded data is written into the topic of the third target identification information in Kafka, wherein the embedded data of business systems corresponding to different identification information are written into the topics of different identification information in Kafka.

[0068] In order to store the buried data in the distributed database of the first target identification information, the electronic device identifies the buried data in Kafka and stores the data in Kafka in the distributed database of the first target identification information.

[0069] In related technologies, in order to collect and analyze various indicators such as system activity, transaction volume, and user behavior analysis, the Change Data Capture (CDC) tool can be used to synchronize the buried data stored in the relational database to the big data computing environment, delete the buried data in the relational database, and perform data analysis in the big data computing environment to obtain various indicators of the analysis. However, this method is more invasive to the business system and may cause database anomalies. In this application, since there is no need to synchronize the buried data in the big data computing environment for data analysis, data analysis can be performed locally on the device deployed by the business system, avoiding the risk of intrusion into the business system caused by synchronizing the buried data in the big data computing environment.

[0070] Figure 2 A schematic diagram of a process of burying data points provided in some embodiments of this application is now directed to Figure 2 Provide explanation.

[0071] The electronic device pre-reads the first sub-file of the configuration file, determines the buried point data stored in the first sub-file, and reads the second sub-file in the configuration file to determine the first target identification information of the distributed database corresponding to the buried point data. The buried point data is then stored in the distributed database corresponding to the first target identification information. Subsequently, the buried point data is cleaned in the device deployed by the business system, that is, in the device where the distributed database is located, and the error-free buried point data obtained is output to the display page of the business system. Among them, the process of data cleaning of the buried point data is a prior art and will not be described in detail here.

[0072] In order to achieve decoupling of the electronic device and the distributed database, based on the above embodiments, in an embodiment of the present application, the method further includes:

[0073] If the buried point data in the distributed database of the first target identification information is lost, then after the identification conditions are met, the buried point data written in the subject of the third target identification information is identified, and the identified buried point data is stored in the distributed database of the first target identification information.

[0074] If the buried data obtained after reading the configuration file is directly stored in the distributed database of the first target identification information, the stored buried data may be lost in the subsequent distributed database. When it is determined that the buried data stored in the distributed database is lost, in order to re-store the buried data in the distributed database, it is necessary to re-read the buried data of the configuration file and the first target identification information corresponding to the buried data stored in the distributed database, and then store the buried data in the distributed database of the first target identification information, which will cause over-coupling between the electronic device and the distributed database.

[0075] In order to decouple the electronic device from the distributed database, since the buried data has been written into the topic of the third target identification information in Kafka in advance, and the topics of the buried data written corresponding to different business systems are known and different, the buried data written in the topic of the third target identification information can be identified after the identification conditions are met, and the identified buried data can be stored in the distributed database of the first target identification information, so that the buried data is stored in the distributed database, avoiding the process of the electronic device re-identifying the buried data in the configuration file and the corresponding first target identification information stored in the distributed database.

[0076] Among them, the recognition condition can be that the burial point data has been lost for a preset length of time or an identification instruction has been received.

[0077] Example 3:

[0078] In order to identify the buried data, based on the above embodiments, in the embodiment of the present application, the buried data written in the subject of the third target identification information is identified, including:

[0079] Identify the embedded data written in the topic of the third target identification information based on Flink; or

[0080] Based on Flink SQL, the embedded data written in the topic of the third target identification information is identified.

[0081] In an embodiment of the present application, in order to identify the embedded data written into the topic of the third target identification information in Kafka, in one possible implementation, the embedded data written into the topic of the third target identification information can be identified based on Flink.

[0082] In another possible implementation, the embedded data written in the topic of the third target identification information can also be identified based on Flink SQL.

[0083] Example 4:

[0084] In order to accurately save each burial point data in the corresponding storage location, based on the above embodiments, in the embodiment of the present application, the configuration file also carries the fourth target identification information of the data table;

[0085] The storing the buried point data in the distributed database of the first target identification information includes:

[0086] According to the fourth target identification information, the burial point data is stored in the target data table of the fourth target identification information in the distributed database of the first target identification information.

[0087] In an embodiment of the present application, multiple data tables are pre-stored in each distributed database, and the data tables are used to store buried point data. In order to determine which data table in the distributed database each buried point data is stored in, the configuration file also carries the fourth target identification information of the data table. Specifically, the data table information can be carried in the third sub-file in the configuration file. Since there may be a situation where multiple buried point data need to be stored in the same distributed database, in order to determine which buried point data is stored in which data table, the third sub-file can also pre-store the corresponding relationship between the identification information of the buried point data and the identification information of the data table in the distributed data for each distributed database. For each buried point data, based on the corresponding relationship between the buried point data stored in the first sub-file and its identification information, the target identification information corresponding to the buried point data is determined, and based on the corresponding relationship between the identification information of the buried point data pre-stored in the second sub-file and the identification information of the distributed database, the first target identification information of the distributed database to which the buried point data corresponding to the target identification information is determined. Finally, based on the corresponding relationship between the identification information of the buried point data pre-stored in the third sub-file and the identification information of the data table in the distributed data for each distributed database, the fourth identification information of the data table to be stored corresponding to the buried point data is determined.

[0088] After determining the fourth target identification information of the data table carried in the third sub-file, the buried point data is stored in the target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information.

[0089] Example 5:

[0090] In order to meet the need for data analysis of buried data, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0091] Read the configuration file to determine whether the configuration file carries information for processing the buried data;

[0092] If so, data analysis is performed on the buried data.

[0093] In an embodiment of the present application, some buried point data may also need to be analyzed. In order to determine whether there is a need for data analysis for each buried point data, the configuration file also carries information on data analysis of the buried point data to be analyzed. Specifically, the fourth sub-file in the configuration file can carry information on processing the buried point data to be analyzed. In order to determine which buried point data to be analyzed, the fourth sub-file also pre-saves the correspondence between the identification information of the buried point data to be analyzed and the information to be processed. For each buried point data, based on the correspondence between the buried point data and its identification information saved in the first sub-file, the target identification information corresponding to the buried point data is determined, and based on the correspondence between the identification information of the buried point data to be analyzed and the information to be processed pre-saved in the fourth sub-file, it is determined whether there is information for processing the buried point data corresponding to the target identification information. If not, it is determined that data analysis is not required for the buried point data. If so, it is determined that data analysis is required for the buried point data.

[0094] When performing data analysis on buried point data, for each buried point data, if it is determined that the configuration file carries information for processing the buried point data, data analysis is performed on the buried point data.

[0095] In order to perform data analysis on the buried data, based on the above embodiments, in the embodiment of the present application, the data analysis on the buried data includes:

[0096] Perform data analysis on the embedded data based on the Flink filter operator; or

[0097] Based on Flink SQL, the tracking data is analyzed.

[0098] In order to perform data analysis on the buried data, in one possible implementation, the buried data can be analyzed based on the Flink filter operator.

[0099] In another possible implementation, data analysis can be performed on the embedded data based on Flink SQL.

[0100] Flink SQL is an open-source, distributed data analysis engine based on the Flink platform. It provides high-throughput, low-latency, and high-performance data analysis capabilities based on platforms such as Yarn and the distributed computing framework (Mesos). It provides SQL-like data analysis and offers both batch and real-time data analysis modes.

[0101] Figure 3A detailed process diagram of a point-of-sale data point-of-sale provided in some embodiments of this application is now directed to Figure 3 Provide explanation.

[0102] First, obtain a pre-configured configuration file. The electronic device reads the configuration file and determines whether the first sub-file in the configuration file exists. If not, it outputs a prompt message that the first sub-file is not found. If so, the embedded point data to be embedded in the first sub-file is written into Kafka, and the embedded point data written in Kafka is identified based on Flink.

[0103] The electronic device then determines whether the second sub-file in the configuration file exists. If not, it outputs a prompt message that the second sub-file is not found. If so, for each buried point data, based on the second sub-file, it determines the first target identification information of the distributed database where the buried point data is stored, and for each buried point data, it determines the fourth target identification information of the data table corresponding to the buried point data carried in the third sub-file in the configuration file, and stores the buried point data in the target data table of the fourth target identification information in the distributed database of the first target identification information. It then determines whether the fourth sub-file in the configuration file exists. If so, for each buried point data, it determines whether the fourth sub-file carries information for processing the buried point data. If so, it performs data analysis on the buried point data to be analyzed based on Flink SQL, and stores the results obtained after the data analysis.

[0104] After receiving the query instruction, the subsequent business system outputs the corresponding result data according to the information to be queried carried in the query instruction.

[0105] Example 6:

[0106] Figure 4 A schematic structural diagram of a data storage device provided in some embodiments of the present application, the device comprising:

[0107] The acquisition module 401 is configured to read a configuration file, obtain the burying point data to be buried carried in the configuration file, and the first target identification information of the distributed database corresponding to the burying point data;

[0108] The storage module 402 is used to store the buried point data in a distributed database of the first target identification information.

[0109] In a possible implementation, the storage module 402 is specifically used to, if the configuration file also carries the second target identification information of the business system to be embedded, determine the third target identification information of the topic in Kafka corresponding to the second identification information according to the correspondence between the second target identification information of the business system to be embedded, the pre-saved identification information of the business system and the identification information of the topic in Kafka, and write the embedding data into the topic of the third target identification information in Kafka; identify the embedding data written in the topic of the third target identification information, and store the identified embedding data in the distributed database of the first target identification information.

[0110] In one possible embodiment, the storage module 402 is also used to identify the buried point data written in the subject of the third target identification information after the identification conditions are met if the buried point data in the distributed database of the first target identification information is lost, and store the identified buried point data in the distributed database of the first target identification information.

[0111] In a possible implementation, the storage module 402 is specifically used to identify the embedded data written in the topic of the third target identification information based on Flink; or to identify the embedded data written in the topic of the third target identification information based on Flink SQL.

[0112] In a possible implementation, the storage module 402 is specifically used to store the burial point data in the target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information if the configuration file also carries the fourth target identification information of the data table.

[0113] In a possible implementation, the device further includes:

[0114] The determination module 403 is configured to read the configuration file and determine whether the configuration file carries information for processing the buried data;

[0115] The analysis module 404 is used to perform data analysis on the buried data if it carries information for processing the buried data.

[0116] In a possible implementation, the analysis module 404 is specifically configured to perform data analysis on the embedded data based on a Flink filter operator; or perform data analysis on the embedded data based on Flink SQL.

[0117] Example 7:

[0118] On the basis of the above embodiments, some embodiments of the present application further provide an electronic device, such as Figure 5 As shown, it includes: a processor 501 , a communication interface 502 , a memory 503 and a communication bus 504 , wherein the processor 501 , the communication interface 502 and the memory 503 communicate with each other via the communication bus 504 .

[0119] The memory 503 stores a computer program. When the program is executed by the processor 501, the processor 501 performs the following steps:

[0120] Read the configuration file, obtain the burying point data to be buried carried in the configuration file and the first target identification information of the distributed database corresponding to the burying point data;

[0121] The buried point data is stored in a distributed database of the first target identification information.

[0122] Furthermore, the processor 501 is specifically used to, if the configuration file also carries the second target identification information of the business system to be embedded, determine the third target identification information of the topic in Kafka corresponding to the second identification information according to the correspondence between the second target identification information of the business system to be embedded, the pre-saved identification information of the business system and the identification information of the topic in Kafka, and write the embedding data into the topic of the third target identification information in Kafka; identify the embedding data written in the topic of the third target identification information, and store the identified embedding data in the distributed database of the first target identification information.

[0123] Furthermore, the processor 501 is also used to identify the buried point data written in the subject of the third target identification information after the identification conditions are met if the buried point data in the distributed database of the first target identification information is lost, and store the identified buried point data in the distributed database of the first target identification information.

[0124] Furthermore, the processor 501 is specifically used to identify the embedded data written in the topic of the third target identification information based on Flink; or to identify the embedded data written in the topic of the third target identification information based on Flink SQL.

[0125] Furthermore, the processor 501 is specifically used to store the burial data in the target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information if the configuration file also carries the fourth target identification information of the data table.

[0126] Furthermore, the processor 501 is also used to read the configuration file and determine whether the configuration file carries information for processing the buried data; if so, perform data analysis on the buried data.

[0127] Furthermore, the processor 501 is specifically configured to perform data analysis on the embedded data based on a Flink filter operator; or perform data analysis on the embedded data based on Flink SQL.

[0128] The communication bus mentioned in the server above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0129] The communication interface 502 is used for communication between the electronic device and other devices.

[0130] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. Alternatively, the memory may be at least one storage device located away from the processor.

[0131] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processing processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0132] Example 8:

[0133] Based on the above embodiments, some embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program is executed on the electronic device, the electronic device implements the following steps:

[0134] The memory stores a computer program, which, when executed by the processor, causes the processor to perform the following steps:

[0135] Read the configuration file, obtain the burying point data to be buried carried in the configuration file and the first target identification information of the distributed database corresponding to the burying point data;

[0136] The buried point data is stored in a distributed database of the first target identification information.

[0137] Furthermore, the configuration file also carries the second target identification information of the business system to be embedded;

[0138] The storing the buried point data in the distributed database of the first target identification information includes:

[0139] According to the correspondence between the second target identification information of the business system to be embedded, the pre-saved identification information of the business system and the identification information of the topic in Kafka, determine the third target identification information of the topic in Kafka corresponding to the second identification information, and write the embedded data to the topic of the third target identification information in Kafka;

[0140] Identify the embedded point data written in the subject of the third target identification information, and store the identified embedded point data in the distributed database of the first target identification information.

[0141] Furthermore, the method further comprises:

[0142] If the buried point data in the distributed database of the first target identification information is lost, then after the identification conditions are met, the buried point data written in the subject of the third target identification information is identified, and the identified buried point data is stored in the distributed database of the first target identification information.

[0143] Furthermore, the identifying the tracking data written in the subject of the third target identification information includes:

[0144] Identify the embedded data written in the topic of the third target identification information based on Flink; or

[0145] Based on Flink SQL, the embedded data written in the topic of the third target identification information is identified.

[0146] Furthermore, the configuration file also carries fourth target identification information of the data table;

[0147] The storing the buried point data in the distributed database of the first target identification information includes:

[0148] According to the fourth target identification information, the burial point data is stored in the target data table of the fourth target identification information in the distributed database of the first target identification information.

[0149] Furthermore, the method further comprises:

[0150] Read the configuration file to determine whether the configuration file carries information for processing the buried data;

[0151] If so, data analysis is performed on the buried data.

[0152] Furthermore, the data analysis of the buried data includes:

[0153] Perform data analysis on the embedded data based on the Flink filter operator; or

[0154] Based on Flink SQL, the tracking data is analyzed.

[0155] Example 9:

[0156] An embodiment of the present application further provides a computer program product, which, when executed by a computer, implements the data storage method described in any one of the above method embodiments applied to an electronic device.

[0157] The above embodiments may be implemented in whole or in part through software, hardware, firmware, or any combination thereof, and may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions that, when loaded and executed on a computer, fully or partially generate the processes or functions described in the embodiments of the present application.

[0158] Because in the implementation of this application, each burial point data can be stored in the distributed database corresponding to the first target identification information of the distributed database where each burial point data is stored. On the one hand, compared with the relational database, the distributed database is more suitable for storing large amounts of data, and there will be no problem of affecting the running speed of the business system after the burial point data reaches a certain scale. Moreover, each burial point data is stored in a separate database, which can effectively reduce the storage pressure of each distributed database. On the other hand, there is no need to write the burial point data into the log to synchronize the burial point data in the big data computing environment, which reduces the workload of developers and reduces the requirements for disk performance and capacity of the business system.

[0159] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0160] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0163] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A data storage method, characterized in that: The method comprises: Read the configuration file, obtain the burying point data to be buried carried in the configuration file and the first target identification information of the distributed database corresponding to the burying point data; Storing the tracking data in a distributed database of the first target identification information; wherein the distributed database is a database in a device deployed by the business system; The configuration file includes a first sub-file for configuring the buried point data and a second sub-file for configuring identification information of the distributed database in which the buried point data is stored; The acquiring of the buried point data to be buried carried in the configuration file and the first target identification information of the distributed database corresponding to the buried point data and stored therein comprises: For each buried point data, based on the correspondence between the buried point data and its identification information stored in the first sub-file, determine the target identification information corresponding to the buried point data, and in the second sub-file, based on the correspondence between the identification information of the buried point data stored in advance and the identification information of the distributed database, determine the first target identification information of the distributed database where the buried point data of the target identification information is stored; The configuration file also carries the second target identification information of the business system to be embedded; The storing the buried point data in the distributed database of the first target identification information includes: According to the correspondence between the second target identification information of the business system to be embedded, the pre-saved identification information of the business system and the identification information of the topic in Kafka, determine the third target identification information of the topic in Kafka corresponding to the second target identification information, and write the embedded data to the topic of the third target identification information in Kafka; Identify the embedded point data written in the subject of the third target identification information, and store the identified embedded point data in the distributed database of the first target identification information.

2. The method according to claim 1, characterized in that The method further comprises: If the buried point data in the distributed database of the first target identification information is lost, then after the identification conditions are met, the buried point data written in the subject of the third target identification information is identified, and the identified buried point data is stored in the distributed database of the first target identification information.

3. The method according to claim 1, characterized in that The tracking data written in the subject of the third target identification information includes: Identify the embedded data written in the topic of the third target identification information based on Flink; or Based on Flink SQL, the embedded data written in the topic of the third target identification information is identified.

4. The method according to claim 1, wherein The configuration file also carries fourth target identification information of the data table; The storing the buried point data in the distributed database of the first target identification information includes: According to the fourth target identification information, the burial point data is stored in the target data table of the fourth target identification information in the distributed database of the first target identification information.

5. The method according to claim 1, characterized in that The method further comprises: Read the configuration file to determine whether the configuration file carries information for processing the buried data; If so, data analysis is performed on the buried data.

6. The method according to claim 5, characterized in that The performing data analysis on the buried data includes: Perform data analysis on the embedded data based on the Flink filter operator; or Based on Flink SQL, the tracking data is analyzed.

7. A data storage device, characterized in that The device comprises: An acquisition module is configured to read a configuration file and obtain the buried point data to be buried carried in the configuration file and the first target identification information of the distributed database in which the buried point data is stored; wherein the configuration file includes a first subfile for configuring the buried point data and a second subfile for configuring the identification information of the distributed database in which the buried point data is stored; A storage module, configured to store the buried data in a distributed database of the first target identification information; wherein the distributed database is a database in a device deployed by the business system; The acquisition module is specifically configured to determine, for each buried point data, the target identification information corresponding to the buried point data based on the correspondence between the buried point data and its identification information stored in the first sub-file, and determine, in the second sub-file, the first target identification information of the distributed database in which the buried point data of the target identification information is stored, based on the correspondence between the identification information of the buried point data stored in advance and the identification information of the distributed database; The storage module is specifically used to determine the third target identification information of the topic in Kafka corresponding to the second target identification information if the configuration file also carries the second target identification information of the business system to be embedded, based on the correspondence between the second target identification information of the business system to be embedded, the pre-saved identification information of the business system and the identification information of the topic in Kafka, and write the embedded data into the topic of the third target identification information in Kafka; identify the embedded data written in the topic of the third target identification information, and store the identified embedded data in the distributed database of the first target identification information.

8. The device according to claim 7, characterized in that The storage module is also used to identify the buried point data written in the subject of the third target identification information after the identification conditions are met if the buried point data in the distributed database of the first target identification information is lost, and store the identified buried point data in the distributed database of the first target identification information.

9. The device according to claim 7, characterized in that The storage module is specifically used to identify the embedded data written in the topic of the third target identification information based on Flink; or to identify the embedded data written in the topic of the third target identification information based on Flink SQL.

10. The device according to claim 7, characterized in that The storage module is specifically used to store the burial point data in the target data table of the fourth target identification information in the distributed database of the first target identification information according to the fourth target identification information if the configuration file also carries the fourth target identification information of the data table.

11. The device according to claim 7, characterized in that The device further comprises: A determination module, configured to read the configuration file and determine whether the configuration file carries information for processing the buried data; The analysis module is used to perform data analysis on the buried data if it carries information for processing the buried data.

12. The device according to claim 11, characterized in that The analysis module is specifically used to perform data analysis on the buried data based on the Flink filter operator; or to perform data analysis on the buried data based on Flink SQL.

13. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to implement the steps of the data storage method according to any one of claims 1 to 6 when executing the computer program stored in the memory.

14. A computer-readable storage medium, characterized in that It stores a computer program, which, when executed by a processor, implements the steps of the data storage method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The computer program product comprises: computer program code, which enables the computer to execute the steps of the data storage method according to any one of claims 1 to 6 when the computer program code is run on the computer.

Images