Data processing method and device, storage medium and electronic equipment
By directly writing data to the database cluster, using the write frequency and rate to determine the interval duration, and writing partition data concurrently, the problem of low data writing efficiency caused by excessive load on the Click House proxy server is solved, and the large amount of data is achieved Fast write and stability.
Patent Information
- Application Number
- CN202310558589.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, the proxy server of Click House is too high, resulting in low data writing efficiency and inability to complete the rapid writing of large data volumes, affecting the application in big data scenarios.
By obtaining user's data writing instructions, the data is directly written to the target database in the database cluster, avoiding the proxy server, using the data writing frequency and rate to determine the write interval, and dividing the data into multiple partitioned data and concurrently writing. , ensure the stability and efficiency of data writing.
It solves the problem of excessive load on the Click House proxy server, improves data writing efficiency, realizes rapid writing of large data volumes, and enhances application capabilities in big data scenarios.
Smart Images

Figure CN119003628A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, device, storage medium and electronic device. Background Art
[0002] With the rapid development of the Internet and the Internet of Things, many companies have accumulated massive amounts of data. The value of data is also reflected in many fields, such as advertising, user recommendations, risk control and other businesses. They all rely on data analysis to improve business results. Click House has been widely favored by the industry for its excellent performance. At present, data writing to Click House mainly relies on the official writing method. The implementation principle is that the client connects to the proxy server of Click House, and after obtaining the data, the data is written to the proxy Click House through the proxy server.
[0003] In the prior art, the client connects to Click House's proxy server, and the proxy server forwards the acquired data to write the data to Click House. In this mode, a proxy server is introduced in the data writing link. When the amount of data to be written is large, it will cause an excessive load on the Click House proxy server, which will cause a single point failure. In addition, the data can only be truly written to Click House after being forwarded by the proxy server, which lengthens the data writing link and reduces the data writing efficiency. Therefore, when writing data to Click House using the current method, it is impossible to quickly write large amounts of data, which affects the further application of Click House in big data scenarios.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] The purpose of the present disclosure is to provide a data processing method, a data processing device, a computer-readable storage medium and an electronic device, thereby overcoming one or more problems caused by the limitations and defects of the related art at least to a certain extent.
[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0007] According to a first aspect of the present disclosure, there is provided a data processing method, comprising:
[0008] Acquire a data writing instruction from a user, where the data writing instruction includes first identification information and second identification information;
[0009] Acquire the data to be processed from the target device according to the first identification information;
[0010] The data to be processed is written into a target database according to the second identification information, and the target database is any database in a database cluster.
[0011] In an exemplary embodiment of the present disclosure, the data writing instruction further includes a data writing frequency and a data writing rate, and writing the to-be-processed data into a target database according to the second identification information includes:
[0012] determining the target database according to the second identification information;
[0013] Determine a data writing interval duration corresponding to the data writing frequency;
[0014] The data to be processed is written into the target database at the data writing rate at every data writing interval.
[0015] In an exemplary embodiment of the present disclosure, determining the data writing interval duration corresponding to the data writing frequency includes:
[0016] Determining whether the data writing frequency / data writing rate is less than or equal to a preset frequency threshold / preset rate threshold;
[0017] If so, determine the data writing interval duration corresponding to the data writing frequency.
[0018] In an exemplary embodiment of the present disclosure, writing the to-be-processed data into a target database according to the second identification information includes:
[0019] Acquire a first number of partitions included in the target database according to the second identification information;
[0020] Dividing the data to be processed into a plurality of partition data, and generating a first serial number for each partition data;
[0021] Determine a second serial number of a target partition to which the partition data is to be written according to the first serial number and the first quantity;
[0022] The partition data is written to the target partition according to the second sequence number.
[0023] In an exemplary embodiment of the present disclosure, the method further includes:
[0024] determining a second number of secondary partitions in the target partition;
[0025] Dividing the partition data according to the second quantity to obtain a plurality of sub-data;
[0026] Each of the sub-data is written into the primary partition and the secondary partition of the target partition respectively.
[0027] In an exemplary embodiment of the present disclosure, the method further includes:
[0028] If it is determined that data loss occurs in the primary partition / the secondary partition, the sub-data corresponding to the primary partition / the secondary partition is written into the secondary partition / the primary partition.
[0029] In an exemplary embodiment of the present disclosure, the partition data further includes a unique identifier, and the method further includes:
[0030] Determine the first partition that has the same unique identifier;
[0031] The partition data in the second partition in the first partition is retained, and the partition data in other partitions except the first partition and the second partition is deleted.
[0032] In an exemplary embodiment of the present disclosure, the method further includes:
[0033] If it is determined that the data writing abnormality judgment condition is met, an abnormality prompt message is sent to the user, and the data writing abnormality judgment condition includes at least one of the following:
[0034] The difference between the end write timestamp of the data to be processed and the start write timestamp of the data to be processed is greater than or equal to the preset duration, the ratio of the total number of partitioned data in the target database to the amount of data of the data to be processed is less than a first preset threshold, and the null value rate of the partitioned data in the target database is greater than or equal to a second preset threshold.
[0035] In an exemplary embodiment of the present disclosure, acquiring the to-be-processed data from the target device according to the first identification information includes:
[0036] Establishing an HCP connection with the target device according to the first identification information;
[0037] If the HCP connection with the target device is successfully established, the data to be processed is obtained from the target device.
[0038] In an exemplary embodiment of the present disclosure, writing the to-be-processed data into a target database according to the second identification information includes:
[0039] Establishing an HCP connection with the target database according to the second identification information;
[0040] If the HCP connection with the target database is successfully established, the data to be processed is written into the target database.
[0041] According to a second aspect of the present disclosure, there is provided a data processing device, comprising:
[0042] A data writing instruction acquisition module, used to acquire a data writing instruction of a user, wherein the data writing instruction includes first identification information and second identification information;
[0043] A to-be-processed data acquisition module, used for acquiring the to-be-processed data from the target device according to the first identification information;
[0044] The module for writing data to be processed is used to write the data to be processed into a target database according to the second identification information, and the target database is any database in the database cluster.
[0045] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are performed.
[0046] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:
[0047] Processor; and
[0048] A memory, configured to store executable instructions of the processor;
[0049] The processor is configured to perform the steps of any one of the methods described in the first aspect by executing the executable instructions.
[0050] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0051] To summarize, the method provided by the present invention obtains a data writing instruction from a user, wherein the data writing instruction includes first identification information of a target device where the data to be processed is located and second identification information of a target database to which the data to be processed is to be written; obtains the data to be processed from the target device according to the first identification information; and writes the data to be processed into the target database.
[0052] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0054] Figure 1 A flowchart of a data processing method in an exemplary embodiment of the present disclosure is schematically shown;
[0055] Figure 2 A flowchart of a data writing process in an exemplary embodiment of the present disclosure is schematically shown;
[0056] Figure 3 A schematic diagram of a database partition in an exemplary embodiment of the present disclosure is schematically shown;
[0057] Figure 4 A block diagram schematically shows a data processing device in an exemplary embodiment of the present disclosure;
[0058] Figure 5 A schematic diagram schematically shows a storage medium in an exemplary embodiment of the present disclosure;
[0059] Figure 6 A block diagram of an electronic device in an exemplary embodiment of the present disclosure is schematically shown.
[0060] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION
[0061] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0062] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. In view of the defects existing in the prior art, this example embodiment first provides a data processing method, which is applied to the client. Figure 1 As shown in , the above data processing method may include the following steps:
[0063] S11. Obtain a data writing instruction from a user, where the data writing instruction includes first identification information and second identification information.
[0064] In an exemplary embodiment of the present disclosure, the first identification information is identification information of a target device where the data to be processed is located, and the second identification information is identification information of a target database to which the data to be processed is to be written.
[0065] S12. Acquire data to be processed from the target device according to the first identification information.
[0066] In an exemplary embodiment of the present disclosure, the target device may be determined according to the first identification information, and then the data to be processed may be acquired from the target device.
[0067] S13. Write the data to be processed into a target database according to the second identification information, where the target database is any database in the database cluster.
[0068] In an exemplary embodiment of the present disclosure, the target database may be determined according to the second identification information, and then the data to be processed may be written into the target database.
[0069] In an exemplary embodiment of the present disclosure, the database cluster may be Click House, Redis or other large-scale distributed databases, which is not specifically limited in this embodiment.
[0070] The method provided by the present disclosure can obtain the user's data writing instruction, and the data writing instruction includes the first identification information and the second identification information; obtain the data to be processed from the target device according to the first identification information; write the data to be processed into the target database according to the second identification information, and the target database is any database in the database cluster. After obtaining the data, the data can be directly written into the database cluster, and the proxy server is removed from the data writing link, avoiding the problem that the existing technology will cause the Click House proxy server to be overloaded, thereby causing a single point of failure, and ensuring the stability of the data writing link; in addition, the method of the present application is adopted to shorten the data writing link, thereby improving the data writing efficiency. Therefore, the method of the present application is adopted to write data into distributed databases such as Click House, which can complete the rapid writing of large amounts of data and accelerate the further application of distributed databases such as Click House in big data scenarios.
[0071] Below, each step of the signal modulation method in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.
[0072] In step S11, a data writing instruction of a user is obtained, wherein the data writing instruction includes first identification information of a target device where the data to be processed is located and second identification information of a target database to which the data to be processed is to be written.
[0073] Based on the above content, in an exemplary embodiment of the present disclosure, the above data writing instruction also includes a data writing frequency and a data writing rate, and writing the to-be-processed data into a target database according to the second identification information includes:
[0074] S111, determining the target database according to the second identification information;
[0075] S112, determining a data writing interval duration corresponding to the data writing frequency;
[0076] S113: Write the to-be-processed data into the target database at the data writing rate at every data writing interval.
[0077] In an exemplary embodiment of the present disclosure, the second identification information may be the number of the target database in the database cluster, the name of the target database, or the Internet Protocol (IP) address of the target database, which can uniquely identify the target database so as to determine the target database when writing the data to be processed.
[0078] Specifically, when the data writing frequency is 0.01 Hz and the data writing rate is 100 Mbps, the data writing interval corresponding to the data writing frequency is 100 seconds. Then, the data to be processed is written to the target database at a data writing rate of 100 Mbps every 100 seconds.
[0079] Based on the above content, in an exemplary embodiment of the present disclosure, the above-mentioned determination of the data writing interval duration corresponding to the data writing frequency includes:
[0080] Determine whether the data writing frequency / data writing rate is less than or equal to a preset frequency threshold / preset rate threshold; if so, determine the data writing interval duration corresponding to the data writing frequency.
[0081] In step S12, the data to be processed is acquired from the target device according to the first identification information.
[0082] In an exemplary embodiment of the present disclosure, the first identification information is the identification information of the target device where the data to be processed is located, and the first identification information may be the network protocol IP address of the target device and the storage directory of the data to be processed in the target device. The client may send a data acquisition request to the target device according to the IP address of the target device and the storage directory of the data to be processed in the target device, so that the target device acquires the data to be processed according to the storage directory and returns the data to be processed to the client.
[0083] In step S13, the data to be processed is written into the target database, where the target database is any database in the database cluster.
[0084] Based on the above content, in an exemplary embodiment of the present disclosure, Figure 2 As shown, the above-mentioned writing the to-be-processed data into the target database according to the second identification information includes:
[0085] S131. Acquire a first number of partitions included in the target database according to the second identification information;
[0086] S132, dividing the data to be processed into multiple partition data, and generating a first serial number for each partition data;
[0087] S133, determining a second sequence number of the target partition to which the partition data is to be written according to the first sequence number and the first quantity;
[0088] S134. Write the partition data to the target partition according to the second sequence number.
[0089] Specifically, refer to Figure 3 As shown, for example, the target database includes N partitions, N is an integer greater than 1, and the serial numbers of N partitions are 1, 2, 3...N. The data to be processed is divided into M partition data, M is an integer greater than 1, and the serial numbers of each partition data are 1, 2, 3...M. Then the serial number of each partition data is modulo N, and the remainder is used as the serial number of the target partition to which the partition data is to be written. For example, N is 3 and M is 8, then the partition data with serial numbers 1, 4 and 7 are written to partition 1, the partition data with serial numbers 2, 5 and 8 are written to partition 2, and the partition data with serial numbers 3 and 6 are written to partition 3.
[0090] By dividing the data to be processed to obtain multiple partition data, and generating a first serial number for each partition data; determining the second serial number of the target partition to which the partition data is to be written based on the first serial number and the first quantity, each partition data can be concurrently written to the corresponding target partition, thereby improving data writing efficiency.
[0091] Based on the above content, in an exemplary embodiment of the present disclosure, the above method further includes:
[0092] S135, determining a second number of sub-partitions in the target partition;
[0093] S136, dividing the partition data according to the second number to obtain a plurality of sub-data;
[0094] S137. Write each sub-data into the primary partition and the secondary partition of the target partition respectively.
[0095] For example Figure 3 As shown, the number of sub-partitions in partition 1 is 2, so the partition data with sequence number 1 is divided into 3 parts, and then the 3 parts of sub-data are written into the primary partition and 2 sub-partitions in partition 1 respectively.
[0096] By dividing the partition data according to the second number to obtain multiple sub-data; each sub-data is written into the primary partition and the secondary partition of the target partition respectively, each sub-data can be written into the partition concurrently, thereby improving data writing efficiency.
[0097] Based on the above content, in an exemplary embodiment of the present disclosure, the method further includes:
[0098] If it is determined that data loss occurs in the primary partition / the secondary partition, the sub-data corresponding to the primary partition / the secondary partition is written into the secondary partition / the primary partition.
[0099] For example, if data is lost in the primary partition of partition 1, sub-data 1 corresponding to the primary partition is written to secondary partition 1 to ensure that partition 1 can store partition data with sequence number 1 completely. For another example, if data is lost in secondary partition 1 of partition 1, sub-data 1 corresponding to secondary partition 1 is written to the primary partition to ensure that partition 1 can store partition data with sequence number 1 completely, thus avoiding data writing abnormality caused by data loss and ensuring normal data writing.
[0100] Based on the above content, in an exemplary embodiment of the present disclosure, the partition data further includes a unique identifier, and the method further includes:
[0101] S138, determining that there is a first partition with the same unique identifier;
[0102] S139: retain the partition data in the second partition in the first partition, and delete the partition data in other partitions except the second partition in the first partition.
[0103] For example, the first partitions with the same unique identifier are partition 3, partition 4, and partition 5, that is, partition 3, partition 4, and partition 5 store the same partition data. However, since the partition data includes the unique identifier, only one copy of the partition data can be written in the target database. Therefore, partition 3 can be used as the second partition, the partition data in partition 3 is retained, and the partition data in partition 4 and partition 5 are deleted.
[0104] Based on the above content, in an exemplary embodiment of the present disclosure, the method further includes:
[0105] If it is determined that the data writing abnormality judgment condition is met, an abnormality prompt message is sent to the user, and the data writing abnormality judgment condition includes at least one of the following:
[0106] The difference between the end write timestamp of the data to be processed and the start write timestamp of the data to be processed is greater than or equal to the preset duration, the ratio of the total number of partitioned data in the target database to the amount of data of the data to be processed is less than a first preset threshold, and the null value rate of the partitioned data in the target database is greater than or equal to a second preset threshold.
[0107] In an exemplary embodiment of the present disclosure, if the difference between the end write timestamp of the data to be processed and the start write timestamp of the data to be processed is greater than or equal to the preset time length, it means that the write time of the data to be processed is too long, that is, it is determined that the data write abnormality judgment condition is met. If the ratio of the total number of partitioned data in the target database to the data volume of the data to be processed is less than the first preset threshold, it means that the write success rate of the data to be processed is too low, that is, it is determined that the data write abnormality judgment condition is met. If the null value rate of the partitioned data in the target database is greater than or equal to the second preset threshold, it means that the null value rate of the data written in the partition is too high, that is, it is determined that the data write abnormality judgment condition is met.
[0108] It should be noted here that after the data to be processed is written into the target database, multi-dimensional data such as data writing duration (the difference between the end writing timestamp and the start writing duration of the data to be processed), the amount of data written (the ratio of the total number of partitioned data in the target database to the amount of data of the data to be processed) and the data quality of the written data (the null value rate of the partitioned data in the target database) can be displayed to the user, and the year-on-year and month-on-month data analysis of multi-dimensional data such as data writing duration, amount of data written and data quality of the written data can be displayed to the user for the user to view.
[0109] Based on the above content, in an exemplary embodiment of the present disclosure, the acquiring the to-be-processed data from the target device according to the first identification information includes:
[0110] S14, establishing an HCP connection with the target device according to the first identification information;
[0111] S15. If the HCP connection with the target device is successfully established, the data to be processed is obtained from the target device.
[0112] In an exemplary embodiment of the present disclosure, before acquiring data, the client first establishes a connection with the host communications processor (HCP) of the target device according to the IP address of the target device. If the HCP connection with the target device is successfully established, the client can communicate with the target device and then acquire the data to be processed from the target device.
[0113] Based on the above content, in an exemplary embodiment of the present disclosure, writing the to-be-processed data into a target database according to the second identification information includes:
[0114] S16. Establishing an HCP connection with the target database according to the second identification information;
[0115] S17: If the HCP connection with the target database is successfully established, the data to be processed is written into the target database.
[0116] In an exemplary embodiment of the present disclosure, before writing data to the target database, the client first establishes an HCP connection with the target database according to the second identification information. If the HCP connection with the target database is successfully established, the client can communicate with the target database and then write the data to be processed into the target database.
[0117] After introducing the data processing method of the exemplary embodiment of the present invention, Figure 4 A data processing apparatus according to an exemplary embodiment of the present invention is described.
[0118] refer to Figure 4 As shown, the data processing device 40 of the exemplary embodiment of the present invention may include: a data writing instruction acquisition module 401, a to-be-processed data acquisition module 402, and a to-be-processed data writing module 403. Among them:
[0119] A data writing instruction acquisition module 401 is used to acquire a data writing instruction of a user, wherein the data writing instruction includes first identification information and second identification information;
[0120] The to-be-processed data acquisition module 402 is used to acquire the to-be-processed data from the target device according to the first identification information;
[0121] The to-be-processed data writing module 403 is used to write the to-be-processed data into a target database according to the second identification information, and the target database is any database in the database cluster.
[0122] In an exemplary embodiment of the present disclosure, the data writing instruction further includes a data writing frequency and a data writing rate, and the to-be-processed data writing module includes:
[0123] a target database determining unit, configured to determine the target database according to the second identification information;
[0124] A data writing interval duration determining unit, used to determine a data writing interval duration corresponding to the data writing frequency;
[0125] A data writing unit is used to write the to-be-processed data into the target database at the data writing rate every the data writing interval.
[0126] In an exemplary embodiment of the present disclosure, the data writing unit includes:
[0127] A write frequency determination module, used to determine whether the data write frequency / data write rate is less than or equal to a preset frequency threshold / preset rate threshold;
[0128] The data writing subunit is used to determine the data writing interval duration corresponding to the data writing frequency.
[0129] In an exemplary embodiment of the present disclosure, the module for writing data to be processed includes:
[0130] A first quantity acquisition unit, configured to acquire a first quantity of partitions included in the target database according to the second identification information;
[0131] A first serial number generating unit, used for dividing the data to be processed into a plurality of partition data, and generating a first serial number for each partition data;
[0132] a second sequence number determining unit, configured to determine, according to the first sequence number and the first quantity, a second sequence number of a target partition to which the partition data is to be written;
[0133] A partition data writing unit is used to write the partition data into the target partition according to the second sequence number.
[0134] In an exemplary embodiment of the present disclosure, the device further includes:
[0135] Partition data partitioning module, used for:
[0136] Determine a second number of sub-partitions in the target partition; divide the partition data according to the second number to obtain a plurality of sub-data; and write each sub-data into the primary partition of the target partition and the sub-partition respectively.
[0137] In an exemplary embodiment of the present disclosure, the device further includes:
[0138] Sub-data writing module, used for:
[0139] If it is determined that data loss occurs in the primary partition / the secondary partition, the sub-data corresponding to the primary partition / the secondary partition is written into the secondary partition / the primary partition.
[0140] In an exemplary embodiment of the present disclosure, the partition data further includes a unique identifier, and the device further includes:
[0141] Sub-data writing module, used for:
[0142] Determine the first partition that has the same unique identifier;
[0143] The partition data in the second partition in the first partition is retained, and the partition data in other partitions except the first partition and the second partition is deleted.
[0144] In an exemplary embodiment of the present disclosure, the device further includes:
[0145] Write an abnormal judgment condition determination module, used to:
[0146] If it is determined that the data writing abnormality judgment condition is met, an abnormality prompt message is sent to the user, and the data writing abnormality judgment condition includes at least one of the following:
[0147] The difference between the end write timestamp of the data to be processed and the start write timestamp of the data to be processed is greater than or equal to the preset duration, the ratio of the total number of partitioned data in the target database to the amount of data of the data to be processed is less than a first preset threshold, and the null value rate of the partitioned data in the target database is greater than or equal to a second preset threshold.
[0148] In an exemplary embodiment of the present disclosure, the module for acquiring data to be processed includes:
[0149] A first HCP connection establishing module, configured to establish an HCP connection with the target device according to the first identification information;
[0150] The to-be-processed data acquisition unit is configured to acquire the to-be-processed data from the target device if the HCP connection with the target device is successfully established.
[0151] In an exemplary embodiment of the present disclosure, the module for writing data to be processed includes:
[0152] A second HCP connection establishing module, configured to establish an HCP connection with the target database according to the second identification information;
[0153] The unit for writing data to be processed is used to write the data to be processed into the target database if the HCP connection with the target database is successfully established.
[0154] Since the functional modules of the data processing device in the embodiment of the present invention are the same as those in the above-mentioned data processing method in the embodiment of the present invention, they will not be described in detail here.
[0155] After introducing the data processing method and data processing device according to the exemplary embodiment of the present invention, Figure 5 A storage medium according to an exemplary embodiment of the present invention is described.
[0156] refer to Figure 5 As shown, a program product 500 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0157] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0158] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0159] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0160] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0161] After introducing the storage medium of the exemplary embodiment of the present invention, next, reference is made to Figure 6 An electronic device according to an exemplary embodiment of the present invention is described.
[0162] Figure 6 The electronic device 60 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0163] like Figure 6 As shown, the electronic device 60 is in the form of a general computing device. The components of the electronic device 60 may include but are not limited to: the at least one processing unit 610, the at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), and a display unit 640.
[0164] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1Steps S11 to S13 shown in FIG.
[0165] The storage unit 620 may include a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203. The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may include the implementation of a network environment.
[0166] The bus 630 may include a data bus, an address bus, and a control bus.
[0167] The electronic device 60 may also communicate with one or more external devices 70 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and such communication may be performed through an input / output (I / O) interface 650. The electronic device 60 also includes a display unit 640, which is connected to the input / output (I / O) interface 650 for display. In addition, the electronic device 60 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 60 through a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 60, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0168] It should be noted that, although several modules or submodules of the data processing device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided into multiple units / modules to be embodied.
[0169] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0170] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.
Claims
1. A data processing method, characterized in that: include: Acquire a data writing instruction from a user, where the data writing instruction includes first identification information and second identification information; Acquire the data to be processed from the target device according to the first identification information; The data to be processed is written into a target database according to the second identification information, and the target database is any database in a database cluster.
2. The method according to claim 1, characterized in that The data writing instruction further includes a data writing frequency and a data writing rate, and writing the to-be-processed data into a target database according to the second identification information includes: determining the target database according to the second identification information; Determine a data writing interval duration corresponding to the data writing frequency; The data to be processed is written into the target database at the data writing rate at every data writing interval.
3. The method according to claim 2, characterized in that The determining of the data writing interval duration corresponding to the data writing frequency comprises: Determining whether the data writing frequency / data writing rate is less than or equal to a preset frequency threshold / preset rate threshold; If so, determine the data writing interval duration corresponding to the data writing frequency.
4. The method according to claim 1, characterized in that: Writing the to-be-processed data into a target database according to the second identification information comprises: Acquire a first number of partitions included in the target database according to the second identification information; Dividing the data to be processed into a plurality of partition data, and generating a first serial number for each of the partition data; Determine a second serial number of a target partition to which the partition data is to be written according to the first serial number and the first quantity; The partition data is written to the target partition according to the second sequence number.
5. The method according to claim 4, characterized in that The method further comprises: determining a second number of secondary partitions in the target partition; Dividing the partition data according to the second quantity to obtain a plurality of sub-data; Each of the sub-data is written into the primary partition and the secondary partition of the target partition respectively.
6. The method according to claim 5, characterized in that The method further comprises: If it is determined that data loss occurs in the primary partition / the secondary partition, the sub-data corresponding to the primary partition / the secondary partition is written into the secondary partition / the primary partition.
7. The method according to claim 4, characterized in that The partition data also includes a unique identifier, and the method further includes: Determine the first partition that has the same unique identifier; The partition data in the second partition in the first partition is retained, and the partition data in other partitions except the first partition and the second partition is deleted.
8. The method according to claim 4, characterized in that The method further comprises: If it is determined that the data writing abnormality judgment condition is met, an abnormality prompt message is sent to the user, and the data writing abnormality judgment condition includes at least one of the following: The difference between the end write timestamp of the data to be processed and the start write timestamp of the data to be processed is greater than or equal to the preset duration, the ratio of the total number of partitioned data in the target database to the amount of data of the data to be processed is less than a first preset threshold, and the null value rate of the partitioned data in the target database is greater than or equal to a second preset threshold.
9. The method according to claim 1, characterized in that: The acquiring the to-be-processed data from the target device according to the first identification information comprises: Establishing a connection with a host communication processor HCP of the target device according to the first identification information; If the HCP connection with the target device is successfully established, the data to be processed is obtained from the target device.
10. The method according to claim 1, characterized in that Writing the to-be-processed data into a target database according to the second identification information comprises: Establishing an HCP connection with the target database according to the second identification information; If the HCP connection with the target database is successfully established, the data to be processed is written into the target database.
11. A data processing device, characterized in that: include: A data writing instruction acquisition module, used to acquire a data writing instruction of a user, wherein the data writing instruction includes first identification information and second identification information; A to-be-processed data acquisition module, used for acquiring the to-be-processed data from the target device according to the first identification information; The module for writing data to be processed is used to write the data to be processed into a target database according to the second identification information, and the target database is any database in the database cluster.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the steps of the method according to any one of claims 1 to 10 by executing the executable instructions.
Citation Information
Patent Citations
Method of operating data storage device and method of operating data processing system
CN104951248A
File write-in method in distributed system and apparatus thereof
CN107493309A
Data writing method and device, data reading method and device, equipment and storage medium
CN111831618A
Data writing method and device, data writing equipment and storage medium
CN114490659A
