Data encoding and decoding method of distributed database and electronic equipment

Through bitmap encoding and decoding technology, the active GTID list of distributed databases is solved, and efficient encoding compression and decoding is achieved, which significantly reduces network traffic and time and optimizes transmission efficiency.

CN116132412BActive Publication Date: 2024-11-19JINZHUAN INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211515468.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-29
Publication Date
2024-11-19
Estimated Expiration
2042-11-29

AI Technical Summary

Technical Problem

The existing distributed database consumes a lot of network traffic when querying active GTID lists. With the increase of TPS and the decrease of validity period of active GTID lists, the encoding compression efficiency is low, making it difficult to effectively reduce network traffic.

Method used

By obtaining the encoding interval corresponding to the data, encoding and decoding the data ID based on bitmap encoding, determining the encoding value using the data density and matching conditions within the encoding interval, storing the encoded data and managing the mapping table, optimizing the encoding and decoding process.

Benefits of technology

It greatly reduces the space occupied by streaming data, provides a higher compression ratio, reduces network traffic, shortens encoding and decoding times, while maintaining lossless characteristics, and improving transmission efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116132412B_ABST
    Figure CN116132412B_ABST
Patent Text Reader

Abstract

The present invention relates to a data encoding and decoding method and electronic device of a distributed database, the method comprising: obtaining stream data to be encoded, the stream data comprising multiple data IDs; obtaining at least one encoding interval corresponding to the stream data, the encoding interval being divided according to the distribution of multiple data IDs; for each encoding interval, determining multiple encoding IDs based on the encoding interval; according to the multiple encoding IDs of each encoding interval, bitmap encoding is performed on the data IDs located in each encoding interval to obtain encoded data. The embodiment of the present application can significantly reduce the space occupied by the stream data, provide a higher compression ratio, thereby further reducing the network traffic required for the transmission of stream data in the distributed database, shortening the encoding compression time, while maintaining the lossless characteristics and optimizing the transmission efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data encoding and decoding method and electronic equipment for a distributed database. Background Art

[0002] A distributed database connects multiple physically dispersed nodes to form a logically unified database. The computing and storage pressures are divided into different computing nodes (Computer Node, CN) and data nodes (Data Node, DN).

[0003] The current distributed database coordinates the commit or rollback of multiple nodes by adding a global transaction node (Global Transaction Node, GTM) and letting GTM use the global transaction ID (GTID). Each GTID corresponds to a distributed transaction. Before the transaction is completed, the GTID is active, and after the transaction is completed, it is inactive. In order to achieve the isolation level in distributed transactions, such as committed reads and consistent reads, it is necessary for the CN node to know the current active GTID list and update it in time after the validity period expires.

[0004] However, in actual applications, the network traffic consumed in querying the active list will be very large. GTID active queries occupy a large amount of network traffic, and with the increase in TPS and the reduction in the validity period of the active GTID list, it will further increase. Therefore, how to better encode and compress the active GTID list is an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a data encoding and decoding method and an electronic device for a distributed database.

[0006] In a first aspect, the present application provides a data encoding method for a distributed database, the method comprising:

[0007] Acquire stream data to be encoded, wherein the stream data includes multiple data IDs;

[0008] Acquire at least one coding interval corresponding to the stream data, where the coding interval is divided according to the distribution of the plurality of data IDs;

[0009] For each of the coding intervals, determining a plurality of coding IDs based on the coding interval;

[0010] According to the multiple coding IDs in each coding interval, bitmap coding is performed on the data IDs in each coding interval to obtain coded data.

[0011] Optionally, according to the multiple coding IDs of each coding interval, bitmap encoding is performed on the data IDs located in each coding interval respectively, including:

[0012] For each of the coding intervals, determining the distribution density of the data ID within the coding interval;

[0013] If the distribution density meets the preset condition, determine the data ID matching status of each coding ID in each coding interval;

[0014] According to the data ID matching situation, each data ID located in the encoding interval is bitmap encoded to obtain encoded data.

[0015] Optionally, bitmap encoding is performed on each of the data IDs in the coding interval according to the data ID matching situation to obtain coded data, including:

[0016] For each coding ID, determine whether there is a data ID matching the coding ID;

[0017] If the coding ID has a data ID that matches it, the coding value corresponding to the coding ID is set to the first coding value;

[0018] If the coding ID does not have a data ID that matches it, the coding value corresponding to the coding ID is set to a second coding value;

[0019] The coded data is determined based on the coded values ​​corresponding to the plurality of coded IDs.

[0020] Optionally, determining the coded data based on coded values ​​corresponding to a plurality of the coded IDs includes:

[0021] The encoding value corresponding to each encoding ID is stored in the encoding array as the encoding data;

[0022] The corresponding relationship between the data ID and the coding ID is stored in a preset mapping table.

[0023] Optionally, bitmap encoding is performed on each of the data IDs in the coding interval according to the data ID matching situation to obtain coded data, including:

[0024] If the distribution density does not meet the preset condition, obtaining the number of data IDs located in the coding interval;

[0025] If the number of IDs is less than a preset threshold, the data IDs within the coding interval are respectively determined as coding values ​​to obtain the coded data.

[0026] Optionally, bitmap encoding is performed on each of the data IDs in the coding interval according to the data ID matching situation to obtain coded data, including:

[0027] If the distribution density does not meet the preset condition, determining the maximum and minimum values ​​of the data IDs within the encoding interval;

[0028] If the maximum value and the minimum value are located in a local sub-interval of the coding interval, a new coding interval is determined using the maximum value as the maximum boundary value and the minimum value as the minimum boundary value;

[0029] Determine a plurality of coding IDs based on the new coding interval;

[0030] The data ID located in the coding interval is bitmap-coded according to the plurality of coding IDs to obtain coded data.

[0031] Optionally, a plurality of coding IDs are determined based on the coding interval, including:

[0032] Based on the coding interval, determining a minimum boundary value and a maximum boundary value of the coding interval;

[0033] A numerical value greater than or equal to the minimum boundary value and less than or equal to the maximum boundary value is determined as the coding ID.

[0034] In a second aspect, the present application provides a method for decoding data in a distributed database, the method comprising:

[0035] Acquire coded data to be decoded, where the coded data is obtained by encoding stream data, and the stream data includes multiple data IDs;

[0036] Extracting at least one coding interval and interval coding data corresponding to each coding interval based on the coding data, wherein the coding interval is divided according to the distribution of multiple data IDs in the stream data;

[0037] For each of the coding intervals, determining a plurality of coding IDs based on the coding interval;

[0038] The section coded data corresponding to the coding section is bitmap decoded according to the plurality of coding IDs to obtain the stream data.

[0039] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0040] Memory, used to store computer programs;

[0041] The processor is used to implement the data encoding method of the distributed database described in any one of the first aspects, or the data decoding method of the distributed database described in the second aspect when executing the program stored in the memory.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program of a data encoding method for a distributed database or a data decoding method for a distributed database is stored. When the program of the data encoding method for a distributed database is executed by a processor, the steps of the data encoding method for a distributed database described in any one of the first aspects are implemented; or, when the program of the data decoding method for a distributed database is executed by a processor, the steps of the data decoding method for a distributed database described in the second aspect are implemented.

[0043] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0044] The embodiment of the present application utilizes the coding ID of at least one coding interval corresponding to the stream data to perform bitmap encoding on the data ID located in each coding interval, thereby significantly reducing the space occupied by the stream data and providing a higher compression ratio, thereby further reducing the network traffic required for streaming data transmission in a distributed database and shortening the coding compression time, while also maintaining lossless characteristics and optimizing transmission efficiency.

[0045] Moreover, the embodiment of the present application can use multiple coding IDs to perform bitmap decoding on interval encoded data to obtain stream data, which greatly reduces the space occupied by the stream data and provides a higher compression ratio, thereby further reducing the network traffic required for streaming data transmission in a distributed database. At the same time, it can restore the encoded data to the original stream data, shorten the decoding and compression time, and at the same time maintain lossless characteristics and optimize transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1A flowchart of a data encoding method for a distributed database provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the distribution of stream data provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of another distribution of stream data provided in an embodiment of the present application;

[0051] Figure 4 An interactive flow chart in a distributed database in a practical application provided by an embodiment of the present application;

[0052] Figure 5 A flowchart of another distributed database data decoding method provided in an embodiment of the present application;

[0053] Figure 6 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] Since the network traffic consumed in querying the active list in actual applications will be very large, GTID active query occupies a large amount of network traffic, and with the increase of TPS and the reduction of the validity period of the active GTID list, it will further increase. Therefore, how to better encode and compress the active GTID list is an urgent problem to be solved. To this end, the embodiment of the present application provides a data encoding and decoding method and electronic device of a distributed database.

[0056] Figure 1 A data encoding method for a distributed database provided in an embodiment of the present application can be applied to:

[0057] Step S101, obtaining stream data to be encoded;

[0058] In an embodiment of the present application, the stream data includes multiple data IDs. In one embodiment of the present application, the stream data may refer to a data stream with time sequence, such as: an active GTID list in a distributed database, etc. The application and release process of the GTID is usually staged in time, that is, the active time of a single GTID is usually proportional to the response time of the transaction. Therefore, the lower limit and upper limit of the unsigned integer list used to represent the active GTID list are usually also within a certain numerical range, and gradually move in the direction of a larger value as time moves backward, but the range does not increase significantly. This temporal stage determines the range of the active GTID list and the high density within the range.

[0059] Step S102, obtaining at least one encoding interval corresponding to the stream data;

[0060] In an embodiment of the present application, the encoding interval is pre-divided according to the distribution of multiple data IDs. For example, for a distributed database, the peak TPS and the average response time of transactions can be understood first, and the data flow can be sampled for a period of time based on the old model. According to the existing sampling results, different interval lengths are used to draw a data distribution diagram, and a suitable interval length is selected according to the data distribution diagram.

[0061] Further, in one embodiment of the present application, if multiple data IDs are distributed relatively evenly, a relatively large encoding interval value may be selected, such as Figure 2 As shown, the data can be divided into sampling intervals such as 0-64, 64-128, 128-192..., and the density of the data ID is counted in each interval. It can be seen that the data density in different sampling intervals is not much different. Therefore, in this data distribution scenario, using a large coding interval to cover all sampling intervals can have a higher coding compression ratio, such as setting 512 as the coding interval length; in another embodiment of the present application, if multiple data IDs are unevenly distributed, such as a relatively concentrated distribution, if a large coding interval is used, the sampling interval with a data density of 0 will also be encoded, such as 0-128, 256-320. This part of the encoding increases the amount of calculation and space usage. In this data distribution scenario, a small coding interval value can be selected for data encoding. For example, if 64 is set as the coding interval value, only two sampling intervals of 128-192 and 192-256 need to be encoded. In the actual coding algorithm, the coding can also be optimized secondary according to some statistical characteristics of the data, such as constructing a coding interval according to the minimum and maximum values ​​of multiple data IDs in the coding interval, such as Figure 3 As shown, when data is concentrated in only some segments, the encoding interval can be determined based on the minimum and maximum values ​​of the concentrated data, for example: 128-256.

[0062] In this step, the encoding intervals pre-divided for the stream data may be obtained.

[0063] Step S103, for each of the coding intervals, determining a plurality of coding IDs based on the coding interval;

[0064] In an embodiment of the present application, since the coding interval itself is composed of a series of numerical values ​​from the minimum boundary value to the maximum boundary value, in one implementation of the present application, each numerical value can be used as a coding ID, that is: based on the coding interval, the minimum boundary value and the maximum boundary value of the coding interval are determined; and a numerical value greater than or equal to the minimum boundary value and less than or equal to the maximum boundary value is determined as the coding ID.

[0065] For example, if the coding interval is 0-5, the coding IDs are 0, 1, 2, 3, 4, 5. If the coding interval is 1000-1100, the coding IDs are 1000, 1001, 1002, 1003...1099, 1100 respectively.

[0066] Step S104: According to the multiple coding IDs in each coding interval, bitmap coding is performed on the data IDs in each coding interval to obtain coded data.

[0067] In this step, for each coding interval, multiple coding IDs of the coding interval may be used to perform bitmap coding on the data ID within the coding interval to obtain coded data.

[0068] The embodiment of the present application utilizes the coding ID of at least one coding interval corresponding to the stream data to perform bitmap encoding on the data ID located in each coding interval, thereby significantly reducing the space occupied by the stream data and providing a higher compression ratio, thereby further reducing the network traffic required for streaming data transmission in a distributed database and shortening the coding compression time, while also maintaining lossless characteristics and optimizing transmission efficiency.

[0069] In another embodiment of the present application, step S104 performs bitmap encoding on the data IDs located in each coding interval according to the multiple coding IDs in each coding interval, including:

[0070] Step S201, for each coding interval, determining the distribution density of the data ID in the coding interval;

[0071] In one embodiment of the present application, the distribution density can be obtained by using the number of data IDs in the coding interval / the length of the coding interval. For example, if the coding interval includes 10 data IDs and the length of the coding interval is 512 bits, the distribution density is 10 / 512=0.0195.

[0072] In another embodiment of the present application, the distribution density can also be obtained by using the number of data IDs in the coding interval / the total number of coding IDs in the coding interval. For example, if the coding interval includes 10 data IDs and the total number of coding IDs in the coding interval is 100, then the distribution density is 10 / 100=0.1.

[0073] Step S202, if the distribution density meets the preset condition, determine the data ID matching status of each coding ID in each coding interval;

[0074] In an embodiment of the present application, the distribution density satisfying the preset conditions may refer to the distribution density being greater than a preset density threshold. If the distribution density is larger and closer to 1 or even equal to 1, it indicates that the distribution of data IDs in the coding interval is very dense. At this time, the coding method provided in the present application is used for encoding, and the coding efficiency is higher.

[0075] Step S203, performing bitmap encoding on each of the data IDs in the encoding interval according to the data ID matching situation to obtain encoded data.

[0076] In one implementation of the present application, step S203 performs bitmap encoding on each of the data IDs in the encoding interval according to the data ID matching situation to obtain encoded data, including:

[0077] Step S2031, for each coding ID, determining whether there is a data ID matching the coding ID;

[0078] In the embodiment of the present application, the data ID that matches the coding ID refers to the data ID that is the same as the coding ID, which is equivalent to using the coding ID as a reference to determine whether there is a data ID that is the same as the coding ID.

[0079] Step S2032: if the coding ID has a data ID that matches it, set the coding value corresponding to the coding ID to a first coding value; illustratively, the first coding value is 1.

[0080] Step S2033: if the coding ID does not have a data ID that matches it, the coding value corresponding to the coding ID is set to a second coding value; illustratively, the second coding value is 0.

[0081] Step S2034: determine the coded data based on the coded values ​​corresponding to the multiple coded IDs.

[0082] For example, the coding interval is 1-10, and the coding IDs of the coding interval include: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10.

[0083] A set of data IDs includes: 1, 3, 5, 6, 7, 10;

[0084] Through steps S2031-S2033, the coded data obtained by bitmap coding is: 10101110 01 (binary), and the starting value of the coding interval is 1.

[0085] In one embodiment of the present application, step S2034 determines the coded data based on the coded values ​​corresponding to the multiple coded IDs, including: storing the coded value corresponding to each coded ID in a coded array as the coded data; storing the data ID in a preset mapping table.

[0086] In the embodiment of the present application, the encoded data can be stored in an array, and for subsequent use, the data ID can also be stored in a preset mapping table.

[0087] In another embodiment of the present application, step S203 performs bitmap encoding on each data ID in the encoding interval according to the data ID matching situation to obtain encoded data, including:

[0088] Step S301, if the distribution density does not meet the preset conditions, obtain the number of IDs of the data IDs located in the coding interval; for the distribution density that does not meet the preset conditions and the distribution density is less than the preset density threshold, the use of bitmap encoding will be inefficient and may occupy more space. Therefore, at this time, the number of IDs of the data IDs in the coding interval can be obtained.

[0089] Step S302: If the number of IDs is less than a preset threshold, the data IDs in the coding interval are respectively determined as coding values ​​to obtain the coded data.

[0090] For example, if the number of IDs is 1, which is less than the preset threshold, the data ID can be directly used as the encoding value. When the number of data IDs in the encoding interval is small, the data ID can be directly used as the encoding value, which takes up less space and avoids the situation where the bitmap takes up a lot of space.

[0091] In another embodiment of the present application, step S203 performs bitmap encoding on each data ID in the encoding interval according to the data ID matching situation to obtain encoded data, including:

[0092] Step S401, if the distribution density does not meet the preset conditions, determine the maximum and minimum values ​​of the data ID in the coding interval; for the distribution density that does not meet the preset conditions and the distribution density is less than the preset density threshold, the use of bitmap encoding will be inefficient and may occupy more space. Therefore, at this time, it can be determined whether the data ID in the coding interval is concentrated in a certain segment.

[0093] Step S402: If the maximum value and the minimum value are located in a local sub-interval of the coding interval, a new coding interval is determined using the maximum value as the maximum boundary value and the minimum value as the minimum boundary value; that is, when the data IDs in the coding interval are concentrated in a certain segment, a new coding interval can be constructed based on the segment, that is, the local sub-interval.

[0094] Step S403, determining a plurality of coding IDs based on the new coding interval;

[0095] Step S404: performing bitmap encoding on the data ID within the encoding interval according to the plurality of encoding IDs to obtain encoded data.

[0096] Exemplarily, if the maximum value or the minimum value is located around the middle of the interval, it means that the data is located in a part of the interval, and the minimum value and the maximum value can be used as the upper and lower limits of the interval for encoding.

[0097] For ease of understanding, the present application also provides an embodiment in practical application, as follows:

[0098] The following are the applicable scenarios of this patent in distributed databases, such as Figure 4 As shown, the computing node obtains the current data from the data node, but there may be some dirty data, so it is necessary to obtain the current active transaction ID list from the global transaction node. The global transaction node uses this patent to encode the active transaction ID list and then transmits it to the computing node through the network. The computing node performs bitmap decoding to obtain the active transaction list and performs subsequent processing.

[0099] Example of active transaction list: 10000 10001 10003 10004 10005 10007 10008 10010 10011 10015 20001 20003 20005 20007 20009 20010 20011 20012 20013 20014

[0102] Total 20*8=160 bytes

[0103] Using segmented bitmap coding

[0104] The first segment starts at 10000

[0105] Encoded binary representation: 11011101 10110001

[0106] The second segment starts at 20001

[0107] Encoded binary representation: 10101010 11111100

[0108] A total of 8 (bytes occupied by the first segment start value) + 2 (+ 8 (bytes occupied by the second segment start value) + 2 bytes occupied by the second segment binary code) = 20 bytes

[0109] The length of the encoded file is 1 / 8 of the original one. The above active list is only used to illustrate the patent principle. The active list in actual use scenarios is often longer, more complex, and more effective.

[0110] For ease of understanding, the present application also provides an embodiment in practical application, as follows:

[0111] After obtaining the interval length (here it is assumed to be 512), on the data producer, i.e. the global transaction node, each time a piece of data is produced, the quotient of the interval length is taken. For example, the quotient of the interval length 512 for data 737 is 1, so it can be obtained that data 737 should be stored in the interval [floor(737 / 512)*512, (floor(737 / 512)+1)*512), i.e. [512,1024). If the data added at this time is the first data in the interval, the interval is generated. A map can be used to record the actual data added to the interval, and a 512 / 8=64-byte array can be used to record the compressed data stream of the interval encoding, and the corresponding bit is set to 1.

[0112] For different data intervals existing on the producer, the length of each interval after encoding is: 8 (interval starting value occupies bytes) + 512 (interval length) / 8 = 72 bytes. For a 64-bit operating system, it is equivalent to 72 / 8 = 9 unsigned long integers, that is, the number of data in a single interval. The size of the set is greater than 9. Assuming the preset density threshold is 8 / 512, 9 / 512>8 / 512. Therefore, the encoding efficiency will be better than direct transmission. The higher the amount of data, the higher the encoding compression efficiency.

[0113] For a single interval with a data quantity less than or equal to 8, the preset density threshold is 8 / 512 = the preset density threshold is 8 / 512. At this time, encoding optimization can be performed based on some statistical data of the interval, such as the number. If the number is 1, the value can be directly recorded and interpreted as an interval with a length of 1. If the maximum or minimum value is located around the middle of the interval, it means that the data is located in a local area of ​​the interval, and the minimum and maximum values ​​can be used as the upper and lower limits of the interval for encoding.

[0114] The above is the basic logic of data distribution and encoding by data producers, which is roughly to exchange time for space and spread the time evenly over non-business time. When the business arrives and needs to obtain the time series data stream, only a small amount of processing is required to send the encoded and compressed data.

[0115] The encoding compression ratio of the active GTID list in the embodiment of the present application is close to 20 times. This method will greatly reduce the network traffic between CN and GTM and improve transmission efficiency. At the same time, the bitmap encoding method is also extremely convenient for CN nodes to determine whether a single GTID is active, and no additional decoding time is required.

[0116] The segmented bitmap coding compression method proposed this time provides a higher compression ratio and faster compression and decompression time while maintaining lossless characteristics.

[0117] In another embodiment of the present application, a data decoding method for a distributed database is also provided. Figure 5 As shown, the method includes:

[0118] Step S501, obtaining coded data to be decoded, wherein the coded data is obtained by encoding stream data, and the stream data includes multiple data IDs;

[0119] Step S502, extracting at least one coding interval and interval coding data corresponding to each coding interval based on the coding data, wherein the coding interval is divided according to the distribution of multiple data IDs in the stream data;

[0120] Step S503, for each of the coding intervals, determining a plurality of coding IDs based on the coding interval;

[0121] Step S504: performing bitmap decoding on the interval coded data corresponding to the coding interval according to the plurality of coding IDs to obtain the stream data.

[0122] In practical applications, based on the embodiments in the aforementioned practical applications, when a piece of data is consumed, the corresponding data can be removed from the set record and the corresponding position in the encoded array can be set to 0. If the set is empty after the data is removed, the set corresponding to the interval and the encoded array need to be removed.

[0123] The embodiment of the present application is a data decoding method that uniquely corresponds to the data encoding method of the aforementioned embodiment. It uses multiple encoding IDs to perform bitmap decoding on interval encoded data to obtain stream data, which greatly reduces the space occupied by the stream data and provides a higher compression ratio, thereby further reducing the network traffic required for streaming data transmission in a distributed database. At the same time, it can restore the encoded data to the original stream data, shorten the decoding and compression time, and at the same time maintain lossless characteristics and optimize transmission efficiency.

[0124] In another embodiment of the present application, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0125] Memory, used to store computer programs;

[0126] The processor is used to implement the data encoding method of the distributed database described in any of the above embodiments, or the data decoding method of the distributed database described in any of the above embodiments when executing the program stored in the memory.

[0127] In the electronic device provided by the embodiment of the present invention, the processor implements bitmap encoding of the data ID located in each coding interval by executing the program stored in the memory by using the coding ID of at least one coding interval corresponding to the stream data, thereby greatly reducing the space occupied by the stream data and providing a higher compression ratio, thereby further reducing the network traffic required for the transmission of stream data in the distributed database, shortening the coding compression time, and maintaining the lossless characteristics and optimizing the transmission efficiency. Moreover, the interval coded data can be bitmap decoded using multiple coding IDs to obtain the stream data, thereby greatly reducing the space occupied by the stream data and providing a higher compression ratio, thereby further reducing the network traffic required for the transmission of stream data in the distributed database, while being able to restore the coded data to the original stream data, shortening the decoding compression time, and maintaining the lossless characteristics and optimizing the transmission efficiency.

[0128] The communication bus 1140 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0129] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0130] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0131] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0132] In another embodiment of the present application, a computer-readable storage medium is further provided, on which a program of a data encoding method for a distributed database or a data decoding method for a distributed database is stored. When the program of the data encoding method for a distributed database is executed by a processor, the steps of the data encoding method for a distributed database described in any of the foregoing embodiments are implemented; or, when the program of the data decoding method for a distributed database is executed by a processor, the steps of the data decoding method for a distributed database described in any of the foregoing embodiments are implemented.

[0133] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0134] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data encoding method for a distributed database, characterized in that: The method comprises: Acquire stream data to be encoded, wherein the stream data includes multiple data IDs; Acquire at least one coding interval corresponding to the stream data, where the coding interval is divided according to the distribution of the plurality of data IDs; For each of the coding intervals, determining a plurality of coding IDs based on the coding interval; According to the multiple coding IDs of each coding interval, bitmap coding is performed on the data IDs located in each coding interval to obtain coded data; According to the plurality of coding IDs in each coding interval, bitmap coding is performed on the data IDs in each coding interval respectively, including: For each of the coding intervals, determining the distribution density of the data ID within the coding interval; If the distribution density meets the preset condition, determine the data ID matching status of each coding ID in each coding interval; According to the data ID matching situation, each data ID located in the coding interval is bitmap encoded to obtain encoded data; If the distribution density does not meet the preset condition, obtaining the number of data IDs located in the coding interval; If the number of IDs is less than a preset threshold, the data IDs within the coding interval are respectively determined as coding values ​​to obtain the coded data.

2. The method according to claim 1, characterized in that According to the data ID matching situation, each data ID located in the coding interval is bitmap encoded to obtain encoded data, including: For each coding ID, determine whether there is a data ID matching the coding ID; If the coding ID has a data ID that matches it, the coding value corresponding to the coding ID is set to the first coding value; If the coding ID does not have a data ID that matches it, the coding value corresponding to the coding ID is set to a second coding value; The coded data is determined based on the coded values ​​corresponding to the plurality of coded IDs.

3. The method according to claim 2, characterized in that Determining the coded data based on the coded values ​​corresponding to the plurality of coded IDs comprises: The encoding value corresponding to each encoding ID is stored in the encoding array as the encoding data; The data ID is stored in a preset mapping table.

4. The method according to claim 1, characterized in that: According to the data ID matching situation, each data ID located in the coding interval is bitmap encoded to obtain encoded data, including: If the distribution density does not meet the preset condition, determining the maximum and minimum values ​​of the data IDs within the encoding interval; If the maximum value and the minimum value are located in a local sub-interval of the coding interval, a new coding interval is determined using the maximum value as the maximum boundary value and the minimum value as the minimum boundary value; Determine a plurality of coding IDs based on the new coding interval; The data ID located in the coding interval is bitmap-coded according to the plurality of coding IDs to obtain coded data.

5. The method according to claim 1, characterized in that: Determining a plurality of coding IDs based on the coding interval includes: Based on the coding interval, determining a minimum boundary value and a maximum boundary value of the coding interval; A numerical value greater than or equal to the minimum boundary value and less than or equal to the maximum boundary value is determined as the coding ID.

6. A data decoding method for a distributed database, characterized in that: The method comprises: Acquire coded data to be decoded, wherein the coded data is obtained by encoding stream data according to the data encoding method for a distributed database according to claim 1, and the stream data includes multiple data IDs; Extracting at least one coding interval and interval coding data corresponding to each coding interval based on the coding data, wherein the coding interval is divided according to the distribution of multiple data IDs in the stream data; For each of the coding intervals, determining a plurality of coding IDs based on the coding interval; The section coded data corresponding to the coding section is bitmap decoded according to the plurality of coding IDs to obtain the stream data.

7. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the data encoding method of the distributed database described in any one of claims 1 to 5, or the data decoding method of the distributed database described in claim 6 when executing the program stored in the memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program of a data encoding method for a distributed database or a data decoding method for a distributed database. When the program of the data encoding method for a distributed database is executed by a processor, the steps of the data encoding method for a distributed database described in any one of claims 1 to 5 are implemented; or, when the program of the data decoding method for a distributed database is executed by a processor, the steps of the data decoding method for a distributed database described in claim 6 are implemented.