Data processing method and device, electronic equipment, storage medium and program product
By determining the persistence strategy based on the key type of key-value pairs, the problem of high storage costs in distributed storage clusters is solved, achieving both flexibility and cost optimization in data storage.
Patent Information
- Application Number
- CN202511057761.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-11-11
AI Technical Summary
In existing technologies, using the same persistence strategy for different types of data through distributed storage clusters results in high storage costs.
Based on the key type of the key-value pair data, the corresponding persistence strategy is determined, and the key-value pair data is written to the storage cluster and disk through a preset mapping relationship. Different persistence strategies are adopted to reduce storage costs.
By differentiating different types of data and adopting different persistence strategies, the storage cost of data is effectively reduced.
Smart Images
Figure CN120929017A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and specifically to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology
[0002] In the field of communication technology, distributed storage clusters (such as Redis clusters) are used for efficient storage and persistence of key-value pairs. In related technologies, storage clusters can employ Append Only File (AOF) methods to achieve data persistence. However, AOF persistence in a storage cluster uses a globally uniform configuration, applying the same persistence strategy to different types of data. Since different types of data may have varying importance, business requirements, and access patterns, using the same persistence strategy leads to high data storage costs.
[0003] It is evident that the relevant technologies suffer from high data storage costs. Summary of the Invention
[0004] This invention provides a data processing method, apparatus, electronic device, storage medium, and program product to address the problem of high data storage costs in related technologies.
[0005] To solve the above problems, the present invention is implemented as follows:
[0006] In a first aspect, embodiments of the present invention provide a data processing method, including:
[0007] Retrieve the key-value pair data to be processed;
[0008] Determine the key type corresponding to the key in the key-value pair data;
[0009] The first persistence strategy corresponding to the key type is determined based on a preset mapping relationship. The preset mapping relationship is used to determine the persistence strategy corresponding to different types. The first persistence strategy is used to determine the way the key-value pair data is written to the disk.
[0010] The key-value pair data is written to the storage cluster and disk based on the first persistence strategy.
[0011] Secondly, embodiments of the present invention also provide a data processing apparatus, comprising:
[0012] The acquisition module is used to acquire key-value pair data to be processed;
[0013] The first determining module is used to determine the key type corresponding to the key in the key-value pair data;
[0014] The second determining module is used to determine the first persistence strategy corresponding to the key type based on a preset mapping relationship. The preset mapping relationship is used to determine the persistence strategy corresponding to different types. The first persistence strategy is used to determine the way the key-value pair data is written to the disk.
[0015] The write module is used to write the key-value pair data to the storage cluster and disk based on the first persistence strategy.
[0016] Thirdly, embodiments of the present invention also provide an electronic device, including a transceiver and a processor.
[0017] The transceiver is used to acquire key-value pair data to be processed;
[0018] The processor is used to determine the key type corresponding to the key in the key-value pair data;
[0019] The processor is further configured to determine a first persistence strategy corresponding to the key type based on a preset mapping relationship, wherein the preset mapping relationship is used to determine the persistence strategy corresponding to different types, and the first persistence strategy is used to determine the way the key-value pair data is written to the disk;
[0020] The processor is further configured to write the key-value pair data to the storage cluster and disk based on the first persistence strategy.
[0021] Fourthly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data processing method described in the first aspect.
[0022] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described in the first aspect.
[0023] In a sixth aspect, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the data processing method described in the first aspect.
[0024] In this embodiment of the invention, the data processing method includes: acquiring key-value pair data to be processed; determining the key type corresponding to the key in the key-value pair data; determining a first persistence strategy corresponding to the key type based on a preset mapping relationship, wherein the preset mapping relationship is used to determine the persistence strategy corresponding to different types, and the first persistence strategy is used to determine the method of writing the key-value pair data to disk; and writing the key-value pair data to a storage cluster and a disk based on the first persistence strategy. In this way, by determining the first persistence strategy through the key type in the key-value pair data, and then writing the key-value pair data to the storage cluster and disk based on the first persistence strategy, different persistence strategies are executed for different key-value pair data, thereby avoiding the problem of high storage costs caused by using the same persistence strategy to store data, and effectively reducing the data storage cost. Attached Figure Description
[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 This is a flowchart of a data processing method provided in an embodiment of the present invention;
[0027] Figure 2 This is a structural diagram of a data processing device provided in an embodiment of the present invention;
[0028] Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0030] This invention provides a data processing method, apparatus, electronic device, storage medium, and program product, which solves the problem of high data storage costs in related technologies by applying different persistence strategies to data with different key values.
[0031] Please see Figure 1 , Figure 1 This is a flowchart of a data processing method provided in an embodiment of the present invention, such as... Figure 1As shown, it includes the following steps:
[0032] Step 101: Obtain the key-value pair data to be processed.
[0033] The key-value pair data to be processed is the key-value pair data to be written to the storage cluster. It should be noted that before writing the key-value pair data to the storage cluster, the persistence strategy corresponding to the key-value pair data needs to be determined first. Then, the key-value pair data is written and synchronized according to the corresponding persistence strategy, thereby achieving the persistence of the key-value pair data.
[0034] In some implementations, the key-value pair data is pre-stored in other temporary storage devices before being written to the storage cluster. When it is necessary to write the key-value pair data to the storage cluster, the key-value pair data is retrieved sequentially from the other temporary storage devices. After determining the persistence policy corresponding to the key-value pair data, the key-value pair data is written to the storage cluster, and the key-value pair data in the other temporary storage devices is deleted.
[0035] Step 102: Determine the key type corresponding to the key in the key-value pair data.
[0036] The key-value pair data mentioned above includes keys and values. For key-value pair data that need to be persisted with the same persistence strategy, the keys they include are the same or belong to the same type. By determining the key type corresponding to the key in the key-value pair, the persistence strategy for writing the key-value pair data can be determined by the key type.
[0037] In some implementations, a mapping table of different key types and different persistence strategies can be pre-configured. The mapping table includes a one-to-one mapping relationship between key types and persistence strategies. The mapping table can be used to quickly determine the key type corresponding to the key in key-value pair data.
[0038] Step 103: Determine the first persistence strategy corresponding to the key type based on the preset mapping relationship. The preset mapping relationship is used to determine the persistence strategy corresponding to different types. The first persistence strategy is used to determine the way the key-value pair data is written to the disk.
[0039] The first persistence strategy described above is a pre-configured strategy for determining how key-value pair data is written to disk. It should be noted that different persistence strategies correspond to different disk writing methods. In some implementations, the disk writing method can be no disk writing, writing to disk simultaneously with writing to the storage cluster, or periodically writing key-value pair data to the storage disk. By using different persistence strategies to persist key-value pair data in different ways, the flexibility of data persistence is improved, thereby avoiding the high storage costs caused by using the same persistence strategy to store data.
[0040] The above-mentioned preset mapping relationship is the mapping relationship between key types and persistence strategies. Specifically, the key type and persistence strategy are in a one-to-one correspondence; that is, the same key type corresponds to the same persistence strategy, and different key types correspond to different persistence strategies.
[0041] In some implementations, key types can be further categorized, in which case multiple key types may correspond to a single persistence strategy. However, each key type is assigned a specific persistence strategy so that the server device can persist data according to that strategy without needing to select from different persistence strategies.
[0042] Step 104: Write the key-value pair data to the storage cluster and disk based on the first persistence strategy.
[0043] In this embodiment of the invention, the data processing method includes: acquiring key-value pair data to be processed; determining the key type corresponding to the key in the key-value pair data; determining a first persistence strategy corresponding to the key type based on a preset mapping relationship, wherein the preset mapping relationship is used to determine the persistence strategy corresponding to different types, and the first persistence strategy is used to determine the method of writing the key-value pair data to disk; and writing the key-value pair data to a storage cluster and a disk based on the first persistence strategy. In this way, by determining the first persistence strategy through the key type in the key-value pair data, and then writing the key-value pair data to the storage cluster and disk based on the first persistence strategy, different persistence strategies are executed for different key-value pair data, thereby avoiding the problem of high storage costs caused by using the same persistence strategy to store data, and effectively reducing the data storage cost.
[0044] In one embodiment, the preset mapping relationship includes at least one of the following:
[0045] When the key type is the first type, the first persistence strategy is to synchronize the data to the disk when writing data;
[0046] When the key type is the second type, the first persistence strategy is to synchronize the data to the disk based on a preset time interval;
[0047] When the key type is the third type, the first persistence strategy is not to synchronize the written data.
[0048] The first type mentioned above is used to characterize key-value pair data with high data consistency requirements, such as business-related data or product order-related data. This type of key-value pair data needs to be stored through synchronous persistence to avoid potential data loss.
[0049] The second type mentioned above is used to characterize data with general data consistency requirements, such as data from distributed locks. This type of key-value pair data does not need to be stored synchronously; it only needs to be stored asynchronously to ensure eventual data consistency.
[0050] The third type mentioned above is used to represent data that does not need to maintain data consistency. This type of data has low importance, such as preference data or payment data. Even if it is lost, it will not have an impact. In this case, it is not necessary to synchronize the key-value pairs of this type of data.
[0051] For example, the preset mapping relationship can be represented by the following formula:
[0052] order_*=Always
[0053] Key_*=Everysec
[0054] Enjoy_*=No
[0055] In the formula, order_* represents the first type, and Always means that data is synchronized to disk when writing data (i.e., synchronous persistence); Key_* represents the second type, and Everysec means that data is synchronized to disk based on a preset time interval (i.e., asynchronous persistence); Enjoy_* represents the third type, and No means that the written data is not synchronized.
[0056] In this embodiment of the invention, when the key type is a first type, the first persistence strategy is to synchronize the data to the disk when writing data; when the key type is a second type, the first persistence strategy is to synchronize the data to the disk based on a preset time interval; and when the key type is a third type, the first persistence strategy is not to synchronize the written data. Thus, a preset mapping relationship is used to determine the persistence strategy corresponding to different key types.
[0057] In one embodiment, prior to obtaining the key-value pair data to be processed, the method further includes:
[0058] Create a configuration file, which includes the preset mapping relationship and other configuration parameters, wherein the other configuration parameters are parameters for obtaining the key-value pair data or parameters for executing the program;
[0059] Based on the configuration file, an Extend Berkeley PacketFilter (EBPF) program is created. The EBPF program is used to execute the steps of obtaining the key-value pair data to be processed and determining the first persistence strategy corresponding to the key type based on a preset mapping relationship.
[0060] The EBPF program is bound to the write function of the storage cluster, which is used to write the key-value pair data to the storage cluster.
[0061] The configuration file described above is used to configure the strategy for persisting key-value pair data. It includes preset mapping relationships and other configuration parameters. The preset mapping relationships determine the persistence strategy, while the other configuration parameters implement steps such as retrieving key-value pair data.
[0062] In some implementations, the configuration file is a lightweight data serialization language (e.g., a YAML file), which may include compilation parameters, loading parameters, additional parameters, and mapping parameters (i.e., parameters with predefined mapping relationships). Compilation parameters may be the header file of the EBPF program, the identifier of the compiler used, etc.; loading parameters may indicate the type of functions in the EBPF program, such as XDP, TC, kprobe, etc.; additional parameters are used to indicate hook points attached to the EBPF program, through which key-value pair data is obtained.
[0063] The aforementioned EBPF program is used to retrieve key-value pair data and determine the persistence strategy. It should be noted that storage clusters (such as Redis clusters) cannot directly implement differentiated processing of key-value pair data through different persistence strategies; an EBPF program is required. EBPF technology is a high-efficiency virtual machine running in the kernel, which can be used for network packet filtering, performance monitoring, and security auditing. EBPF programs can run without modifying kernel code and have minimal impact on system performance. In this embodiment of the invention, by deploying an EBPF program on the Redis server, data packets are filtered and processed. Based on different preset data scenarios (i.e., different key types), the AOF persistence strategy is dynamically adjusted to achieve different persistence strategies for different key types.
[0064] It should be noted that the purpose of the EBPF program is to capture AOF events of the storage cluster and execute predefined custom logic (i.e., execute different persistence strategies based on key types) when these events occur. Specifically, the EBPF program is created through a configuration file and then bound to the AOF write function of the storage cluster. During the process of writing key-value pair data to the storage cluster through the write function, the EBPF program can be called to determine the first persistence strategy, thereby enabling differentiated writing of different key-value pair data to disk.
[0065] In this embodiment of the invention, a configuration file is created, which includes the preset mapping relationship and other configuration parameters. These other configuration parameters are parameters for obtaining the key-value pair data or parameters for executing a program. An Extended Berkeley Packet Filter (EBPF) program is created based on the configuration file. The EBPF program is used to execute the steps of obtaining the key-value pair data to be processed and determining the first persistence strategy corresponding to the key type based on the preset mapping relationship. The EBPF program is bound to the write function of the storage cluster, which is used to write the key-value pair data to the storage cluster. Thus, by creating the EBPF program through the configuration file and binding it to the write function of the storage cluster, the EBPF program can be called to determine the first persistence strategy during the process of writing key-value pair data to the storage cluster via the write function, thereby achieving differentiated writing of different key-value pair data to disk.
[0066] It should be noted that after creating the configuration file, a compilation tool is needed to convert the configuration file into an EBPF program. In some implementations, the toolset in Eunomia can be used to compile the configuration file into bytecode, thereby obtaining the EBPF program.
[0067] Eunomia simplifies the management of EBPF programs. It's an open-source EBPF dynamic loading runtime and development toolchain. YAML files define the configuration and behavior of EBPF programs, describing how they should be compiled, loaded, and attached to specific kernel hooks. Compilation is then performed using compilers within Eunomia (such as clang and LLVM). It's important to note that while Eunomia provides a high-level abstraction for compiling and loading EBPF programs, it still relies on underlying tools and libraries, such as clang, bpftool, or libbpf, to perform the actual compilation and loading operations. Therefore, the necessary environment and tools for compilation are essential.
[0068] The eunomia tool requires defining hook functions (kprobes) and hash mapping functions. Hook functions retrieve key-value pairs to be processed, while hash mapping functions determine the persistence strategy. Specifically, hook functions are used to hook to the AOF write function of the storage cluster, allowing user-space programs to insert breakpoints when kernel functions are called to monitor or modify kernel behavior; hash mapping functions determine the persistence strategy based on the key type.
[0069] In addition, before defining the hook function and hash mapping function, you need to declare the header file in the configuration file, and define the hook function and hash mapping function through the header file.
[0070] In one embodiment, the step of creating the Extended Berkeley Packet Filter (EBPF) procedure based on the configuration file includes:
[0071] Configure the hash mapping function in the EBPF program based on the preset mapping relationship;
[0072] The step of determining the first persistence strategy corresponding to the key type based on a preset mapping relationship includes:
[0073] Based on the hash mapping function in the EBPF program, the first persistence strategy corresponding to the key type is determined.
[0074] The hash mapping function described above is used to store the key type and its corresponding persistence strategy. The hash mapping function determines the first persistence strategy corresponding to the key type. Specifically, the hash mapping function is a data structure in an EBPF program; the mapped key is of type char, and the mapped value is of type int. The hash mapping function enables data sharing between EBPF programs and user-space programs.
[0075] In this embodiment of the invention, a hash mapping function in the EBPF program is configured based on the preset mapping relationship; and a first persistence strategy corresponding to the key type is determined based on the hash mapping function in the EBPF program. Thus, by configuring the hash mapping function through the preset mapping relationship, the first persistence strategy is determined through the hash mapping function.
[0076] In one embodiment, the other configuration parameters include hook parameters, and the process of creating the Extended Berkeley Packet Filter (EBPF) procedure based on the configuration file includes:
[0077] The hook function in the EBPF program is configured based on the hook parameters. The hook function is used to obtain key-value pair data from the structure. The structure is used to store the key-value pair data to be processed.
[0078] The process of obtaining the key-value pair data to be processed includes:
[0079] The key-value pair data to be processed is obtained based on the hook function in the EBPF program.
[0080] The hook function described above is used to obtain key-value pair data to be processed. The hook function is called when the storage cluster performs AOF write operations to obtain key-value pair data and determine the specific first persistence strategy.
[0081] It should be noted that the key-value pair data to be processed is stored in a structure. Before writing the key-value pair data to the storage cluster, it is necessary to retrieve the key-value pair data from the structure. In this embodiment of the invention, a hook function is configured by first declaring a structure (struct pt_regs*ctx). When the hook function is woken up, it receives a context pointer (ctx) sent by the structure, allowing the hook function to retrieve the key-value pair data from the structure through the context pointer. The context pointer includes registers such as the program counter, stack pointer, and return parameters, enabling the hook function to determine the register state of the hook point based on the context pointer, and thus retrieve the key-value pair data to be processed.
[0082] Specifically, the hook function uses two macro functions (e.g., PT_REGS_PARM1(ctx) and PT_REGS_PARM2(ctx)) to retrieve the first and second parameters from the structure, which are the key and value of the key-value pair, respectively. When retrieving the key-value pair data, an auxiliary function (e.g., key_start_with) is used to determine if it starts with a specific string, thus determining the key type. For example, if the key starts with "order", the key type is determined to be type 1. In this case, the function in the EBPF program used to update or insert map elements (e.g., bpf_map_update_elem) is called to set the execution policy corresponding to the key-value pair data in the hash mapping function to always (i.e., synchronous persistence). As another example, if the key starts with "cookie", the key type is determined to be type 3. In this case, bpf_map_update_elem in the EBPF program is called to set the execution policy corresponding to the key-value pair data in the hash mapping function to no (i.e., no persistence).
[0083] In this embodiment of the invention, a hook function in the EBPF program is configured based on the hook parameters. The hook function is used to retrieve key-value pair data from a structure, which stores the key-value pair data to be processed. The hook function in the EBPF program retrieves the key-value pair data to be processed. Thus, by configuring the hook function to retrieve the key-value pair data to be processed, a persistence strategy can be determined through a hash mapping function.
[0084] In one embodiment, after creating the configuration file, the method further includes:
[0085] Create a userspace program, which is used to store the configuration file.
[0086] The aforementioned user-space program stores configuration files, which in turn enables the implementation of a preset mapping relationship (i.e., hash mapping function) between management key types and persistence strategies. This allows the key type to be determined through the key of the key-value pair data, thereby determining the persistence strategy.
[0087] The user-space program can load the bytecode of the compiled EBPF program into the kernel, and then bind the loaded EBPF program with the AOF write function, so that the persistence strategy can be determined and executed before writing key-value pair data to the storage cluster.
[0088] In this embodiment of the invention, by creating a user space program to store configuration files, a preset mapping relationship between management key types and persistence strategies can be achieved.
[0089] In one embodiment, the method further includes:
[0090] Upon receiving update information, the configuration file is updated based on the update information, which includes at least one of the updated preset mapping relationship and the updated other configuration parameters;
[0091] It should be noted that since the user-space program stores configuration files, the EBPF program can be adjusted or updated based on these configuration files. Specifically, upon receiving update information, the configuration file is updated based on the update information, and the EBPF program is adjusted based on the updated configuration file. This allows the adjusted EBPF program to determine its persistence strategy based on new preset mapping relationships or other configuration parameters.
[0092] In this embodiment of the invention, upon receiving update information, the configuration file is updated based on the update information, which includes at least one of the updated preset mapping relationship and the updated other configuration parameters; the EBPF program is then adjusted based on the updated configuration file. Thus, adjusting the EBPF program is achieved by updating the configuration file.
[0093] Please see Figure 2 , Figure 2 This is a structural diagram of a data processing device provided in an embodiment of the present invention, such as... Figure 2 As shown, the data processing device 200 includes:
[0094] The acquisition module 201 is used to acquire key-value pair data to be processed;
[0095] The first determining module 202 is used to determine the key type corresponding to the key in the key-value pair data;
[0096] The second determining module 203 is used to determine the first persistence strategy corresponding to the key type based on a preset mapping relationship. The preset mapping relationship is used to determine the persistence strategy corresponding to different types. The first persistence strategy is used to determine the way the key-value pair data is written to the disk.
[0097] The writing module 204 is used to write the key-value pair data to the storage cluster and disk based on the first persistence strategy.
[0098] In one embodiment, the preset mapping relationship includes at least one of the following:
[0099] When the key type is the first type, the first persistence strategy is to synchronize the data to the disk when writing data;
[0100] When the key type is the second type, the first persistence strategy is to synchronize the data to the disk based on a preset time interval;
[0101] When the key type is the third type, the first persistence strategy is not to synchronize the written data.
[0102] In one embodiment, the data processing apparatus 200 further includes:
[0103] The first creation module is used to create a configuration file, which includes the preset mapping relationship and other configuration parameters, wherein the other configuration parameters are parameters for obtaining the key-value pair data or parameters for executing the program;
[0104] The second creation module is used to create an Extended Berkeley Packet Filter (EBPF) program based on the configuration file. The EBPF program is used to execute the steps of obtaining key-value pair data to be processed and determining the first persistence strategy corresponding to the key type based on a preset mapping relationship.
[0105] A binding module is used to bind the EBPF program to the write function of the storage cluster, wherein the write function is used to write the key-value pair data to the storage cluster.
[0106] In one embodiment, the second creation module includes:
[0107] The first configuration unit is used to configure the hash mapping function in the EBPF program based on the preset mapping relationship;
[0108] The second determining module 203 includes:
[0109] The determining unit is used to determine the first persistence strategy corresponding to the key type based on the hash mapping function in the EBPF program.
[0110] In one embodiment, the other configuration parameters include hook parameters, and the second creation module includes:
[0111] The second configuration unit is used to configure the hook function in the EBPF program based on the hook parameters. The hook function is used to obtain key-value pair data from the structure, and the structure is used to store the key-value pair data to be processed.
[0112] The acquisition module 201 includes:
[0113] The acquisition unit is used to acquire the key-value pair data to be processed based on the hook function in the EBPF program.
[0114] In one embodiment, the data processing apparatus 200 further includes:
[0115] The third creation module is used to create a user space program, which is used to store the configuration file.
[0116] In one embodiment, the data processing apparatus 200 further includes:
[0117] An update module is configured to update the configuration file based on update information received, wherein the update information includes at least one of the updated preset mapping relationship and the updated other configuration parameters;
[0118] An adjustment module is used to adjust the EBPF program based on the updated configuration file.
[0119] The data processing apparatus provided in this embodiment of the invention can implement each process of each embodiment of the above data processing method, with one-to-one correspondence of technical features and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0120] It should be noted that the data processing device in the embodiments of the present invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.
[0121] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.
[0122] For details, see Figure 3 As shown, this embodiment of the invention also provides an electronic device, including a bus 301, a transceiver 302, an antenna 303, a bus interface 304, a processor 305, and a memory 306.
[0123] The transceiver 302 is used to acquire key-value pair data to be processed;
[0124] The processor 305 is used to determine the key type corresponding to the key in the key-value pair data;
[0125] The processor 305 is further configured to determine a first persistence strategy corresponding to the key type based on a preset mapping relationship. The preset mapping relationship is used to determine the persistence strategy corresponding to different types, and the first persistence strategy is used to determine the way the key-value pair data is written to the disk.
[0126] The processor 305 is further configured to write the key-value pair data to the storage cluster and disk based on the first persistence strategy.
[0127] In one embodiment, the preset mapping relationship includes at least one of the following:
[0128] When the key type is the first type, the first persistence strategy is to synchronize the data to the disk when writing data;
[0129] When the key type is the second type, the first persistence strategy is to synchronize the data to the disk based on a preset time interval;
[0130] When the key type is the third type, the first persistence strategy is not to synchronize the written data.
[0131] In one embodiment, the processor 305 is further configured to create a configuration file, the configuration file including the preset mapping relationship and other configuration parameters, the other configuration parameters being parameters for obtaining the key-value pair data or parameters for executing a program;
[0132] The processor 305 is further configured to create an Extended Berkeley Packet Filter (EBPF) program based on the configuration file. The EBPF program is configured to execute the steps of obtaining key-value pair data to be processed and determining the first persistence strategy corresponding to the key type based on a preset mapping relationship.
[0133] The processor 305 is further configured to bind the EBPF program to the write function of the storage cluster, the write function being configured to write the key-value pair data to the storage cluster.
[0134] In one embodiment, the step of creating the Extended Berkeley Packet Filter (EBPF) procedure based on the configuration file includes:
[0135] Configure the hash mapping function in the EBPF program based on the preset mapping relationship;
[0136] The step of determining the first persistence strategy corresponding to the key type based on a preset mapping relationship includes:
[0137] Based on the hash mapping function in the EBPF program, the first persistence strategy corresponding to the key type is determined.
[0138] In one embodiment, the other configuration parameters include hook parameters, and the process of creating the Extended Berkeley Packet Filter (EBPF) procedure based on the configuration file includes:
[0139] The hook function in the EBPF program is configured based on the hook parameters. The hook function is used to obtain key-value pair data from the structure. The structure is used to store the key-value pair data to be processed.
[0140] The process of obtaining the key-value pair data to be processed includes:
[0141] The key-value pair data to be processed is obtained based on the hook function in the EBPF program.
[0142] In one embodiment, the processor 305 is further configured to create a user-space program for storing the configuration file.
[0143] In one embodiment, the processor 305 is further configured to update the configuration file based on the update information received, wherein the update information includes at least one of the updated preset mapping relationship and the updated other configuration parameters;
[0144] The processor 305 is also configured to adjust the EBPF program based on the updated configuration file.
[0145] exist Figure 3 In this document, a bus architecture (represented by bus 301) is used. Bus 301 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 305 and memory represented by memory 306. Bus 301 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 304 provides an interface between bus 301 and transceiver 302. Transceiver 302 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 305 is transmitted over a wireless medium via antenna 303, which further receives data and transmits it to processor 305.
[0146] Processor 305 manages bus 301 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 306 can be used to store data used by processor 305 during operation.
[0147] Optionally, the processor 305 can be a CPU, ASIC, FPGA, or CPLD.
[0148] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, further details are omitted here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0149] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The various processes corresponding to the data processing method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.
[0150] In the embodiments of this invention, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, indicating the inclusion of A alone, B alone, C alone, and the three cases of A and B being present, B and C being present, A and C being present, and A, B, and C being present.
[0151] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of this application.
[0153] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A data processing method, characterized in that, include: Retrieve the key-value pair data to be processed; Determine the key type corresponding to the key in the key-value pair data; The first persistence strategy corresponding to the key type is determined based on a preset mapping relationship. The preset mapping relationship is used to determine the persistence strategy corresponding to different types. The first persistence strategy is used to determine the way the key-value pair data is written to the disk. The key-value pair data is written to the storage cluster and disk based on the first persistence strategy.
2. The method as described in claim 1, characterized in that, The preset mapping relationship includes at least one of the following: When the key type is the first type, the first persistence strategy is to synchronize the data to the disk when writing data; When the key type is the second type, the first persistence strategy is to synchronize the data to the disk based on a preset time interval; When the key type is the third type, the first persistence strategy is not to synchronize the written data.
3. The method as described in claim 1 or 2, characterized in that, Before obtaining the key-value pair data to be processed, the method further includes: Create a configuration file, which includes the preset mapping relationship and other configuration parameters, wherein the other configuration parameters are parameters for obtaining the key-value pair data or parameters for executing the program; Based on the configuration file, an Extended Berkeley Packet Filter (EBPF) program is created. The EBPF program is used to execute the steps of obtaining key-value pair data to be processed and determining the first persistence strategy corresponding to the key type based on a preset mapping relationship. The EBPF program is bound to the write function of the storage cluster, which is used to write the key-value pair data to the storage cluster.
4. The method as described in claim 3, characterized in that, The procedure for creating the Extended Berkeley Packet Filter (EBPF) based on the configuration file includes: Configure the hash mapping function in the EBPF program based on the preset mapping relationship; The step of determining the first persistence strategy corresponding to the key type based on a preset mapping relationship includes: Based on the hash mapping function in the EBPF program, the first persistence strategy corresponding to the key type is determined.
5. The method as described in claim 3, characterized in that, The other configuration parameters include hook parameters, and the creation of the Extended Berkeley Packet Filter (EBPF) procedure based on the configuration file includes: The hook function in the EBPF program is configured based on the hook parameters. The hook function is used to obtain key-value pair data from the structure. The structure is used to store the key-value pair data to be processed. The process of obtaining the key-value pair data to be processed includes: The key-value pair data to be processed is obtained based on the hook function in the EBPF program.
6. The method as described in claim 3, characterized in that, After creating the configuration file, the method further includes: Create a userspace program, which is used to store the configuration file.
7. The method as described in claim 3, characterized in that, The method further includes: Upon receiving update information, the configuration file is updated based on the update information, which includes at least one of the updated preset mapping relationship and the updated other configuration parameters; The EBPF program is adjusted based on the updated configuration file.
8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire key-value pair data to be processed; The first determining module is used to determine the key type corresponding to the key in the key-value pair data; The second determining module is used to determine the first persistence strategy corresponding to the key type based on a preset mapping relationship. The preset mapping relationship is used to determine the persistence strategy corresponding to different types. The first persistence strategy is used to determine the way the key-value pair data is written to the disk. The write module is used to write the key-value pair data to the storage cluster and disk based on the first persistence strategy.
9. An electronic device, characterized in that, Including transceivers and processors, The transceiver is used to acquire key-value pair data to be processed; The processor is used to determine the key type corresponding to the key in the key-value pair data; The processor is further configured to determine a first persistence strategy corresponding to the key type based on a preset mapping relationship, wherein the preset mapping relationship is used to determine the persistence strategy corresponding to different types, and the first persistence strategy is used to determine the way the key-value pair data is written to the disk; The processor is further configured to write the key-value pair data to the storage cluster and disk based on the first persistence strategy.
10. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1 to 7.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 7.
12. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1 to 7.