Data reading and writing method, device, equipment and storage medium

Abstract

The application discloses a data read-write method, relates to the technical field of data processing, and comprises the following steps: receiving an operation instruction; determining a metadata module corresponding to a business object name according to the business object name; determining a first metadata unit according to a starting address of an instruction address interval and a plurality of metadata units in the metadata module, and determining at least one target metadata unit in the metadata module according to the first metadata unit, wherein the first metadata unit is a metadata unit with the largest address greater than the starting address and the smallest difference from the starting address; and performing a read operation or a write operation on the metadata unit according to an operation type, the instruction address interval and the at least one target metadata unit, so that the technical problem that metadata management faces the trade-off between performance and flexibility is solved, the metadata module is divided into a plurality of metadata units, atomic operation can be performed on the metadata management, and the efficiency of metadata management is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data reading and writing method, apparatus, device and storage medium. Background Technology

[0002] When implementing a data control separation scheme in current storage systems, metadata management is typically handled on a per-business-object basis. When accessed business objects are of uniform size, the system can directly locate and access the corresponding metadata information through the business address, achieving efficient data management. However, in scenarios where business object sizes vary, it is often necessary to first find the closest metadata and then use certain computational logic to ultimately determine the location of the target metadata, which increases access complexity and overhead to some extent. In small I / O (Input / Output) scenarios (such as 4KB random writes), if each 4KB data block corresponds to a metadata object, the system will face the challenge of managing massive amounts of metadata as storage capacity continues to grow. This huge amount of metadata information will put enormous pressure on the backend KV (Key-Value) storage system, causing a significant decrease in the efficiency of metadata query and update, thereby affecting overall performance. Conversely, if storage management is done using larger granular objects, such as treating 4MB of business data as a single object with only one key-value metadata, the amount of metadata can be greatly reduced, demonstrating excellent performance advantages in large I / O scenarios. However, this design is not flexible enough when dealing with small I / O operations. For example, when only 4KB of data needs to be written, the metadata corresponding to the entire 4MB object still needs to be updated, causing unnecessary resource consumption and seriously affecting the efficiency of small I / O operations.

[0003] Therefore, in view of the shortcomings of existing technical solutions, the present invention provides a data reading and writing method. Summary of the Invention

[0004] This application provides data read / write methods, apparatus, devices, and storage media to at least address the trade-off between performance and flexibility in metadata management under varying business object sizes and low I / O scenarios in related technologies.

[0005] This application provides a data read / write method, the method comprising: receiving an operation instruction, wherein the operation instruction includes an operation type, a business object name, and an instruction address range; determining a metadata module corresponding to the business object name based on the business object name; determining a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and determining at least one target metadata unit in the metadata module based on the first metadata unit, wherein the first metadata unit is the metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; and performing a read operation or a write operation on the metadata unit based on the operation type, the instruction address range, and at least one target metadata unit.

[0006] This application also provides a data read / write device, comprising: a first processing module for receiving an operation instruction, wherein the operation instruction includes an operation type, a business object name, and an instruction address range; a second processing module for determining a metadata module corresponding to the business object name based on the business object name; a third processing module for determining a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and determining at least one target metadata unit in the metadata module based on the first metadata unit, wherein the first metadata unit is the metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; and a fourth processing module for performing read or write operations on the metadata unit based on the operation type, the instruction address range, and at least one target metadata unit.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program, which implements the following steps: receiving an operation instruction, wherein the operation instruction includes an operation type, a business object name, and an instruction address range; determining a metadata module corresponding to the business object name based on the business object name; determining a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and determining at least one target metadata unit in the metadata module based on the first metadata unit, wherein the first metadata unit is the metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; and performing a read operation or a write operation on the metadata unit based on the operation type, the instruction address range, and at least one target metadata unit.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the following steps: receiving an operation instruction, wherein the operation instruction includes an operation type, a business object name, and an instruction address range; determining the metadata module corresponding to the business object name based on the business object name; determining a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and determining at least one target metadata unit in the metadata module based on the first metadata unit, wherein the first metadata unit is the metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; and performing a read operation or a write operation on the metadata unit based on the operation type, the instruction address range, and at least one target metadata unit.

[0009] This application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the following steps: receiving an operation instruction, wherein the operation instruction includes an operation type, a business object name, and an instruction address range; determining the metadata module corresponding to the business object name based on the business object name; determining a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and determining at least one target metadata unit in the metadata module based on the first metadata unit, wherein the first metadata unit is the metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; and performing a read operation or a write operation on the metadata unit based on the operation type, the instruction address range, and at least one target metadata unit.

[0010] This application utilizes an application to receive operation instructions, which include an operation type, a business object name, and an instruction address range. Based on the business object name, it determines the metadata module corresponding to that name. Based on the starting address of the instruction address range and multiple metadata units within the metadata module, it determines a first metadata unit and, based on the first metadata unit, at least one target metadata unit within the metadata module. The first metadata unit is the metadata unit whose address is greater than the starting address and has the smallest difference from the starting address. Based on the operation type, instruction address range, and at least one target metadata unit, it performs read or write operations on the metadata units. Therefore, by dividing the metadata module into multiple metadata units, atomic operations can be performed during metadata management, improving the efficiency of metadata management. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a data read / write method provided in an embodiment of this application;

[0013] Figure 2 A schematic diagram of the cache space update process of a data read / write method provided in this application embodiment;

[0014] Figure 3 A structural block diagram of a data read / write device provided in an embodiment of this application;

[0015] Figure 4 This is an internal structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0017] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0018] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The embodiments of this application provide a data read and write method, and the method is described in detail in conjunction with the execution flow of the data read and write method.

[0021] S101: Receive operation instructions, which include operation type, business object name and instruction address range.

[0022] Here, the operation type includes read instructions or write instructions.

[0023] Here, the instruction address range can be represented by the start address and end address, or by the offset and length. For example, if the offset is 4 and the length is 20, then the instruction address range is 0x1004-0x1024.

[0024] Here, the business object name includes the metadata module.

[0025] S102: Determine the metadata module corresponding to the business object name based on the business object name.

[0026] Here, the business object name and metadata module are stored in key-value pairs.

[0027] The metadata module is stored in the metadata database, which can be a KV (Key-Value) database. The metadata database stores multiple metadata modules.

[0028] S103: Based on the starting address of the instruction address range and multiple metadata units in the metadata module, determine the first metadata unit, and based on the first metadata unit, determine at least one target metadata unit in the metadata module, wherein the first metadata unit is the metadata unit whose address is greater than the starting address and whose difference from the starting address is the smallest.

[0029] Here, a metadata module includes multiple metadata units.

[0030] Here, the target metadata unit includes the first metadata unit; the target metadata unit may include one metadata unit or multiple metadata units.

[0031] Here, a metadata module includes multiple metadata units.

[0032] Each metadata unit includes an address range, with the starting address of the address range serving as the address of the metadata unit.

[0033] For example, assuming the starting address is 0x0017, and the metadata unit includes three address ranges: 0x0000-0x0015, 0x0020-0x0030, and 0x0035-0x0045, then the metadata unit in the address range of 0x0020-0x0030 is selected as the first metadata unit.

[0034] S104: Perform a read or write operation on the metadata unit according to the operation type, instruction address range, and at least one target metadata unit.

[0035] Specifically, overlapping address ranges are determined based on the instruction address range and the target metadata unit.

[0036] Specifically, based on the operation type, it is determined whether it is a write operation or a read operation. Based on the instruction address range and the target metadata unit, the overlapping address range is determined, and the overlapping address range is read or written.

[0037] It should be noted that this application divides the metadata module into multiple metadata units, which allows for atomic operations during metadata management, thereby improving the efficiency of metadata management.

[0038] In some specific implementations, the method further includes:

[0039] Receive business objects;

[0040] Based on the single data throughput of the metadata database, the business object is divided into at least one business unit;

[0041] According to the mapping rules, at least one business unit is mapped to at least one corresponding metadata.

[0042] Obtain the starting metadata unit from the metadata module, and store at least one metadata unit sequentially in the corresponding metadata unit.

[0043] Here, the business object is the name of the business data stored in the backend storage at the protocol layer.

[0044] Here, the single data throughput of the metadata database can be defined as business I / O (Input / Output).

[0045] Here, the mapping rule can be sparse mapping, where multiple business units are mapped into metadata through sparse mapping.

[0046] The starting metadata unit can be the first metadata unit in the metadata module that does not store metadata.

[0047] The metadata is stored sequentially into one or more corresponding metadata units.

[0048] A business object can be stored in one metadata module or multiple metadata modules, depending on the size of the business object.

[0049] Specifically, if business objects are distributed across metadata modules for writing, such as a business object with an offset of 200K and a len of 128K, then when storing forward relationships, the business object should be divided into two segments, i.e., two metadata module shards, and then the forward relationships should be updated.

[0050] In this way, fault isolation can be achieved, improving the system's fault tolerance and availability, and allowing individual operation on each metadata unit.

[0051] In some specific implementations, at least one metadata element is stored sequentially in a corresponding metadata unit, including:

[0052] The number of preset metadata units is determined based on the size of the metadata module and the size of the metadata.

[0053] The preset number of metadata units is compared with 2. n Alignment yields the number of metadata units, where 2 n Let n be the power of 2, and n take values ​​such that 2 n The value is less than or equal to the maximum integer of the preset number of metadata units;

[0054] The metadata module is divided into multiple metadata units based on the number of elements.

[0055] At least one metadata element is stored sequentially in the metadata unit of the metadata module.

[0056] Here, the size of the metadata module can be a preset size, such as 2k (i.e., 2048 bytes), and the size of the metadata can also be a preset size, such as 20 bytes.

[0057] For example, the content of metadata can be as follows:

[0058] uint32_t offset;

[0059] uint32_t len;

[0060] uint64_t chunk_id;

[0061] uint32_t inner_offset;

[0062] Among them, offset, len, and chunk_id each occupy 4 bytes of space, while inner_offset occupies 8 bytes, thus a metadata entry occupies 20 bytes.

[0063] Specifically, assuming the size of the metadata module is 2KB (i.e., 2048 bytes), and the size of the metadata is 20 bytes, then 2048 / 20 = 102 (i.e., the preset number of metadata units).n Aligning downwards, we get 64; a metadata module stores 64 metadata units.

[0064] In one embodiment, a heartbeat mechanism is set up to monitor the status of each metadata unit. When an anomaly is detected in a metadata unit, a failover process is triggered. Specifically, if metadata unit a in metadata module A is detected to be unavailable, due to the storage of the metadata module and 2 n Alignment is performed if some metadata modules have free space. Free space is found in metadata module B, so metadata unit a is moved to metadata module B for storage. The index is updated, and metadata module A is marked as unavailable. Once metadata module A recovers, metadata unit a is migrated back to metadata module A. This improves the system's fault tolerance and resource utilization.

[0065] Thus, according to 2 n Alignment is beneficial for low-level operations such as CPU caching, memory alignment, and bitwise operation optimization, thereby improving access speed.

[0066] In some specific implementations, a first metadata unit is determined based on the starting address of the instruction address range and multiple metadata units in the metadata module, and at least one target metadata unit in the metadata module is determined based on the first metadata unit, including:

[0067] Determine whether the instruction address range passes the validity test;

[0068] In response to the instruction address range passing the validity test, obtain the starting address of the instruction address range;

[0069] Based on the starting address, the first metadata unit is determined by detecting metadata units in the metadata module using a binary search algorithm.

[0070] Detect the left adjacent metadata unit of the first metadata unit and determine whether the left adjacent metadata unit includes the starting address;

[0071] In response to the fact that the left adjacent metadata unit includes the starting address, the left adjacent metadata unit is obtained as the left metadata unit;

[0072] Get the end address of the instruction address range;

[0073] Sequentially detect multiple right-adjacent metadata units of the first metadata unit;

[0074] In response to the detection of a metadata unit including a termination address, the detection is stopped and at least one right adjacent metadata unit is obtained as the right metadata unit.

[0075] The target metadata unit is obtained based on the first metadata unit, the left metadata unit, and the right metadata unit.

[0076] Here, the binary search algorithm quickly locates the target value by halving the search range.

[0077] The validity test may include determining whether the length of the instruction address range is 0 or whether the instruction address range exceeds the maximum capacity of the storage area.

[0078] Specifically, when a query request is received, the system first uses a binary search algorithm to quickly locate the candidate unit (i.e., the first metadata unit) near the starting address. Then, it checks the logical range of each adjacent unit in front of the initially located candidate unit to see if it covers the query starting point, until the leftmost potential covering unit is found, which is then used as the left metadata unit.

[0079] Thus, by combining the binary search algorithm with a dynamic backoff mechanism, the inefficiency of brute-force traversal of all metadata units can be avoided, while ensuring accuracy without omissions. By dynamically balancing fast location and accurate coverage detection, millisecond-level response can still be achieved under ultra-large-scale mapping relationships, while strictly guaranteeing the integrity of the results, thus solving the problem of missed or false detections caused by traditional query methods ignoring boundary coverage.

[0080] In some specific implementations, the operation instructions include read instructions and update instructions. Based on the operation type, instruction address range, and at least one target metadata unit, read or write operations are performed on the metadata unit, including:

[0081] In response to the operation instruction being a read instruction, the offset of the target metadata unit is corrected based on the target metadata unit and the instruction address range to obtain the corrected target metadata unit.

[0082] Read the corrected target metadata unit;

[0083] In response to the operation instruction being an update instruction, the non-overlapping interval is obtained based on the target metadata unit and the instruction address range, wherein the non-overlapping interval includes the offset;

[0084] Delete the target metadata unit and write the non-overlapping range and the content corresponding to the update instruction into the target metadata unit.

[0085] Here, non-overlapping intervals can include left non-overlapping intervals and right non-overlapping intervals, where the left non-overlapping interval exists in the leftmost target metadata unit and the right non-overlapping interval exists in the rightmost target metadata unit.

[0086] Specifically, the offset of the target metadata unit can be corrected as follows: assuming the target data unit is 0x0000-0x0020 and the instruction address range is 0x0010-0x0025, the offset can be corrected to 0x0010-0x0020.

[0087] Specifically, the overlapping address range between each instruction address range and each target metadata unit is calculated one by one, and the overlapping address range is corrected (such as adjusting the offset within the block), and the corrected address range is read.

[0088] Specifically, the target metadata unit is split into left fragments (i.e., left non-overlapping intervals) and / or right fragments (i.e., right non-overlapping intervals). The split fragments retain their original offsets. All target metadata units are deleted in batches. The split fragments and the updated content are then inserted into the metadata module together.

[0089] In this way, the shortcomings of traditional sequential overlay methods, such as data loss or logical overlay chaos, are resolved. A physical-to-logical mapping management system is built that can support millisecond-level accurate queries and cope with high-frequency concurrent updates. While improving storage performance, the system anomaly risk is reduced through data integrity protection strategies.

[0090] In some specific implementations, after determining the metadata module corresponding to the business object name based on the business object name, the method further includes:

[0091] Check whether the metadata module corresponding to the business object name in the operation instruction is stored in the cache space;

[0092] In response to the fact that the metadata module is not stored in the cache space, the metadata module is retrieved from the metadata database and stored in the cache space;

[0093] In response to the fact that the metadata module is stored in the cache space, the metadata module is retrieved from the cache space.

[0094] Here, cache space is a fast-access storage area used for temporary data storage. The main purpose of caching is to reduce the number of accesses to slower main memory or databases, thereby speeding up data retrieval.

[0095] The metadata database stores multiple metadata modules, and the cache space stores multiple metadata modules. The metadata database includes metadata modules in the cache space, as well as metadata modules outside the cache space.

[0096] In the cache space, data is stored in the form of key-value pairs, and data can be located and retrieved using a hash table.

[0097] In this way, the efficiency of data management can be improved by using caching space.

[0098] In some specific implementations, in response to the metadata module being stored in a cache space, the method further includes:

[0099] Based on the access frequency of the metadata module, adjust the queue of the metadata module in the cache space, where the queue includes a first queue and a second queue;

[0100] Store the metadata module at the end of the queue;

[0101] According to the caching rules corresponding to the queue, delete the metadata module at the head of the queue.

[0102] Here, the first queue can be an LRU (Least Recently Used) queue, and the second queue can be a FIFO (First In, First Out) queue.

[0103] The metadata in the first queue is accessed more frequently than the metadata in the second queue. Specifically, when the access frequency of the metadata in the second queue is greater than or equal to a preset threshold, the metadata in the second queue is moved to the first queue.

[0104] Among them, the metadata modules that are further back in the queue are more likely to be accessed again.

[0105] The caching rules include the preset queue length.

[0106] Specifically, when a metadata module is accessed for the first time, it is added directly to the cache. If the cached metadata module is accessed again, it is moved to the end of the queue. When a metadata module needs to be evicted, it is removed from the head of the queue.

[0107] Thus, by introducing an optimized LRU algorithm, we can ensure that the cache is utilized effectively, whether it is a random write or a sequential write, without wasting cache space and hitting the cache when necessary.

[0108] In one embodiment, Figure 2 This is a schematic diagram of the cache space update process in an embodiment of this application, as shown below. Figure 2As shown, the cache space update process in this application includes: receiving an operation instruction, determining whether the metadata module corresponding to the business object name is in the hash table; if the metadata module is in the hash table, determining whether the metadata module is in the first queue or the second queue; when the metadata module is in the first queue, writing the metadata module to the tail of the first queue, updating the hash table pointer, checking the size of the first queue, and deleting the head of the first queue when the limit is exceeded, releasing the cache space; when the metadata module is in the second queue, incrementing the access count of the metadata module by one, determining whether the access count is greater than or equal to k; when it is greater than or equal to k, writing the metadata module to the tail of the first queue; when it is less than k, keeping it in the second queue, writing the metadata module to the tail of the second queue, checking the size of the second queue, and deleting the head of the second queue when the limit is exceeded; when the metadata module is not in the hash table, writing the metadata module to the hash table and then to the tail of the second queue.

[0109] It should be understood that, although Figure 1 and 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 and 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0111] Embodiments of this application also provide a data read / write device, comprising: a first processing module 301 for receiving an operation instruction, wherein the operation instruction includes an operation type, a business object name, and an instruction address range; a second processing module 302 for determining a metadata module corresponding to a business object name based on the business object name; a third processing module 303 for determining a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and determining at least one target metadata unit in the metadata module based on the first metadata unit, wherein the first metadata unit is a metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; and a fourth processing module 304 for performing a read operation or a write operation on the metadata unit based on the operation type, the instruction address range, and at least one target metadata unit.

[0112] In a preferred embodiment of this application, the device further includes a storage module, which is specifically used for: receiving a business object; dividing the business object into at least one business unit according to the single data throughput of the metadata database; mapping the at least one business unit into at least one corresponding metadata according to the mapping rules; obtaining the starting metadata unit in the metadata module; and storing the at least one metadata in the corresponding metadata unit in sequence.

[0113] In a preferred embodiment of this application, the storage module is further configured to: determine a preset number of metadata units based on the size of the metadata module and the size of the metadata; and multiply the preset number of metadata units by 2... n Alignment yields the number of metadata units, where 2 n Let n be the power of 2, and n take values ​​such that 2 n The value is less than or equal to the maximum integer of the preset number of metadata units; the metadata module is divided into multiple metadata units according to the number of units; at least one metadata unit is stored sequentially in the metadata unit of the metadata module.

[0114] In a preferred embodiment of this application, the third processing module 303 is specifically used for: determining whether the instruction address range passes the validity test; in response to the instruction address range passing the validity test, obtaining the starting address of the instruction address range; based on the starting address, detecting metadata units in the metadata module using a binary search algorithm to determine the first metadata unit; detecting the left adjacent metadata unit of the first metadata unit and determining whether the left adjacent metadata unit includes the starting address; in response to the left adjacent metadata unit including the starting address, obtaining the left adjacent metadata unit as the left metadata unit; obtaining the ending address of the instruction address range; sequentially detecting multiple right adjacent metadata units of the first metadata unit; in response to detecting a metadata unit including the ending address, stopping the detection and obtaining at least one right adjacent metadata unit as the right metadata unit; and obtaining the target metadata unit based on the first metadata unit, the left metadata unit, and the right metadata unit.

[0115] In a preferred embodiment of this application, the fourth processing module 304 is specifically configured to: in response to an operation instruction being a read instruction, correct the offset of the target metadata unit according to the target metadata unit and the instruction address range to obtain the corrected target metadata unit; read the corrected target metadata unit; in response to an operation instruction being an update instruction, obtain the non-overlapping range according to the target metadata unit and the instruction address range, wherein the non-overlapping range includes the offset; delete the target metadata unit, and write the content corresponding to the non-overlapping range and the update instruction into the target metadata unit.

[0116] In a preferred embodiment of this application, the device further includes a caching module, which is specifically used to: detect whether the metadata module corresponding to the business object name in the operation instruction is stored in the cache space; in response to the metadata module not being stored in the cache space, retrieve the metadata module from the metadata database and store the metadata module in the cache space; and in response to the metadata module being stored in the cache space, retrieve the metadata module from the cache space.

[0117] As a preferred implementation, in this embodiment of the application, the caching module is specifically used to: adjust the queue of the metadata module in the cache space according to the access frequency of the metadata module, wherein the queue includes a first queue and a second queue; store the metadata module at the tail of the queue; and delete the metadata module at the head of the queue according to the caching rules corresponding to the queue.

[0118] For a description of the features in the embodiment corresponding to the data read / write device, please refer to the relevant description in the embodiment corresponding to the data read / write method, which will not be repeated here.

[0119] Embodiments of this application also provide an electronic device, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data read / write method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0120] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0121] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: S1: receiving an operation instruction, wherein the operation instruction includes an operation type, a business object name, and an instruction address range; S2: determining the metadata module corresponding to the business object name based on the business object name; S3: determining a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and determining at least one target metadata unit in the metadata module based on the first metadata unit, wherein the first metadata unit is the metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; S4: performing a read operation or a write operation on the metadata unit based on the operation type, the instruction address range, and at least one target metadata unit.

[0122] In one embodiment, when the processor executes the computer program, it further performs the following steps: receiving a business object; dividing the business object into at least one business unit according to the single data throughput of the metadata database; mapping the at least one business unit to at least one corresponding metadata according to the mapping rules; obtaining the starting metadata unit in the metadata module, and storing the at least one metadata unit in the corresponding metadata unit in sequence.

[0123] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a preset number of metadata units based on the size of the metadata module and the size of the metadata; and multiplying the preset number of metadata units by 2. n Alignment yields the number of metadata units, where 2 n Let n be the power of 2, and n take values ​​such that 2 n The value is less than or equal to the maximum integer of the preset number of metadata units; the metadata module is divided into multiple metadata units according to the number of units; at least one metadata unit is stored sequentially in the metadata unit of the metadata module.

[0124] In one embodiment, when the processor executes a computer program, it further performs the following steps: determining whether the instruction address range passes a validity test; in response to the instruction address range passing the validity test, obtaining the starting address of the instruction address range; based on the starting address, detecting metadata units in the metadata module using a binary search algorithm to determine a first metadata unit; detecting the left adjacent metadata unit of the first metadata unit and determining whether the left adjacent metadata unit includes the starting address; in response to the left adjacent metadata unit including the starting address, obtaining the left adjacent metadata unit as the left metadata unit; obtaining the ending address of the instruction address range; sequentially detecting multiple right adjacent metadata units of the first metadata unit; in response to detecting a metadata unit including the ending address, stopping the detection and obtaining at least one right adjacent metadata unit as the right metadata unit; and obtaining the target metadata unit based on the first metadata unit, the left metadata unit, and the right metadata unit.

[0125] In one embodiment, when the processor executes a computer program, it further performs the following steps: in response to an operation instruction being a read instruction, correcting the offset of the target metadata unit based on the target metadata unit and the instruction address range to obtain a corrected target metadata unit; reading the corrected target metadata unit; in response to an operation instruction being an update instruction, obtaining a non-overlapping range based on the target metadata unit and the instruction address range, wherein the non-overlapping range includes the offset; deleting the target metadata unit, and writing the non-overlapping range and the content corresponding to the update instruction into the target metadata unit.

[0126] In one embodiment, when the processor executes the computer program, it further performs the following steps: detecting whether the metadata module corresponding to the business object name in the operation instruction is stored in the cache space; in response to the metadata module not being stored in the cache space, retrieving the metadata module from the metadata database and storing the metadata module in the cache space; in response to the metadata module being stored in the cache space, retrieving the metadata module from the cache space.

[0127] In one embodiment, when the processor executes the computer program, it further performs the following steps: adjusting the queue of metadata modules in the cache space according to the access frequency of the metadata modules, wherein the queue includes a first queue and a second queue; storing the metadata modules at the tail of the queue; and deleting the metadata modules at the head of the queue according to the caching rules corresponding to the queue.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: S1: receiving an operation instruction, wherein the operation instruction includes an operation type, a business object name, and an instruction address range; S2: determining the metadata module corresponding to the business object name based on the business object name; S3: determining a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and determining at least one target metadata unit in the metadata module based on the first metadata unit, wherein the first metadata unit is the metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; S4: performing a read operation or a write operation on the metadata unit based on the operation type, the instruction address range, and at least one target metadata unit.

[0129] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving a business object; dividing the business object into at least one business unit according to the single data throughput of the metadata database; mapping the at least one business unit to at least one corresponding metadata according to the mapping rules; obtaining the starting metadata unit in the metadata module, and storing the at least one metadata unit in the corresponding metadata unit in sequence.

[0130] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a preset number of metadata units based on the size of the metadata module and the size of the metadata; and multiplying the preset number of metadata units by 2. n Alignment yields the number of metadata units, where 2 n Let n be the power of 2, and n take values ​​such that 2 n The value is less than or equal to the maximum integer of the preset number of metadata units; the metadata module is divided into multiple metadata units according to the number of units; at least one metadata unit is stored sequentially in the metadata unit of the metadata module.

[0131] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining whether the instruction address range passes the validity test; in response to the instruction address range passing the validity test, obtaining the starting address of the instruction address range; based on the starting address, detecting metadata units in the metadata module using a binary search algorithm to determine the first metadata unit; detecting the left adjacent metadata unit of the first metadata unit and determining whether the left adjacent metadata unit includes the starting address; in response to the left adjacent metadata unit including the starting address, obtaining the left adjacent metadata unit as the left metadata unit; obtaining the ending address of the instruction address range; sequentially detecting multiple right adjacent metadata units of the first metadata unit; in response to detecting a metadata unit including the ending address, stopping the detection and obtaining at least one right adjacent metadata unit as the right metadata unit; and obtaining the target metadata unit based on the first metadata unit, the left metadata unit, and the right metadata unit.

[0132] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: in response to an operation instruction being a read instruction, correcting the offset of the target metadata unit according to the target metadata unit and the instruction address range to obtain a corrected target metadata unit; reading the corrected target metadata unit; in response to an operation instruction being an update instruction, obtaining a non-overlapping range according to the target metadata unit and the instruction address range, wherein the non-overlapping range includes the offset; deleting the target metadata unit, and writing the non-overlapping range and the content corresponding to the update instruction into the target metadata unit.

[0133] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: detecting whether the metadata module corresponding to the business object name in the operation instruction is stored in the cache space; in response to the metadata module not being stored in the cache space, retrieving the metadata module from the metadata database and storing the metadata module in the cache space; in response to the metadata module being stored in the cache space, retrieving the metadata module from the cache space.

[0134] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: adjusting the queue of metadata modules in the cache space according to the access frequency of the metadata modules, wherein the queue includes a first queue and a second queue; storing the metadata modules at the tail of the queue; and deleting the metadata modules at the head of the queue according to the caching rules corresponding to the queue.

[0135] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data read / write method, characterized in that, The method includes: Receive an operation instruction, wherein the operation instruction includes an operation type, a business object name, and an instruction address range; Based on the business object name, determine the metadata module corresponding to the business object name; Based on the starting address of the instruction address range and multiple metadata units in the metadata module, a first metadata unit is determined, and at least one target metadata unit in the metadata module is determined based on the first metadata unit, wherein the first metadata unit is a metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; Based on the operation type, the instruction address range, and the at least one target metadata unit, perform a read operation or a write operation on the metadata unit; The method further includes: Receive business objects; Based on the single data throughput of the metadata database, the business object is divided into at least one business unit; According to the mapping rules, the at least one business unit is mapped to at least one corresponding metadata. Obtain the starting metadata unit in the metadata module, and store the at least one metadata unit sequentially in the corresponding metadata unit; The step of determining a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and determining at least one target metadata unit in the metadata module based on the first metadata unit, includes: Determine whether the instruction address range passes the validity test; In response to the instruction address range passing the validity test, the starting address of the instruction address range is obtained; Based on the starting address, the first metadata unit is determined by detecting the metadata units in the metadata module using a binary search algorithm. Detect the left adjacent metadata unit of the first metadata unit, and determine whether the left adjacent metadata unit includes the starting address; In response to the fact that the left adjacent metadata unit includes the starting address, the left adjacent metadata unit is obtained as the left metadata unit; Obtain the termination address of the instruction address range; Sequentially detect multiple right adjacent metadata units of the first metadata unit; In response to the detection of a metadata unit including the termination address, the detection is stopped, and at least one right adjacent metadata unit is obtained as the right metadata unit. The target metadata unit is obtained based on the first metadata unit, the left metadata unit, and the right metadata unit.

2. The data read / write method according to claim 1, characterized in that, The step of storing the at least one metadata in the corresponding metadata unit in sequence includes: The number of preset metadata units is determined based on the size of the metadata module and the size of the metadata. Align the preset number of metadata units with 2n to obtain the number of metadata units, where 2n represents the power of 2, and n is the largest integer that makes the value of 2n less than or equal to the preset number of metadata units. The metadata module is divided into multiple metadata units according to the number of elements; The at least one metadata is stored sequentially in the metadata unit of the metadata module.

3. The data read / write method according to claim 1, characterized in that, The operation instructions include read instructions and update instructions. The step of performing a read or write operation on the metadata unit based on the operation type, the instruction address range, and the at least one target metadata unit includes: In response to the operation instruction being the read instruction, the offset of the target metadata unit is corrected according to the target metadata unit and the instruction address range to obtain the corrected target metadata unit; Read the corrected target metadata unit; In response to the operation instruction being the update instruction, a non-overlapping interval is obtained based on the target metadata unit and the instruction address range, wherein the non-overlapping interval includes an offset; Delete the target metadata unit and write the content corresponding to the non-overlapping interval and update instruction into the target metadata unit.

4. The data read / write method according to claim 1, characterized in that, After determining the metadata module corresponding to the business object name based on the business object name, the method further includes: Detect whether the metadata module corresponding to the business object name in the operation instruction is stored in the cache space; In response to the fact that the metadata module is not stored in the cache space, the metadata module is retrieved from the metadata database and stored in the cache space; In response to the fact that the metadata module is stored in the cache space, the metadata module is retrieved from the cache space.

5. The data read / write method according to claim 4, characterized in that, The method further includes responding to the fact that the metadata module is stored in the cache space: Based on the access frequency of the metadata module, adjust the queue of the metadata module in the cache space, wherein the queue includes a first queue and a second queue; The metadata module is stored at the tail of the queue; According to the caching rules corresponding to the queue, delete the metadata module at the head of the queue.

6. A data reading and writing apparatus for implementing the data reading and writing method according to any one of claims 1 to 5, characterized in that, The device includes: The first processing module is used to receive operation instructions, wherein the operation instructions include operation type, business object name and instruction address range; The second processing module is used to determine the metadata module corresponding to the business object name based on the business object name. The third processing module is used to determine a first metadata unit based on the starting address of the instruction address range and multiple metadata units in the metadata module, and to determine at least one target metadata unit in the metadata module based on the first metadata unit, wherein the first metadata unit is a metadata unit whose address is greater than the starting address and has the smallest difference from the starting address; The fourth processing module is used to perform read or write operations on the metadata unit according to the operation type, the instruction address range, and the at least one target metadata unit.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the steps of the data read / write method as claimed in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the data read / write method as claimed in any one of claims 1 to 5.

Images