A database lock method and device based on data instance

Abstract

The invention discloses a database lock method and device based on data instances, and relates to the technical field of database locks. The invention adopts hash distribution based on instance ID combined with dynamic adjustment of task queue length, and the distribution thread can realize load balancing of write threads, avoiding delay caused by uneven task distribution; at the same time, a dynamic thread pool expansion mechanism is introduced to automatically adjust the number of threads according to the real-time load, improve the processing capacity under high load, save resources under low load, and significantly enhance the adaptability of the device; in addition, a lock-free writing and local locking mechanism is introduced to solve the dilemma of granularity selection of traditional lock methods, and direct writing is performed in a conflict-free scenario to avoid the performance overhead of lock operations; in the event of a conflict, only the target instance is locked, the influence range of the lock is limited, and the overall concurrency efficiency is improved; in addition, the dynamic control of the lock state can effectively ensure data consistency and reduce the impact of the globalization of traditional locks.

Description

Technical Field

[0001] The present invention relates to the technical field of database locks, and in particular to a database lock method and device based on data instances. Background Art

[0002] Database locks are the core mechanism for ensuring data consistency and integrity in high-concurrency environments. However, with the widespread application of distributed systems, in scenarios such as the Internet of Things and time series data, high-frequency data writing and concurrent read and write operations have led to increasingly fierce competition for lock resources, which can easily lead to performance bottlenecks when processing large-scale instances.

[0003] At present, the three commonly used data lock mechanisms in the industry include row lock, data page lock and data table lock. Row lock uses data entry as granularity to effectively solve the problem of mutual exclusion of access, but its switch lock operation consumes high resources; table lock operates at the table level, which reduces the cost of lock operation, but increases the probability of mutual exclusion conflict and affects concurrency performance; page lock combines the advantages of row lock and table lock, manages through page storage units, and takes into account granularity and performance; but with the continuous growth of IoT data concurrency, the effect of page lock is gradually weakened; to deal with these problems, some solutions adopt multi-granularity lock, optimistic lock and pessimistic lock strategies for optimization, multi-granularity lock in table, page or Locks are used flexibly at different levels of rows to reduce the probability of conflicts; optimistic locking assumes that there are fewer conflicts, reduces the use of locks, and resolves conflicts through version control; pessimistic locking actively avoids conflicts and ensures that no other transaction operations occur during the lock period. This type of solution alleviates lock conflicts to a certain extent, but at the same time it causes new problems: it is difficult to select the appropriate granularity level for multi-granular locks, and lock conflicts may still occur under high concurrency conditions; optimistic locking requires complex conflict detection logic, and performance actually decreases in high-conflict scenarios; and pessimistic locking increases system waiting time due to long lock holding time, resulting in a decrease in overall throughput.

[0004] In addition, traditional methods also have shortcomings in dynamic load adaptability. The peak and trough fluctuations of write load lead to uneven distribution of lock resources. Under high load, fixed threads and lock mechanisms are difficult to respond quickly, while under low load, it is easy to waste system resources, further dragging down system performance. Therefore, there is an urgent need for a database lock method and device based on data instances to solve such problems. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] The present invention provides a database lock method and device based on data instances to solve the problems of large resource consumption and low data reading and writing efficiency in data reading and writing.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a database lock device based on a data instance, which includes:

[0009] Distribute threads, distribute write tasks according to the hash value of the instance ID in the instance data key, and dynamically adjust the allocation weight of the write thread;

[0010] The instance data key is a set of identifiers used to uniquely identify a data instance, consisting of an instance ID, a timestamp, and a data value;

[0011] Different devices or data instances are distinguished by instance IDs, the time point when specific data is written is marked by timestamps, and the actual content of the storage is represented by data values;

[0012] The write thread group receives the write tasks of the distribution thread, executes the write operation of the instance data, and detects read-write conflicts:

[0013] If there is no conflict, write directly;

[0014] If there is a conflict, the instance ID is locked before writing;

[0015] The storage module organizes data storage by instance, supports dynamic expansion, and links data nodes in chronological order to form a chain storage structure; the storage module contains multiple data units SliceTable;

[0016] In the data unit SliceTable, SliceTable is the basic unit of the storage module, similar to a small time series database table; each SliceTable is responsible for storing the data of an instance within a specific time period, including the instance data content and the previous and next node pointers Prev and Next.

[0017] As a preferred solution of the database lock device based on data instance described in the present invention, the step of distributing the writing task according to the hash value of the instance ID in the instance data key value and dynamically adjusting the allocation weight of the writing thread is as follows:

[0018] Initialize the task queue and initial allocation weight for each write thread. The initialization formula is:

[0019]

[0020] in, is the initial allocation weight of thread i, N is the total number of write threads,

[0021] Initialize the task queue length of each writer thread to zero: in, is the task queue length of thread i at the beginning;

[0022] Assume that the instance data key value is K(I,T,V), and perform hash calculation on the instance IDI of the instance data key value. The calculation formula is: H(I)=Hash(I)modN, where H(I) is the hash calculation result of the instance IDI, indicating the index of the initial distribution target thread, Hash(I) is the hash function calculation of the instance IDI, and N is the total number of write threads. According to the result of H(I), add the task K(I,T,V) to the queue of the target thread. The addition formula is:

[0023]

[0024] in, The length of the task queue for the initial distribution target thread is increased by 1 after hash distribution;

[0025] The allocation weight is adjusted in real time according to the current task queue length of the write thread. The adjustment formula is:

[0026]

[0027] Among them, W i (t) is the dynamic allocation weight of thread i at time t, Q i (t) is the current task queue length of thread i, α is the adjustment factor, 0<α≤1, ∈ is the adjustment factor;

[0028] For each task, combined with the remaining processing capacity R of the write thread i (t) and weight W i (t) Select the target thread. The selection formula is:

[0029]

[0030] Among them, T i Write thread for the selected target, R i (t) is the remaining processing capacity of thread i, defined as:

[0031] R i (t) = C i -Q i (t), where C i is the maximum processing capacity of thread i, Q i (t) is the current task queue length of thread i;

[0032] Assign tasks to the selected thread T i After that, update the task queue length, the update formula is:

[0033]

[0034] in, For the selected thread T i The task queue length updated after the task is assigned, For the selected thread T i The length of the task queue at the current time.

[0035] As a preferred solution of the data instance-based database lock device described in the present invention, wherein: the write thread group is a thread set consisting of multiple write threads responsible for executing data write operations, and dynamic thread pool expansion is performed in the write thread group:

[0036] Automatically adjust the size of the write thread pool based on the load: reduce the number of threads when the load is low, and increase the number of threads when the load is high;

[0037] The write thread is responsible for receiving data tasks from the distribution thread, detecting whether there is a read-write conflict in the instance data, and performing lock-free write or locked write operations on the instance data as needed.

[0038] As a preferred solution of the database lock device based on data instance described in the present invention, the step of performing dynamic thread pool expansion in the write thread group is:

[0039] Initialize the initial number of threads in the write thread pool. The initialization formula is: N 0 = InitialThreads, where N 0 is the initial number of write threads, InitialThreads is the number of initially configured write threads,

[0040] Set the high load and low load thresholds to

[0041] L high and L low ,

[0042] Among them, L high is the high load threshold. When the load exceeds this value, thread expansion is triggered. low is the low load threshold. When the load is lower than this value, thread reduction is triggered.

[0043] Define the load U(t) as the average task queue length of all current threads, the formula is:

[0044]

[0045] Among them, U(t) is the average load at time t, N(t) is the current number of threads at time t, Q i (t) is the task queue length of thread i at time t;

[0046] If U(t)>L high, then increase the number of threads,

[0047] If U(t)<L low , then reduce the number of threads,

[0048] If L low ≤U(t)≤L high , then keep the number of threads unchanged;

[0049] When additional threads are needed, the number of threads after expansion is calculated using the formula:

[0050] N(t+1)=N(t)+ΔN add ,

[0051] Among them, N(t+1) is the number of threads in the adjusted thread pool, N(t) is the number of threads in the current thread pool, ΔN add The number of threads for each expansion is configured as a fixed value;

[0052] When threads need to be reduced, the number of threads after reduction is calculated using the following formula:

[0053] N(t+1)=max(N min , N(t)-ΔN reduce ),

[0054] Among them, max ensures that the number of threads is not less than the minimum thread limit N min , ΔN reduce The number of threads to be reduced each time;

[0055] Initialize the task queue for the newly added thread. The initialization queue formula is:

[0056] Q j (t+1)=0forj∈[N(t)+1, N(t+1)],

[0057] Among them, Q j (t+1) is the task queue length of the new thread j, initialized to 0,

[0058] Evenly distribute tasks to all threads in the current thread pool. The distribution formula is:

[0059]

[0060] Among them, Q i (t+1) is the length of the task queue reallocated to thread i after adjusting the thread pool. The total number of tasks for all current threads.

[0061] In a second aspect, the present invention provides a database lock method based on a data instance, comprising:

[0062] Step S1, the distribution thread generates a specific write task queue according to the hash value of the instance ID, and distributes the data task to the corresponding write thread;

[0063] Step S2, the write thread receives the task, extracts instance data from the task, detects whether there is a read-write conflict, and outputs the conflict detection result; the instance data is the state or data information reported by a single instance (IoT device, sensor, etc.) at a specific time point, and the instance data includes instance ID, timestamp and data value;

[0064] Step S3, updating the write status according to the conflict detection result;

[0065] If there is no conflict, the lock-free write operation is performed directly.

[0066] If there is a conflict, the local lock mechanism is called to lock the instance data and write it;

[0067] Step S4, the write thread updates the SliceTable linked list structure, inserts the new instance data in chronological order, and forms an ordered storage linked list;

[0068] Step S5: The storage module links the data nodes in chronological order and uses a hot-cold migration mechanism to store hot data and cold data in layers.

[0069] As a preferred solution of the database lock method based on data instance described in the present invention, the steps of: the write thread receiving the task, extracting the instance data from the task, detecting whether there is a read-write conflict, and outputting the conflict detection result are as follows:

[0070] The write thread extracts instance data D from the received task queue k , the instance data format is:

[0071] D k = {I k , T k , V k},

[0072] Among them, D k is the instance data in the kth task, I k is the instance ID, T k is the timestamp, V k is the data value; maps the instance data to the corresponding SliceTable node S(I k ), this node is instance I k The storage location of S(I k )=Locate(I k ), where S(I k ) is Example I kIn the target SliceTable node in the storage module, Locate(I k ) is a positioning function used to determine its storage location according to the instance ID;

[0073] The write thread checks the target SliceTable node S(I k ) is in a read-write conflict state:

[0074] Define the read-write conflict flag as Conflict(S(I k )):

[0075] If S(I k ) is being read or written by other threads, then Conflict(S(I k ))=1,

[0076] If no other thread operates S(I k ), then Conflict(S(I k ))=0, where Conflict(S(I k )) is the conflict flag of the current target SliceTable node, S(I k ) is the target SliceTable node;

[0077] If the SliceTable node is locked, then L(S(I k ))=1, if the SliceTable node is not locked, then L(S(I k ))=0, where L(S(I k )) is the SliceTable node S(I k )'s lock status;

[0078] Combine the lock status and conflict flag to update the conflict detection result. According to the detection result, output the conflict detection status of the instance data. The status formula is: R k =Conflict(S(I k )), where R k is the conflict detection result of the kth task, with a value of 0 or 1. k )) is the conflict flag for detection;

[0079] The writing thread passes the detection results to the subsequent steps.

[0080] If R k =0, indicating no conflict, directly enter the lock-free write,

[0081] If R k =1, indicating that there is a conflict and locking is required.

[0082] As a preferred solution of the database lock method based on data instance described in the present invention, the lock-free writing and local lock mechanism in step S3 is specifically as follows:

[0083] When there is no read-write conflict, directly update SliceTable;

[0084] When there is a read-write conflict, only the conflicting instance ID is locked.

[0085] As a preferred solution of the database lock method based on data instance described in the present invention, the lock-free writing and local lock mechanism is specifically as follows:

[0086] Input instance data and conflict detection results D k = {I k , T k , V k}, input the conflict detection result R k =Conflict(S(I k ));

[0087] If R k = 0, indicating no conflict, directly executing lock-free write, and writing the instance data D k Write the corresponding SliceTable node S(I k ), the formula is: S(I k )=S(I k )∪{D k}, where S(I k ) is Example I k The storage node in SliceTable, ∪ represents the new data D k Insert into the target node and update the SliceTable status to "written": W k =1, where W k is the instance data D k The write status, 1 means the write is successful;

[0088] If R k =1, indicating a conflict, a local lock operation is performed on the target instance. The locking mechanism is:

[0089] For target instance I k Locking is performed to prevent other threads from operating during data writing. The locking formula is:

[0090] L(S(I k ))=1, where L(S(I k )) is the lock flag of the SliceTable node. 1 means it is locked. In the locked state, the data D kWrite to the target SliceTable node, the writing formula is: S(I k )=S(I k )∪{D k}, after writing is completed, release the lock: L(S(I k ))=0;

[0091] Write completion mark, same as lock-free write operation, mark write status as W k =1;

[0092] Output write completion flag and updated SliceTable, the output result is O k = {W k ,S(I k )}, where O k The write result set includes the write status and the updated SliceTable.

[0093] As a preferred solution of the database lock method based on data instance described in the present invention, the step of the writing thread updating the SliceTable linked list structure is:

[0094] The write thread extracts the instance data from the task, finds the corresponding SliceTable linked list based on the instance ID, and finds the insertion position in the target SliceTable linked list based on the timestamp in the instance data:

[0095] If the timestamp is newer than all the nodes in the linked list, it is inserted at the end of the linked list.

[0096] If the timestamp is older than all the nodes in the linked list, insert it to the head of the linked list.

[0097] Otherwise, insert into the middle node of the linked list in chronological order;

[0098] Insert a new node. The insertion process is:

[0099] Create a new linked list node to store the current instance data.

[0100] Update the front and back pointers of the node. The update method is:

[0101] If inserted at the head of the linked list, update the pointer of the head of the linked list.

[0102] If inserted at the end of the linked list, update the pointer at the end of the linked list.

[0103] If the insertion is in the middle, adjust the pointers of the previous and next nodes;

[0104] After the insertion operation, check whether the front and back pointers of the linked list are correctly connected, record the updated linked list status to the storage module, and return the status information of successful update.

[0105] As a preferred solution of the database lock method based on data instance described in the present invention, the step of using the hot-cold migration mechanism to store hot data and cold data in layers is as follows:

[0106] The access frequency of each instance data is defined as the access frequency of each instance data in the time window [t 1 , t 2 ], the formula is:

[0107]

[0108] Among them, F k is the instance data D k The access frequency, A k (t) is whether the instance data D is accessed at time t k , the value is 0 for not visited or 1 for visited, t 1 , t 2 The start and end time of the time window for counting access frequency;

[0109] Define the cold and hot threshold T of access frequency cold , if F k >T cold , then it is judged as hot data. If F k ≤T cold , then it is judged as cold data. If F k >T cold , then D k ∈Hot, if F k ≤T cold , then D k ∈Cold, where Hot is the hot data set and Cold is the cold data set;

[0110] Hot Data D k ∈Hot is stored in high-speed storage media, and the storage area is a high-speed storage module.

[0111] Cold Data k ∈Cold is migrated to the low-speed storage medium storage area as a low-speed storage module;

[0112] The storage module records the migration locations of hot data and cold data. k ∈Hot, then Storage(D k )=HighSpeed,if D k ∈Cold, then Storage(D k)=LowSpeed,where Storage(D k ) is the instance data D k The storage location of HighSpeed ​​is the high-speed storage area, and LowSpeed ​​is the low-speed storage area;

[0113] At a fixed time interval T update Recalculate the access frequency F of all data within k ,

[0114] If the proportion of high-frequency access data to the total storage exceeds the set range, the hot and cold boundary threshold T is dynamically adjusted. cold , the adjustment formula is:

[0115] T cold,new =T cold,old +ΔT,

[0116] Among them, T cold,new is the updated cold-hot boundary threshold, T cold,old is the previous demarcation threshold, ΔT is the adjustment factor; according to the updated hot and cold classification results, the hot data and cold data are migrated to the corresponding storage areas.

[0117] The beneficial effects of the present invention are as follows: the present invention adopts hash distribution based on instance ID combined with dynamic adjustment of task queue length, and the distribution thread can achieve load balancing of write threads and avoid delays caused by uneven task distribution; at the same time, a dynamic thread pool expansion mechanism is introduced to automatically adjust the number of threads according to the real-time load, thereby improving processing capacity under high load, saving resources under low load, and significantly enhancing the adaptability of the device.

[0118] The present invention introduces lock-free writing and local locking mechanisms, which solves the dilemma of granularity selection in traditional locking methods. It directly writes in conflict-free scenarios to avoid the performance overhead of lock operations. When there is a conflict, only the target instance is locked to limit the scope of influence of the lock and improve the overall concurrency efficiency. In addition, the dynamic control of the lock state can effectively ensure data consistency and reduce the impact of the globalization of traditional locks.

[0119] The present invention adopts the chain storage structure of SliceTable, which provides dynamic scalability and efficient and orderly storage capabilities for time series data; inserts new data nodes in chronological order, and the chain table is updated efficiently and supports distributed storage requirements, which solves the problem of insufficient scalability of traditional storage methods; at the same time, a hot and cold data tiered storage mechanism is adopted to distinguish hot data from cold data based on dynamic access frequency, and high-frequency data is preferentially stored in high-speed media, which greatly improves access performance, and low-frequency data is migrated to low-cost media to optimize resource allocation.

[0120] In summary, this solution comprehensively optimizes the limitations of traditional methods in lock granularity selection, load adaptability, and storage efficiency, and provides dynamic adjustment and efficient storage mechanisms for high-concurrency distributed scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0122] Figure 1 It is a SliceTable chain storage structure diagram based on an example of the present invention;

[0123] Figure 2 An example-based task distribution flow chart for the present invention;

[0124] Figure 3 A flow chart is written for the example-based tasks of the present invention. DETAILED DESCRIPTION

[0125] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0126] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0127] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0128] Example 1, reference Figure 1 , Figure 2 and Figure 3 , this embodiment provides a database lock device based on a data instance, including:

[0129] Distribute threads, distribute write tasks according to the hash value of the instance ID in the instance data key, and dynamically adjust the allocation weight of the write thread;

[0130] The steps of distributing write tasks according to the hash value of the instance ID in the instance data key value and dynamically adjusting the allocation weight of the write thread are as follows:

[0131] Initialize the task queue and initial allocation weight for each write thread. The initialization formula is:

[0132]

[0133] in, is the initial allocation weight of thread i, N is the total number of write threads,

[0134] Initialize the task queue length of each writer thread to zero: in, is the task queue length of thread i at the beginning;

[0135] Assume that the instance data key value is K(I,T,V), and perform hash calculation on the instance IDI of the instance data key value. The calculation formula is: H(I)=Hash(I)modN, where H(I) is the hash calculation result of the instance IDI, indicating the index of the initial distribution target thread, Hash(I) is the hash function calculation of the instance IDI, and N is the total number of write threads. According to the result of H(I), add the task K(I,T,V) to the queue of the target thread. The addition formula is:

[0136]

[0137] in, The length of the task queue for the initial distribution target thread is increased by 1 after hash distribution;

[0138] The allocation weight is adjusted in real time according to the current task queue length of the write thread. The adjustment formula is:

[0139]

[0140] Among them, W i (t) is the dynamic allocation weight of thread i at time t, Q i (t) is the current task queue length of thread i, α is the adjustment factor, 0<α≤1, ∈ is the adjustment factor;

[0141] For each task, combined with the remaining processing capacity R of the write thread i (t) and weight W i (t) Select the target thread. The selection formula is:

[0142]

[0143] Among them, T i Write thread for the selected target, R i(t) is the remaining processing capacity of thread i, defined as:

[0144] R i (t) = C i -Q i (t), where C i is the maximum processing capacity of thread i, Q i (t) is the current task queue length of thread i;

[0145] Assign tasks to the selected thread T i After that, update the task queue length, the update formula is:

[0146]

[0147] in, For the selected thread T i The task queue length updated after the task is assigned, For the selected thread T i The length of the task queue at the current time;

[0148] Specifically, by calculating the task queue length and weight in real time, the distribution strategy is dynamically adjusted to ensure load balancing of the write thread. At the same time, the remaining processing capacity is calculated and initially hashed and distributed, combining the current queue length and the maximum processing capacity of the thread to improve distribution efficiency.

[0149] The instance data key is a set of identifiers used to uniquely identify a data instance, consisting of an instance ID, a timestamp, and a data value;

[0150] Different devices or data instances are distinguished by instance IDs, the time point when specific data is written is marked by timestamps, and the actual content of the storage is represented by data values;

[0151] The write thread group receives the write tasks of the distribution thread, executes the write operation of the instance data, and detects read-write conflicts:

[0152] If there is no conflict, write directly;

[0153] If there is a conflict, the instance ID is locked before writing;

[0154] The write thread group is a thread collection consisting of multiple write threads responsible for performing data writing operations. Dynamic thread pool expansion is performed in the write thread group:

[0155] Automatically adjust the size of the write thread pool based on the load: reduce the number of threads when the load is low, and increase the number of threads when the load is high;

[0156] The write thread is responsible for receiving data tasks from the distribution thread, detecting whether there is a read-write conflict in the instance data, and performing lock-free or locked write operations on the instance data as needed;

[0157] The steps for dynamic thread pool expansion in the write thread group are:

[0158] Initialize the initial number of threads in the write thread pool. The initialization formula is: N 0 = InitialThreads, where N 0 is the initial number of write threads, InitialThreads is the number of initially configured write threads,

[0159] Set the high load and low load thresholds to

[0160] L high and L low ,

[0161] Among them, L high is the high load threshold. When the load exceeds this value, thread expansion is triggered. low is the low load threshold. When the load is lower than this value, thread reduction is triggered.

[0162] Define the load U(t) as the average task queue length of all current threads, the formula is:

[0163]

[0164] Among them, U(t) is the average load at time t, N(t) is the current number of threads at time t, Q i (t) is the task queue length of thread i at time t;

[0165] If U(t)>L high , then increase the number of threads,

[0166] If U(t)<L low , then reduce the number of threads,

[0167] If L low ≤U(t)≤L high , then keep the number of threads unchanged;

[0168] When additional threads are needed, the number of threads after expansion is calculated using the formula:

[0169] N(t+1)=N(t)+ΔN add ,

[0170] Among them, N(t+1) is the number of threads in the adjusted thread pool, N(t) is the number of threads in the current thread pool, ΔN add The number of threads for each expansion is configured as a fixed value;

[0171] When threads need to be reduced, the number of threads after reduction is calculated using the following formula:

[0172] N(t+1)=max(N min , N(t)-ΔN reduce ),

[0173] Among them, max ensures that the number of threads is not less than the minimum thread limit N min , ΔN reduce The number of threads to be reduced each time;

[0174] Initialize the task queue for the newly added thread. The initialization queue formula is:

[0175] Q j (t+1)=0forj∈[N(t)+1, N(t+1)],

[0176] Among them, Q j (t+1) is the task queue length of the new thread j, initialized to 0,

[0177] Evenly distribute tasks to all threads in the current thread pool. The distribution formula is:

[0178]

[0179] Among them, Q i (t+1) is the length of the task queue reallocated to thread i after adjusting the thread pool. The total number of tasks for all current threads;

[0180] Specifically, by calculating the average load in real time, the thread pool expansion is closely integrated with actual needs. After adjusting the thread pool, the load balance among threads is ensured by averaging the task queues. The expansion / reduction range is combined with the high-load and low-load threshold settings to avoid frequent adjustments to the thread pool.

[0181] The storage module organizes data storage by instance, supports dynamic expansion, and links data nodes in chronological order to form a chain storage structure; the storage module contains multiple data units SliceTable;

[0182] In the data unit SliceTable, SliceTable is the basic unit of the storage module, similar to a small time series database table; each SliceTable is responsible for storing the data of an instance in a specific time period, including the instance data content and the previous and next node pointers (Prev and Next);

[0183] This embodiment also provides a database lock method based on a data instance, including:

[0184] Step S1, the distribution thread generates a specific write task queue according to the hash value of the instance ID, and distributes the data task to the corresponding write thread;

[0185] Step S2, the write thread receives the task, extracts instance data from the task, detects whether there is a read-write conflict, and outputs the conflict detection result; instance data is the state or data information reported by a single instance (IoT device, sensor, etc.) at a specific time point, and the instance data consists of instance ID, timestamp and data value;

[0186] The writing thread receives the task, extracts the instance data from the task, detects whether there is a read-write conflict, and outputs the conflict detection result in the following steps:

[0187] The write thread extracts instance data D from the received task queue k , the instance data format is:

[0188] D k = {I k , T k , V k},

[0189] Among them, D k is the instance data in the kth task, I k is the instance ID, T k is the timestamp, V k is the data value; maps the instance data to the corresponding SliceTable node S(I k ), this node is instance I k The storage location of S(I k )=Locate(I k ), where S(I k ) is Example I k In the target SliceTable node in the storage module, Locate(I k ) is a positioning function used to determine its storage location according to the instance ID;

[0190] The write thread checks the target SliceTable node S(I k ) is in a read-write conflict state:

[0191] Define the read-write conflict flag as Conflict(S(I k )):

[0192] If S(I k ) is being read or written by other threads, then Conflict(S(I k ))=1,

[0193] If no other thread operates S(Ik ), then Conflict(S(I k ))=0, where Conflict(S(I k )) is the conflict flag of the current target SliceTable node, S(I k ) is the target SliceTable node;

[0194] If the SliceTable node is locked, then L(S(I k ))=1, if the SliceTable node is not locked, then L(S(I k ))=0, where L(S(I k )) is the SliceTable node S(I k )'s lock status;

[0195] Combine the lock status and conflict flag to update the conflict detection result. According to the detection result, output the conflict detection status of the instance data. The status formula is: R k =Conflict(S(I k )), where R k is the conflict detection result of the kth task, with a value of 0 or 1. k )) is the conflict flag for detection;

[0196] The writing thread passes the detection results to the subsequent steps.

[0197] If R k =0, indicating no conflict, directly enter the lock-free write,

[0198] If R k =1, indicating that there is a conflict and locking is required;

[0199] Specifically, a combination of conflict flags and lock states is used to improve the accuracy of conflict detection; the mapping function provides the ability to quickly access the target SliceTable node, reducing query latency.

[0200] Step S3, updating the write status according to the conflict detection result;

[0201] If there is no conflict, the lock-free write operation is performed directly.

[0202] If there is a conflict, the local lock mechanism is called to lock the instance data and write it;

[0203] The lock-free writing and local lock mechanism in step S3 is specifically as follows:

[0204] When there is no read-write conflict, directly update SliceTable;

[0205] In case of read-write conflict, only the conflicting instance ID is locked;

[0206] The lock-free write and local lock mechanisms are as follows:

[0207] Input instance data and conflict detection results D k = {I k , T k , V k}, input the conflict detection result R k =Conflict(S(I k ));

[0208] If R k = 0, indicating no conflict, directly executing lock-free write, and writing the instance data D k Write the corresponding SliceTable node S(I k ), the formula is: S(I k )=S(I k )∪{D k}, where S(I k ) is Example I k The storage node in SliceTable, ∪ represents the new data D k Insert into the target node and update the SliceTable status to "written": W k =1, where W k is the instance data D k The write status, 1 means the write is successful;

[0209] If R k =1, indicating a conflict, a local lock operation is performed on the target instance. The locking mechanism is:

[0210] For target instance I k Locking is performed to prevent other threads from operating during data writing. The locking formula is:

[0211] L(S(I k ))=1, where L(S(I k )) is the lock flag of the SliceTable node. 1 means it is locked. In the locked state, the data D k Write to the target SliceTable node, the writing formula is: S(I k )=S(I k )∪{D k}, after writing is completed, release the lock: L(S(I k ))=0;

[0212] Write completion mark, same as lock-free write operation, mark write status as W k =1;

[0213] Output write completion flag and updated SliceTable, the output result is O k = {W k ,S(I k )}, where O k Write result set, including write status and updated SliceTable;

[0214] Specifically, direct writing in non-conflicting scenarios improves writing efficiency and avoids the overhead caused by lock operations; only the conflicting instances are locked to avoid the impact of global locks on performance; the lock flag is used to dynamically control the writing status to ensure data consistency and writing order. This mechanism can effectively improve writing performance in high-concurrency scenarios while ensuring data consistency and integrity.

[0215] Step S4, the write thread updates the SliceTable linked list structure, inserts the new instance data in chronological order, and forms an ordered storage linked list;

[0216] The steps for the write thread to update the SliceTable linked list structure are:

[0217] The write thread extracts the instance data from the task, finds the corresponding SliceTable linked list based on the instance ID, and finds the insertion position in the target SliceTable linked list based on the timestamp in the instance data:

[0218] If the timestamp is newer than all the nodes in the linked list, it is inserted at the end of the linked list.

[0219] If the timestamp is older than all the nodes in the linked list, insert it to the head of the linked list.

[0220] Otherwise, insert into the middle node of the linked list in chronological order;

[0221] Insert a new node. The insertion process is:

[0222] Create a new linked list node to store the current instance data.

[0223] Update the front and back pointers of the node. The update method is:

[0224] If inserted at the head of the linked list, update the pointer of the head of the linked list.

[0225] If inserted at the end of the linked list, update the pointer at the end of the linked list.

[0226] If the insertion is in the middle, adjust the pointers of the previous and next nodes;

[0227] After the insertion operation, check whether the front and back pointers of the linked list are correctly connected, record the updated linked list status to the storage module, and return the status information of successful update;

[0228] Step S5, the storage module links the data nodes in chronological order and uses a hot-cold migration mechanism to store hot data and cold data in layers;

[0229] The steps for tiered storage of hot and cold data using the hot-cold migration mechanism are:

[0230] The access frequency of each instance data is defined as the access frequency of each instance data in the time window [t 1 , t 2 ], the formula is:

[0231]

[0232] Among them, F k is the instance data D k The access frequency, A k (t) is whether the instance data D is accessed at time t k , the value is 0 for not visited or 1 for visited, t 1 , t 2 The start and end time of the time window for counting access frequency;

[0233] Define the cold and hot threshold T of access frequency cold , if F k >T cold , then it is judged as hot data. If F k ≤T cold , then it is judged as cold data. If F k >T cold , then D k ∈Hot, if F k ≤T cold , then D k ∈Cold, where Hot is the hot data set and Cold is the cold data set;

[0234] Hot Data D k ∈Hot is stored in high-speed storage media, and the storage area is a high-speed storage module.

[0235] Cold Data k ∈Cold is migrated to the low-speed storage medium storage area as a low-speed storage module;

[0236] The storage module records the migration locations of hot data and cold data. k ∈Hot, then Storage(D k )=HighSpeed,if D k ∈Cold, then Storage(Dk )=LowSpeed,where Storage(D k ) is the instance data D k The storage location of HighSpeed ​​is the high-speed storage area, and LowSpeed ​​is the low-speed storage area;

[0237] At a fixed time interval T update Recalculate the access frequency F of all data k ,

[0238] If the proportion of high-frequency access data to the total storage exceeds the set range, the hot and cold boundary threshold T is dynamically adjusted. cold , the adjustment formula is:

[0239] T cold,new =T cold,old +ΔT,

[0240] Among them, T cold,new is the updated cold-hot boundary threshold, T cold,old is the previous demarcation threshold, ΔT is the adjustment factor; according to the updated hot and cold classification results, hot data and cold data are migrated to the corresponding storage areas;

[0241] Specifically, dynamic access frequency statistics are access frequency statistics based on time windows that can accurately reflect the hot and cold changes of data; and threshold adaptive adjustment enables flexible migration of hot and cold data and optimizes storage resource utilization; in addition, tiered storage is used to prioritize the storage of high-frequency access data in high-speed media to improve storage efficiency, greatly improve access performance, and save storage costs for low-frequency data.

[0242] In summary, the present invention targets the storage characteristics of distributed databases and the urban IoT data scenarios it is targeting. Urban IoT devices upload IoT device status data in a certain time period (10 seconds, 60 seconds, 120 seconds, etc.); the concurrency of IoT device instances uploading data is very large, but the same instance only uploads one piece of data within the same period; such data uploading characteristics create opportunities for us to innovate the database lock mechanism, changing the lock overhead of database tables (Table), pages (Page), and data rows (Row), and instead determining whether to lock the data instance, which greatly reduces the lock overhead. Generally, no lock is required for database writing.

[0243] At the same time, when applied to Maxtable scenarios, since Maxtable data is written sequentially and does not move data, that is, writing a new piece of data does not move the location of the old data, there is no need to lock in the case of mixed reading and writing. This significantly improves efficiency. If the industry's general methods are used without considering the characteristics of the storage system itself, writing to the database will consume a large amount of lock resources, thus affecting the quality of online services.

[0244] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A database lock device based on a data instance, characterized in that: include, Distribute threads, distribute write tasks according to the hash value of the instance ID in the instance data key, and dynamically adjust the allocation weight of the write thread; The instance data key is a set of identifiers used to uniquely identify a data instance, consisting of an instance ID, a timestamp, and a data value; Different devices or data instances are distinguished by instance IDs, the time point when specific data is written is marked by timestamps, and the actual content of the storage is represented by data values; The write thread group receives the write tasks of the distribution thread, executes the write operation of the instance data, and detects read-write conflicts: If there is no conflict, write directly; If there is a conflict, the instance ID is locked before writing; The storage module organizes data storage by instance, supports dynamic expansion, and links data nodes in chronological order to form a chain storage structure; the storage module contains multiple data units SliceTable; In the data unit SliceTable, SliceTable is the basic unit of the storage module; each SliceTable is responsible for storing the data of an instance within a specific time period, including the instance data content and the previous and next node pointers.

2. A database lock device based on data instances as claimed in claim 1, characterized in that: The steps of distributing the writing tasks according to the hash value of the instance ID in the instance data key value and dynamically adjusting the allocation weight of the writing threads are: Initialize the task queue and initial allocation weight for each write thread. The initialization formula is: in, is the initial allocation weight of thread i, N is the total number of write threads, Initialize the task queue length of each writer thread to zero: in, is the task queue length of thread i at the beginning; Assume that the instance data key value is K(I,T,V), and perform hash calculation on the instance IDI of the instance data key value. The calculation formula is: H(I)=Hash(I)modN, where H(I) is the hash calculation result of the instance IDI, indicating the index of the initial distribution target thread, Hash(I) is the hash function calculation of the instance IDI, and N is the total number of write threads. According to the result of H(I), add the task K(I,T,V) to the queue of the target thread. The addition formula is: in, The length of the task queue for the initial distribution target thread is increased by 1 after hash distribution; The allocation weight is adjusted in real time according to the current task queue length of the write thread. The adjustment formula is: Among them, W i (t) is the dynamic allocation weight of thread i at time t, Q i (t) is the current task queue length of thread i, α is the adjustment factor, 0<α≤1, ∈ is the adjustment factor; For each task, combined with the remaining processing capacity R of the write thread i (t) and weight W i (t) Select the target thread. The selection formula is: Among them, T i Write thread for the selected target, R i (t) is the remaining processing capacity of thread i, defined as: R i (t) = C i -Q i (t), where C i is the maximum processing capacity of thread i, Q i (t) is the current task queue length of thread i; Assign tasks to the selected thread T i After that, update the task queue length, the update formula is: in, For the selected thread T i The task queue length updated after the task is assigned, For the selected thread T i The length of the task queue at the current time.

3. A database lock device based on data instances as claimed in claim 2, characterized in that: The write thread group is a thread set consisting of multiple write threads responsible for executing data writing operations, and dynamic thread pool expansion is performed in the write thread group: Automatically adjust the size of the write thread pool based on the load: reduce the number of threads when the load is low, and increase the number of threads when the load is high; The write thread is responsible for receiving data tasks from the distribution thread, detecting whether there is a read-write conflict in the instance data, and performing lock-free write or locked write operations on the instance data as needed.

4. A database lock device based on data instances as claimed in claim 3, characterized in that: The steps of performing dynamic thread pool expansion in the write thread group are: Initialize the initial number of threads in the write thread pool. The initialization formula is: N0 = InitialThreads, where N0 is the initial number of write threads and InitialThreads is the number of initially configured write threads. Set the high load and low load thresholds to L high and L low , Among them, L high is the high load threshold. When the load exceeds this value, thread expansion is triggered. low is the low load threshold. When the load is lower than this value, thread reduction is triggered. Define the load U(t) as the average task queue length of all current threads, the formula is: Among them, U(t) is the average load at time t, N(t) is the current number of threads at time t, Q i (t) is the task queue length of thread i at time t; If U(t)>L high , then increase the number of threads, If U(t)<L low , then reduce the number of threads, If L low ≤U(t)≤L high , then keep the number of threads unchanged; When additional threads are needed, the number of threads after expansion is calculated using the formula: N(t+1)=N(t)+ΔN add , Among them, N(t+1) is the number of threads in the adjusted thread pool, N(t) is the number of threads in the current thread pool, ΔN add The number of threads for each expansion is configured as a fixed value; When threads need to be reduced, the number of threads after reduction is calculated using the following formula: N(t+1)=max(N min ,N(t)-ΔN reduce ), Among them, max ensures that the number of threads is not less than the minimum thread limit N min , ΔN reduce The number of threads to be reduced each time; Initialize the task queue for the newly added thread. The initialization queue formula is: Q j (t+1)=0forj∈[N(t)+1,N(t+1)], Among them, Q j (t+1) is the task queue length of the new thread j, initialized to 0, Evenly distribute tasks to all threads in the current thread pool. The distribution formula is: Among them, Q i (t+1) is the length of the task queue reallocated to thread i after adjusting the thread pool. The total number of tasks for all current threads.

5. A database locking method based on a data instance, based on a database locking device based on a data instance according to any one of claims 1 to 4, characterized in that: include: Step S1, the distribution thread generates a specific write task queue according to the hash value of the instance ID, and distributes the data task to the corresponding write thread; Step S2, the write thread receives the task, extracts instance data from the task, detects whether there is a read-write conflict, and outputs the conflict detection result; The instance data is the status or data information reported by a single instance at a specific time point, and the instance data includes an instance ID, a timestamp and a data value; Step S3, updating the write status according to the conflict detection result; If there is no conflict, the lock-free write operation is performed directly. If there is a conflict, the local lock mechanism is called to lock the instance data and write it; Step S4, the write thread updates the SliceTable linked list structure, inserts the new instance data in chronological order, and forms an ordered storage linked list; Step S5: The storage module links the data nodes in chronological order and uses a hot-cold migration mechanism to store hot data and cold data in layers.

6. A database lock method based on data instance as claimed in claim 5, characterized in that: The writing thread receives the task, extracts the instance data from the task, detects whether there is a read-write conflict, and outputs the conflict detection result in the following steps: The write thread extracts instance data D from the received task queue k , the instance data format is: D k ={I k ,T k ,V k }, Among them, D k is the instance data in the kth task, I k is the instance ID, T k is the timestamp, V k is the data value; maps the instance data to the corresponding SliceTable node S(I k ), this node is instance I k The storage location of S(I k )=Locate(I k ), where S(I k ) is Example I k In the target SliceTable node in the storage module, Locate(I k ) is a positioning function used to determine its storage location according to the instance ID; The write thread checks the target SliceTable node S(I k ) is in a read-write conflict state: Define the read-write conflict flag as Conflict(S(I k )): If S(I k ) is being read or written by other threads, then Conflict(S(I k ))=1, If no other thread operates S(I k ), then Conflict(S(I k ))=0, where Conflict(S(I k )) is the conflict flag of the current target SliceTable node, S(I k ) is the target SliceTable node; If the SliceTable node is locked, then L(S(I k ))=1, if the SliceTable node is not locked, then L(S(I k ))=0, where L(S(I k )) is the SliceTable node S(I k )'s lock status; Combine the lock status and conflict flag to update the conflict detection result. According to the detection result, output the conflict detection status of the instance data. The status formula is: R k =Conflict(S(I k )), where R k is the conflict detection result of the kth task, with a value of 0 or 1. k )) is the conflict flag for detection; The writing thread passes the detection results to the subsequent steps. If R k =0, indicating no conflict, directly enter the lock-free write, If R k =1, indicating that there is a conflict and locking is required.

7. A database lock method based on data instances as claimed in claim 6, characterized in that: The lock-free writing and local lock mechanism in step S3 is specifically as follows: When there is no read-write conflict, directly update SliceTable; When there is a read-write conflict, only the conflicting instance ID is locked.

8. A database lock method based on data instances as claimed in claim 7, characterized in that: The lock-free writing and local lock mechanism is specifically as follows: Input instance data and conflict detection results D k = {I k , T k , V k }, input the conflict detection result R k =Conflict(S(I k )); If R k = 0, indicating no conflict, directly executing lock-free write, and writing the instance data D k Write the corresponding SliceTable node S(I k ), the formula is: S(I k )=S(I k )∪{D k }, where S(I k ) is Example I k The storage node in SliceTable, ∪ represents the new data D k Insert into the target node and update the SliceTable status to "written": W k =1, where W k is the instance data D k The write status, 1 means the write is successful; If R k =1, indicating a conflict, a local lock operation is performed on the target instance. The locking mechanism is: For target instance I k Locking is performed to prevent other threads from operating during data writing. The locking formula is: L(S(I k ))=1, where L(S(I k )) is the lock flag of the SliceTable node. 1 means it is locked. In the locked state, the data D k Write to the target SliceTable node, the writing formula is: S(I k )=S(I k )∪{D k }, after writing is completed, release the lock: L(S(I k ))=0; Write completion mark, same as lock-free write operation, mark write status as W k =1; Output write completion flag and updated SliceTable, the output result is O k = {W k ,S(I k )}, where O k The write result set includes the write status and the updated SliceTable.

9. A database lock method based on data instances as claimed in claim 8, characterized in that: The steps of the write thread updating the SliceTable linked list structure are: The write thread extracts the instance data from the task, finds the corresponding SliceTable linked list based on the instance ID, and finds the insertion position in the target SliceTable linked list based on the timestamp in the instance data: If the timestamp is newer than all the nodes in the linked list, it is inserted at the end of the linked list. If the timestamp is older than all the nodes in the linked list, insert it to the head of the linked list. Otherwise, insert into the middle node of the linked list in chronological order; Insert a new node. The insertion process is: Create a new linked list node to store the current instance data. Update the front and back pointers of the node. The update method is: If inserted at the head of the linked list, update the pointer of the head of the linked list. If inserted at the end of the linked list, update the pointer at the end of the linked list. If the insertion is in the middle, adjust the pointers of the previous and next nodes; After the insertion operation, check whether the front and back pointers of the linked list are correctly connected, record the updated linked list status to the storage module, and return the status information of successful update.

10. A database lock method based on data instances as claimed in claim 9, characterized in that: The steps of using the hot-cold migration mechanism to store hot data and cold data in layers are: The access frequency of each instance data is defined as the number of accesses within the time window [t1, t2], and the formula is: Among them, F k is the instance data D k The access frequency, A k (t) is whether the instance data D is accessed at time t k , the value is 0 for not visited or 1 for visited, t1, t2 are the start and end time of the time window for statistical access frequency; Define the cold and hot threshold T of access frequency cold , if F k >T cold , then it is judged as hot data. If F k ≤T cold , then it is judged as cold data. If F k >T cold , then D k ∈Hot, if F k ≤T cold , then D k ∈Cold, where Hot is the hot data set and Cold is the cold data set; Hot Data D k ∈Hot is stored in high-speed storage media, and the storage area is a high-speed storage module. Cold Data k ∈Cold is migrated to the low-speed storage medium storage area as a low-speed storage module; The storage module records the migration locations of hot data and cold data. k ∈Hot, then Storage(D k )=HighSpeed,if D k ∈Cold, then Storage(D k )=LowSpeed,where Storage(D k ) is the instance data D k The storage location of HighSpeed ​​is the high-speed storage area, and LowSpeed ​​is the low-speed storage area; At a fixed time interval T update Recalculate the access frequency F of all data k , If the proportion of high-frequency access data to the total storage exceeds the set range, the hot and cold boundary threshold T is dynamically adjusted. cold , the adjustment formula is: T cold,new =T cold,old +ΔT, Among them, T cold,new is the updated cold-hot boundary threshold, T cold,old is the previous demarcation threshold, ΔT is the adjustment factor; according to the updated hot and cold classification results, the hot data and cold data are migrated to the corresponding storage areas.

Images