Data traversal method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a data traversal method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: receiving a first request sent by a client; the first request represents a data adding request; in response to the first request, starting a timed task cluster; the timed task cluster comprises one or more timed tasks, and the timed tasks comprise timed tasks used for data traversal; the one or more timed tasks are distributed to one or more task instance nodes through a polling scheduling strategy, so that the one or more task instance nodes execute the one or more timed tasks in series; the one or more timed tasks respectively correspond to data under different vernier values.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data traversal method, apparatus, device, storage medium, and computer program product. Background Art

[0002] Data traversal is a very common scenario in computer systems. How to traverse the data in the database quickly and accurately is the difficulty of this problem. Currently, common data traversal methods include: memory iteration, paging iteration, cursor iteration, group iteration, etc.

[0003] However, memory iteration and paging iteration themselves have slow traversal efficiency; traditional cursor iteration can only run on a single thread. Although the improved cursor iteration can achieve multi-threaded parallel operation, it either requires the query handle to be allocated in advance and maintained when data is inserted, which has a certain impact on data insertion efficiency and requires additional memory, or it requires judgment on the queries of the same cursor by different threads to filter out multiple queries, which also has low traversal efficiency; although group iteration can achieve multi-threaded traversal, if the data changes unevenly, the data allocated to each thread will also be uneven, which also has the problem of low traversal efficiency. Summary of the Invention

[0004] In order to solve the technical problems existing in the related art, the embodiments of the present application provide a data traversal method, device, equipment, storage medium and computer program product.

[0005] To achieve the above objectives, the technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a data traversal method, the method comprising:

[0007] Receive a first request sent by a client; the first request represents a request for adding new data;

[0008] In response to the first request, starting a scheduled task cluster; the scheduled task cluster includes one or more scheduled tasks, and the scheduled tasks include scheduled tasks for data traversal;

[0009] The one or more scheduled tasks are assigned to one or more task instance nodes through a polling scheduling strategy, so that the one or more task instance nodes execute the one or more scheduled tasks serially; the one or more scheduled tasks correspond to data under different cursor values ​​respectively.

[0010] In a second aspect, an embodiment of the present application further provides a data traversal device, the device comprising:

[0011] A receiving unit, configured to receive a first request sent by a client; the first request represents a request for adding new data;

[0012] A first starting unit, configured to start a scheduled task cluster in response to the first request; the scheduled task cluster includes one or more scheduled tasks, and the scheduled tasks include scheduled tasks for data traversal;

[0013] An allocation unit is used to allocate the one or more scheduled tasks to one or more task instance nodes through a polling scheduling strategy, so that the one or more task instance nodes execute the one or more scheduled tasks serially; the one or more scheduled tasks correspond to data under different cursor values ​​respectively.

[0014] In a third aspect, an embodiment of the present application further provides a data traversal device, comprising: a processor and a memory for storing a computer program capable of running on the processor;

[0015] Wherein, when the processor is used to run the computer program, it executes the steps of the data traversal method described in the embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data traversal method described in the embodiment of the present application are implemented.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the data traversal method described in the embodiment of the present application.

[0018] The data traversal method, apparatus, device, storage medium and computer program product provided in the embodiment of the present application receive a first request sent by a client; the first request represents a request for adding new data; in response to the first request, a scheduled task cluster is started; the scheduled task cluster includes one or more scheduled tasks, and the scheduled tasks include scheduled tasks for data traversal; the one or more scheduled tasks are assigned to one or more task instance nodes through a polling scheduling strategy, so that the one or more task instance nodes execute the one or more scheduled tasks serially; the one or more scheduled tasks correspond to data under different cursor values ​​respectively. The technical solution of the embodiment of the present application is adopted, and one or more scheduled tasks for data traversal are assigned to each task instance node through a polling scheduling strategy, so that the traversal process of the cursor iteration can be executed serially on different task instance nodes, and after the data traversal is completed, the related business after the data iteration is processed in parallel, which not only retains the query efficiency of the cursor iteration, but also makes up for the single-node bottleneck problem of the cursor iteration, and can greatly improve the efficiency of data traversal. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The process diagram of the data traversal method of the embodiment of the present application is as follows Figure 1 ;

[0020] Figure 2 The process diagram of the data traversal method of the embodiment of the present application is as follows Figure 2 ;

[0021] Figure 3 This is a schematic diagram of the implementation architecture of the data traversal method according to an embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of the structure of a data traversal device according to an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of the hardware composition structure of the data traversal device in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0026] Data traversal is a very common scenario in the field of distributed computing in computer systems. How to quickly and accurately traverse data in a database is a difficult problem. Currently, commonly used data traversal methods in related technologies include: memory iteration, paging iteration, cursor iteration, group iteration, etc.

[0027] In-memory iteration, all matching data in the database is retrieved at once, and then iterated in the server's memory. However, this method consumes a lot of memory and can lead to memory overflow when the data volume is large.

[0028] Pagination iteration involves querying and traversing pages one by one, following certain sorting rules and with a certain amount of data per page. While this approach ensures a constant amount of data per query, it can lead to slower query speeds for larger pages as data further back in the query progresses.

[0029] Cursor iteration involves selecting a field as the cursor, sorting by that field, and, given an initial cursor value, querying the first n records greater than or less than the cursor value. The cursor value of the last record found is then used as the new cursor value, and the query continues until no more data is found. This method ensures stable query speeds, but it relies on the cursor value of the previous query and can only be performed in a single thread. This can lead to low traversal efficiency when the data volume is large.

[0030] With the continuous development of computer technology, multi-threaded parallel processing methods have also emerged for cursor iteration, such as allocating query handles in advance, assigning a thread to each handle to traverse the handle range, and multi-threaded atomic cursor iteration, an improved method that terminates by determining if the cursor is read repeatedly.

[0031] Grouped iteration is achieved by finding the maximum and minimum values ​​of the data, then determining a step size to group the data and query the data within each group. Although this method can implement multi-threaded traversal, if the data changes unevenly, the data allocated to each thread will also be uneven.

[0032] It can be seen that the above data traversal method in the related art has the following main shortcomings:

[0033] 1. Slow iteration efficiency: Memory iteration and paging iteration inherently have slow query efficiency. Traditional cursor iteration can only run in a single thread. Although improved cursor iteration can achieve multi-threaded parallel operation, it either requires pre-allocation of query handles and maintenance during data insertion, which has a certain impact on data insertion efficiency and consumes additional memory. Or it requires judgment on queries of the same cursor by different threads to filter out multiple queries, which is also inefficient. Although group iteration can achieve multi-threaded traversal, if the data changes unevenly, the data allocated to each thread will also be uneven, resulting in low traversal efficiency.

[0034] 2. Not applicable in distributed environments: Traditional cursor iteration cannot directly decompose tasks to distributed nodes due to the sequential dependencies. Paging iteration and group iteration can theoretically decompose tasks to different nodes, but they themselves have problems with low query efficiency or uneven distribution.

[0035] 3. Poor stability: If the service crashes during data traversal, re-iteration is required.

[0036] Based on this, an embodiment of the present application proposes a data traversal method. In various embodiments of the present application, one or more timed tasks for data traversal are assigned to each task instance node by adopting a polling scheduling strategy, so that the traversal process of the cursor iteration can be executed serially on different task instance nodes, and after the data traversal is completed, the related business after the data iteration is processed in parallel, which not only retains the query efficiency of the cursor iteration, but also makes up for the single-node bottleneck problem of the cursor iteration, and can greatly improve the efficiency of data traversal.

[0037] The embodiment of the present application provides a data traversal method, which is applied to a data traversal device. Figure 1 The process diagram of the data traversal method of the embodiment of the present application is as follows Figure 1 ,like Figure 1 As shown, the data traversal method includes:

[0038] Step 101: Receive a first request sent by a client; the first request represents a request for adding new data.

[0039] Here, the client initiates a first request, which represents a data addition request. The data addition request can also be called a data storage request, or a business data storage request. The embodiment of the present application does not limit the name of the first request.

[0040] Here, the data traversal device is equipped on the server, wherein the server may include a server cluster, the server cluster includes one or more server nodes, and the one or more server nodes are used to create one or more scheduled tasks; that is, the embodiment of the present application can execute the data traversal method by one or more server nodes.

[0041] Step 102: In response to the first request, start a scheduled task cluster; the scheduled task cluster includes one or more scheduled tasks, and the scheduled tasks include a scheduled task for data traversal.

[0042] In an embodiment of the present application, the one or more scheduled tasks are distributedly deployed, and the scheduled task cluster further includes one or more task instance nodes. Task instance nodes can also be referred to as task execution nodes, that is, the scheduled tasks are executed by the task execution nodes.

[0043] Here, the scheduled task cluster may include a distributed scheduled task cluster, specifically, a Kubernetes (abbreviated as K8s) distributed scheduled task cluster.

[0044] In actual application, in one embodiment, after the data traversal device starts the scheduled task cluster, the method further includes: registering the one or more task instance nodes to a task center. The task center can be understood as a task execution module set up on the server.

[0045] Step 103: Allocate the one or more scheduled tasks to one or more task instance nodes through a polling scheduling strategy, so that the one or more task instance nodes execute the one or more scheduled tasks serially; the one or more scheduled tasks correspond to data under different cursor values ​​respectively.

[0046] Here, in actual application, after registering the one or more task instance nodes to the task center, the task center can trigger and call the task instance nodes to execute the scheduled tasks according to the execution frequency. Specifically, a round-robin strategy can be used for calling to ensure that only one scheduled task is allowed to be executed at the same time, and to ensure that the tasks of each task instance node are evenly distributed.

[0047] It should be noted that the execution frequency can be pre-configured according to actual needs. For example, the execution frequency can be pre-configured to 1 second, which is not limited in this embodiment of the present application.

[0048] Here, the one or more timed tasks correspond to data under different cursor values ​​respectively. The cursor in the embodiment of the present application is a range cursor, that is, the data under different cursor values ​​is data within a range. For example, the cursor value corresponding to the timed task T1 is cursor t1, and the data corresponding to cursor t1 is data 0-100.

[0049] In actual application, before the data traversal device allocates the one or more scheduled tasks to one or more task instance nodes through the polling scheduling strategy, it is necessary to first determine the polling scheduling strategy.

[0050] Based on this, in one embodiment, the method further includes:

[0051] Determine the cursor fields and query conditions;

[0052] The polling scheduling strategy is determined based on the cursor field and the query condition.

[0053] Here, in actual application, the cursor field is determined based on the business data in the business database. The business database can also be called a database, or a business data table, or a data table. The embodiment of this application does not limit the name of the business database.

[0054] Based on this, in one embodiment, determining the cursor field includes:

[0055] Determine business data relevance;

[0056] The cursor field is determined based on the business data relevance.

[0057] Here, the cursor field can be a single field or a combination of fields. The embodiment of the present application does not limit the type of the cursor field. In actual application, in order to improve the iteration efficiency of the iteration scenario, an index can be added to the cursor field.

[0058] Here, after determining the business data relevance, the size of the business data relevance can be sorted, and fields with smaller business data relevance can be selected as cursor fields. For example, the cursor fields can include fields such as serial number, data modification time, and data creation time.

[0059] In actual application, in one embodiment, the method further includes: determining an iteration status field; the iteration status field is used to indicate a status value of whether data has been read during this iteration.

[0060] Here, the data traversal device can distinguish whether the data has been read in an iteration process (i.e., the data traversal process) by determining the iteration status field. The iteration status field is not a business field and only changes during the traversal process. Ordinary insertion and update operations do not need to pay attention to the iteration status field.

[0061] In one embodiment, determining the query condition includes:

[0062] Determining a traversal condition, and determining a cursor condition based on the cursor field;

[0063] The query condition is determined based on a combination of the traversal condition and the cursor condition.

[0064] Here, it should be noted that if there are fields in the traversal condition that are updated as the traversal business is processed, so that the updated fields no longer meet the traversal conditions, there is no need to add traversal condition fields. Otherwise, the iteration status field needs to be added to the traversal conditions.

[0065] In actual application, in one embodiment, determining the cursor condition based on the cursor field includes:

[0066] Determine a first cursor value; the first cursor value represents the cursor value of this iteration;

[0067] A condition that the cursor field is greater than the first cursor value is determined as the cursor condition.

[0068] Specifically, in one embodiment, determining the first cursor value includes:

[0069] Query whether the elements in the ordered set are empty and obtain the first query result;

[0070] If the first query result indicates that the elements in the ordered set are non-empty, selecting a maximum cursor value from the ordered set, and determining the maximum cursor value as the first cursor value;

[0071] When the first query result indicates that the elements in the ordered set are empty, a default initial cursor value is determined as the first cursor value.

[0072] In an embodiment of the present application, the elements in the ordered set include one or more cursor values ​​for iteration; wherein, the one or more cursor values ​​for iteration are stored in a middleware cache, and the cursor values ​​can be shared among distributed scheduled task clusters to realize distributed iterative traversal and be isolated from the service to avoid cursor loss due to service downtime.

[0073] Here, a sorted set, or Zset (also known as a sorted set), is a very powerful and flexible data structure that can efficiently perform insertion, deletion, and range queries. A sorted set is a data type in the Redis (Remote Dictionary Server) database. Each element in the set is associated with a floating-point score, allowing the elements in the set to be sorted based on the score. In other words, a sorted set stores a unique, ordered collection of elements, each with a score that is used to sort the elements.

[0074] In actual application, before querying whether the elements in the ordered set are empty and obtaining the first query result, the method further includes: determining whether the task instance node thread pool is fully occupied. Accordingly, querying whether the elements in the ordered set are empty and obtaining the first query result includes: if the task instance node thread pool is not fully occupied, querying whether the elements in the ordered set are empty and obtaining the first query result.

[0075] Here, when the task instance node thread pool is fully occupied, the task center is waiting to trigger the next scheduled task to ensure that there is no risk of memory overflow due to excessive traversal of data.

[0076] Here, if the first query result indicates that the element in the ordered set is empty, it indicates that this iteration is the first iteration, and the default initial cursor value is used as the cursor value for this iteration. It should be noted that the default initial cursor value can be preset according to actual needs.

[0077] In one embodiment, allocating the one or more scheduled tasks to one or more task instance nodes through a round-robin scheduling strategy so that the one or more task instance nodes execute the one or more scheduled tasks serially includes:

[0078] The one or more scheduled tasks are assigned to one or more task instance nodes through a round-robin scheduling strategy, so that the one or more task instance nodes iteratively process the one or more scheduled tasks according to the following operations:

[0079] Based on the query condition, query the business data in the business database indicated by the cursor field to obtain a second query result;

[0080] If the second query result indicates that the business data in the business database is not empty, determining a second cursor value and writing the second cursor value into the ordered set; the second cursor value indicates the cursor value of the last business data in this iteration;

[0081] The above steps are continuously executed until the thread of the scheduled task is executed and the iterative process is completed.

[0082] Here, the data traversal device queries the business database using the query conditions, sorts the cursor field in ascending order, and retrieves the first n business data items, where n is the step size and is a positive integer greater than or equal to 1. n can be determined based on the complexity of the business executed after the data traversal. If no business data is found, it indicates that there is no latest business data to be traversed. The cursor value of the last business data item of this iteration is written to the ordered set Zset as the latest cursor value.

[0083] In one embodiment, the method further includes: after the current iteration process is completed, deleting the second cursor value from the ordered set.

[0084] In actual application, the data traversal device can use a thread pool to execute business processing after iteration, without waiting for the business processing to be completed before starting the next iteration process, thereby speeding up the iteration speed and realizing the decoupling of traversal query and business processing after traversal.

[0085] Based on this, in one embodiment, the method further includes:

[0086] Select a target thread and start the target thread;

[0087] The target thread is used to asynchronously process the related services after this iteration.

[0088] Here, the data traversal device can select an idle thread from the task instance node thread pool as the target thread, start the target thread, and use the target thread to asynchronously process related services after this iteration. In actual application, the data traversal device can also add the cursor value of this iteration (i.e., the first cursor value) to the thread name of the target thread, so that the target thread is bound to the cursor value of this iteration.

[0089] By adopting the technical solution of the embodiment of the present application, one or more timed tasks for data traversal are assigned to each task instance node through a polling scheduling strategy, so that the traversal process of the cursor iteration can be executed serially on different task instance nodes, and after the data traversal is completed, the related business after the data iteration is processed in parallel, which not only retains the query efficiency of the cursor iteration, but also makes up for the single-node bottleneck problem of the cursor iteration, and can greatly improve the efficiency of data traversal.

[0090] The present application is described below in conjunction with application examples.

[0091] The solution of this application is mainly based on cursor iteration, combining technologies such as scheduled tasks, registration centers, ordered sets, and thread pools, so that cursor iteration can be executed on different nodes (i.e., task instance nodes) and related businesses after data iteration can be processed in parallel after completing data traversal. This not only retains the query efficiency of cursor iteration, but also makes up for the single-node bottleneck problem of cursor iteration, which can greatly improve data traversal efficiency. In addition, the cursor value is stored in the middleware cache and can be shared on different nodes. When the service goes down, the cursor value of the last iteration can still be saved without the need for re-traversal.

[0092] Figure 2 The process diagram of the data traversal method of the embodiment of the present application is as follows Figure 2 ,like Figure 2 As shown, the data traversal method specifically includes the following steps:

[0093] Step 1: Create relevant business data tables (i.e. business databases) and determine the cursor fields.

[0094] Here, the cursor field can be a single field or a composite field, and an index is added to the cursor field. Adding an index is not a necessary operation and only speeds up iteration efficiency for common iteration scenarios.

[0095] It should be noted that the cursor field can be selected as a field with less business relevance, such as the serial number, data modification time, and other fields.

[0096] In practical applications, you can also determine the iteration status field, which can be used to distinguish whether the data has been read in a traversal. The iteration status field is not a business field and only changes during the traversal process. It does not need to be paid attention to during normal insert and update operations.

[0097] Step 2: The business storage request enters the system and saves the business data into the business database.

[0098] It should be noted that steps 1 and 2 are Figure 2 Not shown in .

[0099] Step 3: Start the scheduled task cluster and register the scheduled task instance (corresponding to the task instance node) to the task center. The task center calls the scheduled task instance according to the configured execution frequency to execute the scheduled task, that is, the distributed scheduled task starts polling. Specifically, the round-robin + single-machine serial load balancing method can be used to call it, ensuring that only one scheduled task is allowed to be executed at the same time and that the tasks of each scheduled task instance are evenly distributed. The specific steps of polling scheduling are as follows:

[0100] Step 3.1: Determine whether the node business thread pool is full (that is, determine whether the node business thread pool is fully occupied). If not, continue to step 3.2. Otherwise, the execution ends and waits for the task center to trigger the next task to ensure that there is no risk of memory overflow due to excessive traversal of data.

[0101] Step 3.2: Query whether the elements in the ordered set Zset in the middleware cache are empty. If the ordered set is not empty, pop up the last element in the ordered set Zset, that is, the largest cursor value, as the cursor value of this iteration (corresponding to the first cursor value); if the ordered set is empty, it means that this iteration is the first iteration, and the preset default start cursor is used as the cursor value of this iteration.

[0102] Here, since the non-maximum cursor values ​​have been processed by other nodes or other threads of the node, iteration can only continue from the maximum cursor value backward.

[0103] Step 3.3: Determine the query conditions.

[0104] Here, the traversal condition (corresponding to the traversal condition) and the condition that the cursor field is greater than the current cursor value (corresponding to the cursor condition) are used as query conditions. The database is queried, sorted in ascending order by the cursor field, and the first n pieces of data are found. n is the step size, which depends on the complexity of the business executed after the data traversal. If no data is found, it means that there is no latest data to traverse, and the process returns to step 3.

[0105] If a field in the traversal condition is updated as the traversal business is processed, so that the updated field no longer meets the traversal condition, there is no need to add a traversal condition field. Otherwise, the iteration status field needs to be added to the traversal condition.

[0106] Generally speaking, cursor fields should not be used as condition fields for traversal. If the business needs to use cursor fields as conditions for query, unless the cursor field in the query condition is a range condition and the range is large enough, otherwise, the cursor field needs to be replaced.

[0107] Step 3.4: Write the cursor value of the last data item (corresponding to the second cursor value) into the ordered set Zset, which is the latest cursor value.

[0108] Step 3.5: Take a thread (corresponding to the target thread) from the thread pool of the node, start the thread to asynchronously process the related business of the query result of this iteration, and add the cursor value of this iteration to the thread name of the thread to bind the thread to the cursor.

[0109] Step 3.6: After the main thread of the scheduled task is executed, it waits for the task center to continue triggering the next round of tasks, and the cycle repeats, returning to step 3.

[0110] In this way, while this node processes the current batch of business data, other nodes can simultaneously iterate and process the next batch of business data, achieving dual parallel processing of multi-threaded parallel processing and multi-node parallel processing. The more complex the business processing process, the more obvious the advantage of parallel processing.

[0111] Step 4: After the asynchronous thread completes processing the data in step 3.5, if there is an iteration status field, the iteration status field is updated so that the data in the iteration status field no longer meets the traversal condition and no longer appears in subsequent traversals, ensuring that each piece of data is traversed only once.

[0112] Step 5: Delete the cursor value of the last data in this iteration from the ordered set Zset.

[0113] At this point, the asynchronous thread processing ends, the thread is recycled, and the thread is unbound from the cursor.

[0114] The following is an example of an implementation of the data traversal method of the present application, with reference to an application scenario.

[0115] Suppose a carrier issues a new batch of cards and needs to assign a specific label to them. The logic for determining which label a card belongs to is complex, necessitating regular traversal of newly issued cards and labeling them. Following the aforementioned data traversal method, newly issued card information is written to the database with a creation time field, denoted as t. The creation time field is used as a cursor field, and the query condition for cursor iteration is whether the label field is empty. Since the label field is populated during the iteration process and can be used to distinguish whether a card has been traversed, the iteration status field is no longer used.

[0116] Figure 3 This is a schematic diagram of the implementation architecture of the data traversal method of the embodiment of the present application, such as Figure 3 As shown in the figure, assuming the default start cursor is one month before the current time, denoted as t0, the scheduled task cluster is started with an execution frequency of 1 second, an iteration step of 100, and a query condition of cards with empty tags. When a card is added, scheduled task T1 will query the first 100 cards with empty tags and t> t0, and write the creation time t1 of the last card to the sorted set.

[0117] The node where scheduled task T1 resides obtains an idle thread from thread pool P1 to perform the tagging task on the 100 cards, and the scheduled task's main thread terminates. After waiting for the asynchronous thread to complete the tagging operation, the tag field of the 100 cards in the database is no longer empty. The creation time t1 of the last card is then deleted from the sorted set. At this point, even if there are cursor values ​​in the sorted set that are less than t1, duplicate traversal will not occur because these cards no longer meet the traversal criteria after the tag field is no longer empty.

[0118] After the main thread of scheduled task T1 ends, the task center continues to trigger the node of scheduled task T2 to execute scheduled task T2. The above process is carried out in sequence and repeatedly. In this way, whenever new cards enter the database, the scheduled task can quickly traverse these cards and mark them to ensure that there is no duplication or omission.

[0119] The key technical points of the solution of this application are:

[0120] 1. To address the problem that existing cursor iteration cannot run in parallel with multiple threads and needs to be re-iterated when the service is down, this application proposes a new cursor iteration method. This method uses a K8s distributed timed task cluster, registers the nodes in the distributed timed task cluster with the task center, and uses a round-robin strategy to assign the scheduled tasks of data traversal to each node, so that the traversal process can be executed on multiple nodes;

[0121] 2. Use the Zset data structure cached by Redis middleware to store iterative cursors, which can be shared among distributed scheduled task clusters to implement distributed iterative traversal and isolated from the service to avoid cursor loss due to service downtime;

[0122] 3. Use the thread pool to execute the business processing after the iteration, without waiting for the business processing to be completed before starting the next iteration, speeding up the iteration speed and realizing the decoupling of the traversal query and the business processing after the traversal.

[0123] Compared with the solutions of the related art, the solution of this application has the following beneficial effects:

[0124] 1) It can be applied to distributed systems, parallel computing, and improved traversal efficiency. This application uses middleware to cache and store cursor values, so that each node in the distributed system can share the cursor, thereby iterating at each node, greatly improving parallel efficiency.

[0125] 2) Decoupling of traversal and business processing: This application uses asynchronous threads to process business, and can continue to the next round of traversal without waiting for data processing to complete. Therefore, the iterative process is not strictly sequential.

[0126] 3) High fault tolerance. Even if a node crashes, the cursor is still stored in the middleware cache, and iteration can continue. The data that crashed can be processed again in the next round of traversal.

[0127] This application can be applied to any scenario that requires data traversal, especially scenarios that require real-time continuous traversal and polling, such as order retry, asynchronous processing requests, full data cache loading, etc. For example, in a high-concurrency scenario, the system call data volume is large and the business processing is complex. In this case, the call information can be stored in the database table first, and then the unprocessed call requests can be traversed through the method of this application to achieve asynchronous processing. Especially in a distributed system, the efficiency of traversal processing can be greatly improved.

[0128] In order to implement the data traversal method of the embodiment of the present application, the embodiment of the present application also provides a data traversal device, Figure 4 This is a schematic diagram of the structure of the data traversal device according to an embodiment of the present application. Figure 4 As shown, the data traversal device includes:

[0129] The receiving unit 41 is configured to receive a first request sent by a client; the first request represents a request for adding new data;

[0130] A first starting unit 42 is configured to start a scheduled task cluster in response to the first request; the scheduled task cluster includes one or more scheduled tasks, and the scheduled tasks include a scheduled task for data traversal;

[0131] The allocation unit 43 is used to allocate the one or more scheduled tasks to one or more task instance nodes through a polling scheduling strategy, so that the one or more task instance nodes execute the one or more scheduled tasks serially; the one or more scheduled tasks correspond to data under different cursor values ​​respectively.

[0132] In one embodiment, the one or more scheduled tasks are distributedly deployed, and the scheduled task cluster further includes one or more task instance nodes; the device further includes: a registration unit; wherein,

[0133] The registration unit is used to register the one or more task instance nodes with the task center.

[0134] In one embodiment, the apparatus further includes: a first determining unit and a second determining unit; wherein,

[0135] The first determining unit is used to determine the cursor field and the query condition;

[0136] The second determining unit is configured to determine the polling scheduling strategy based on the cursor field and the query condition.

[0137] In one embodiment, the first determining unit is specifically configured to:

[0138] Determine business data relevance;

[0139] The cursor field is determined based on the business data relevance.

[0140] In one embodiment, the first determining unit includes a first determining subunit, a second determining subunit and a third determining subunit; wherein,

[0141] The first determining subunit is used to determine the traversal condition;

[0142] The second determining subunit is configured to determine a cursor condition based on the cursor field;

[0143] The third determining subunit is configured to determine the query condition based on a combination of the traversal condition and the cursor condition.

[0144] In one embodiment, the second determining subunit includes a fourth determining subunit and a fifth determining subunit; wherein,

[0145] The fourth determining subunit is configured to determine a first vernier value; the first vernier value represents the vernier value of this iteration;

[0146] The fifth determining subunit is configured to determine the condition that the cursor field is greater than the first cursor value as the cursor condition.

[0147] In one embodiment, the fourth determining subunit is specifically configured to:

[0148] Query whether the elements in the ordered set are empty and obtain the first query result;

[0149] If the first query result indicates that the elements in the ordered set are non-empty, selecting a maximum cursor value from the ordered set, and determining the maximum cursor value as the first cursor value;

[0150] When the first query result indicates that the elements in the ordered set are empty, a default initial cursor value is determined as the first cursor value.

[0151] In one embodiment, the elements in the ordered set include one or more cursor values ​​for iteration;

[0152] The one or more cursor values ​​used for iteration are stored in a middleware cache.

[0153] In one embodiment, the allocation unit 43 is specifically configured to:

[0154] The one or more scheduled tasks are assigned to one or more task instance nodes through a round-robin scheduling strategy, so that the one or more task instance nodes iteratively process the one or more scheduled tasks according to the following operations:

[0155] Based on the query condition, query the business data in the business database indicated by the cursor field to obtain a second query result;

[0156] If the second query result indicates that the business data in the business database is not empty, determining a second cursor value and writing the second cursor value into the ordered set; the second cursor value indicates the cursor value of the last business data in this iteration;

[0157] The above steps are continuously executed until the thread of the scheduled task is executed and the iterative process is completed.

[0158] In one embodiment, the apparatus further includes: a deletion unit; wherein,

[0159] The deleting unit is used to delete the second cursor value from the ordered set after the current iteration process is completed.

[0160] In one embodiment, the device further includes: a selection unit, a second starting unit, and an asynchronous processing unit; wherein,

[0161] The selection unit is used to select a target thread;

[0162] The second starting unit is used to start the target thread;

[0163] The asynchronous processing unit is used to asynchronously process related services after this iteration through the target thread.

[0164] In actual application, the receiving unit 41 can be implemented by a communication interface in the data traversal device; the first starting unit 42 and the allocation unit 43 can be implemented by a processor in the data traversal device.

[0165] It should be noted that the data traversal device provided in the above embodiment only uses the division of the above program modules as an example to illustrate data traversal. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the above-described processing. In addition, the data traversal device provided in the above embodiment and the data traversal method embodiment are based on the same concept. The specific implementation process is detailed in the data traversal method embodiment and will not be repeated here.

[0166] Based on the hardware implementation of the above program modules, and in order to implement the data traversal method of the embodiment of the present application, the embodiment of the present application also provides a data traversal device, Figure 5 This is a schematic diagram of the hardware structure of the data traversal device according to an embodiment of the present application. Figure 5 As shown, the data traversal device 50 includes:

[0167] Communication interface 51, capable of exchanging information with other devices (such as clients);

[0168] The processor 52 is connected to the communication interface 51 to implement information interaction with other devices and is used to execute the above-mentioned data traversal method when running a computer program, and the computer program is stored in the memory 53.

[0169] It should be noted that the specific processing process of the communication interface 51 and the processor 52 can be understood by referring to the above-mentioned data traversal method.

[0170] Of course, in actual application, the various components in the data traversal device 50 are coupled together through the bus system 54. It is understood that the bus system 54 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 54 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 54 .

[0171] The memory 53 in the embodiment of the present application is used to store various types of data to support the operation of the data traversal device 50. Examples of such data include: any computer program used to operate on the data traversal device 50.

[0172] The data traversal method disclosed in the above-mentioned embodiment of the present application can be applied to the processor 52 or implemented by the processor 52. The processor 52 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-mentioned data traversal method can be completed by the hardware integrated logic circuit or software instructions in the processor 52. The above-mentioned processor 52 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 52 can implement or execute the various data traversal methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the data traversal method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 53. The processor 52 reads the information in the memory 53 and completes the steps of the above-mentioned data traversal method in combination with its hardware.

[0173] In an exemplary embodiment, the data traversal device 50 can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned data traversal method.

[0174] It can be understood that the memory 53 of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 53 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memory.

[0175] In an exemplary embodiment, the present application further provides a storage medium, namely, a computer storage medium, specifically, a computer-readable storage medium, such as a memory 53 storing a computer program. The computer program can be executed by a processor 52 in a data traversal device 50 to complete the steps of the data traversal method described in the aforementioned embodiment of the present application. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0176] In an exemplary embodiment, the present application also provides a computer program product, including a computer program, which can be executed by the processor 52 in the data traversal device 50 to complete the steps of the data traversal method described in the aforementioned embodiment of the present application.

[0177] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0178] In addition, the technical solutions described in the embodiments of the present application can be arbitrarily combined without conflict.

[0179] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data traversal method, characterized in that: The method comprises: Receive a first request sent by a client; the first request represents a request for adding new data; In response to the first request, starting a scheduled task cluster; the scheduled task cluster includes one or more scheduled tasks, and the scheduled tasks include scheduled tasks for data traversal; The one or more scheduled tasks are assigned to one or more task instance nodes through a polling scheduling strategy, so that the one or more task instance nodes execute the one or more scheduled tasks serially; the one or more scheduled tasks correspond to data under different cursor values ​​respectively.

2. The method according to claim 1, characterized in that The one or more scheduled tasks are distributedly deployed, and the scheduled task cluster further includes one or more task instance nodes; the method further includes: Register the one or more task instance nodes with the task center.

3. The method according to claim 1, characterized in that The method further comprises: Determine the cursor fields and query conditions; The polling scheduling strategy is determined based on the cursor field and the query condition.

4. The method according to claim 3, characterized in that Determining the cursor field includes: Determine business data relevance; The cursor field is determined based on the business data relevance.

5. The method according to claim 3, characterized in that Determining the query condition includes: Determining a traversal condition, and determining a cursor condition based on the cursor field; The query condition is determined based on a combination of the traversal condition and the cursor condition.

6. The method according to claim 5, characterized in that Determining a cursor condition based on the cursor field includes: Determine a first cursor value; the first cursor value represents the cursor value of this iteration; A condition that the cursor field is greater than the first cursor value is determined as the cursor condition.

7. The method according to claim 6, characterized in that Determining the first cursor value includes: Query whether the elements in the ordered set are empty and obtain the first query result; If the first query result indicates that the elements in the ordered set are non-empty, selecting a maximum cursor value from the ordered set, and determining the maximum cursor value as the first cursor value; When the first query result indicates that the elements in the ordered set are empty, a default initial cursor value is determined as the first cursor value.

8. The method according to claim 7, characterized in that The elements in the ordered set include one or more cursor values ​​for iteration; The one or more cursor values ​​used for iteration are stored in a middleware cache.

9. The method according to claim 1, characterized in that The step of allocating the one or more scheduled tasks to one or more task instance nodes through a round-robin scheduling strategy so that the one or more task instance nodes execute the one or more scheduled tasks serially includes: The one or more scheduled tasks are assigned to one or more task instance nodes through a round-robin scheduling strategy, so that the one or more task instance nodes iteratively process the one or more scheduled tasks according to the following operations: Based on the query condition, query the business data in the business database indicated by the cursor field to obtain a second query result; If the second query result indicates that the business data in the business database is not empty, determining a second cursor value and writing the second cursor value into the ordered set; the second cursor value indicates the cursor value of the last business data in this iteration; The above steps are continuously executed until the thread of the scheduled task is executed and the iterative process is completed.

10. The method according to claim 9, characterized in that The method further comprises: After this iteration process is completed, the second cursor value is deleted from the ordered set.

11. The method according to claim 9, characterized in that The method further comprises: Select a target thread and start the target thread; The target thread is used to asynchronously process the related services after this iteration.

12. A data traversal device, characterized in that: The device comprises: A receiving unit, configured to receive a first request sent by a client; the first request represents a request for adding new data; A first starting unit, configured to start a scheduled task cluster in response to the first request; the scheduled task cluster includes one or more scheduled tasks, and the scheduled tasks include scheduled tasks for data traversal; An allocation unit is used to allocate the one or more scheduled tasks to one or more task instance nodes through a polling scheduling strategy, so that the one or more task instance nodes execute the one or more scheduled tasks serially; the one or more scheduled tasks correspond to data under different cursor values ​​respectively.

13. A data traversal device, characterized in that: include: a processor and a memory for storing a computer program capable of running on said processor; Wherein, when the processor is used to run the computer program, it executes the steps of the method according to any one of claims 1 to 11.

14. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 11.

Images