Incremental data acquisition method and device, storage medium, and processor
By monitoring the generation of incremental data in cloud services and using a dataset cursor to find incremental data, the problem of low efficiency in preparing incremental data caused by differences in client conditions is solved, and efficient acquisition and real-time synchronization of incremental data are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2022-07-29
- Publication Date
- 2026-07-31
AI Technical Summary
In cloud service scenarios, the different conditions under which multiple clients obtain datasets mean that the server needs to provide incremental data separately for each client. This results in low efficiency in preparing incremental data, and the timing of client acquisition cannot be perfectly matched with the timing of server preparation.
When incremental data is generated by monitoring target data, a retrieval signal is sent to the client to receive the client's incremental data request. The incremental data is then searched and retrieved based on the dataset cursor. The parameters included in the dataset cursor are used to improve query efficiency and ensure that the actions of the client and the server match.
It improves the efficiency of incremental data query, ensures the real-time and accuracy of incremental data preparation, and solves the problem of low efficiency caused by the server providing incremental data separately to each client.
Smart Images
Figure CN115455110B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data synchronization, and more specifically, to an incremental data acquisition method and apparatus, a storage medium, and a processor. Background Technology
[0002] In cloud service scenarios, there is a cloud server and multiple clients. These clients request dataset synchronization from the cloud server as needed, including full synchronization or incremental synchronization. However, when there are a large number of clients, the server needs to push incremental data via message queues or long-lived connections. This approach needs to consider the different conditions each client has when obtaining the dataset, resulting in long incremental data preparation cycles and difficulty in guaranteeing the idempotency of incremental data for each client. On the other hand, the full data retrieval solution faces memory bottlenecks during client-side parsing and puts pressure on the server's memory and disk resources when querying data.
[0003] There is currently no effective solution to the problem that when a client requests data synchronization from a server, the different conditions under which multiple clients obtain the dataset cause the server to need to provide incremental data separately for each client, resulting in low efficiency in incremental data preparation. Summary of the Invention
[0004] The main purpose of this application is to provide an incremental data acquisition method and system to solve the problem in related technologies where, when a client requests data synchronization from a server, the different conditions for multiple clients to obtain datasets lead to the server needing to provide incremental data separately for each client, resulting in low efficiency in incremental data preparation.
[0005] To achieve the above objectives, according to one aspect of this application, an incremental data acquisition method is provided, comprising: upon detecting that incremental data is generated from target data, sending an incremental data acquisition signaling to a client, wherein the client is configured to request the incremental data; receiving an incremental data request from the client, wherein the incremental data request includes a cursor of the dataset from the client's previous acquisition of incremental data; responding to the incremental data request, searching for the incremental data according to the dataset cursor, and acquiring the incremental data, wherein the dataset cursor includes multiple parameter items for searching the incremental data.
[0006] Optionally, in response to the incremental data request, before searching for the incremental data according to the dataset cursor and obtaining the incremental data, the method further includes: determining whether the dataset identifier in the dataset cursor is consistent with the dataset identifier currently used by the client, wherein the incremental data request further includes: the dataset identifier currently used by the client, the dataset identifier being used to identify the modified version of the dataset; if the dataset identifier in the dataset cursor is consistent with the dataset identifier currently used by the client, initializing the offset cursor of the target data, wherein the dataset cursor further includes the offset cursor of the target data, the offset cursor being used to find the incremental data of the target data; if the dataset identifier in the dataset cursor is inconsistent with the dataset identifier currently used by the client, replacing the dataset identifier in the dataset cursor with the dataset identifier currently used by the client, and initializing the offset cursor of the target data.
[0007] Optionally, initializing the offset cursor for the target data includes: setting the search start time to the minimum creation time of the target data, setting the search end time to the current time, and determining whether the primary key identifier between the start time and the end time is null. The offset cursor includes the search start time and end time, and the search batch, which corresponds to the primary key identifier. If the primary key identifier between the start time and the end time is null, the search batch is set to the initial batch number, and the target data is searched. If the primary key identifier between the start time and the end time is not null, the batch number to be searched is determined based on the primary key identifier in the dataset cursor, and the search continues along the dimensions of the target data based on the batch number. The dataset cursor also includes the primary key identifier of the dataset containing the incremental data previously acquired by the client.
[0008] Optionally, after obtaining search data in multiple dimensions by traversing the offset cursor as incremental data, the process includes: monitoring whether new incremental data has been generated for the target data after the current time; sending the search results to the client, and sending information on whether new incremental data has been generated to the client, wherein the client is used to determine whether to continue generating the next batch of incremental data acquisition requests based on the current dataset cursor, and sending them to the server to acquire the new incremental data.
[0009] Optionally, responding to the incremental data request and searching the incremental data according to the dataset cursor to obtain the incremental data includes: responding to the incremental data request and searching the incremental data of the corresponding target data according to the offset cursor to obtain the search results for the current batch, wherein the dataset cursor includes multiple offset cursors, and the offset cursors correspond to the dimensions to which the target data belongs; determining the search data of the dimension to which the target data belongs based on the search results; and obtaining the search data of multiple dimensions by traversing the offset cursors as the incremental data.
[0010] Optionally, in response to the incremental data request, after searching the incremental data of the corresponding target data according to the offset cursor to obtain the search results of the current batch, the method further includes: determining whether the amount of data in the search results is less than the preset amount of data to be searched in each batch as set in the dataset cursor; if the amount of data is not less than the preset amount of data, continuing the search for the next batch, determining the search results for the next batch, until the amount of data is less than the preset amount of data; if the amount of data is less than the preset amount of data, the search ends, and the search data of the dimension to which the target data belongs is determined according to the search results of multiple batches.
[0011] Optionally, before initializing the offset cursor of the target data, the method further includes: serializing the dataset cursor to generate a dataset cursor object, wherein the dataset cursor object includes multiple offset cursor objects, and the offset cursor objects are used to perform initialization processing.
[0012] To achieve the above objectives, according to another aspect of this application, an incremental data acquisition apparatus is provided. The apparatus includes: a monitoring module, configured to send an incremental data acquisition signal to a client when incremental data is detected to be generated from target data, wherein the client requests the incremental data; a receiving module, configured to receive the incremental data request sent by the client, wherein the incremental data request includes a cursor of the dataset from the client's previous acquisition of incremental data; and an acquisition module, configured to respond to the incremental data request, search for the incremental data based on the dataset cursor, and acquire the incremental data, wherein the dataset cursor includes multiple parameter items for searching the incremental data.
[0013] According to another aspect of this application, a computer-readable storage medium is also provided, the storage medium being used to store a program, wherein the program executes the incremental data acquisition method described in any one of the foregoing.
[0014] According to another aspect of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the incremental data acquisition method described in any one of the foregoing.
[0015] This application achieves a solution where, upon detecting the generation of incremental data for target data, a signaling message for acquiring incremental data is sent to the client, whereby the client requests the incremental data. The server receives the incremental data request from the client, which includes a cursor of the dataset from the client's previous incremental data acquisition. In response to the incremental data request, the server searches for and acquires the incremental data based on the dataset cursor, where the dataset cursor includes multiple parameters for searching the incremental data. After detecting the generation of incremental data on the server side, a signaling message is sent to the client to notify them that incremental data has been generated. If the client needs to acquire the data, they send a data request. The server then searches for and acquires the incremental data based on the dataset cursor in the data request. This improves the efficiency of incremental data querying by utilizing the dataset cursor information. Furthermore, by triggering the incremental data acquisition process in real-time through the signaling message, the actions of the client and server are accurately matched, thereby improving the efficiency of incremental data preparation. This solves the problem in related technologies where, when a client requests data synchronization from the server, the different conditions for acquiring datasets by multiple clients lead to the server needing to provide incremental data separately for each client, and the timing of client acquisition and server preparation cannot be perfectly matched, resulting in low efficiency in incremental data preparation. Attached Figure Description
[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0017] Figure 1 This is a flowchart of an incremental data acquisition method provided according to an embodiment of this application;
[0018] Figure 2 This is a flowchart of an incremental data acquisition method provided according to an embodiment of this application;
[0019] Figure 3 This is a schematic diagram illustrating the acquisition of batches based on incremental data provided in the embodiments of this application;
[0020] Figure 4 This is a schematic diagram of an incremental data acquisition device according to an embodiment of this application;
[0021] Figure 5 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation
[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0025] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent information from the aforementioned user or organization.
[0026] Example
[0027] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of an incremental data acquisition method provided according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:
[0028] Step S101: If incremental data is detected to be generated from the target data, an incremental data acquisition signaling is sent to the client, wherein the client is used to request the incremental data.
[0029] Step S102: Receive an incremental data request from the client, wherein the incremental data request includes the cursor of the dataset from the last time the client acquired incremental data;
[0030] Step S103: Respond to the incremental data request, search for incremental data according to the dataset cursor, and obtain the incremental data. The dataset cursor includes multiple parameter items for searching for incremental data.
[0031] Through the above steps, when incremental data is detected to be generated from the target data, an incremental data retrieval signal is sent to the client, whereby the client requests the incremental data. The server receives the incremental data request from the client, which includes a cursor of the dataset from the client's previous incremental data retrieval. In response to the incremental data request, the server searches for and retrieves the incremental data based on the dataset cursor, which includes multiple parameters for searching the incremental data. After the server detects the generation of incremental data, it sends a retrieval signal to the client, notifying the client that incremental data has been generated. If the client needs to retrieve it, it sends a data request, and further searches and retrieves the incremental data based on the dataset cursor in the data request. This improves the efficiency of incremental data querying by utilizing the dataset cursor information. Furthermore, by triggering the incremental data retrieval process in real time through the retrieval signal, the server accurately matches the actions of the client and server, thereby improving the efficiency of incremental data preparation. This solves the problem in related technologies where, when a client requests data synchronization from the server, the different conditions for multiple clients retrieving datasets lead to the server needing to provide incremental data separately for each client, and the timing of client retrieval and server preparation cannot be perfectly matched, resulting in low efficiency in incremental data preparation.
[0032] The entity executing the above steps can be a server-side device, which has data processing and computation capabilities, and can be a server, processor, calculator, etc. In the cloud service scenario of this embodiment, the server-side device can provide services to multiple clients simultaneously. Multiple clients request incremental data from the server for synchronization as needed. However, due to the differences in the conditions under which multiple clients obtain datasets when requesting data synchronization from the server, the server needs to provide incremental data separately to each client. Moreover, the timing of client acquisition and the timing of server preparation cannot be perfectly matched, resulting in low efficiency in incremental data preparation.
[0033] In this embodiment, the server autonomously monitors incremental data. Upon detecting incremental data generation for the target data, it sends an incremental data retrieval signal to the client. Upon receiving this signal, the client knows that the server has generated incremental data. The client can configure the triggering method for incremental data retrieval according to user needs. It can automatically send an incremental data request to the server upon receiving the retrieval signal, thus maximizing data consistency between the client and server and ensuring real-time synchronization. Alternatively, it can determine whether to initiate an incremental data request based on requirements after receiving the retrieval signal.
[0034] The aforementioned dataset cursor can include `dataSetId`, which is the ID of the dataset from the last incremental data retrieval; `cursor_i`, which is the offset cursor i of the target data in a certain dimension; `cursorTbId`, which is the primary key cursor ID of the table within the time window; `fromTime`, which is the start time of the time window; and `toTime`, which is the end time of the time window. These parameters of the data cursor may all play a role in subsequent incremental data retrieval processes on the server side.
[0035] This embodiment, by proactively sending incremental data generation notifications to the client, can concentrate the client's acquisition of incremental data within a short period of time after the incremental data is sent, thus matching the timing of the client's request for incremental data with the timing of the server's generation of incremental data.
[0036] In step S102 above, the server receives an incremental data request from the client. This incremental data request includes a cursor of the dataset from the client's previous incremental data retrieval. This dataset cursor contains parameters for acquiring incremental data. Upon receiving the incremental data request, the server responds by searching for incremental data using the dataset cursor and retrieving the incremental data. The dataset cursor includes multiple parameter items used for searching for incremental data.
[0037] Optionally, in response to an incremental data request, the method further includes searching for incremental data based on the dataset cursor. Before retrieving the incremental data, the method also includes determining whether the dataset identifier in the dataset cursor is consistent with the dataset identifier currently used by the client. The incremental data request also includes the dataset identifier currently used by the client, which is used to identify the modified version of the dataset. The purpose is to determine whether the dataset identifier recorded in the dataset cursor is consistent with the dataset identifier currently used by the client.
[0038] If they match, it means the dataset identifier in the dataset cursor is accurate, and the dataset currently used by the client can be determined based on it. If they don't match, it means the dataset identifier in the dataset cursor is incorrect, and the dataset currently used by the client cannot be determined based on it. Therefore, in the case of inconsistency, the dataset identifier in the dataset cursor can be replaced with the dataset identifier currently used by the client as described above.
[0039] It should be noted that the step described above, which checks whether the dataset identifier in the dataset cursor matches the dataset identifier currently used by the client, requires determining the dataset identifier currently used by the client. This can be performed by the client or the server. If performed by the client, the client should request the currently used dataset identifier from the server beforehand.
[0040] If the dataset identifier in the dataset cursor is consistent with the dataset identifier currently used by the client, the offset cursor of the target data is initialized. The dataset cursor also includes the offset cursor of the target data, which is used to find the incremental data of the target data. If the dataset identifier in the dataset cursor is inconsistent with the dataset identifier currently used by the client, the dataset identifier in the dataset cursor is replaced with the dataset identifier currently used by the client, and the offset cursor of the target data is initialized.
[0041] The initialization of the offset cursor for the target data described above is to prepare for searching incremental data. Optionally, initializing the offset cursor for the target data includes: setting the search start time to the minimum creation time of the target data, setting the search end time to the current time, and determining whether the primary key identifier between the start and end times is null. Here, the offset cursor includes the search start and end times, as well as the search batch, and the batch corresponds to the primary key identifier.
[0042] If the primary key identifier between the start and end times is null, it indicates that the dataset containing the target data has not been retrieved before. In this case, all data in the dataset containing the target data should be included as incremental data in this retrieval. It should be noted that when the client requests incremental data retrieval, the client sends an incremental data request containing the maximum amount of data to be retrieved in a single batch, such as `limit`. Therefore, incremental data needs to be retrieved in multiple batches sequentially.
[0043] If the dataset containing the target data has never been retrieved since its creation, it needs to be retrieved starting from the initial batch, i.e., from the beginning. In other words, the search batch should be set to the very first batch number, and the target data should be searched for.
[0044] If the primary key is not empty, it means that data has already been retrieved from the start and end times of the dataset containing the target data. Therefore, the incremental data to be retrieved in this dataset should be the data excluding the already retrieved data. The primary key in the dataset cursor determines the batch number to be searched, identifying which batches of data have already been retrieved. Then, the search continues along the dimensions of the target data based on the batch number, starting from the batches that have already been retrieved. The dataset cursor also includes the primary key of the dataset from which the client last retrieved incremental data, i.e., dataSetId, used to identify data that has already been retrieved.
[0045] Optionally, after obtaining search data in multiple dimensions by traversing the offset cursor and using it as incremental data, the process includes: monitoring whether new incremental data has been generated for the target data after the current time; sending the search results to the client, and sending information on whether new incremental data has been generated to the client, wherein the client is used to determine whether to continue generating the next batch of incremental data acquisition requests based on the current dataset cursor, and sending them to the server to acquire new incremental data.
[0046] It should be noted that incremental data generation may be continuous. Acquiring incremental data immediately after its generation might not capture all of it, leading to a re-triggering of the acquisition signaling. This is wasteful of resources from a process perspective and also presents room for efficiency optimization. Therefore, in this embodiment, after acquiring incremental data, it automatically checks if new incremental data has been generated. If so, it sends the acquired incremental data to the client while simultaneously triggering the acquisition of new incremental data in parallel. This automatically and continuously acquires the generated incremental data, improving acquisition efficiency and optimizing the incremental data acquisition process.
[0047] Optionally, in response to an incremental data request, the incremental data is searched based on the dataset cursor to obtain the incremental data. This includes: responding to the incremental data request, searching for the incremental data of the corresponding target data based on the offset cursor to obtain the search results for the current batch. The dataset cursor includes multiple offset cursors, which correspond to the dimensions of the target data. The search data for the dimension of the target data is determined based on the search results. The search data for multiple dimensions is obtained by traversing the offset cursors and used as incremental data.
[0048] The dimensions of the data mentioned above can be understood as the type of target data. Because business systems are often complex and involve numerous business processes, the target data, as incremental data, may represent different types of data from multiple different business processes. The dimensions of the data characterize information such as the dataset or usage scenario of the target data, and its data type.
[0049] The aforementioned offset cursor can be understood as a search tool used to search within the dataset containing the target data. Combined with the search batch size and the maximum amount of data retrieved per batch, it searches the dataset, retrieving all data from the last retrieved batch onwards, and using this as incremental data for the target data.
[0050] Optionally, in response to an incremental data request, after searching the incremental data of the corresponding target data according to the offset cursor and obtaining the search results for the current batch, the method further includes: determining whether the amount of data in the search results is less than the preset amount of data to be searched in each batch as set in the dataset cursor; if the amount of data is not less than the preset amount of data, continuing the search for the next batch, determining the search results for the next batch, until the amount of data is less than the preset amount of data; if the amount of data is less than the preset amount of data, the search ends, and the search data of the dimension to which the target data belongs is determined based on the search results of multiple batches.
[0051] It should be noted that incremental data searching is performed in batches. After each batch is searched, a search result is generated, including the data found in that batch. However, the generation of incremental data may be continuous, and there is no marker indicating that the incremental data search has ended. This embodiment determines whether the incremental data search is complete based on the amount of data in the search result and the maximum amount of data acquired in a single batch.
[0052] If the data volume is not less than the preset data volume, it is usually equal to the preset data volume. This is because the data volume of the query results will not exceed the maximum data volume of a single batch. This indicates that the incremental data retrieved in this batch cannot be determined to be complete, so the search continues in the next batch to determine the search results of the next batch, until the data volume is less than the preset data volume. If the data volume is less than the preset data volume, it means that the incremental data search is complete, and the remaining data volume is insufficient to reach the maximum data volume of this batch. In this case, the search ends, and the search data of the dimension to which the target data belongs is determined based on the search results of multiple batches.
[0053] Optionally, before initializing the offset cursor of the target data, the method further includes: serializing the dataset cursor to generate a dataset cursor object, wherein the dataset cursor object includes multiple offset cursor objects, which are used to perform initialization processing.
[0054] Considering that the format of a dataset cursor is a field, which may not be suitable for the server's usage scenario, the dataset cursor can be processed appropriately so that the server can effectively use it. For example, the server can serialize the dataset cursor into a dataset cursor object, such as a JSON string saved by the client. If the backend is implemented in Java, a component that serializes JSON into an object can be considered.
[0055] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0056] It should be noted that this application also provides an optional implementation method, which will be described in detail below.
[0057] This implementation provides an incremental data acquisition scheme that supports multi-dimensional, time-range-based, primary-key cursor, sliding window, and batch acquisition. It helps ensure the real-time acquisition of incremental data by the client and improves the performance of the server in calculating incremental data. This scheme can trigger incremental updates via signaling, ensuring both real-time data acquisition and query performance.
[0058] Figure 2 This is a flowchart of the incremental data acquisition method provided according to the embodiments of this application, such as... Figure 2 As shown, the incremental data acquisition process steps are as follows.
[0059] 1) The server determines that certain dimensions have triggered the addition, update, or deletion of data records, and then sends a signaling message to obtain incremental data for the specified dimensions;
[0060] 2) The client receives the incremental data retrieval signal from the server and prepares to initiate a dataset query. The query parameter format is shown in Table 1, which is a description of the client's query parameters:
[0061] Table 1. Description of Client Query Parameters
[0062] noun explain myDataSetId Client current dataset id limit Query batch size dataSetCursor Dataset cursor, generated on the server, saved on the client.
[0063] The following is an example of the client query parameter structure:
[0064]
[0065] 3) The client passes in the dataset cursor (dataSetCursor) to query the incremental dataset. It should be noted that for the first query, or if the dataset ID (dataSetId) changes, the data offset cursor i (cursor_i) for a certain dimension is set to null.
[0066] The data structure of a dataset cursor includes: dataSetId: dataset ID (last time incremental data was retrieved); cursor_i: cursor i representing the data offset for a specific dimension; cursorTbId: primary key cursor ID within the time window; fromTime: start time of the time window; and toTime: end time of the time window. See Table 2 for details, which describes the data structure of the dataset cursor.
[0067] Table 2. Data Structure Description of Data Cursors
[0068]
[0069] The following is an example of the data structure for a dataset cursor:
[0070]
[0071] 4) The server receives the dataset query request from the client and prepares to query the data;
[0072] 5) The server serializes the dataset cursor (dataSetCursor) into a dataset cursor object (dataSetCursorObj). For example, if the client saves a JSON string, the backend is implemented in Java. Consider using a component that serializes JSON into an object. Steps 6) and 7) Execute the initialization of the properties of the dataset cursor object (dataSetCursorObj).
[0073] 6) Check if the dataset id (dataSetId) in the dataset cursor object (dataSetCursorObj) is equal to the dataset id (myDataSetId) in the parameter. If they are not equal, set the data offset cursor i (cursor_i) of a certain dimension to an object with the attribute null.
[0074] 7) Initialize the data offset cursor i (cursor_i) for a certain dimension;
[0075] The initialization algorithm can be referenced in the following pseudocode:
[0076] hasNext:=false
[0077] if fromTime==null then fromTime:=#{min_time}
[0078] if toTime==null then toTime:=#{current_t ime}
[0079] if cursorTbId==null then cursorTbId:=0L else hasNext:=true
[0080] Figure 3 This is a schematic diagram illustrating the incremental data acquisition batch method provided in the embodiments of this application, such as... Figure 3 As shown, when cursorTbId in the query request parameters is not null, it indicates that the time window is being moved, so hasNext:=true. Figure 3 The second batch, the third batch; otherwise, hasNext:=false, determining hasNext based on the number of records in the query results, such as Figure 3 The first batch.
[0081] The above initialization algorithm contains some server-side attribute information, as shown in Table 3. Table 3 is the server-side attribute information table.
[0082] Table 3 Server-side attribute information table
[0083] noun explain minTime Minimum time to start business currentTime Current time
[0084] 8) Execute an incremental query, using the query syntax to search the database. Example query syntax:
[0085] select tb.*
[0086] from tb
[0087] where 1=1
[0088] and tb.data_set_id=#{dataSetId}
[0089] and tb.modify_time>#{fromTime}
[0090] and tb.modify_time<=#{endTime}
[0091] and tb.tb_id>#{cursorTbId}
[0092] order by tb.tb_id asc
[0093] limit 0, #{limit};
[0094] The data tables in the database mentioned above have multiple columns, the specific meanings of which are shown in Table 4. Table 4 is a table defining the columns of the data tables in the database.
[0095] Table 4 defines the columns of the data tables in the database.
[0096] noun explain tb Data table tb_id Table primary key id modify_time Record update time
[0097] 9) Analyze the query results, such as Figure 2 As shown, there are two branches, a and b:
[0098] a. If there are query results, check if the number of records in the query results (sizeOf(dataSet)) is greater than or equal to #{limit}; if it is true, that is, the number of records in the query results is greater than or equal to #{limit}, it means that there may still be data in the current time window, and the next query will still use the current time window to get a batch; Pseudocode for assigning the attribute of the custom cursor i:
[0099] fromTime remains unchanged, toTime remains unchanged, cursorTbId:=max(tb_id), hasNext:=true;
[0100] If the result is false, meaning the number of records in the query result is less than #{limit}, then proceed to step b.
[0101] b. This indicates that there is no data in the current time window, and the next query will need to move the time window and reset cursorTbId; Pseudocode for assigning properties to custom cursor i:
[0102] fromTime:=toTime, toTime:=null, cursorTbId:=0L, hasNext:=hasNext||false; (such as Figure 3 (Third batch)
[0103] 10) The serialized dataset cursor object (dataSetCursorObj) is a dataset cursor (dataSetCursor). Consider using a string compression algorithm to make the dataset cursor as concise as possible.
[0104] 11) Return the query data, the structure of which is shown in Table 5. Table 5 is a description table of the query result data structure, including: dataset id (dataSetId), dataset cursor (dataSetCursor), and query dataset (dataSet_i);
[0105] Table 5: Data Structure Description of Query Results
[0106] noun explain hasNext Does the client want to continue querying the next batch? dataSet_i Incremental dataset of a certain dimension
[0107] The following is an example of the data structure for the query results:
[0108]
[0109] 12) The client processes the query results and stores the dataset cursor;
[0110] 13) The client checks the hasNext attribute in the query result. If it is true, it jumps to step 3); otherwise, the query ends.
[0111] This implementation uses an asynchronous communication signaling mechanism between the server and client, a dataset cursor data structure for storing query progress, a table structure supporting incremental queries, SQL syntax for executing incremental queries, and an algorithm for determining incremental data query batches. The table structure supporting incremental queries implements the SQL syntax for executing incremental queries, and, based on signaling, ensures the real-time acquisition of datasets by the client. The dataset cursor data structure is designed, and the algorithm for determining incremental data query batches compares the result set to determine if a next batch is available, thus reducing the number of queries.
[0112] This application also provides an incremental data acquisition device. It should be noted that the incremental data acquisition device of this application can be used to execute the incremental data acquisition method provided in this application. The incremental data acquisition device provided in this application is described below.
[0113] Figure 4 This is a schematic diagram of an incremental data acquisition device according to an embodiment of this application, such as... Figure 4 As shown, the device includes: a monitoring module 42, a receiving module 44, and an acquisition module 46. The device will be described in detail below.
[0114] The monitoring module 42 is used to send an incremental data acquisition signal to the client when incremental data is detected to be generated from the target data, wherein the client is used to request the incremental data; the receiving module 44 is used to be connected to the monitoring module 42 and to receive the incremental data request sent by the client, wherein the incremental data request includes the dataset cursor of the client's last acquisition of incremental data; the acquisition module 46 is used to be connected to the receiving module 44 and to respond to the incremental data request, and to search for the incremental data according to the dataset cursor to acquire the incremental data, wherein the dataset cursor includes multiple parameter items for searching the incremental data.
[0115] The incremental data acquisition device provided in this application embodiment, upon detecting the generation of incremental data for target data, sends an incremental data acquisition signaling to the client, whereby the client requests incremental data; receives the incremental data request from the client, wherein the incremental data request includes a cursor of the dataset from the client's previous incremental data acquisition; responds to the incremental data request, searches for incremental data based on the dataset cursor, and acquires the incremental data, wherein the dataset cursor includes multiple parameter items for searching the incremental data. After the server detects the generation of incremental data, it sends an acquisition signaling to the client, reminding the client that incremental data has been generated. If acquisition is required, a data request is sent, and the incremental data is further searched and acquired based on the dataset cursor in the data request. This improves the efficiency of incremental data querying through the information of the dataset cursor, and triggers the incremental data acquisition process in real time through the acquisition signaling, accurately matching the actions of the client and the server, thereby improving the efficiency of incremental data preparation. This solves the problem in related technologies where, when a client requests data synchronization from the server, the conditions for multiple clients acquiring datasets differ, requiring the server to provide incremental data separately for each client, and the timing of client acquisition and server preparation cannot be perfectly matched, resulting in low incremental data preparation efficiency.
[0116] The incremental data acquisition device includes a processor and a memory. The monitoring module 42, receiving module 44, acquisition module 46, etc., are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.
[0117] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, the problem in related technologies where multiple clients request data synchronization from the server, due to differences in the conditions of the datasets obtained by different clients, necessitates the server providing incremental data separately to each client, resulting in low efficiency in incremental data preparation.
[0118] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0119] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements an incremental data acquisition method.
[0120] This invention provides a processor for running a program, wherein the program executes an incremental data acquisition method during runtime.
[0121] Figure 5This is a schematic diagram of an electronic device provided according to an embodiment of this application, such as... Figure 5 As shown, this application embodiment provides an electronic device 50, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the above methods.
[0122] The device in this application may be a server, PC, PAD, mobile phone, etc.
[0123] This application also provides a computer program product that, when executed on an incremental data acquisition device, is suitable for executing a program that initializes any of the above-described method steps.
[0124] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0125] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable incremental data acquisition device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable incremental data acquisition device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable incremental data acquisition device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0127] These computer program instructions can also be loaded onto a computer or other programmable incremental data acquisition device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0129] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0130] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0131] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0132] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0133] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. An incremental data acquisition method, characterized by, include: When incremental data is detected to be generated from target data, a signaling message for obtaining incremental data is sent to the client, wherein the client is used to request the incremental data; Receive an incremental data request from the client, wherein the incremental data request includes the cursor of the dataset from the client's last acquisition of incremental data; Determine whether the dataset identifier in the dataset cursor is consistent with the dataset identifier currently used by the client, wherein the incremental data request further includes: the dataset identifier currently used by the client, the dataset identifier being used to identify the modified version of the dataset; If the dataset identifier in the dataset cursor is consistent with the dataset identifier currently used by the client, the offset cursor of the target data is initialized. The dataset cursor also includes the offset cursor of the target data, which is used to find the incremental data of the target data. In response to the incremental data request, the incremental data is searched according to the dataset cursor to obtain the incremental data, wherein the dataset cursor includes multiple parameter items for searching the incremental data; The initialization of the offset cursor for the target data includes: The start time of the search is set to the minimum creation time of the target data, the end time of the search is set to the current time, and it is determined whether the primary key identifier between the start time and the end time is empty. The offset cursor includes the start time and end time of the search, as well as the search batch, and the batch corresponds to the primary key identifier. If the primary key identifier between the start time and the end time is not empty, the batch number to be searched is determined according to the primary key identifier in the dataset cursor, and the dimension of the target data is searched according to the batch number. The dataset cursor also includes the primary key identifier of the dataset of incremental data previously obtained by the client.
2. The method of claim 1, wherein, In response to the incremental data request, before retrieving the incremental data using the dataset cursor, the method further includes: If the dataset identifier in the dataset cursor is inconsistent with the dataset identifier currently used by the client, the dataset identifier in the dataset cursor is replaced with the dataset identifier currently used by the client, and the offset cursor of the target data is initialized.
3. The method according to claim 1 or 2, characterized in that, Initializing the offset cursor for the target data includes: If the primary key identifier between the start time and the end time is empty, the batch to be searched is set to the initial batch number, and the target data is searched.
4. The method of claim 3, wherein, After obtaining search data in multiple dimensions by traversing the offset cursor, and using this as the incremental data, the following are included: Monitor whether new incremental data has been generated for the target data since the current time; The search results are sent to the client, along with information on whether new incremental data has been generated. The client then uses this information to determine whether to continue generating the next batch of incremental data retrieval requests based on the current dataset cursor and send them to the server to retrieve the new incremental data.
5. The method according to claim 1 or 2, characterized in that, In response to the incremental data request, the incremental data is retrieved by searching the incremental data using the dataset cursor, including: In response to the incremental data request, the incremental data of the corresponding target data is searched according to the offset cursor to obtain the search results of the current batch. The dataset cursor includes multiple offset cursors, and the offset cursors correspond to the dimensions of the target data. Based on the search results, determine the search data for the dimension to which the target data belongs; The search data in multiple dimensions is obtained by traversing the offset cursor, and is used as the incremental data.
6. The method of claim 4, wherein, In response to the incremental data request, after searching for the incremental data corresponding to the target data according to the offset cursor and obtaining the search results for the current batch, the method further includes: Determine whether the amount of data in the search results is less than the preset amount of data to be searched in each batch in the dataset cursor; If the amount of data is not less than the preset amount of data, continue the search for the next batch, determine the search results for the next batch, until the amount of data is less than the preset amount of data; If the amount of data is less than the preset amount of data, the search ends, and the search data of the dimension to which the target data belongs is determined based on the search results of multiple batches.
7. The method according to claim 1 or 2, characterized in that, Before initializing the offset cursor of the target data, the method further includes: The dataset cursor is serialized to generate a dataset cursor object, wherein the dataset cursor object includes multiple offset cursor objects, which are used to perform initialization processing.
8. An incremental data acquisition device, characterized in that, The incremental data acquisition device is used to execute the incremental data acquisition method according to any one of claims 1 to 7, including: The monitoring module is used to send an incremental data acquisition signal to the client when incremental data is detected to be generated from the target data, wherein the client is used to request the incremental data; A receiving module is used to receive an incremental data request sent by the client, wherein the incremental data request includes a cursor of the dataset from the client's last acquisition of incremental data; The acquisition module is used to respond to the incremental data request, search for the incremental data according to the dataset cursor, and acquire the incremental data, wherein the dataset cursor includes multiple parameter items for searching the incremental data.
9. A computer-readable storage medium, characterized in that, The storage medium is used to store a program, wherein the program executes the incremental data acquisition method according to any one of claims 1 to 7.
10. An electronic device, comprising: It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the incremental data acquisition method according to any one of claims 1 to 7.