Method, device and system for confirming data status

Abstract

The present disclosure relates to a method, device and system for confirming data status. The method comprises storing an identification of a data page in response to receiving a request for checking the data page from a source agent. The method further comprises obtaining a check status of the data page based on the identification. The check status is used to confirm whether the data page transmitted in a log signal is written into a database. The method further comprises sending an indication to the source agent for requesting a data collector to retransmit the data page in response to the check status indicating that the data page is not written into the database for a predetermined time duration. In this way, the problem that the data collector cannot confirm whether the data page is written into the database can be solved. When data loss occurs in a transmission channel, the present solution can achieve effective retransmission of the data page.

Description

Technical Field

[0001] This disclosure generally relates to the field of data storage, and more specifically to methods, apparatus, and systems for verifying the status of data. Background Technology

[0002] A distributed data collection system is a system based on distributed computing and storage technologies used to collect, process, store, and analyze data from data sources located in different locations. Due to its high reliability, high scalability, high performance, and flexibility, distributed data collection systems are widely used in scenarios such as IoT data acquisition, log data acquisition, data warehouse construction, and real-time data processing (e.g., network traffic data).

[0003] OpenTelemetry, an open-source observability framework, is a collection of tools, application programming interfaces (APIs), and software development kits (SDKs) that can be used to detect, generate, collect, and export telemetry data (e.g., metrics, logs, and traces) to help gain deeper insights into the entire lifecycle of requests in a system and analyze their performance and behavior. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, and system for verifying data status.

[0005] In a first aspect of the embodiments of this disclosure, a method for confirming data status is provided. The method includes storing an identifier of a data page in response to receiving a request from a source agent for verifying a data page. The method further includes obtaining a verification status of the data page based on the identifier. The verification status is used to confirm whether the data page, transmitted in the form of a log signal, has been written to a database. The method further includes sending an instruction to the source agent to request a data collector to retransmit the data page in response to the verification status indicating that the data page has not been written to the database for a predetermined period of time.

[0006] In a second aspect of embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for verifying data status. The method includes storing an identifier of a data page in response to receiving a request from a source agent for verifying a data page. The method further includes obtaining a verification status of the data page based on the identifier, wherein the verification status is used to verify whether the data page, transmitted in the form of a log signal, has been written to a database. The method further includes sending an instruction to a source agent to request a data collector to retransmit the data page in response to a verification status indicating that the data page has not been written to the database for a predetermined period of time.

[0007] In a third aspect of embodiments of this disclosure, a system for verifying data status is provided, including a data collector, a source agent, and a data status manager. The data collector is located near a data source and configured to collect data pages from the data source. The source agent is located near the data collector and configured to send a request to verify the data pages. The source agent is further configured to request the data collector to retransmit the data pages in response to a received verification status indication that a data page has not been written to a database for a predetermined period of time. The data status manager is configured to store an identifier of the data page in response to receiving a request for verification from the source agent. The data status manager is also configured to obtain a verification status of the data page based on the identifier, wherein the verification status is used to confirm whether the data page, transmitted in the form of a log signal, has been written to the database. The data status manager is further configured to send an instruction to the source agent requesting the data collector to retransmit the data page in response to a verification status indication that a data page has not been written to the database for a predetermined period of time.

[0008] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which some embodiments of this disclosure may be implemented is shown;

[0011] Figure 2 A flowchart illustrating the processing of the verification status according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of a system for confirming data status according to some embodiments of the present disclosure is shown;

[0013] Figure 4 An interactive flowchart illustrating a process for confirming data status according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A flowchart illustrating a process for determining a predetermined duration according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A flowchart of a method for confirming data status according to some embodiments of the present disclosure is shown; and

[0016] Figure 7 A block diagram of a device according to some embodiments of the present disclosure is shown. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] Modern enterprises rely heavily on data-driven insights, which requires robust data collection systems to gather diverse business data from various sources and reliably transmit it to a central repository.

[0020] In related technologies, a lightweight data collector is employed to gather data page by page from the data source. This data is then transmitted to an OpenTelemetry pipeline for processing and finally sent to a backend database. While the data collector acknowledges receipt of data, this acknowledgment only guarantees delivery to the OpenTelemetry pipeline, not data persistence in the database. This asynchronous persistence model presents two major challenges: first, the lack of persistence acknowledgment means the data collector cannot confirm whether data has been successfully persisted to the database; second, if data is lost during transmission or processing within the OpenTelemetry pipeline, recovering a specific data page from the collection point becomes extremely difficult. Furthermore, the dynamic nature of the data at the data source, changing over time, exacerbates the data recovery challenge.

[0021] Therefore, embodiments of this disclosure propose a scheme for verifying data status, and more specifically, introduce a lightweight scheme for verifying data status in distributed data collection systems to ensure data integrity.

[0022] In embodiments of this disclosure, when a source agent (e.g., located near the data collector) receives a data page from the data collector, it sends a request to the data status manager to verify the data page. Upon receiving the request, the data status manager obtains the verification status based on information such as the identifier of the data page contained in the request. The verification status is used to confirm whether the data page transmitted in the form of an Otel Log Format has been written to the database. If the verification status indicates that the data page transmission time exceeds a predetermined duration, i.e., the data page has not been written to the database within the predetermined duration, then the data status manager sends an instruction to the source agent to request the data collector to retransmit the data page. In this way, the problem that the data collector cannot confirm whether the data page has been written to the database can be solved. When data loss occurs in the transmission channel, the present solution enables effective retransmission of data pages.

[0023] Figure 1 A schematic diagram of an example environment 100 in which some embodiments of this disclosure may be implemented is shown. For example... Figure 1 As shown, environment 100 includes a client system 102, a data collector 104, and a source agent 106 deployed on the data source side. Among them, as... Figure 1The client systems 102-1, 102-2, 102-3, 102-4, ..., 102-N shown are collectively referred to as client system 102 or individually; the data collectors 104-1, 104-2, ..., 104-N are collectively referred to as data collector 104 or individually; and the source agents 106-1, 106-2, ..., 106-N are collectively referred to as source agent 106 or individually.

[0024] Client system 102 (also referred to as data source) can be distributed across different geographical locations, and this disclosure does not limit this. Data collector 104 is positioned close to client system 102 to collect data from one or more nearby client systems 102. In some embodiments, data collector 104 collects data in units of data pages. Source agent 106 is positioned close to data collector 104. In other words, source agent 106, like data collector 104, is positioned close to the data source. Source agent 106 obtains the data pages it has collected, as well as one or more identifiers of those data pages, from the nearby data collector 104. Source agent 106 can also further calculate other identifiers (e.g., digital fingerprints) of the data page based on these identifiers. According to embodiments of this disclosure, by positioning data collector 104 and source agent 106 close to client system 102, advantages such as reduced network latency and utilization of edge computing are achieved.

[0025] Continue to refer to Figure 1 Environment 100 also includes a destination agent 110 and a database 112 deployed on the destination side. Among them, such as... Figure 1 The destination agents 110-1, ..., and destination agents 110-N shown are collectively referred to as or individually referred to as destination agent 110; the databases 112-1, ..., and databases 112-N are collectively referred to as or individually referred to as database 112.

[0026] In some embodiments, the destination agent 110 may be deployed close to the database 112. The destination agent 110 communicates with the nearby database 112 via a network, reducing network latency and accelerating computation by shortening the deployment distance between the two.

[0027] Furthermore, in environment 100, the data pages collected by data collector 104 are transmitted to the background database 112 in the form of log signals, and the transmission channel is as follows: Figure 1As shown in 120-1, 120-2, ..., 120-N. According to embodiments of this disclosure, data from different data sources in different geographical locations can be received and output to multiple destinations, meeting the diverse needs of a data collection system. Simultaneously, data can be formatted into different formats within transmission channel 120, or aggregation can be performed before writing to the database.

[0028] Continue to refer to Figure 1 Environment 100 also includes a data status manager 108 positioned as a central node. The data status manager 108 is communicatively connected to both the source agent 106 and the destination agent 110 to obtain the reconciled status of data pages in a timely manner. The reconciled status is used to confirm whether the data pages transmitted in the form of log signals (i.e., the data pages collected by the data collector 104) have been written to the database 112. In some embodiments, the data status manager 108 is preferably positioned closer to the destination.

[0029] In some embodiments, source agent 106 periodically sends a request to data state manager 108 to verify data pages. This request includes an identifier for the data page, which may include, for example, at least one of the following: a data collector identifier (denoted as Data CollectorID), a data page identifier (denoted as Data PageID), a resource type (denoted as Resource Type), or a digital fingerprint generated based on the data page (also referred to as a first digital fingerprint). In some embodiments, the digital fingerprint is generated by calculating the MD5 hash value of the data page. The request may also include the data collection time (denoted as Data CollectionTime).

[0030] Upon receiving a request from source proxy 106, data state manager 108 checks with destination proxy 110 whether the data page indicated by the identifier exists in database 112 (i.e., whether the data page has been written to the database). Based on the identifier, destination proxy 110 queries the data page in nearby database 112 and generates a digital fingerprint (also called a second digital fingerprint) of the queried data page. This digital fingerprint is generated by calculating the MD5 hash value of the queried data page. Then, destination proxy 110 confirms whether the data page has been written to database 112 by determining whether the second digital fingerprint matches the first digital fingerprint; and returns the confirmation result (i.e., the verification status) to data state manager 108.

[0031] The verification status obtained by the data status manager 108 from the destination agent 110 includes: the data page has been written to the database (referred to as "verified") and the data page has not been written to the database (referred to as "unverified"). The data status manager 108 performs further processing based on the obtained verification status.

[0032] Figure 2 A flowchart of a process 200 for checking the status according to some embodiments of the present disclosure is shown. (In conjunction with...) Figure 2 In box 202, process 200 obtains the verification status. At this time, the verification status includes verified and unverified. If the data page has been written to database 112, the obtained verification status is verified; if the data page has not been written to database 112, the obtained verification status is unverified.

[0033] In box 204, process 200 determines whether a data page has been verified based on the verification status. If the data page has been verified, the process proceeds to box 206, where data status manager 108 notifies source agent 106 that the data page has been verified. Simultaneously, data status manager 108 may also send an instruction to delete the data page. The verification process for that data page then ends.

[0034] If the data page is not verified, proceed to frame 208, where the data status manager 108 needs to further determine whether the transmission time of the data page exceeds the predetermined duration. Specifically, it determines whether the time interval between the current time and the data collection time (i.e., the difference between the current time and the data collection time) is greater than the predetermined duration: if the time interval is greater than the predetermined duration, it indicates a timeout; if it is not greater than the predetermined duration, it indicates no timeout.

[0035] According to embodiments of this disclosure, the predetermined duration can ensure the efficiency and reliability of the data reconciliation process. If the predetermined duration is set too short, it may trigger error alarms and unnecessary data retransmissions, causing resource overload on the transmission channel. If the predetermined duration is set too long, there is a risk of excessive local data storage, potentially exceeding storage capacity limits. In some embodiments, the predetermined duration is dynamically determined based on the identifier of the data page. The specific method for determining the predetermined duration will be detailed below and will not be repeated here.

[0036] If a timeout is confirmed after assessment, the process proceeds to box 210, where the data status manager 108 notifies the source agent 106: "Retransmission required." That is, the data status manager 108 sends an instruction to the source agent 106 requesting the data collector 104 to retransmit the data page. Therefore, the verification status, in addition to "verified" and "not verified," also includes instructing the source agent 106 to retransmit the data page when it has exceeded a predetermined time limit and has not been written to the database (referred to as "retransmission required"). In this case, the source agent 106, responding to the "retransmission required" instruction, requests the data collector 104 to retransmit the data page. Afterward, the data status manager 108 repeats the aforementioned process 200, proceeding to box 202, until the verification status is "verified," at which point process 200 ends.

[0037] If the timeout is confirmed after judgment, the system returns to box 202, and the data status manager 108 continues to obtain the verification status until the verification status is "verified", at which point the process ends at 200.

[0038] According to the scheme disclosed herein, it is possible to determine whether the data collected by the data collector has been written to the database. This scheme can be deployed in environments that rely on high-throughput data collection from distributed client systems, ensuring end-to-end data integrity and minimizing data transfer volume while verifying data integrity, thereby improving efficiency.

[0039] Figure 3 A schematic diagram of a storage system 300 according to some embodiments of the present disclosure is shown. (See reference...) Figure 3 As shown, the storage system 300 includes a system for verifying data status, a destination proxy 304, and a database 305. The system for verifying data status includes a data collector 310, a source proxy 320, and a data status manager 330. In some embodiments, the destination proxy 304 may also be deployed within the system for verifying data status. In summary, the destination proxy 304 is located near the database 305 at the destination (e.g., a third-party database), communicates with the database 305, and is responsible for verifying with the database whether data pages have been written. It should be noted that... Figure 3 The components shown are illustrative and this disclosure does not limit the number of components in system 300.

[0040] Data collector 310 is positioned close to the data source to collect data pages from it. Source agent 320 is positioned near data collector 310. Data collector 310 and source agent 320 can be deployed on separate hardware or on the same hardware; this embodiment does not impose any restrictions on this. Data state manager 330, as the central component, is responsible for managing and monitoring the data pages. Figure 4An interactive flowchart of a process 400 for confirming data status according to some embodiments of this disclosure is shown. The following is in conjunction with... Figure 4 This further explains the interaction flow of each component within the system 300.

[0041] At position 402, the data collector 310 sends the collected data pages and one or more identifiers to the source agent 320. In some embodiments, the data collector 310 transmits the collected data pages to the database in the form of log signals, and sends a backup of the data page (hereinafter collectively referred to as a "data page") and the identifier of the data page to the source agent 320. The identifier may be one or more of the following: data collector identifier (a unique identifier of the data collector 310), data page identifier (a unique identifier of the data page), and resource type (identifying the resource of the data page). In some embodiments, the data collector 310 may also send the data collection time (the timestamp of the data collector 310 collecting the data page from the client system) to the source agent 320.

[0042] At position 404, source proxy 320 stores the data page and generates a digital fingerprint for the data page. In some embodiments, the digital fingerprint also serves as an identifier to identify each data page.

[0043] At 406, source agent 320 sends a request to data state manager 330 to verify a data page. This request includes at least one of the identifiers of the data page described at 402, or a digital fingerprint, or the data collection time. This triggers data state manager 330 to begin the data verification process.

[0044] According to some embodiments, the source agent 320 may periodically (e.g., every minute, but not limited thereto) send a request to the data state manager 330 to verify a data page until it obtains a verification status from the data state manager 330 indicating that the data page has been written to the database.

[0045] At 408, the data state manager 330 stores the identifier of the data page. In some embodiments, the data state manager 330 uses a data table to track the verification status of the data page. This data table may contain the following key elements: data collector identifier, data page identifier, resource type, data collection time, digital fingerprint, verification status, and data verification time. The data collector identifier, data page identifier, resource type, data collection time, and digital fingerprint—the information carried in the aforementioned request—are associated and stored in the data table here.

[0046] In addition, the data table includes verification status and data verification time. Verification status indicates whether a data page has been verified, or if not, whether retransmission is necessary. Data verification time records the timestamp when the data page's verification status was "verified."

[0047] Table 1 exemplarily illustrates a data table in the Data Status Manager. As shown in Table 1, this data table contains 5 data records, each representing a data page. Within a data record, one or more identifiers of the stored data page are associated. It should be understood that when the verification status of a data page is "Unverified" or "Requires Retransmission," the data verification time is empty (e.g., data records 1, 4, and 5).

[0048] Table 1 Data Tables in Data Status Manager

[0049]

[0050]

[0051] Using the above method, the data table can minimize data storage requirements and effectively track and manage the data verification status of the entire system.

[0052] At 410, the data state manager 330 sends a request to the destination agent 304 to confirm whether a data page has been written to the database. According to embodiments of this disclosure, this instruction may also include an identifier of the corresponding data page. In some embodiments, the instruction includes a data page identifier and a digital fingerprint. In still other embodiments, the instruction includes a data page identifier, a digital fingerprint, and a data collector identifier.

[0053] At position 412, destination agent 304 queries database 305 for data pages based on the aforementioned identifier. In some embodiments, destination agent 304 queries database 305 based on data page identifiers. To optimize query efficiency, in some further embodiments, destination agent 304 queries database 305 based on data page identifiers and data collection time. The data collection time is used as a filter condition to query data page identifiers after that time.

[0054] At point 414, database 305 returns the query results to destination agent 304. Database 305 may find more than one matching data page, or it may not find any matching data pages. If a data page is found, it returns it to destination agent 304; if no data page is found, it notifies destination agent 304 that no relevant data was found.

[0055] At point 416, destination proxy 304 generates a digital fingerprint (i.e., a second digital fingerprint) of the retrieved data page. Then, destination proxy 340 determines whether this digital fingerprint matches the digital fingerprint (i.e., a first digital fingerprint) stored in the aforementioned data table. If they match, it is confirmed that the data page has been written to database 305; if they do not match, it is confirmed that the data page has not been written to database 305. Assuming destination proxy 304 retrieves multiple data pages, it generates a digital fingerprint for each data page. Then, it determines whether each digital fingerprint matches the digital fingerprint in the aforementioned data table and confirms the corresponding result. According to some other embodiments, if destination proxy 304 does not retrieve a data page, it is confirmed that the data page has not been written to database 305.

[0056] At 418, the destination agent 304 sends an indication to the data status manager 330 to confirm whether the data page has been written to the database (in other words, returning an indication). As mentioned earlier, this indication represents the verification status of the data page. The verification status obtained by the data status manager 330 from the destination agent 304 includes: the data page has been written to the database (“verified”) or the data page has not been written to the database (“unverified”).

[0057] At point 420, the data state manager 330 performs further processing based on the confirmation result. Then, at point 422, based on the processing result, it sends the corresponding notification instruction to the source agent 320.

[0058] If the verification status is "verified", then in step 420, the data status manager 330 determines the current time as the data verification time of the data page and stores it in the data table. At the same time, the verification status in the data table is updated to "verified".

[0059] At step 422, the data status manager 330 notifies the source agent 320: verification has been completed. Then, proceeding to step 424-1, the source agent 320 deletes the data page. Furthermore, the source agent 320 no longer periodically executes step 406.

[0060] If the verification status is "unverified," further analysis is required in step 420: determine whether the data page's transmission time exceeds the predetermined duration. The transmission time is determined based on the difference between the current time and the data collection time. The predetermined duration needs to be determined based on the data page's identifier.

[0061] As mentioned earlier, setting a predetermined duration allows for a balance between the efficiency and reliability of data verification. Many factors influence the predetermined duration setting, such as the performance of the transmission channel, network latency, database write speed, and the geographical distance between the data collector 310 and the database 305, among others. Figure 5 A flowchart of an example process 500 for determining a predetermined duration according to some embodiments of the present disclosure is shown.

[0062] like Figure 5 As shown in box 510, multiple data collection times and multiple data verification times associated with a data collector identifier within a recent period are retrieved. In other words, based on the data collector identifier, the associated data collection time and data verification time are retrieved from the data table. Further, the search scope is limited to a recent period (e.g., the past hour). It should be noted that the data collection times and data verification times retrieved here appear in pairs, meaning they come from the same data record in the data table. It should be understood that the retrieved data record should have a verification status of "verified". In some other embodiments, the associated resource type is also retrieved.

[0063] Table 2 provides an example of eight data records obtained in the manner described above, including resource type, data collection time, and data verification time.

[0064] Table 2 Data records associated with data collector identifiers

[0065]

[0066]

[0067] In box 520, multiple historical verification durations are determined based on multiple data collection times and multiple data verification times. Specifically, for each pair of data collection times and data verification times, the difference between the data verification time and the data collection time is used as the corresponding historical verification duration.

[0068] Based on the data obtained in Table 2, the corresponding historical verification time was calculated, as shown in Table 3.

[0069] Table 3 Historical Verification Duration

[0070] Resource types Historical verification time asset_v2 2 minutes 20 seconds resource_group_v2 1 minute 50 seconds upgrade_package_v2 1 minute 30 seconds license_v2 2 minutes and 31 seconds alert_v2 3 minutes and 15 seconds activity_v3 3 minutes and 50 seconds activity_v3 3 minutes and 30 seconds activity_v3 3 minutes and 31 seconds

[0071] In box 530, specific numerical points of multiple historical verification durations are identified. In some embodiments, the specific numerical point is the 90th percentile.

[0072] Continuing with the previous example, calculate the 90th percentile for the eight historical verification times in Table 3. First, sort the historical verification times: 1:30, 1:50, 2:20, 2:31, 3:15, 3:30, 3:31, 3:50. Then, calculate the 90th percentile as 3:31, which is 3 minutes and 31 seconds.

[0073] In box 540, the scaling factor is determined based on the resource type of the data page. According to this embodiment, considering the timeliness of resources, different scaling factors are set for different resource types. Smaller scaling factors are set for resource types with high timeliness; conversely, larger scaling factors are set for resource types with low timeliness. In other words, the higher the timeliness of a resource type, the smaller the scaling factor. Continuing with the previous example, corresponding scaling factors are set for each resource type in Table 3, as shown in Table 4. It should be noted that this example shows several resource types, such as assets, resource groups, upgrade packages, license certificates, alarm events, and activity events, and does not limit the resource types. In practical applications, the type of resource and the value of its scaling factor can be determined according to the application scenario.

[0074] Table 4 Scale factors for each resource type

[0075] Resource types scale factor asset_v2 1.5 resource_group_v2 1.5 upgrade_package_v2 1.5 license_v2 1.6 alert_v2 1.0 activity_v3 1.0

[0076] In box 550, the predetermined duration of a data page is determined based on a specific numerical point and a scaling factor. In some embodiments, the product of the specific numerical point and the scaling factor is used as the predetermined duration of the data page. Continuing with the previous example, Table 5 shows the finally calculated predetermined duration.

[0077] Table 5. Booking Duration for Each Resource Type

[0078] Resource types Calculation process Reservation duration asset_v2 (3 minutes 30 seconds) * 1.5 5 minutes and 15 seconds resource_group_v2 (3 minutes 30 seconds) * 1.5 5 minutes and 15 seconds upgrade_package_v2 (3 minutes 30 seconds) * 1.5 5 minutes and 15 seconds license_v2 (3 minutes 30 seconds) * 1.6 5 minutes and 36 seconds alert_v2 (3 minutes 30 seconds) * 1.0 3 minutes and 30 seconds activity_v3 (3 minutes 30 seconds) * 1.0 3 minutes and 30 seconds

[0079] According to process 500 of this disclosure, by considering the historical verification time of the data collector where the data page is located and selecting specific numerical points from it, typical data verification times can be accurately captured during the dynamic determination of the predetermined time, effectively eliminating extreme outliers. Furthermore, considering the timeliness of resources, setting scaling factors for different resource types and further optimizing specific numerical points can balance the efficiency and reliability of data verification.

[0080] Returning to 420, if it is determined that the data page transmission time exceeds the predetermined duration, then at 422, the data status manager 330 notifies the source agent 320 that a retransmission is required. Then, proceeding to 424-2, the source agent 320 requests the data collector 310 to retransmit the data page.

[0081] If it is determined that the data page transmission time has not exceeded the predetermined duration, the source agent 320 continues to execute 406 and periodically sends a request to the data status manager 330 to verify the data page.

[0082] Figure 6A flowchart of a method 600 for confirming data status according to some embodiments of the present disclosure is shown. Method 600 is typically executed in a data status manager. It should be noted that method 600 complements the foregoing process 400, and related details will not be repeated.

[0083] like Figure 6 As shown, in block 610, in response to receiving a request from the source agent for verifying a data page, the identifier of the data page is stored. In some embodiments, the request may include the identifier of the data page and the data collection time, the identifier including at least one of the following: a data collector identifier, a data page identifier, a resource type, or a digital fingerprint generated based on the data page.

[0084] In box 620, the verification status of the data page is obtained based on the identifier. The verification status is used to confirm whether the data page transmitted in the form of log signals has been written to the database.

[0085] Based on the data page identifier, data collection time, and digital fingerprint, the Data State Manager sends an instruction to the destination agent to confirm whether the data page has been written to the database. The destination agent then queries nearby databases for the data page and generates a digital fingerprint for the retrieved page. The destination agent determines whether the data page has been written to the database by comparing the two digital fingerprints and returns the result (i.e., the verification status) to the Data State Manager.

[0086] In response to an indication received from the destination agent that the confirmation data page has not been written to the database, the data status manager needs to further determine whether the time period between the current time and the data collection time is greater than the predetermined duration, and if the time period is greater than the predetermined duration, obtain the verification status as "retransmission required".

[0087] In response to an indication received from the destination agent that the confirmed data page has been written to the database, the verification status is set to "verified". Simultaneously, the current time is determined as the data verification time for the data page.

[0088] In box 630, in response to a check status indicating that a data page has not been written to the database for a predetermined period of time, an instruction is sent to the source agent to request the data collector to retransmit the data page.

[0089] Figure 7A schematic block diagram of an example device 700 that can be used to implement embodiments of the present disclosure is shown. As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 702 or loaded from storage unit 708 into random access memory (RAM) 703. Various programs and data required for the operation of device 700 may also be stored in RAM 703. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0090] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0091] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as method 300. For example, in some embodiments, method 300 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of method 600 described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform method 300 by any other suitable means (e.g., by means of firmware).

[0092] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0093] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0094] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0095] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for confirming data status, comprising: In response to receiving a request from the source proxy for verifying a data page, the identifier of the data page is stored; Based on the identifier, the verification status of the data page is obtained, and the verification status is used to confirm whether the data page transmitted in the form of log signals has been written to the database. as well as In response to the verification status indicating that the data page has not been written to the database for a predetermined period of time, an instruction is sent to the source agent to request the data collector to retransmit the data page.

2. The method of claim 1, wherein the request includes an identifier of the data page and a data collection time, the identifier including at least one of the following: a data collector identifier, a data page identifier, a resource type, or a first digital fingerprint generated based on the data page.

3. The method according to claim 2, wherein obtaining the verification status of the data page based on the identifier includes: Based on the data page identifier, the data collection time, and the first digital fingerprint, a request is sent to the destination agent to confirm whether the data page has been written to the database; Based on the confirmation received from the destination agent that the data page has not been written to the database, it is determined whether the time period between the current time and the data collection time is greater than the predetermined duration. as well as In response to determining that the time period is greater than the predetermined duration, the verification status is determined to be that the data page has exceeded the predetermined duration and has not been written to the database.

4. The method according to claim 3, wherein determining whether the time period between the current time and the data collection time is greater than the predetermined duration includes: Based on the identifier, the predetermined duration of the data page is determined; as well as Determine whether the time interval between the current time and the data collection time is greater than the predetermined duration.

5. The method of claim 4, wherein determining the predetermined duration of the data page based on the identifier comprises: Obtain multiple data collection times and multiple data verification times within a recent period associated with the data collector identifier; Based on multiple data collection times and multiple data verification times, multiple historical verification durations are determined; Determine specific numerical points for the duration of the multiple historical verifications; Determine the scaling factor based on the resource type of the data page; as well as The predetermined duration of the data page is determined based on the specific numerical point and the scaling factor.

6. The method according to claim 2, wherein obtaining the verification status of the data page based on the identifier includes: Based on the data page identifier, the data collection time, and the first digital fingerprint, a request is sent to the destination agent to confirm whether the data page has been written to the database; as well as Based on the confirmation received from the destination agent that the data page has been written to the database, the verification status is determined to be that the data page has been written to the database.

7. The method according to claim 1, further comprising: In response to the verification status indicating that the data page has been written to the database, an instruction to delete the data page is sent to the source agent.

8. The method according to claim 5, further comprising: In response to the verification status indicating that the data page has been written to the database, the current time is determined as the data verification time of the data page.

9. An electronic device, comprising: At least one processor; as well as Coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, cause the electronic device to perform actions, the actions including: In response to receiving a request from the source proxy for verifying a data page, the identifier of the data page is stored; Based on the identifier, the verification status of the data page is obtained, wherein the verification status is used to confirm whether the data page transmitted in the form of a log signal has been written to the database; and In response to the verification status indicating that the data page has not been written to the database for a predetermined period of time, an instruction is sent to the source agent to request the data collector to retransmit the data page.

10. The device of claim 9, wherein the request includes an identifier of the data page and a data collection time, the identifier including at least one of the following: a data collector identifier, a data page identifier, a resource type, or a first digital fingerprint generated based on the data page.

11. The device of claim 10, wherein obtaining the verification status of the data page based on the identifier includes: Based on the data page identifier, the data collection time, and the first digital fingerprint, a request is sent to the destination agent to confirm whether the data page has been written to the database; Based on the confirmation received from the destination agent that the data page has not been written to the database, it is determined whether the time period between the current time and the data collection time is greater than the predetermined duration. as well as In response to determining that the time period is greater than the predetermined duration, the verification status is determined to be that the data page has exceeded the predetermined duration and has not been written to the database.

12. The device of claim 11, wherein determining whether the time period between the current time and the data collection time is greater than the predetermined duration includes: Based on the identifier, the predetermined duration of the data page is determined; as well as Determine whether the time interval between the current time and the data collection time is greater than the predetermined duration.

13. The device of claim 12, wherein determining the predetermined duration of the data page based on the identifier comprises: Obtain multiple data collection times and multiple data verification times within a recent period associated with the data collector identifier; Based on the multiple data collection times and multiple data verification times, multiple historical verification durations are determined; Determine specific numerical points for the duration of the multiple historical verifications; Determine the scaling factor based on the resource type of the data page; as well as The predetermined duration of the data page is determined based on the specific numerical point and the scaling factor.

14. The device of claim 10, wherein obtaining the verification status of the data page based on the identifier includes: Based on the data page identifier, the data collection time, and the first digital fingerprint, a request is sent to the destination agent to confirm whether the data page has been written to the database; as well as Based on the confirmation received from the destination agent that the data page has been written to the database, the verification status is determined to be that the data page has been written to the database.

15. The device according to claim 9, wherein the action further includes: In response to the verification status indicating that the data page has been written to the database, an instruction to delete the data page is sent to the source agent.

16. The device according to claim 13, wherein the action further includes: In response to the verification status indicating that the data page has been written to the database, the current time is determined as the data verification time of the data page.

17. A system for confirming data status, comprising: A data collector, positioned close to the data source, is configured to collect data pages from the data source; A source proxy, located close to the data collector, is configured to send a request for verifying a data page; the source proxy is further configured to request the data collector to retransmit the data page in response to a received verification status indicating that the data page has not been written to the database for a predetermined period of time. as well as A data state manager is configured to store the identifier of the data page in response to receiving a request from the source agent for verifying the data page; Based on the identifier, the verification status of the data page is obtained, wherein the verification status is used to confirm whether the data page transmitted in the form of a log signal has been written to the database; the data status manager is further configured to send an instruction to the source agent to request the data collector to retransmit the data page in response to the verification status indicating that the data page has not been written to the database for a predetermined period of time.

18. The system of claim 17, wherein the source agent is further configured to store data pages from the data collector and generate a first digital fingerprint of the data pages.

19. The system of claim 18, further comprising: A destination proxy, located close to the database, is configured to obtain a request from the data state manager to confirm whether the data page has been written to the database. The request includes a data page identifier, data collection time, and the first digital fingerprint. The destination proxy is further configured to confirm whether the data page has been written to the database.

20. The system of claim 19, wherein confirming whether the data page has been written to the database comprises: Based on the data page identifier and the data collection time, query the data page in the database; Generate a second digital fingerprint of the retrieved data page; In response to the second digital fingerprint matching the first digital fingerprint, it is confirmed that the data page has been written to the database; as well as In response to the inconsistency between the second digital fingerprint and the first digital fingerprint, it is confirmed that the data page has not been written to the database.

Images