Data tracing method and device, electronic equipment and non-instantaneous machine readable medium

Abstract

The invention relates to a data tracing method and device, electronic equipment and a non-instantaneous machine readable medium. The data tracing method comprises the following steps: triggering a plurality of same query requests according to a preset time sequence, and obtaining a plurality of log fragments corresponding to the preset time sequence; wherein the preset time sequence comprises a plurality of non-periodically set moments, and each moment corresponds to one log fragment; acquiring first log entries existing in the plurality of log fragments; based on the query execution time corresponding to the first log entry, removing log entries with periodic query execution time in the first log entry to obtain a second log entry; and performing data tracing on the query request based on the second log entry. Through the data tracing method and device, the technical problem that traditional data tracing is poor in accuracy is solved, and the technical effect of improving data tracing accuracy and data tracing efficiency is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data tracing method, device, electronic device and non-transitory machine-readable medium. Background Art

[0002] During the digital transformation process, the original data applications will be migrated and replaced to the big data technology products on the cloud. It is necessary to trace the data of the original data applications to accurately locate the metadata information of the data source involved in the migration, that is, the database name, table name, field name, etc. The traditional data traceability is mainly done manually, including interviews with employees in multiple positions such as data application system users, service modelers, application developers, and database administrators. This approach is very dependent on the accumulation of original knowledge within the enterprise, such as product documents, service modeling documents, development documents, etc., and the completeness and accuracy of these documents are difficult to guarantee. Therefore, in many cases, developers are even required to cooperate with the traceability through the source code. It can be seen that the traditional data traceability has the disadvantages of heavy investment, reliance on manual experience, long implementation cycle, and prone to errors. Summary of the invention

[0003] The embodiments of the present application provide a data tracing method, device, electronic device and non-transitory machine-readable medium to at least solve the problem of poor accuracy in traditional data tracing.

[0004] According to one aspect of an embodiment of the present application, a data tracing method is provided, including: triggering multiple identical query requests according to a preset time sequence, and obtaining multiple log fragments corresponding to the preset time sequence; wherein the preset time sequence includes multiple non-periodic moments, each moment corresponding to a log fragment; obtaining a first log entry that exists in multiple log fragments; based on the query execution time corresponding to the first log entry, removing the log entries with periodic query execution time in the first log entry to obtain a second log entry; and tracing the query request based on the second log entry.

[0005] According to another aspect of an embodiment of the present application, a data tracing device is provided, comprising: a first processing module, configured to trigger a plurality of identical query requests according to a preset time sequence, and obtain a plurality of log fragments corresponding to the preset time sequence; wherein the preset time sequence comprises a plurality of non-periodic set moments, each moment corresponding to a log fragment;

[0006] The second processing module is used to obtain the first log entry that exists in multiple log fragments; the third processing module is used to remove the log entries with periodic query execution time in the first log entries based on the query execution time corresponding to the first log entry, so as to obtain the second log entry; the fourth processing module is used to trace the query request based on the second log entry.

[0007] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the method described above.

[0008] According to another aspect of an embodiment of the present application, a non-transitory machine-readable medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method described above.

[0009] In an embodiment of the present application, multiple identical query requests are triggered according to a preset time sequence to obtain multiple log fragments corresponding to the preset time sequence; then a first log entry that exists in multiple log fragments is obtained; based on the query execution time corresponding to the first log entry, the log entries with periodic query execution time in the first log entry are removed to obtain a second log entry; finally, data traceability is performed on the query request based on the second log entry, thereby solving the technical problem of poor accuracy of traditional data traceability and achieving the technical effect of improving the accuracy and efficiency of data traceability. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0011] Figure 1 A flowchart of a data tracing method provided in an embodiment of the present application;

[0012] Figure 2 A schematic diagram of the data tracing process provided for the preferred implementation mode of this application;

[0013] Figure 3 A schematic diagram of a data tracing device provided in an embodiment of the present application;

[0014] Figure 4 A hardware structure block diagram of an electronic device is shown. DETAILED DESCRIPTION

[0015] Embodiments of the present embodiment will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present embodiment are shown in the accompanying drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, which are instead provided for a more thorough and complete understanding of the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not intended to limit the scope of protection of the present embodiment.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0018] RPA (Robotic Process Automation): A robotic process automation system is an application that provides an alternative way to automate end-user manual operations by mimicking the manual operations of end users on computers.

[0019] Data application: Provides software applications based on data content for different service scenarios. From the user's perspective, the application mainly focuses on reading data, such as traditional reports, business intelligence analysis, and data large screens.

[0020] Metadata (or metadata information): Metadata is information about the organization of data, data domains, and their relationships; it is divided into technical metadata and service metadata according to its purpose. First, metadata can provide user-based information, such as metadata that records service description information of data items to help users use data. Second, metadata can support the system's management and maintenance of data, such as metadata about the storage method of data items to support the system in accessing data in the most efficient way.

[0021] An embodiment of the present application provides a data tracing method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0022] Figure 1 A flowchart of a data tracing method provided in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0023] Step S102, triggering multiple identical query requests according to a preset time sequence, and acquiring multiple log fragments corresponding to the preset time sequence; wherein the preset time sequence includes multiple non-periodic set moments, each moment corresponding to a log fragment;

[0024] Step S104, obtaining a first log entry that exists in multiple log segments;

[0025] Step S106, based on the query execution time corresponding to the first log entry, remove the log entries with periodic query execution time from the first log entry to obtain a second log entry;

[0026] Step S108: tracing the data source of the query request based on the second log entry.

[0027] It should be noted that, for the first log entry and the second log entry, the number of log entries contained in the two depends on the query request in the application scenario, and no restriction is imposed here.

[0028] In an embodiment of the present application, multiple identical query requests are triggered according to a preset time sequence to obtain multiple log fragments corresponding to the preset time sequence; then a first log entry that exists in multiple log fragments is obtained; based on the query execution time corresponding to the first log entry, the log entries with periodic query execution time in the first log entry are removed to obtain a second log entry; finally, data traceability is performed on the query request based on the second log entry, thereby solving the technical problem of poor accuracy of traditional data traceability and achieving the technical effect of improving the accuracy and efficiency of data traceability.

[0029] As an optional embodiment, triggering multiple identical query requests according to a preset time sequence includes: configuring corresponding preset time sequences for the multiple identical query requests, and triggering the multiple identical query requests according to the preset time sequence.

[0030] Optionally, you can use robotic process automation tools, human-computer interaction scripts, and other process automation tools to configure corresponding preset time sequences for multiple identical query requests, which can simulate natural persons to trigger query requests for data applications, thereby improving data traceability efficiency and reducing the time and cost of manual intervention. For example, use robotic process automation tools to configure corresponding preset time sequences for multiple identical query requests, and then the robotic process automation tools will trigger multiple identical query requests according to the preset time sequence.

[0031] As an optional embodiment, configuring corresponding preset time sequences for multiple identical query requests includes: configuring corresponding preset time sequences for multiple identical query requests using a random time setting method; or configuring corresponding preset time sequences for multiple identical query requests using a regular time setting method.

[0032] For example, the preset time sequence obtained by configuring the random time setting method can be: 1:20, 1:35, 1:42, 1:51, etc. (the time intervals are not periodic and regular); the preset time sequence obtained by configuring the regular time setting method can be: 1:20, 1:25, 1:31, 1:38, etc. (the time intervals are not periodic, and the subsequent time interval will increase by 1 minute based on the previous time interval). It should be noted that in the specific implementation process, the preset time sequence includes but is not limited to the content described above.

[0033] Whether a random time setting method is used to configure a corresponding preset time sequence for multiple identical query requests, or a regular time setting method is used to configure a corresponding preset time sequence for multiple identical query requests, the final preset time sequence is still a plurality of non-periodic set moments.

[0034] In the embodiments of the present application, random or regular time settings can better adapt to real scenarios and improve the accuracy of data tracing.

[0035] As an optional embodiment, obtaining multiple log fragments corresponding to a preset time series includes: obtaining the preset time series and a query completion time series corresponding to the preset time series, wherein each query completion moment in the query completion time series corresponds to a moment in the preset time series; obtaining the log when the query request is executed; and filtering the log fragments of the log according to the preset time series and the query completion time series to obtain multiple log fragments.

[0036] Each moment in the above preset time sequence is used as the query start time of the query request, which is used to trigger the query request at this moment; the above query completion time is the time returned after the query request is executed; for the query request, the query request is triggered at the query start time, and then when the candidate database is accessed according to the query request, the query execution time of the query request is recorded, and finally after the candidate database is accessed according to the query request, a time point is returned, which is the query completion time. Due to the influence of factors such as transmission delay, the query execution time is between the query start time and the query completion time.

[0037] The log when executing the query request is all log entries of the candidate database when executing the query request.

[0038] Furthermore, log fragments of the log are filtered according to a preset time sequence and a query completion time sequence to obtain multiple log fragments, including: obtaining the query execution time when the query request is executed in the log; if the query execution time is greater than or equal to the moment in the preset time sequence, and less than or equal to the query completion moment in the query completion time sequence, then obtaining the log fragment between the moment in the preset time sequence and the query completion moment in the query completion time sequence to obtain multiple log fragments.

[0039] In an embodiment of the present application, multiple time ranges are formed by a preset time series and a query completion time series corresponding to the preset time series, and log fragments in the log are screened within the multiple time ranges to obtain multiple log fragments. Since the time range covers the query execution time, the obtained log fragments can ensure that the log entries corresponding to the query execution time can be obtained, thereby narrowing the scope of data tracing and greatly improving the accuracy of data tracing.

[0040] As an optional embodiment, obtaining a first log entry that exists in multiple log fragments includes: performing string matching on the multiple log fragments respectively to obtain the first log entry that exists in the multiple log fragments.

[0041] For example, any two log fragments in multiple log fragments are: log fragment 1 = {log 1.1, log 1.2, log 1.3}, log fragment 2 = {log 2.1, log 2.2, log 2.3, log 2.4}, and by performing string matching on log fragment 1 and log fragment 2, it can be determined whether each log in log fragment 1 also exists in log fragment 2. The same method is used to perform string matching on other log fragments in multiple log fragments, and finally the first log entry that exists in multiple log fragments can be obtained.

[0042] Since the query parameters are the same under the same query request, the generated log entries should also be the same. For example, a first log entry existing in multiple log segments is found, and the first log entry is a log entry generated based on the query request.

[0043] In the embodiment of the present application, the first log entry existing in multiple log segments can be accurately obtained by string matching.

[0044] As an optional embodiment, based on the query execution time corresponding to the first log entry, the log entries with periodic query execution time in the first log entry are removed to obtain the second log entry, including: determining the time interval of the query execution time; filtering the query execution time according to the time interval of the query execution time to obtain the query execution time with periodicity; removing the log entries corresponding to the query execution time with periodicity from the first log entry to obtain the second log entry.

[0045] The log entries corresponding to the above-mentioned periodic query execution time include but are not limited to the log entries generated by the execution system's periodic query requests. The log entries corresponding to the periodic query execution time are not the logs required for data tracing, and their existence will seriously affect the accuracy of data tracing.

[0046] Furthermore, the time interval of the query execution time may be determined by the following implementation: sorting the query execution time to obtain a query execution time sequence; and calculating the time interval of the query execution time according to the query execution time sequence.

[0047] It should be noted that the query execution times can be sorted in chronological order. Since the chronological order in the query execution time sequence is fixed, the time interval between any two adjacent query execution times in the query execution time sequence can be calculated respectively.

[0048] Furthermore, the query execution time is screened according to the time interval of the query execution time to obtain the query execution time with periodicity. The following implementation method can be adopted: according to the time interval of the query execution time, the time interval variance is calculated; when the time interval variance is less than a preset time interval variance threshold, the query execution time is determined to be a query execution time with periodicity.

[0049] In an embodiment of the present application, by comparing the time interval variance with a preset time interval variance threshold, the query execution time with periodicity is accurately screened out, thereby avoiding the adverse impact of periodic corresponding log entries on data traceability and further improving the accuracy of data traceability.

[0050] As an optional embodiment, data tracing is performed on the query request based on the second log entry, including: parsing the second log entry to obtain metadata information on which the second log entry depends; and filtering out metadata information corresponding to the query request based on the metadata information on which the second log entry depends.

[0051] Optionally, the second log entry is parsed using a log parsing tool to obtain metadata information on which the second log entry depends. The log parsing tool is a software tool for parsing and analyzing log entries.

[0052] The above metadata information includes but is not limited to database name, table name, field information, etc.

[0053] Furthermore, based on the metadata information on which the second log entry depends, the metadata information corresponding to the query request is screened out, and the following implementation method may be adopted: the metadata information on which the second log entry depends is deduplicated and merged to obtain the metadata information corresponding to the query request.

[0054] In an embodiment of the present application, the metadata information on which the query request depends can be deduplicated and merged to implement the screening of the metadata information, thereby better managing and utilizing the metadata information and improving the efficiency of data tracing.

[0055] Based on the above embodiments and preferred embodiments, a preferred implementation is provided.

[0056] Figure 2 A schematic diagram of the data tracing process provided for the preferred implementation of this application is shown in FIG. Figure 2 As shown, through RPA (or other automation tools, human-computer interaction scripts) tools, natural persons are simulated to trigger query requests for data applications, and (automatic exploration system) obtains and generates access logs of the database at the same time, and accurately filters out metadata information (database name, table name, field, etc.) of the data source corresponding to the query request from the log, thereby realizing data traceability. The overall architecture based on the data traceability process can realize the traceability of query requests from the application side to the access logs in the back-end database. Compared with traditional manual data traceability work, this implementation method has the advantages of low cost, unattended, short implementation cycle, and accurate traceability, which can greatly improve the implementation efficiency.

[0057] Furthermore, the specific implementation of the data traceability process is as follows:

[0058] If the data application scenario is a report, the report has N query fields, and the list of all candidate databases is known, but it is not known which databases, tables, and fields are related to this query.

[0059] 1) Collect query requests; since there are N query fields, these query fields can be combined arbitrarily for querying. Theoretically, there are at most 2 N kinds of query combinations. Assign values to each query request to form a list of query requests Q(0 to 2 N ).

[0060] 2) Configure and execute RPA; for example, for each group of query requests Q i in Q, configure a set of preset time series T i in RPA, such as 1:10, 1:20, 2:05, etc. RPA will execute the query requests according to each moment T i-j-start (equivalent to the query start time) and let RPA record the query completion time.

[0061] It should be noted that for each query start time set in T i , try to keep them scattered and irregularly spaced, which can minimize the interference caused by some system instabilities, such as system jitter, etc. When a group of query requests is completed, for each query start time T i-j-start in T, obtain a query completion time T i-j-end .

[0062] In addition, 1 < i <= length(Q), where i represents any record in Q, denoted as Q i , and each Q i will be configured with a set of query start times, denoted as T i (such as time 1, time 2, time 3, etc.); 1 < j <= length(T i ), where j represents any record in the T i set. Since this is the query start time, it is denoted as T i-j-start , and correspondingly, there is also a matching query completion time T i-j-end .

[0063] 3) Obtain logs; before executing step 2), turn on the log switch of the above database to facilitate obtaining the access logs of the candidate database DB during the RPA execution task period (taking the mysql database as an example, its binlog can be subscribed). This log includes the SQL logs corresponding to the query requests, the query execution time, and all metadata information of the database (database name, table name, field information, etc.).

[0064] 4) Log filtering and interference exclusion; use the results of steps 2) and 3) for data traceability, and accurately find the data source information corresponding to each group of query requests Q i from the logs in step 3) through an algorithm. The algorithm implementation process is as follows:

[0065] Set result set Result = [];

[0066] Traverse each database DB in the candidate database DB k Log: Set Log k =DB k Access log records (that is, SQL logs line by line); traverse T i Each query starts at time T i-j-start , query completion time T i-j-end ; Among them, for Log k Filter the SQL logs in the query, and the filtering condition is that the query execution time is greater than or equal to T i-j-start , and less than or equal to T i -j-end , forming a log fragment (multi-line SQL log) and adding it to the temporary log set TmpLog; for the log fragments in the temporary log set TmpLog, using the string matching algorithm, find out the SQL logs that exist in these log fragments and input them into the temporary result set TmpResult;

[0067] It should be noted that K represents a specific database in the candidate database DB set; accordingly, Log k Indicates the access log of the specific database.

[0068] Traverse each SQL log in TmpResult: k Perform global search and matching in , check all log records matched by the matched TmpResult, sort the corresponding query execution time, and calculate whether the time interval is uniform, that is, if the variance of the calculated time interval is less than the specified threshold, it is likely that there is a periodic query request in the system, which is an interference item, and it is removed from TmpResult; TmpResult is added to Result.

[0069] Finally, the query request Q i , accurately find the corresponding SQL log from the previous candidate database log.

[0070] 5) SQL parsing: In 4) for each set of query requests Q i A Result is obtained, which includes the corresponding SQL logs selected by the algorithm. Then, the SQL parsing tool is used to automatically parse the database name, table name, and field name that each SQL log depends on, and then remove and merge them. In this way, the automatic data tracing in the report scenario is completed.

[0071] It should be noted that the report scenario exemplified above is for easier understanding. In fact, it is the same in other data scenarios. There is no need to pay attention to the internal logic of the data application, and it is not like the traditional data tracing work that follows a routine and investigates the implementation logic of the data application to find the data source. The user only needs to input a query request, and the data tracing method / device provided in the embodiment of the present application can complete the automatic data tracing.

[0072] According to another aspect of the embodiments of the present application, a data tracing device is also provided.

[0073] Figure 3 A schematic diagram of a data tracing device provided in an embodiment of the present application, such as Figure 3 As shown, the data tracing device includes: a first processing module 32, a second processing module 34, a third processing module 36 and a fourth processing module 38. The data tracing device is described in detail below.

[0074] A first processing module 32 is used to trigger multiple identical query requests according to a preset time sequence, and obtain multiple log fragments corresponding to the preset time sequence; wherein the preset time sequence includes multiple non-periodic set moments, each moment corresponding to a log fragment;

[0075] A second processing module 34, connected to the first processing module 32, for obtaining first log entries existing in multiple log segments;

[0076] A third processing module 36, connected to the second processing module 34, is used to remove the log entries with periodic query execution time from the first log entries based on the query execution time corresponding to the first log entry, to obtain the second log entry;

[0077] The fourth processing module 38 is connected to the third processing module 36 and is used to perform data tracing on the query request based on the second log entry.

[0078] In an embodiment of the present application, the data tracing device triggers multiple identical query requests according to a preset time sequence to obtain multiple log fragments corresponding to the preset time sequence; then obtains a first log entry that exists in multiple log fragments; then, based on the query execution time corresponding to the first log entry, removes the log entries with periodic query execution time in the first log entries to obtain second log entries; finally, based on the second log entry, data tracing is performed on the query request, thereby solving the technical problem of poor accuracy in traditional data tracing and achieving the technical effect of improving the accuracy and efficiency of data tracing.

[0079] It should be noted here that the above-mentioned first processing module 32, second processing module 34, third processing module 36 and fourth processing module 38 correspond to steps S102 to S108 in the method embodiment, and the examples and application scenarios implemented by the above-mentioned modules are the same as the corresponding steps, but are not limited to the contents disclosed in the above-mentioned method embodiment.

[0080] As an optional embodiment, the first processing module 32 includes: a first processing unit, configured to configure corresponding preset time sequences for multiple identical query requests, and trigger multiple identical query requests according to the preset time sequences.

[0081] As an optional embodiment, the above-mentioned first processing unit includes: a first processing sub-unit, used to configure corresponding preset time sequences for multiple identical query requests using a random time setting method; or, to configure corresponding preset time sequences for multiple identical query requests using a regular time setting method.

[0082] As an optional embodiment, the above-mentioned first processing module 32 includes: a second processing unit, used to obtain a preset time series and a query completion time series corresponding to the preset time series, wherein each query completion moment in the query completion time series corresponds to a moment in the preset time series; a third processing unit, used to obtain the log when the query request is executed; and a fourth processing unit, used to filter the log fragments of the log according to the preset time series and the query completion time series to obtain multiple log fragments.

[0083] As an optional embodiment, the above-mentioned fourth processing unit includes: a second processing sub-unit, used to obtain the query execution time when the query request is executed in the log; a third processing sub-unit, used to obtain the log fragment between the moment in the preset time sequence and the query completion moment in the query completion time sequence if the query execution time is greater than or equal to the moment in the preset time sequence and less than or equal to the query completion moment in the query completion time sequence, so as to obtain multiple log fragments.

[0084] As an optional embodiment, the second processing module 34 includes: a fifth processing unit, configured to perform string matching on the multiple log segments respectively to obtain the first log entry existing in the multiple log segments.

[0085] As an optional embodiment, the above-mentioned third processing module 36 includes: a sixth processing unit, used to determine the time interval of the query execution time; a seventh processing unit, used to filter the query execution time according to the time interval of the query execution time to obtain a periodic query execution time; an eighth processing unit, used to remove the log entry corresponding to the periodic query execution time from the first log entry to obtain a second log entry.

[0086] As an optional embodiment, the sixth processing unit includes: a fourth processing subunit, used to sort the query execution time to obtain a query execution time sequence; and a fifth processing subunit, used to calculate the time interval of the query execution time according to the query execution time sequence.

[0087] As an optional embodiment, the above-mentioned seventh processing unit includes: a sixth processing sub-unit, used to calculate the time interval variance according to the time interval of the query execution time; and a seventh processing sub-unit, used to determine that the query execution time is a periodic query execution time when the time interval variance is less than a preset time interval variance threshold.

[0088] As an optional embodiment, the fourth processing module 38 includes: a ninth processing unit, used to parse the second log entry to obtain metadata information on which the second log entry depends; and a tenth processing unit, used to filter out metadata information corresponding to the query request based on the metadata information on which the second log entry depends.

[0089] As an optional embodiment, the tenth processing unit includes: an eighth processing sub-unit, configured to perform deduplication and merging processing on metadata information on which the second log entry depends, to obtain metadata information corresponding to the query request.

[0090] The embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program executable by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the present application when executed by the at least one processor.

[0091] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network. The electronic device includes but is not limited to a computer device.

[0092] Optionally, in this embodiment, the computer device may include: a memory and a processor, the memory storing a computer program; the processor, configured to execute the computer program stored in the memory, wherein when the computer program is executed, the processor executes any one of the above method steps.

[0093] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the data tracing method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned data tracing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0094] Optionally, the processor may call the information and application stored in the memory through a transmission device to perform the following steps: trigger multiple identical query requests according to a preset time sequence, and obtain multiple log fragments corresponding to the preset time sequence; wherein the preset time sequence includes multiple non-periodic moments, each moment corresponding to a log fragment; obtain a first log entry that exists in multiple log fragments; based on the query execution time corresponding to the first log entry, remove the log entries with periodic query execution time in the first log entry to obtain a second log entry; and trace the query request based on the second log entry.

[0095] Optionally, in this embodiment, the processor may also execute program code of the following steps: triggering multiple identical query requests according to a preset time sequence, including: configuring corresponding preset time sequences for multiple identical query requests, and triggering multiple identical query requests according to the preset time sequence.

[0096] Optionally, in this embodiment, the processor may also execute the program code of the following steps: configuring corresponding preset time sequences for multiple identical query requests, including: configuring corresponding preset time sequences for multiple identical query requests using a random time setting method; or, configuring corresponding preset time sequences for multiple identical query requests using a regular time setting method.

[0097] Optionally, in this embodiment, the processor may also execute program code for the following steps: obtaining multiple log fragments corresponding to a preset time sequence, including: obtaining a preset time sequence and a query completion time sequence corresponding to the preset time sequence, wherein each query completion moment in the query completion time sequence corresponds to a moment in the preset time sequence; obtaining a log when executing a query request; filtering the log fragments of the log according to the preset time sequence and the query completion time sequence to obtain multiple log fragments.

[0098] Optionally, in this embodiment, the above-mentioned processor can also execute the program code of the following steps: filter the log fragments in the log according to the preset time sequence and the query completion time sequence to obtain multiple log fragments, including: obtain the query execution time when the query request is executed in the log; if the query execution time is greater than or equal to the moment in the preset time sequence, and less than or equal to the query completion moment in the query completion time sequence, then obtain the log fragment between the moment in the preset time sequence and the query completion moment in the query completion time sequence to obtain multiple log fragments.

[0099] Optionally, in this embodiment, the processor may also execute program code of the following steps: obtaining a first log entry existing in multiple log fragments, including: performing string matching on the multiple log fragments respectively to obtain the first log entry existing in the multiple log fragments.

[0100] Optionally, in this embodiment, the processor may also execute program code for the following steps: based on the query execution time corresponding to the first log entry, removing the log entries with periodic query execution time in the first log entry to obtain a second log entry, including: determining the time interval of the query execution time; filtering the query execution time according to the time interval of the query execution time to obtain a query execution time with periodicity; removing the log entries corresponding to the query execution time with periodicity from the first log entry to obtain a second log entry.

[0101] Optionally, in this embodiment, the processor may also execute program code of the following steps: determining the time interval of query execution time, including: sorting the query execution time to obtain a query execution time sequence; and calculating the time interval of query execution time according to the query execution time sequence.

[0102] Optionally, in this embodiment, the processor may also execute program code of the following steps: filtering the query execution time according to the time interval of the query execution time to obtain a query execution time with periodicity, including: calculating the time interval variance according to the time interval of the query execution time; and determining that the query execution time is a query execution time with periodicity when the time interval variance is less than a preset time interval variance threshold.

[0103] Optionally, in this embodiment, the processor may also execute program code of the following steps: tracing the data source of the query request based on the second log entry, including: parsing the second log entry to obtain metadata information on which the second log entry depends; and filtering out metadata information corresponding to the query request based on the metadata information on which the second log entry depends.

[0104] Optionally, in this embodiment, the processor may also execute the following program code: based on the metadata information on which the second log entry depends, filtering out metadata information corresponding to the query request, including: deduplicating and merging the metadata information on which the second log entry depends to obtain metadata information corresponding to the query request.

[0105] The embodiment of the present application further provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application. The non-transitory machine-readable medium includes a storage medium.

[0106] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0107] Optionally, in this embodiment, the storage medium may be used to store the program code corresponding to the domain name query method provided in the above embodiment, and control the processor to execute the above data tracing method when the program code is executed by the processor.

[0108] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: triggering multiple identical query requests according to a preset time sequence, and obtaining multiple log fragments corresponding to the preset time sequence; wherein the preset time sequence includes multiple non-periodic moments, each moment corresponding to a log fragment; obtaining a first log entry that exists in multiple log fragments; based on the query execution time corresponding to the first log entry, removing the log entries with periodic query execution time in the first log entry to obtain a second log entry; and tracing the query request based on the second log entry.

[0109] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program codes for executing the following steps: triggering multiple identical query requests according to a preset time sequence, including: configuring corresponding preset time sequences for multiple identical query requests, and triggering multiple identical query requests according to the preset time sequence.

[0110] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program codes for executing the following steps: configuring corresponding preset time sequences for multiple identical query requests, including: configuring corresponding preset time sequences for multiple identical query requests using a random time setting method; or, configuring corresponding preset time sequences for multiple identical query requests using a regular time setting method.

[0111] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program codes for executing the following steps: obtaining multiple log fragments corresponding to the preset time sequence, including: obtaining the preset time sequence and the query completion time sequence corresponding to the preset time sequence, wherein each query completion moment in the query completion time sequence corresponds to a moment in the preset time sequence; obtaining the log when executing the query request; filtering the log fragments of the log according to the preset time sequence and the query completion time sequence to obtain multiple log fragments.

[0112] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program codes for executing the following steps: filtering the log fragments in the log according to the preset time sequence and the query completion time sequence to obtain multiple log fragments, including: obtaining the query execution time when the query request is executed in the log; if the query execution time is greater than or equal to the moment in the preset time sequence, and less than or equal to the query completion moment in the query completion time sequence, then obtaining the log fragment between the moment in the preset time sequence and the query completion moment in the query completion time sequence to obtain multiple log fragments.

[0113] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program code for executing the following steps: obtaining a first log entry that exists in multiple log fragments, including: performing string matching on multiple log fragments respectively to obtain the first log entry that exists in multiple log fragments.

[0114] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program code for executing the following steps: based on the query execution time corresponding to the first log entry, removing the log entries in the first log entry whose query execution time is periodic to obtain a second log entry, including: determining the time interval of the query execution time; filtering the query execution time according to the time interval of the query execution time to obtain a query execution time with periodicity; removing the log entry corresponding to the query execution time with periodicity from the first log entry to obtain the second log entry.

[0115] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program codes for executing the following steps: determining the time interval of the query execution time, including: sorting the query execution time to obtain a query execution time series; and calculating the time interval of the query execution time based on the query execution time series.

[0116] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program codes for executing the following steps: filtering the query execution time according to the time interval of the query execution time to obtain a query execution time with periodicity, including: calculating the time interval variance according to the time interval of the query execution time; when the time interval variance is less than a preset time interval variance threshold, determining that the query execution time is the query execution time with periodicity.

[0117] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program codes for executing the following steps: tracing the data source of the query request based on the second log entry, including: parsing the second log entry to obtain metadata information on which the second log entry depends; based on the metadata information on which the second log entry depends, filtering out metadata information corresponding to the query request.

[0118] Optionally, in this embodiment, the above-mentioned storage medium is configured to store program code for executing the following steps: based on the metadata information on which the second log entry depends, filtering out the metadata information corresponding to the query request, including: deduplicating and merging the metadata information on which the second log entry depends to obtain the metadata information corresponding to the query request.

[0119] The embodiment of the present application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0120] refer to Figure 4 , the structural block diagram of the electronic device that can be used as the server or client of the embodiment of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0121] like Figure 4As shown, the electronic device includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the electronic device can also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0122] Multiple components in the electronic device are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 can be any type of device that can input information to the electronic device, and the input unit 406 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 can include but is not limited to a disk, an optical disk. The communication unit 409 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0123] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present application may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 402 and / or a communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

[0124] The computer program for implementing the method of the embodiment of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable signal medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0126] It should be noted that the term "including" and its variations used in the embodiments of the present application are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present application are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0127] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0128] The various steps described in the method implementation methods provided in the embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present application is not limited in this respect.

[0129] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0130] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A data tracing method, comprising: Trigger multiple identical query requests according to a preset time sequence, and obtain multiple log fragments corresponding to the preset time sequence; wherein the preset time sequence includes multiple non-periodic set moments, each moment corresponding to a log fragment; Obtaining a first log entry that exists in a plurality of the log fragments; Based on the query execution time corresponding to the first log entry, remove the log entries whose query execution time is periodic from the first log entries to obtain a second log entry; Data tracing is performed on the query request based on the second log entry.

2. The method according to claim 1, wherein: Trigger multiple identical query requests according to a preset time sequence, including: A corresponding preset time sequence is configured for multiple identical query requests, and multiple identical query requests are triggered according to the preset time sequence.

3. The method according to claim 2, wherein: Configuring corresponding preset time sequences for a plurality of identical query requests includes: A random time setting method is used to configure a corresponding preset time sequence for multiple identical query requests; or a regular time setting method is used to configure a corresponding preset time sequence for multiple identical query requests.

4. The method according to claim 1, wherein: Acquiring multiple log segments corresponding to the preset time series includes: Acquire the preset time sequence and a query completion time sequence corresponding to the preset time sequence, wherein each query completion moment in the query completion time sequence corresponds to a moment in the preset time sequence; Obtaining the log when executing the query request; The log segments of the log are screened according to the preset time sequence and the query completion time sequence to obtain a plurality of the log segments.

5. The method according to claim 4, wherein: The log fragments of the log are screened according to the preset time sequence and the query completion time sequence to obtain a plurality of the log fragments, including: Obtain the query execution time when the query request is executed in the log; If the query execution time is greater than or equal to the moment in the preset time sequence, and less than or equal to the query completion moment in the query completion time sequence, then obtain the log fragment between the moment in the preset time sequence and the query completion moment in the query completion time sequence to obtain multiple log fragments.

6. The method according to claim 1, wherein: Obtaining a first log entry that exists in a plurality of log fragments includes: String matching is performed on each of the multiple log segments to obtain the first log entries that exist in the multiple log segments.

7. The method according to claim 1, wherein: Based on the query execution time corresponding to the first log entry, removing the log entries whose query execution time is periodic from the first log entries to obtain the second log entry, including: Determining a time interval for executing the query; The query execution time is filtered according to the time interval of the query execution time to obtain the query execution time with periodicity; The log entry corresponding to the periodic query execution time is removed from the first log entry to obtain the second log entry.

8. The method according to claim 7, wherein: Determining the time interval of the query execution time includes: Sorting the query execution time to obtain a query execution time series; The time interval of the query execution time is calculated according to the query execution time series.

9. The method according to claim 7, wherein: The query execution time is filtered according to the time interval of the query execution time to obtain the query execution time with periodicity, including: Calculate the time interval variance according to the time interval of the query execution time; When the time interval variance is less than a preset time interval variance threshold, the query execution time is determined to be the query execution time with periodicity.

10. The method according to claim 1, wherein: Performing data tracing on the query request based on the second log entry includes: Parsing the second log entry to obtain metadata information on which the second log entry depends; Based on the metadata information on which the second log entry depends, the metadata information corresponding to the query request is filtered out.

11. The method according to claim 10, wherein: Filtering out metadata information corresponding to the query request based on the metadata information on which the second log entry depends includes: De-duplicate and merge the metadata information on which the second log entry depends to obtain metadata information corresponding to the query request.

12. A data tracing device, comprising: A first processing module is used to trigger multiple identical query requests according to a preset time sequence, and obtain multiple log fragments corresponding to the preset time sequence; wherein the preset time sequence includes multiple non-periodic set moments, each moment corresponding to a log fragment; A second processing module is used to obtain a first log entry that exists in a plurality of the log fragments; A third processing module, configured to remove the log entries whose query execution time is periodic from the first log entries based on the query execution time corresponding to the first log entry, to obtain a second log entry; A fourth processing module is used to trace the data source of the query request based on the second log entry.

13. An electronic device comprising: A processor, and a memory storing a program, wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 11.

14. A non-transitory machine-readable medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 11.

Images