Configuration result blood relationship accelerated query method, system, equipment and medium

Abstract

The invention discloses a configuration result blood relationship accelerated query method, system and device and a medium, and the method comprises the steps: when receiving an event key-based query request, carrying out word segmentation on the queried event key to generate a plurality of sub-strings with fixed lengths; mapping each sub-character string to the position of a bit array by using a Bloom filter to obtain a queried bit array; comparing the queried bit array with a bit array in a result table to obtain a candidate record; character string matching confirmation is conducted on the candidate records, the candidate records which are successfully matched and confirmed serve as query results and are output, and the method, the system, the equipment and the medium can improve query efficiency and accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of data analysis, and relates to a method, system, device and medium for accelerating query of blood relationship of configuration results. Background Art

[0002] In large-scale data processing and analysis systems, calculation results are usually stored in JSON format to record complex data dependencies and operation results in a structured manner. Especially in configuration tools, each calculation result of a canvas often contains unique identifiers of multiple input and output operators (i.e., eventkeys), which are used to track data flows and dependencies between canvases. However, with the increase in the number of canvases and calculation results, the amount of JSON data stored in the result table grows rapidly, resulting in a decrease in the performance of eventkey-based dependency queries. In traditional methods, string matching (such as LIKE operations) is usually used to find specific eventkeys, but this approach will lead to serious performance bottlenecks on large data sets, long query response times, and is difficult to meet the needs of real-time queries. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method, system, device and medium for accelerated query of configuration result bloodline, which can improve the efficiency and accuracy of the query.

[0004] To achieve the above object, the present invention discloses a configuration result lineage accelerated query method, comprising:

[0005] When receiving a query request based on eventkey, the eventkey of the query is segmented to generate several substrings of fixed length;

[0006] Use Bloom filter to map each substring to the position of the bit array to obtain the queried bit array;

[0007] Compare the queried bit array with the bit array in the result table to obtain candidate records;

[0008] Perform string matching on candidate records, and output candidate records that have successfully matched as query results.

[0009] The configuration result blood relationship accelerated query method of the present invention is further improved in that:

[0010] Furthermore, before comparing the queried bit array with the bit array in the result table, the process further includes:

[0011] Get JSON data text;

[0012] Segment the JSON data text to obtain several substrings of fixed length;

[0013] Use a Bloom filter to map each substring to a position in a bit array, generating a bit array representing each JSON data text;

[0014] The bit array and the JSON data text are stored in a result table.

[0015] Furthermore, the mapping of the Bloom filter is implemented through multiple hash functions.

[0016] Furthermore, the method further includes: finding the best balance point between the misjudgment rate and the storage cost of the Bloom filter by adjusting the length of the bit array and the number of hash functions, so as to obtain the optimal Bloom filter.

[0017] The present invention discloses a configuration result blood relationship accelerated query system, comprising:

[0018] The first word segmentation module is used for, when receiving a query request based on an eventkey, segmenting the query eventkey to generate a plurality of substrings of a fixed length;

[0019] A query module, used for mapping each substring to a position of a bit array using a Bloom filter to obtain a queried bit array;

[0020] A comparison module, used to compare the queried bit array with the bit array in the result table to obtain candidate records;

[0021] The confirmation module is used to perform string matching confirmation on candidate records, and output the candidate records that have been successfully matched as query results.

[0022] The configuration result blood relationship accelerated query system of the present invention is further improved in that:

[0023] Furthermore, it also includes:

[0024] Get module, used to get JSON data text;

[0025] The second word segmentation module is used to segment the JSON data text to obtain several substrings of fixed length;

[0026] A generation module, used for mapping each substring to a position of a bit array using a Bloom filter, and generating a bit array representing each JSON data text;

[0027] The storage module is used to store the bit array and the JSON data text in a result table.

[0028] Furthermore, the mapping of the Bloom filter is implemented through multiple hash functions.

[0029] Furthermore, it also includes:

[0030] The optimization module is used to find the best balance point between the misjudgment rate and storage cost of the Bloom filter by adjusting the length of the bit array and the number of hash functions, so as to obtain the optimal Bloom filter.

[0031] The present invention discloses a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the configuration result bloodline accelerated query method are implemented.

[0032] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the configuration result bloodline accelerated query method are implemented.

[0033] The present invention has the following beneficial effects:

[0034] The configuration result lineage accelerated query method, system, device and medium described in the present invention, in specific operation, segment JSON data into substrings of fixed length, and use Bloom filter to generate bit arrays corresponding to each record, so as to construct a fast query index. In the query stage, the input eventkey is also segmented and converted into a bit array, and the records that may contain the eventkey are screened out by comparing with the bit array recorded in the result table, and then string matching is confirmed to avoid direct search of the entire JSON text, significantly improve query efficiency, and be suitable for complex data environments that require frequent execution of dependency search and data lineage tracking. Through the present invention, fast dependency query can be implemented on large data sets, reducing response time and improving the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0036] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0039] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0040] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0041] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0042] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0045] Embodiment 1

[0046] refer to Figure 1 The configuration result blood relationship accelerated query method of the present invention comprises the following steps:

[0047] 1) Word segmentation and index construction of JSON data;

[0048] When storing JSON data, a Bloom filter-based bit array index is generated for each record. The specific operation process is as follows:

[0049] 11) Participle;

[0050] Use the word segmentation tool to split the JSON data text into several substrings of fixed length. If the word segmentation length is set to L, the entire JSON string will be divided into several substrings of length L, ensuring that each substring covers part of the JSON content. For example, the JSON string {"key1":"value1","key2":"value2"}, when the length is 4, will generate a substring set {"{"key","key1",":"va","valu","alue","ue1,","key2"}.

[0051] The specific pseudo code is:

[0052]

[0053] 12) Bloom filter generates a bit array;

[0054] For each word segmentation result, a Bloom filter is used to generate a bit array. Specifically, the Bloom filter maps each substring to a position in a bit array to generate a bit array representing the entire JSON record. The mapping of the Bloom filter is implemented through multiple hash functions to ensure that the bit array is evenly distributed.

[0055] The pseudo code is:

[0056]

[0057] 13) Index storage;

[0058] Store the bit array of each JSON record together with the original JSON data in the result table to avoid rebuilding the index on each query.

[0059] 2) Processing of query requests;

[0060] When an eventkey-based query request is received, the query is processed as follows to quickly find dependencies.

[0061] 21) Query the word segmentation and bit array generation of eventkey;

[0062] The eventkey of the query is segmented to generate a set of substrings of the same fixed length as that used in the index. The substrings are mapped to a bit array using a Bloom filter to generate the bit array of the query.

[0063] The pseudo code is:

[0064]

[0065] 22) Bit array comparison;

[0066] Compare the query bit array with the bit array of each record in the result table one by one. When the value of the bit array of the record in the result table is 1 in all query bits, it is determined that the record may contain the target eventkey, and the candidate record is obtained, thereby quickly filtering out the unmatched records.

[0067] The pseudo code is:

[0068]

[0069] 23) String matching confirmation;

[0070] Since the bit array comparison of the Bloom filter may produce misjudgment, it is necessary to perform string matching confirmation on the candidate records. Specifically, for the screened candidate records, string matching (such as regular expressions) will be used to directly search for eventkey in the JSON field to ensure accurate results.

[0071] The pseudo code is:

[0072]

[0073] 3) The result is returned;

[0074] Returns all confirmed JSON records containing the target eventkey for the system to track and analyze dependencies.

[0075] 4) Bloom filter parameter tuning;

[0076] To further optimize the solution, the parameters of the Bloom filter should be configured according to the actual needs of the system. For example, the length of the bit array and the number of hash functions directly affect the false positive rate of the Bloom filter. The specific configuration depends on the average size of the JSON data and the length of the segmentation. The formula for calculating the false positive rate is:

[0077]

[0078] Where k is the number of hash functions, n is the number of bits in the bit array, and m is the number of substrings after word segmentation of the JSON string.

[0079] By adjusting the bit array length m and the number of hash functions k, the optimal balance between the false positive rate and the storage cost is found to obtain the optimal Bloom filter.

[0080] 5) Algorithm complexity analysis;

[0081] 51) Index construction;

[0082] The complexity of index construction is O(N·M·k), where N is the number of records in the JSON data, M is the number of tokens in each record, and k is the number of hash functions. Due to the high computational efficiency of Bloom filters, this process is suitable for batch processing and offline index construction.

[0083] 52) Query processing;

[0084] During the query process, the complexity of bit array comparison is O(Q·M·k), where Q is the number of candidate records. Since Bloom filters can quickly exclude records that do not contain eventkey, they can significantly reduce the number of direct string matches, making queries more efficient.

[0085] It should be noted that the present invention has the following characteristics:

[0086] The present invention avoids direct search of the entire JSON string through Bloom filter and word segmentation technology, significantly reducing the query response time of the system. In addition, since the bit array design of Bloom filter occupies less memory and supports parallel processing, it is suitable for large-scale data environment. Compared with the traditional LIKE string search, the present invention not only improves the query performance, but also has good scalability and flexibility, and is suitable for complex data lineage tracking scenarios.

[0087] The present invention combines Bloom filter and word segmentation technology to quickly locate records containing specific eventkeys, effectively reducing the number of direct string searches, thereby significantly improving query efficiency, and is particularly suitable for dependency relationship searches in large-scale JSON data.

[0088] The bit array of the Bloom filter in the present invention occupies little memory and is easy to store. Compared with the traditional LIKE search, the demand for CPU and memory is greatly reduced, and the overall performance of the system is improved.

[0089] The present invention is suitable for frequent dependency queries and complex cross-canvas data tracking needs, and can be flexibly adjusted according to different word segmentation lengths and hash parameters to meet various data scale and accuracy requirements.

[0090] The Bloom filter in the present invention controls the misjudgment rate within an acceptable range by tuning parameters, thereby ensuring query efficiency without sacrificing query accuracy.

[0091] Embodiment 2

[0092] The configuration result blood relationship accelerated query system of the present invention comprises:

[0093] The first word segmentation module is used for, when receiving a query request based on an eventkey, segmenting the query eventkey to generate a plurality of substrings of a fixed length;

[0094] A query module, used for mapping each substring to a position of a bit array using a Bloom filter to obtain a queried bit array;

[0095] A comparison module, used to compare the queried bit array with the bit array in the result table to obtain candidate records;

[0096] The confirmation module is used to perform string matching confirmation on candidate records, and output the candidate records that have been successfully matched as query results.

[0097] As an embodiment of the present invention, it also includes:

[0098] Get module, used to get JSON data text;

[0099] The second word segmentation module is used to segment the JSON data text to obtain several substrings of fixed length;

[0100] A generation module, used for mapping each substring to a position of a bit array using a Bloom filter, and generating a bit array representing each JSON data text;

[0101] The storage module is used to store the bit array and the JSON data text in a result table.

[0102] As an implementation manner of the present invention, the mapping of the Bloom filter is implemented by multiple hash functions.

[0103] As an embodiment of the present invention, it also includes:

[0104] The optimization module is used to find the best balance point between the misjudgment rate and storage cost of the Bloom filter by adjusting the length of the bit array and the number of hash functions, so as to obtain the optimal Bloom filter.

[0105] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0106] Embodiment 3

[0107] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the configuration result lineage acceleration query method are implemented, for example, comprising: obtaining JSON data text; segmenting the JSON data text to obtain a plurality of substrings of fixed length; mapping each substring to a position of a bit array using a Bloom filter to generate a bit array representing each JSON data text; storing the bit array and the JSON data text in a result table; when receiving a query request based on an eventkey, segmenting the eventkey of the query to generate a plurality of substrings of fixed length; mapping each substring to a position of a bit array using a Bloom filter to obtain a queried bit array; comparing the queried bit array with the bit array in the result table to obtain candidate records; performing string matching confirmation on the candidate records, and outputting the candidate records with successful matching confirmation as query results. The memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc. The processor, network interface, and memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0108] Embodiment 4

[0109] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the configuration result lineage acceleration query method are implemented, for example, including: obtaining a JSON data text; segmenting the JSON data text to obtain a number of substrings of fixed length; mapping each substring to a position of a bit array using a Bloom filter to generate a bit array representing each JSON data text; storing the bit array and the JSON data text in a result table; when receiving a query request based on an eventkey, segmenting the eventkey of the query to generate a number of substrings of fixed length; using a Bloom filter to map each substring to a position of a bit array to obtain a query bit array; comparing the query bit array with the bit array in the result table to obtain a candidate record; performing string matching confirmation on the candidate record, and outputting the candidate record with successful matching confirmation as a query result. Specifically, the computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory, etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, and the like.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0115] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0116] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A configuration result lineage accelerated query method, characterized in that: include: When receiving a query request based on eventkey, the eventkey of the query is segmented to generate several substrings of fixed length; Use Bloom filter to map each substring to the position of the bit array to obtain the queried bit array; Compare the queried bit array with the bit array in the result table to obtain candidate records; Perform string matching on candidate records, and output candidate records that have successfully matched as query results.

2. The configuration result lineage accelerated query method according to claim 1 is characterized in that: Before comparing the queried bit array with the bit array in the result table, the method further includes: Get JSON data text; Segment the JSON data text to obtain several substrings of fixed length; Use a Bloom filter to map each substring to a position in a bit array, generating a bit array representing each JSON data text; The bit array and the JSON data text are stored in a result table.

3. The configuration result lineage accelerated query method according to claim 1 is characterized in that: The mapping of Bloom filters is implemented through multiple hash functions.

4. The configuration result lineage accelerated query method according to claim 3 is characterized in that: Also includes: By adjusting the length of the bit array and the number of hash functions, the optimal balance between the misjudgment rate and storage cost of the Bloom filter is found to obtain the optimal Bloom filter.

5. A configuration result lineage accelerated query system, characterized in that: include: The first word segmentation module is used for, when receiving a query request based on an eventkey, segmenting the query eventkey to generate a plurality of substrings of a fixed length; A query module, used for mapping each substring to a position of a bit array using a Bloom filter to obtain a queried bit array; A comparison module, used to compare the queried bit array with the bit array in the result table to obtain candidate records; The confirmation module is used to perform string matching confirmation on candidate records, and output the candidate records that have been successfully matched as query results.

6. The configuration result blood relationship accelerated query system according to claim 5, characterized in that: Also includes: Get module, used to get JSON data text; The second word segmentation module is used to segment the JSON data text to obtain several substrings of fixed length; A generation module, used for mapping each substring to a position of a bit array using a Bloom filter, and generating a bit array representing each JSON data text; The storage module is used to store the bit array and the JSON data text in a result table.

7. The configuration result blood relationship accelerated query system according to claim 5, characterized in that: The mapping of Bloom filters is implemented through multiple hash functions.

8. The configuration result blood relationship accelerated query system according to claim 7, characterized in that: Also includes: The optimization module is used to find the best balance point between the misjudgment rate and storage cost of the Bloom filter by adjusting the length of the bit array and the number of hash functions, so as to obtain the optimal Bloom filter.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the configuration result bloodline accelerated query method as described in any one of claims 1-4 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the configuration result bloodline accelerated query method as described in any one of claims 1-4 are implemented.

Images