Data query method and device, equipment, storage medium and product

Abstract

The invention relates to the technical field of databases, in particular to a data query method, device and equipment, a storage medium and a product, and the method comprises the following steps: setting a query request and a filtering condition, defining a multi-level joint query logic, and specifying a result storage address space; executing a first-level query operation according to the query request; implementing first-stage filtering treatment according to the filtering conditions; expanding a second-level query process based on the first-level query result; carrying out second-stage filtering and screening on the second-stage query result; and obtaining and effectively organizing a query result. According to the method, the multi-stage joint query and filtering functions and a special packaging method of the graphic data structure are added, so that the effects of more efficient data interaction, network resource saving and obvious improvement of the processing speed are realized.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data query method, device, equipment, storage medium and product. Background Art

[0002] In today's big data era, performance has always been the focus of database developers. In recent years, NoSQL (not only SQL) databases, as a type of high-performance, non-relational, open-source memory database, have been favored by enterprises and developers. A typical representative of NoSQL memory databases is redis, which is a key-value type database product that has been widely used in big data platforms and cache systems. Compared with relational databases, it has the advantages of fast service response, high concurrent access, easy expansion, and easy development. How to further improve the performance of this type of database has become a hot topic for research in the industry.

[0003] However, existing NoSQL databases still have certain limitations in some aspects. In complex data query scenarios, multiple network interactions are often required to obtain the required data. Such frequent network interactions not only increase system latency and reduce the efficiency of data interaction, but may also occupy more network bandwidth resources due to the transmission of large amounts of data.

[0004] In addition, the traditional database processing method is usually to transmit all the queried data to the caller's local area, and then filter and process it. This method not only increases the burden of network transmission, but also consumes more computing resources when filtering at the caller's local area.

[0005] For data structure processing, traditional methods require deserialization operations before using data, which is not only cumbersome but also brings additional performance overhead. At the same time, data copying operations will also occupy a certain amount of memory space, affecting the overall performance of the system.

[0006] As data scale continues to expand and application scenarios become increasingly complex, higher requirements are placed on the performance and efficiency of database systems. How to reduce the number of network interactions, reduce network bandwidth usage, increase data processing speed, and optimize the use of data structures have become urgent issues to be solved in the field of database technology. Summary of the invention

[0007] The present invention provides a data query method, device, equipment, storage medium and product for improving data query efficiency.

[0008] In a first aspect, the present invention provides a data query method, the method comprising:

[0009] Set query requests and filter conditions, define multi-level cascade query logic, and specify the result storage address space;

[0010] Execute the first level query operation according to the query request;

[0011] Implement the first level of filtering according to the filtering conditions;

[0012] Launch the second-level query process based on the first-level query results;

[0013] Perform second-level filtering on the second-level query results;

[0014] Obtain and efficiently organize query results.

[0015] The query request accurately specifies the data range to be queried through a specific combination of key identifiers; the query operation will quickly locate and return one or more groups of data that meet the conditions based on the key-value storage structure; the key-value storage structure introduces the C++ standard template library STL to express more complex value relationships; the second-level query process is based on the information contained in the previous level query result, according to the defined multi-level joint query logic, and uses the key identifiers related to the second-level data to continue the query operation in the database; the second-level query result can be further joint-queried with other types of data to continue more levels of query association.

[0016] In a second aspect, the present invention provides a data query device, comprising:

[0017] The filtering module quickly filters the results based on the filtering parameters on the database side;

[0018] The multi-level joint query module performs multi-level key-value pair queries on the database side according to the custom joint query logic, parses and filters data values, and finally generates a graphical data structure result to achieve deep data mining and association analysis;

[0019] The graphics data structure result storage module efficiently allocates and stores the graphics data structure results in memory, using continuous memory addresses to facilitate direct access by the caller through memory mapping.

[0020] In a third aspect, the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the data query method as described in the first aspect.

[0021] In a fourth aspect, the present invention provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the data query method as described in the first aspect.

[0022] In a fifth aspect, the present invention provides a computer program product, comprising: a computer program, when the computer program is executed by a processor of an electronic device, the processor is caused to execute the data query method as described in the first aspect.

[0023] The beneficial technical effects of the technical solution of the present invention include at least:

[0024] The present invention is based on the existing NoSQL database design, and adds multi-level cascade query and filtering functions, as well as a special packaging (serialization) method for graph data structures. In the scenario where the existing NoSQL database requires multiple network interactions, the present invention only requires one network interaction, and the data interaction is more efficient; the present invention greatly reduces the network bandwidth occupancy, directly filters invalid data on the database side, without transmitting these invalid data to the caller for local filtering, saving network resources; the graph data structure returned by the database side of the present invention can be used on the caller without deserialization, and no additional data copy is required, and can be directly accessed by pointer through memory mapping, thereby improving processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flow chart of a data query method provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a key-value type database storage structure according to an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the structure of a data query device provided by an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of key-value type database filtering according to an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a multi-level cascade query of a key-value type database according to an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of a graphical data structure result according to an embodiment of the present invention;

[0031] Figure 7 A schematic diagram of a memory mirror of a graphics data structure result according to an embodiment of the present invention;

[0032] Figure 8A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The specific implementation of the present invention is described in detail below. The present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used are only for describing specific embodiments and are not intended to limit the present invention.

[0035] Figure 1 The flowchart of a data query method provided by an embodiment of the present invention can be executed by a data query device, which can be implemented in the form of hardware and / or software, and can be configured in a computer terminal, server, etc. Figure 1 As shown, the method includes:

[0036] S101: Set query request and filter conditions, define multi-level cascade query logic, and specify result storage address space.

[0037] Clarify the type of data targeted by this query, and use a specific combination of key identifiers to accurately specify the data range to be queried. These key identifiers may cover multiple data dimensions, such as business type, regional scope, numbering rules, etc., which together constitute the conditions for uniquely identifying a set of data.

[0038] Set a series of conditions for filtering data. These conditions can be based on various attributes of the data, such as time range, numerical value, specific value of a specific field, etc. Through these filtering conditions, you can accurately filter out a subset of data that meets specific business needs.

[0039] Define the complex relationships between different data types and the order and flow of multi-level queries. This involves determining the dependencies and hierarchical structures between data and how to guide subsequent levels of queries through the results of the previous level of queries to achieve associative analysis of data.

[0040] Plan the memory address range used to store query results to ensure that there is enough continuous memory space to efficiently store and manage the queried data, improve data access speed and processing efficiency, and reduce the impact of memory fragmentation on system performance.

[0041] S102: Execute a first-level query operation according to the query request.

[0042] It should be noted that key-value is a common data structure and data storage method. In the key-value mode, the "key" is an identifier used to uniquely identify data, similar to a label or index. The "value" is the data content associated with the key. Figure 2 This is a key-value query situation, each key has a corresponding value.

[0043] Based on the key combination key in the query request, an accurate search operation is performed in the database to obtain the first-level data that matches the query condition. The database will quickly locate and return one or more sets of data that meet the conditions based on the key-value storage structure.

[0044] S103: Implement the first level filtering process according to the filtering conditions.

[0045] According to the pre-set filtering conditions, the data obtained from the first-level query is strictly screened and filtered. This process will check whether each piece of data meets the filtering conditions one by one, remove the data that does not meet the conditions from the result set, and only retain the data that meets all the set conditions to ensure the accuracy and effectiveness of the first-level query results.

[0046] S104: launching a second-level query process based on the first-level query results.

[0047] Based on the valid data results obtained after the first-level filtering, the query operation is continued in the database according to the defined multi-level joint query logic, using the key identifiers related to the second-level data. This process will further mine the second-level data related to it based on the information contained in the previous-level query results to achieve multi-level related query of data.

[0048] S105: Perform second-level filtering on the second-level query results.

[0049] The data obtained from the second-level query is filtered again according to the set filtering conditions. These filtering conditions may involve the logical relationship between multiple fields, the matching of specific data patterns, and more advanced data validation rules. Through a strict screening process, it is ensured that every piece of data in the final result set meets all the filtering conditions of the first and second levels at the same time, thereby ensuring the accuracy and completeness of the results.

[0050] It should be noted that the secondary query results can be further linked with other types of data to continue query associations at more levels.

[0051] S106: Obtain and effectively organize query results.

[0052] After multiple levels of query and filtering, a data set that fully meets the query requirements and all filtering conditions is obtained. This data is orderly sorted and organized according to the pre-designed data structure for subsequent processing and use. Finally, the results are returned to the caller in an efficient and easy-to-understand and easy-to-use way, such as encapsulating and returning in a specific data format or data container.

[0053] Figure 3 A schematic diagram of the structure of a data query device provided by an embodiment of the present invention. Figure 3 As shown, the device comprises:

[0054] The filtering module quickly filters the results based on the filtering parameters on the database side.

[0055] The purpose of the filtering module is to quickly filter the value results of a single key-value query and delete invalid values ​​according to the filtering parameters, which can greatly reduce the amount of data for subsequent joint key queries.

[0056] Take value1 as an example, its content includes key3, key4, key5, key6, key7, key8 and other keys for subsequent data query. In the general query method, the caller needs to obtain the values ​​corresponding to all these keys, and then filter invalid values ​​locally on the caller. This method is too inefficient and occupies a high network bandwidth. The filtering module in the present invention can filter directly on the database side, such as Figure 4 As shown, value4, value7, value8 and other values ​​do not need to be transmitted to the caller, but are directly filtered and discarded on the database side. It should be noted that in some business scenarios, there are tens of thousands of key values ​​in some values, and the values ​​corresponding to these key values ​​contain the valid time range. After time filtering, there may be only a few valid data remaining. In this scenario, the filtering mechanism of the present invention avoids the network transmission of tens of thousands of value values.

[0057] The multi-level joint query module performs multi-level key-value pair queries on the database side according to the custom joint query logic, parses and filters data values, and finally generates a graphical data structure result to achieve deep data mining and association analysis.

[0058] like Figure 5 As shown in the figure, in a single data query process, multi-level key-value data query work is completed on the database side, and the intermediate results do not need to be returned to the caller. Finally, a graph data structure consisting of value values ​​is generated, such as Figure 6 .

[0059] The multi-level joint query module directly expands the value on the database side, parses the value and obtains multiple key values ​​according to the user-defined joint query function and joint query parameters; then obtains new value locally on the database side through these newly obtained key values; if there are filter functions and filter parameters, these new value values ​​are filtered; and so on, the filtered value is parsed and queried again.

[0060] The graphics data structure result storage module efficiently allocates and stores the graphics data structure results in memory, using continuous memory addresses to facilitate direct access by the caller through memory mapping.

[0061] The function of this module is to efficiently allocate and store the graph data structure results in memory. The purpose is to store the graph data structure in the form of continuous memory addresses within the address space range specified in the query request.

[0062] like Figure 7 As shown, the returned result is an array of value+value relationship array pointers. The value relationship array pointer is a pointer to the next value pointer array structure. In the figure, the value1 relationship array pointer points to the pointer array composed of valu3 pointer, valu5 pointer, and valu6 pointer. The same background color indicates this pointing relationship.

[0063] It should be noted that the above pointer stores the memory address of another variable. Through the pointer, the memory space pointed to can be directly operated to achieve efficient access and processing of data.

[0064] Compared with the general query process, the input parameters include not only the query request, but also new multi-level cascade query and filtering parameters and result address space, etc.; in addition to the query module, the database side also adds a multi-level cascade query module and a filtering module. This embodiment not only reduces the number of network interactions, but also greatly reduces the amount of network data transmission; the multi-level cascade query result is presented as a graphical data structure. In order to improve the efficiency of the caller in reading the results, the structure is stored in the specified free address space. The calling process can directly memory-map the data structure without deserialization, thereby reducing memory usage.

[0065] It should be noted that the present invention introduces the C++ Standard Template Library STL (Standard Template Library) to express more complex value relationships. It consists of six major components, including generic containers (including multiple basic data structures and can be nested), generic algorithms, iterators, function objects, adapters, and memory allocators. The generic containers of STL provide commonly used data structures, and the default memory allocator is defective. Through a custom memory allocator, shared C++ objects can be allocated from a specified address space, and private objects are still allocated from the process heap space. Finally, the continuous memory block data of the specified memory area is returned to the caller.

[0066] In one possible design, the above Figure 3 The structure of the device shown can be implemented as an electronic device. Figure 8 As shown, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable codes, and when the executable codes are executed by the processor 21, the processor 21 can at least implement the data query method provided in the above embodiments.

[0067] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the data query method provided in the aforementioned embodiment.

[0068] The device embodiments described above are merely illustrative, wherein the network elements described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0069] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0070] Example:

[0071] The use case data uses two types of data, Record1 and Record2. The specific data is as follows:

[0072] The key of Record1 in the database is airline + rule area + rule number + fare basis, and the corresponding value is one or more Record1s. The reason for this one-to-many relationship is that the effective time intervals are different.

[0073] The key of Record2 in the database is airline + rule area + rule number, and the corresponding value is one or more Record2. The freight rate basis is not a key member because there is a wildcard in this field. The freight rate type can be empty, which means matching all; if it is not empty, it needs to strictly match the freight rate type of Record1.

[0074] It should be noted that a wildcard is a special character or symbol used to represent an uncertain or variable part in a query condition. It can be used to match multiple possible values, thereby increasing the flexibility and versatility of the query. For example, when querying the freight basis, if "B*" is used as a wildcard, it will match any freight basis value starting with "B", such as "B1", "B22", "BXYZ", etc.

[0075] The data content is as follows:

[0076]

[0077]

[0078] S101: Set query request and filter conditions, define multi-level cascade query logic, and specify result storage address space.

[0079] Query request: The key of Record1, that is, airline + rule area + rule number + tariff basis. In this example, it is: CA008CN01BE70NR.

[0080] Filter parameters: The current date is 20230116, which needs to match the valid time range of Record1 / Record2.

[0081] Multi-level linked query parameters: The business process of associating Record1 with Record2.

[0082] Address space: The first address is 0X000001000000000, which is a continuous 100M virtual address space.

[0083] S102: Execute a first-level query operation according to the query request.

[0084] According to the query request CA008CN01BE70NR, two Record1s are found as follows

[0085]

[0086] S103: Implement the first level filtering process according to the filtering conditions.

[0087] According to the filter parameters (the current date is 20230116), filter out an invalid Record1

[0088]

[0089] S104: launching a second-level query process based on the first-level query results.

[0090] According to the matched Record1, query Record2, and the query key (i.e. the key of Record2: airline + rule area + rule number) is CA008CN01. Found 4 Record2 data.

[0091]

[0092] S105: Perform second-level filtering on the second-level query results.

[0093] The filtering process is divided into three steps: freight rate basis, effective time interval, and freight rate type. The filtering results are as follows:

[0094]

[0095] S106: Obtain and effectively organize query results.

[0096] The final result data is as follows:

[0097]

[0098] It should be noted that this example is simplified for the convenience of explanation. A Record1 only matches one Record2, but in actual situations, dozens of Record2s will be matched. In addition, Record2 needs to be linked with other types of data, which will not be explained in detail due to space constraints.

Claims

1. A data query method, characterized in that: include: Set query requests and filter conditions, define multi-level cascade query logic, and specify the result storage address space; Execute the first level query operation according to the query request; Implement the first level of filtering according to the filtering conditions; Launch the second-level query process based on the first-level query results; Perform second-level filtering on the second-level query results; Obtain and efficiently organize query results.

2. The data query method according to claim 1, characterized in that: The query request accurately specifies the data range to be queried through a specific key identifier combination.

3. The data query method according to claim 1, characterized in that: The query operation will quickly locate and return one or more groups of data that meet the conditions based on the key-value storage structure.

4. The data query method according to claim 3, characterized in that: The key-value storage structure introduces the C++ standard template library STL to express more complex value relationships.

5. The data query method according to claim 1, characterized in that: The second-level query process is to continue querying in the database according to the information contained in the previous-level query result, in accordance with the defined multi-level joint query logic, and using key identifiers related to the second-level data.

6. The data query method according to claim 1, characterized in that: The second-level query results can be further linked with other types of data to continue more levels of query association.

7. A data query device, characterized in that: include: The filtering module quickly filters the results based on the filtering parameters on the database side; The multi-level joint query module performs multi-level key-value pair queries on the database side according to the custom joint query logic, parses and filters data values, and finally generates a graphical data structure result to achieve deep data mining and association analysis; The graphics data structure result storage module efficiently allocates and stores the graphics data structure results in memory, using continuous memory addresses to facilitate direct access by the caller through memory mapping.

8. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the data query method as described in any one of claims 1 to 6.

9. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the data query method according to any one of claims 1 to 6.

10. A computer program product, characterized in that include: A computer program, when executed by a processor of an electronic device, causes the processor to execute the data query method according to any one of claims 1 to 6.

Images