A data query method and device

Abstract

The present invention discloses a data query method and device, and relates to the field of computer technology. A specific implementation of the method includes: determining a data query identifier according to a data query request, wherein the data query request is used to query multidimensional data; using multiple threads to read the multidimensional data in a database, and using the data query identifier as a cache identifier for the multidimensional data to cache the multidimensional data; querying the multidimensional data in the cache according to the data query identifier. This implementation can improve the query speed, spread the database pressure, avoid the risk of large SQL queries on the main database, and reduce the pressure of data queries.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data query method and device. Background Art

[0002] In the scenario of multidimensional data query, for statistical data of different dimensions, because each data is of different dimensions, the traditional method is to query the data of each dimension one by one in sequence, and form entity objects to return to the page or required logic. The existing query method for multidimensional data is single-threaded statistics one by one, and uses the main database for data query.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] The query speed is slow and the database pressure cannot be evenly distributed. Statistics and other data put a lot of pressure on the database. When a connection is occupied for a long time without being released when the call volume is slightly high, it is a risk point. In addition, when the query pressure is high, it will also affect other functions of the system. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data query method and device, which can improve the query speed, distribute the database pressure, avoid the risk of large SQL (Structured Query Language) querying the main database, and reduce the pressure of data query.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a data query method is provided.

[0007] A data query method comprises: determining a data query identifier according to a data query request, wherein the data query request is used to query multidimensional data; using multiple threads to read the multidimensional data in a database, and using the data query identifier as a cache identifier for the multidimensional data to cache the multidimensional data; and querying the multidimensional data in the cache according to the data query identifier.

[0008] Optionally, before the step of using multiple threads to read the multidimensional data in the database, the step includes: confirming that there is no data in the cache with the data query identifier as the cache identifier.

[0009] Optionally, the step of determining the data query identifier according to the data query request includes: generating a number corresponding to the submission time period according to the submission time period of the data query request, and determining the number as the data query identifier.

[0010] Optionally, multiple threads are used to read the multidimensional data in the database, and the data query identifier is used as a cache identifier for the multidimensional data. The step of caching the multidimensional data includes: using multiple threads to read the multidimensional data from multiple slave databases, wherein each thread reads data of one dimension from a slave database; organizing the multidimensional data returned by each thread, and associating the organized multidimensional data through the data query identifier; and caching the multidimensional data using the data query identifier as a cache identifier for the multidimensional data.

[0011] According to another aspect of an embodiment of the present invention, a data query device is provided.

[0012] A data query device includes: an identification determination module, used to determine a data query identification according to a data query request, wherein the data query request is used to query multidimensional data; a data cache module, used to read the multidimensional data in a database using multiple threads, and use the data query identification as a cache identification of the multidimensional data to cache the multidimensional data; the data query module is used to query the multidimensional data in the cache according to the data query identification.

[0013] Optionally, it further includes a judgment module, which is used to confirm that there is no data with the data query identifier as the cache identifier in the cache.

[0014] Optionally, the identifier determination module is further used to: generate a number corresponding to the submission time period according to the submission time period of the data query request, and determine the number as the data query identifier.

[0015] Optionally, the data cache module is also used to: use multiple threads to read the multidimensional data from multiple slave databases, wherein each thread reads data of one dimension from a slave database; organize the multidimensional data returned by each thread, and associate the organized multidimensional data through the data query identifier; and use the data query identifier as a cache identifier of the multidimensional data to cache the multidimensional data.

[0016] According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

[0017] An electronic device includes: one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the data query method provided by the present invention.

[0018] According to yet another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0019] A computer readable medium stores a computer program, which implements the data query method provided by the present invention when executed by a processor.

[0020] An embodiment of the above invention has the following advantages or beneficial effects: determining a data query identifier according to a data query request, using multiple threads to read multidimensional data in a database, each thread reading data of one dimension from a slave database, and using the data query identifier as a cache identifier for the multidimensional data, caching the multidimensional data, and querying the multidimensional data in the cache according to the data query identifier. It can improve the query speed, make full use of the slave database, realize the distribution of database pressure, avoid the risk of large SQL queries on the master database, and reduce the pressure of data queries.

[0021] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0023] Figure 1 is a schematic diagram of main steps of a data query method according to an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of a data query process according to an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of main modules of a data query device according to an embodiment of the present invention;

[0026] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0027] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0029] Figure 1 It is a schematic diagram of main steps of a data query method according to an embodiment of the present invention.

[0030] like Figure 1As shown, the data query method of the embodiment of the present invention mainly includes the following steps S101 to S103.

[0031] Step S101: determining a data query identifier according to a data query request, where the data query request is used to query multidimensional data.

[0032] Among them, a number corresponding to the submission time period can be generated according to the submission time period of the data query request, and the number is determined as the data query identifier.

[0033] The above number is generated according to the submission time period of the data query request. Specifically, a number corresponding to the time period to which the submission time of the data query request belongs can be generated. The data query requests submitted within the time period have the same number, that is, the data query identifier is the same. The number generation rule is, for example: a time period is pre-set, and the numbers carried by all data query requests within the time period are the number of milliseconds at the starting time point of the time period. Taking the time period of 10 minutes as an example, the data query requests submitted between 14:20:01 on June 10, 2018 and 14:29:59 on June 10, 2018, all have numbers determined to be equal to the number of milliseconds at 14:20:00 on June 10, 2018. In other embodiments, the number can also be the number of seconds, minutes, hours, etc. at the starting time point of the time period, or a time series whose number is equal to the starting time point, such as 20180610142000.

[0034] Step S102: Use multiple threads to read the multidimensional data in the database, and use the data query identifier as a cache identifier of the multidimensional data to cache the multidimensional data.

[0035] Before step S102, it can be determined whether there is data with the data query identifier as the cache identifier in the cache. If it is confirmed that there is no data with the data query identifier as the cache identifier in the cache, step S102 is executed, and then the cache expiration time is set. Otherwise, the multidimensional data with the data query identifier as the cache identifier is read from the cache.

[0036] Step S102 may include: using multiple threads to read multidimensional data from multiple slave databases, wherein each thread reads data of one dimension from a slave database; organizing the multidimensional data returned by each thread, and associating the organized multidimensional data through a data query identifier; using the data query identifier as a cache identifier for the multidimensional data, and caching the multidimensional data.

[0037] The multidimensional data returned by each thread is organized. Specifically, the multidimensional data returned by each thread can be counted according to the dimension, and the statistical multidimensional data is associated through the data query identifier. For example, statistics are collected from the dimensions of "today's acceptance quantity", "today's shelf quantity", "today's acceptance to be shelf quantity", "historical unshelf quantity", "should be shelf quantity", etc., and the statistical data of each dimension is associated through the data query identifier.

[0038] Use multiple threads to read multidimensional data from multiple slave databases. For example, when calculating the statistical data of a certain dimension, use the first slave database to query the required data. When calculating the second statistical data, use the second slave database to query the required data. And so on. If there are fewer slave databases, the first slave database can be used again after all slave databases are used in a loop. In the embodiment of the present invention, CountDownLatch (a latch in a java thread) can be used to block the main thread. When all child threads return data, the main thread continues to execute, organizes the multidimensional data returned by each thread, and associates these multidimensional data through data query identifiers. When organizing data, a local variable can be used to temporarily store data in the format of Map<number, Map<business KEY, statistical quantity>. Finally, the temporarily stored data is stored in a cache (such as redis (REmote DIctionary Server, a key-value storage system)) at one time.

[0039] Step S103: query the multidimensional data in the cache according to the data query identifier.

[0040] The data query process of the embodiment of the present invention can be as follows: Figure 2 As shown, after receiving the data query request, firstly, the multidimensional data (i.e. Figure 2 If multidimensional data with the number as the cache identifier is obtained from the cache, the obtained multidimensional data is returned to the caller who submitted the data query request (this step Figure 2 (not shown), if the multidimensional data is not obtained from the cache, then the multidimensional data is obtained from slave libraries 1 to 3 respectively through processing thread 1, processing thread 2, and processing thread 3 (this embodiment takes three processing threads (child threads) and three slave libraries as an example). After all processing threads return data of the corresponding dimensions, the multidimensional data returned by these processing threads are written into the cache with the above-generated number as the cache identifier, so that the multidimensional data can be obtained from the cache according to the number. The embodiment of the present invention can improve the query speed, make full use of the slave library, realize the distribution of database pressure, avoid the risk of large SQL querying the main library, reduce the pressure of data query, and will not affect other functions of the system when the query pressure is large.

[0041] Figure 3 It is a schematic diagram of main modules of a data query device according to an embodiment of the present invention.

[0042] like Figure 3 As shown, the data query device 300 according to the embodiment of the present invention mainly includes: an identification determination module 301 , a data cache module 302 , and a data query module 303 .

[0043] The identifier determination module 301 is used to determine a data query identifier according to a data query request, where the data query request is used to query multidimensional data.

[0044] The identifier determination module 301 may be specifically configured to: generate a number corresponding to the submission time period according to the submission time period of the data query request, and determine the number as the data query identifier.

[0045] The data cache module 302 is used to read the multidimensional data in the database using multiple threads, and use the data query identifier as a cache identifier of the multidimensional data to cache the multidimensional data.

[0046] The data cache module 302 can be specifically used to: use multiple threads to read multidimensional data from multiple slave databases, wherein each thread reads data of one dimension from a slave database; organize the multidimensional data returned by each thread, and associate the organized multidimensional data through a data query identifier; use the data query identifier as a cache identifier for the multidimensional data to cache the multidimensional data.

[0047] The data query module 303 is used to query the multidimensional data in the cache according to the data query identifier.

[0048] The data query device 300 may further include a judgment module, which is used to confirm that there is no data with the data query identifier as a cache identifier in the cache.

[0049] In addition, the specific implementation content of the data query device in the embodiment of the present invention has been described in detail in the data query method described above, so the repeated content will not be described again here.

[0050] Figure 4 An exemplary system architecture 400 is shown to which the data query method or data query device according to the embodiment of the present invention can be applied.

[0051] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, 403, network 404 and server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0052] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0053] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0054] The server 405 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 401, 402, and 403. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0055] It should be noted that the data query method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the data query device is generally disposed in the server 405 .

[0056] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0057] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device or server suitable for implementing an embodiment of the present application. Figure 5 The terminal device or server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0058] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0059] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0060] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are executed.

[0061] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0062] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0063] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including an identification determination module, a data cache module, and a data query module. The names of these modules do not, in some cases, constitute limitations on the modules themselves, for example, the identification determination module may also be described as a "module for determining a data query identification according to a data query request".

[0064] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: determining a data query identifier according to a data query request, the data query request is used to query multidimensional data; using multiple threads to read the multidimensional data in the database, and using the data query identifier as a cache identifier of the multidimensional data to cache the multidimensional data; querying the multidimensional data in the cache according to the data query identifier.

[0065] According to the technical solution of the embodiment of the present invention, the data query identifier is determined according to the data query request, and the multidimensional data in the database is read using multiple threads, each thread reads data of one dimension from a slave database, and uses the data query identifier as a cache identifier for the multidimensional data, caches the multidimensional data, and queries the multidimensional data in the cache according to the data query identifier. It can improve the query speed, make full use of the slave database, realize the distribution of database pressure, avoid the risk of large SQL queries on the master database, reduce the pressure of data queries, and will not affect other functions of the system when the query pressure is high.

[0066] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data query method, characterized in that: include: Generate a number corresponding to the time period to which the submission time of the data query request belongs, and determine the number as a data query identifier, wherein the data query request is used to query multidimensional data; Read the multidimensional data in the database using multiple threads, and use the data query identifier as a cache identifier of the multidimensional data to cache the multidimensional data; The multidimensional data in the cache is queried according to the data query identifier.

2. The method according to claim 1, characterized in that Before the step of using multiple threads to read the multidimensional data in the database, the method includes: Confirm that there is no data in the cache with the data query identifier as the cache identifier.

3. The method according to claim 1, characterized in that The step of using multiple threads to read the multidimensional data in the database, using the data query identifier as a cache identifier of the multidimensional data, and caching the multidimensional data includes: Using multiple threads to read the multidimensional data from multiple slave databases, wherein each thread reads data of one dimension from one slave database; Organizing the multidimensional data returned by each thread, and establishing associations between the organized multidimensional data through the data query identifier; The multidimensional data is cached using the data query identifier as a cache identifier of the multidimensional data.

4. A data query device, characterized in that: include: An identifier determination module, used to generate a number corresponding to the time period to which the submission time of the data query request belongs, and determine the number as the data query identifier, wherein the data query request is used to query multidimensional data; A data cache module, used to read the multidimensional data in the database using multiple threads, and use the data query identifier as a cache identifier of the multidimensional data to cache the multidimensional data; A data query module is used to query the multidimensional data in the cache according to the data query identifier.

5. The device according to claim 4, characterized in that It also includes a judgment module for: Confirm that there is no data in the cache with the data query identifier as the cache identifier.

6. The device according to claim 4, characterized in that The data cache module is also used for: Using multiple threads to read the multidimensional data from multiple slave databases, wherein each thread reads data of one dimension from one slave database; Organizing the multidimensional data returned by each thread, and establishing associations between the organized multidimensional data through the data query identifier; The multidimensional data is cached using the data query identifier as a cache identifier of the multidimensional data.

7. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 3.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images