A data query system, data query method, device and server

Abstract

The present invention discloses a data query system and a data query method, device and server. The system includes: an online query library, a data update queue and an online query service. The online query library stores the online data set to be queried and is deployed in the local memory of the online query service. The data update queue stores the update log of the online data set. The online query service updates the online data set in the online query library according to the update log in the data update queue, and queries the online data set in the online query library through the local memory when receiving the data query instruction. The technical solution provided by the present invention ensures the accuracy of the query data in the online query library, avoids the query performance bottleneck problem caused by network bandwidth limitation when accessing the data to be queried through the network, converts the network access query into the local memory access query, greatly reduces the access delay of the data query, and thus improves the efficiency of the data query.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data query system and a data query method, device and server. Background Art

[0002] With the rapid development of Internet technology, there is a large amount of Internet data in different business scenarios in the Internet field to recommend high-quality content to other users, such as the push of multimedia resources such as dynamic, video and audio released online. At present, in the recommendation system, a key-value pair is usually used, and the candidate identifier of each recommendation candidate (such as the candidate ID number, etc.) is used as the key value, and all the descriptive information of the recommendation candidate (such as the publisher of the recommendation candidate, the release time, the current number of views or views, etc.) is used as the specific value content under the key value, thereby generating a corresponding positive ranking data set. By querying the positive ranking data set later, the recommended content can be accurately recalled, filtered and sorted.

[0003] In existing data query systems, an independent query service is usually separated. The query service has a mechanism for data synchronization and updating. At this time, external devices uniformly access the query service to access each positive data set in the query service through the network to obtain the corresponding positive data for query under the corresponding recommendation conditions.

[0004] However, with the continuous development of the business, the query service's access to positive-ranked data has also continued to increase. At this time, due to the hardware limitations of the network bandwidth, it is impossible to successfully access a large amount of positive-ranked data with low latency. As a result, when the access volume to positive-ranked data is too large, it will cause a corresponding query performance bottleneck. Summary of the invention

[0005] The embodiments of the present invention provide a data query system, a data query method, an apparatus and a server, which avoid query performance bottlenecks caused by network bandwidth limitations and improve the efficiency and accuracy of data query.

[0006] In a first aspect, an embodiment of the present invention provides a data query system, the system comprising: an online query library, a data update queue and an online query service, the online query library stores an online data set to be queried and is deployed in the local memory of the online query service, the data update queue stores an update log of the online data set; wherein,

[0007] The online query service updates the online data set in the online query library according to the update log in the data update queue, and queries the online data set in the online query library through the local memory when receiving a data query instruction.

[0008] In a second aspect, an embodiment of the present invention provides a data query method, which is applied to the data query system described in the first aspect, and the method includes:

[0009] According to the update log stored in the data update queue for the online data set to be queried, the online data set in the online query library is updated accordingly, and the online query library is deployed in the local memory of the online query service;

[0010] If a data query instruction is received through the online query service, the online data set in the online query library is queried through the local memory.

[0011] In a third aspect, an embodiment of the present invention provides a data query device, which is configured in the data query system described in the first aspect, and the device includes:

[0012] A data update module, used to update the online data set in the online query library according to the update log stored in the data update queue for the online data set to be queried, and the online query library is deployed in the local memory of the online query service;

[0013] The data query module is configured to query the online data set in the online query library through the local memory if a data query instruction is received through the online query service.

[0014] In a fourth aspect, an embodiment of the present invention provides a server, the server comprising:

[0015] one or more processors;

[0016] A storage device for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the data query method described in any embodiment of the present invention.

[0018] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data query method described in any embodiment of the present invention.

[0019] The embodiment of the present invention provides a data query system, a data query method, an apparatus and a server, which deploy an online query library on the local memory of an online query service, and are used to store an online data set to be queried, and store an update log of the online data set through a data update queue, so that the online query service can update the online data set in the online query library accordingly by reading the update log added in the data update queue in real time, thereby ensuring the accuracy of the query data in the online query library. At the same time, when the online query service receives a data query instruction, it can directly query the online data set in the online query library through the local memory, avoiding the query performance bottleneck problem caused by network bandwidth limitation when accessing the data to be queried through the network, converting the network access query into the local memory access query, greatly reducing the access delay of the data query, and thus improving the efficiency of the data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0021] Figure 1 A schematic diagram of a data query system provided in the first embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the structure of a data query system provided in Embodiment 2 of the present invention;

[0023] Figure 3 A flowchart of a data query method provided in Embodiment 3 of the present invention;

[0024] Figure 4 A flowchart of a data query method provided in Embodiment 4 of the present invention;

[0025] Figure 5 A schematic diagram of the structure of a data query device provided in Embodiment 5 of the present invention;

[0026] Figure 6 A schematic diagram of the structure of a server provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0028] Embodiment 1

[0029] Figure 1 This is a schematic diagram of the principle architecture of a data query system provided in the first embodiment of the present invention. This embodiment can be applied to the situation of querying the published data in any business scenario, such as the need to query various multimedia resources that have been published by each user in the resource recommendation service. Figure 1 The data query system may include: an online query library 110 , a data update queue 120 and an online query service 130 .

[0030] Among them, the online query library 110 stores the online data set to be queried and is deployed in the local memory of the online query service 130, and the data update queue 120 stores the update log of the online data set; specifically, the online query service 130 updates the online data set in the online query library 110 according to the update log in the data update queue 120, and when receiving the data query instruction, queries the online data set in the online query library 110 through the local memory.

[0031] Optionally, as various multimedia resources are widely disseminated on the Internet, there may be a need to query corresponding multimedia resources in different business scenarios. For example, in a video recommendation scenario, it is necessary to perform corresponding recommendation queries on multimedia resources that have been published on the network. In this case, this embodiment will specifically separate an independent online query service 130, and subsequently perform corresponding data query operations through the online query service 130.

[0032] At this time, in order to ensure the efficiency and accuracy of data query, the data query system in this embodiment needs to meet the following basic conditions: 1) Since the current large-scale data query services are distributed, each query request may be processed by different machines in a machine cluster, so the data queried by different machines through the data query system must be consistent; 2) Since the specific content of resource data will change with the various resource operations performed by the user, the online data set to be queried is required to support dynamic updates after storage, and the real-time nature of the dynamic updates must be guaranteed; 3) It is required to be able to meet the needs of successfully accessing a large amount of online data to be queried with low latency.

[0033] However, existing data query methods usually store the data to be queried in a redis cluster. When the online query service 130 performs a data query operation, it is necessary to access the redis cluster through the network to obtain the corresponding data to be queried for querying under the target query conditions. At this time, due to the hardware limitation of the network bandwidth, it is impossible to successfully access a large amount of data to be queried under low latency. Therefore, this embodiment needs to solve the problem of query performance bottleneck caused by network bandwidth limitation.

[0034] Specifically, in order to avoid query performance bottlenecks caused by network bandwidth limitations, this embodiment deploys a corresponding online query library 110 in the local memory of the machine where the separated online query service 130 is located, and uses the online query library 110 to store various types of resource data as the online data set to be queried in this embodiment. At this time, when the corresponding data query operation is performed through the online query service 130, a large number of online data sets to be queried can be directly accessed from the local memory, thereby converting access through the network into access through the local memory, avoiding access to the online data set to be queried through the network, and at the same time utilizing the low latency performance of the local memory to ensure the high efficiency of data query.

[0035] It should be noted that, in order to ensure the efficiency of data query, the online query library 110 of this embodiment can use a key-value pair method to store each online data to be queried, and use the data identifier of each online data to be queried as the key value of the online data, and use all descriptive information of the online data (such as the publisher of the online data, the publishing time, the current number of views or views, and the number of likes and reposts, etc.) as the specific value content under the key value, so that the online data sets in the online query library 110 can be queried in a forward-ordered manner, thereby ensuring the comprehensiveness of the data query.

[0036] In addition, since the specific value content of each online data to be queried, such as the current number of views or likes and forwardings, will continue to change with the corresponding resource operations performed by the user, the online data set in the online query library 110 is dynamically stored. In order to ensure the accuracy of the dynamic storage of the online data set, this embodiment will also additionally set a corresponding data update queue 120. When the user performs a corresponding resource update operation on a certain online data, the update log generated for the online data is stored through the data update queue 120. The update log records the identifier of the currently updated online data and the specific operation information of the current update, so that the online data set in the online query library 110 can be accurately updated later.

[0037] It should be noted that the data update queue in this embodiment can be set up using Kafka message middleware to ensure orderly writing and efficient reading of update logs.

[0038] In this embodiment, according to the data query requirements in different business scenarios, a large amount of resource data that needs to be queried in the business scenario is first found, and a corresponding positive-ranked data set is generated in a key-value pair manner as the online data set to be queried in this embodiment, and stored in the online query library 110 deployed in the local memory of the online query service 130. At the same time, according to the various resource operations performed by the user on each online data, such as likes and forwarding, it is determined whether the online data set in the online query library 110 needs to be changed. When a resource operation performed by the user causes a certain online data to be changed, an update log of the online data is generated and written into the data update queue 120 of this embodiment accordingly. Therefore, in order to ensure the timely update of the online data to be queried, this embodiment reads the corresponding update log from the data update queue 120 in real time through the online query service 130, and updates the online data pointed to in the online database 110 deployed in the local memory according to the update log, thereby ensuring the accuracy of the online data set in the online database 110. At the same time, since the external devices will uniformly access the online query service 130 to perform the corresponding data query operation, when the corresponding data query instruction is received through the online query service 130, the present embodiment can directly read the corresponding online data set from the online query library 110 deployed in the local memory, and perform query analysis on each online data according to the query conditions in the data query instruction, thereby obtaining the corresponding query result. At this time, in the data query process, the access through the network is converted to the access through the local memory, avoiding the query performance bottleneck problem caused by the network bandwidth limitation when accessing the data to be queried through the network, and greatly reducing the access delay of the data query.

[0039] At the same time, since various query conditions are usually set according to the business scenario of the current query when querying data to filter the online data set in the online query library 110, the increasing query conditions will make the readability of the execution code of the query process worse, and since the execution code is a black box at the architectural level, it is difficult to support the maintenance, reuse and code optimization of the execution code. Therefore, this embodiment will pre-split various query conditions to obtain various query sub-conditions under the lowest unit, and set a corresponding query operator for each query sub-condition. The query operator only supports data query that meets the corresponding query sub-condition, and the query operators under various query sub-conditions are respectively registered in the online query service 130 of this embodiment, so that the query operators under various query sub-conditions are pre-registered in the online query service 130. When a query operation needs to be performed later, the corresponding query total condition can be generated through different combination logics between the query operators, and then the corresponding query of the online data set is executed.

[0040] Specifically, the present embodiment may develop a corresponding operator configuration interface 131 on the online query service 130. Each time the querying party executes a query operation, the querying party may first set the target query operator to be used in the current query and the combination logic between the target query operators through the operator configuration interface 131, thereby generating operator configuration information adapted to the current query. At this time, the online query service 130 may receive the operator configuration information adapted to the current query through the operator configuration interface 131 developed thereon, and determine the target query operator and the combination logic between the target query operators used in the current query by parsing the operator configuration information, so as to subsequently utilize the target query operator and the combination logic between the target query operators to perform accurate query of the data in the online query library 110. At this time, the present embodiment may freely combine the query logic to be executed by dynamically configuring the query operators, thereby improving the reusability and maintainability of the data query code.

[0041] Exemplarily, when the corresponding data query operation is executed through the online query service 130, a FilterPipeline object is first created, and then the target query operator used by the current query and the combination logic between the target query operators are determined by parsing the operator configuration information adapted to the current query. At this time, select() is first used to read the data identifier of each online data to be queried in the online query library 110, and then a series of target query operators are added through filter(). Each target query operator will query the specific value content of the online data based on the data identifier of each online data read as the key value, and then use the combination logic between the target query operators to determine whether the online data needs to be filtered. Finally, export() is used to export all the positive row information corresponding to the online data that passes the query conditions to obtain the corresponding query results.

[0042] It should be noted that the online query service 130 in this embodiment can support operator registration under any query sub-condition to ensure the richness of the query operators. At the same time, the design and semantics of the query operators can refer to some commonly used operator types in the big data processing pipeline, and there is no limitation on this.

[0043] The technical solution provided in this embodiment deploys an online query library on the local memory of the online query service to store the online data set to be queried, and stores the update log of the online data set through the data update queue, so that the online query service can update the online data set in the online query library accordingly by reading the update log added in the data update queue in real time, thereby ensuring the accuracy of the query data in the online query library. At the same time, when the online query service receives the data query instruction, it can directly query the online data set in the online query library through the local memory, avoiding the query performance bottleneck problem caused by network bandwidth limitation when accessing the data to be queried through the network, converting the network access query into the local memory access query, greatly reducing the access delay of the data query, thereby improving the efficiency of the data query.

[0044] Embodiment 2

[0045] Figure 2 This is a schematic diagram of the structure of a data query system provided by the second embodiment of the present invention. This embodiment is optimized based on the technical solution provided by the above embodiment. Figure 2 The data query system may include an online query library 210 , a data update queue 220 and an online query service 230 .

[0046] Among them, the online query library 210 stores the online data set to be queried and is deployed in the local memory of the online query service 230, and the data update queue 220 stores the update log of the online data set; specifically, the online query service 230 updates the online data set in the online query library 210 according to the update log in the data update queue 220, and when receiving the data query instruction, queries the online data set in the online query library 210 through the local memory.

[0047] In addition, query operators under various query sub-conditions are pre-registered in the online query service 230 to implement the query of the online data set through different combination logics of the query operators. An operator configuration interface 231 is developed on the online query service 230 to receive operator configuration information adapted to the current query, and the operator configuration information records the target query operator used by the current query and the combination logic between the target query operators.

[0048] Meanwhile, the data query system in this embodiment may include an offline data source end 240 and an offline construction process 250 in addition to the online query library 210 , the data update queue 220 and the online query service 230 .

[0049] Among them, the offline data source end 240 is used to store the offline data set to be queried; specifically, when the offline data set is updated, the offline data source end 240 generates a corresponding update log and writes it into the data update queue 220; the offline construction process 250 regularly reads the corresponding offline data set from the offline data source end to construct the online data set, and distributes the constructed online data set to the online query service 230 for storage in the online query library 210.

[0050] Optionally, in order to ensure the storage performance of various resource data published on the Internet, this embodiment will specially set up an offline data source terminal 240 to store the various resource data that have been published, that is, the offline data set in this embodiment. At this time, the offline data set can use a key-value pair method to generate corresponding positive row data for storage.

[0051] Specifically, a wide table will be maintained offline on the offline data source end 240, and each row in the wide table stores the positive row information of an offline data (such as the publisher, publishing time, current viewing or browsing number, and number of likes and forwarding, etc. of the offline data), and serves as the data source of the online data set stored in the online query library 210 in this embodiment. This embodiment only requires the offline data source end 240 to have high storage capacity, and does not need to have the high concurrent reading performance required by the machine where the online query service 230 is located. Exemplarily, the offline data source end 240 in this embodiment can use the tidb database for offline data set storage.

[0052] In addition, when the user performs corresponding resource operations on any offline data in the offline data set, it may cause the positive row information of the offline data in the offline data source end 240 to change. Therefore, when it is detected that a certain offline data in the offline data set stored in the offline data source end 240 is updated, an update log of the offline data will be generated. For example, the tidb database will generate a corresponding binlog log for the updated offline data, and write the update log to the data update queue 220 of this embodiment, so as to perform real-time update of the online data set in the online query library 210 later.

[0053] At the same time, in order to ensure the dynamic update of the online data set stored in the online query library 210, this embodiment will periodically trigger the execution of a batch reading task through the offline construction process 250 to periodically read the corresponding offline data set from the offline data set stored in the offline data source end 240, and convert the read offline data set into a format that meets the query high performance, so as to construct a corresponding online data set, and distribute the constructed online data set to the online query service 230, and the online query service 230 will store the constructed online data set in the online query library 210 deployed in the local memory.

[0054] Specifically, the online data set constructed by the offline construction process 250 can be stored using a hash table data structure, and a data offset is used in the storage format to replace a pointer to a memory address, so that the entire online data set can be stored as a file. At this time, the construction timestamp of the online data set can also be marked, so that the update log written after the construction timestamp in the data update queue 220 can be used to timely update the online data set in the online query library 210, thereby ensuring the update efficiency of the online data set.

[0055] Furthermore, in order to ensure the update accuracy of the online data set in the online query library 210, this embodiment will also configure the corresponding online loading process 232 and online update process 233 on the online query service 230; specifically, the online loading process 232 loads the online data set distributed by the offline construction process 250 and stores it in the online query library 210; the online update process 233 reads the update log of the online data set from the data update queue in real time according to the construction timestamp of the online data set, and updates the online data set accordingly in the online query library 210.

[0056] Exemplarily, the online loading process 232 continuously obtains the offline constructed data set file from the offline construction process 250, that is, the online data set distributed by the offline construction process 250 in this embodiment, and adopts a memory mapping file method (such as mmap method) to load and store the online data set into the online query library 210 in the local memory; at the same time, because the online data set in the online query library 210 will be dynamically updated, it is necessary to read the update log of each online data written after the construction timestamp from the data update queue 220 in real time according to the construction timestamp marked when the online data set is constructed by the offline construction process 250 through the online update process 233, that is, continuously read the latest update log, and update the positive row content of the online data set pointed to in the online query library 210 according to the read update log during the reading process, so as to adopt a combination of the full online data set constructed offline and the incremental update performed online through the data update queue 220 to ensure the persistence and real-time update of the online data set in the online query library 210.

[0057] The technical solution provided in this embodiment adopts a combination of offline construction of the full online data set and online incremental update through a data update queue to timely update the online data set in the online query library, thereby ensuring the comprehensiveness and accuracy of the online data set in the local memory. At the same time, the online query service uses local memory for independent storage and maintains a complete online data set, replacing network access with local memory access, which greatly reduces the access delay of data query, thereby improving the efficiency of data query.

[0058] Embodiment 3

[0059] Figure 3 This is a flow chart of a data query method provided in the third embodiment of the present invention. This embodiment can be applied to the situation of querying published data in any business scenario, such as the need to query various types of multimedia resources that have been published by each user in the resource recommendation service, and can be applied to the data query system provided by any embodiment of the present invention. A data query method provided in this embodiment can be executed by a data query device provided in an embodiment of the present invention, which can be implemented by software and / or hardware and integrated in a server that executes this method.

[0060] Specifically, refer to Figure 3 , the method may include the following steps performed non-sequentially:

[0061] S310: According to the update log stored in the data update queue for the online data set to be queried, the online data set in the online query library is updated accordingly.

[0062] Optionally, in order to avoid query performance bottlenecks caused by network bandwidth limitations, this embodiment deploys a corresponding online query library in the local memory of the machine where the separated online query service is located, and uses the online query library to store various types of resource data as the online data set to be queried in this embodiment. At this time, when the corresponding data query operation is performed through the online query service, a large number of online data sets to be queried can be directly accessed from the local memory, thereby converting network access to local memory access, avoiding access to the online data set to be queried through the network, and at the same time utilizing the low latency performance of the local memory to ensure the high efficiency of data query.

[0063] At this time, since the specific content of each online data to be queried, such as the current number of views or likes and forwardings, will continue to change with the corresponding resource operations performed by the user, the online data set in the online query library is dynamically stored. In order to ensure the accuracy of the dynamic storage of the online data set, this embodiment will also additionally set a corresponding data update queue. When the user performs a corresponding resource update operation on a certain online data, the update log generated for the online data is stored through the data update queue. The update log records the identifier of the currently updated online data and the specific operation information of the current update, so as to accurately update the online data set in the online query library later.

[0064] Specifically, this embodiment determines whether the online data set in the online query library needs to be changed based on various resource operations performed by the user on various online data, such as likes and forwarding. When a resource operation performed by the user causes a certain online data to change, an update log of the online data will be generated and written into the data update queue of this embodiment accordingly. Therefore, in order to ensure the timely update of the online data to be queried, it is necessary to read the corresponding update log from the data update queue in real time through the online query service, and then find the online data pointed to by the update log from the online data set stored in the online database deployed in the local memory according to the update log, and perform corresponding updates, thereby ensuring the accuracy of the online data set in the online database.

[0065] S320: If a data query instruction is received through the online query service, query the online data set in the online query library through the local memory.

[0066] Optionally, since the external devices will uniformly access the online query service to perform the corresponding data query operation, when the corresponding data query instruction is received through the online query service, this embodiment can directly read the corresponding online data set from the online query library deployed in the local memory, and perform query analysis on each online data according to the query conditions in the data query instruction, thereby obtaining the corresponding query result. At this time, in the data query process, the access through the network is converted to the access through the local memory, avoiding the query performance bottleneck problem caused by the network bandwidth limitation when accessing the data to be queried through the network, and greatly reducing the access delay of the data query.

[0067] For example, when querying data, various query conditions are usually set according to the business scenario of the current query to filter the online data set in the online query library. At this time, the increasing number of query conditions will make the readability of the execution code of the query process worse. Moreover, since the execution code is a black box at the architectural level, it is difficult to support the maintenance, reuse and code optimization of the execution code. Therefore, this embodiment will pre-split various query conditions to obtain various query sub-conditions under the lowest unit, and set a corresponding query operator for each query sub-condition. The query operator only supports data queries that meet the corresponding query sub-condition, and the query operators under various query sub-conditions are registered in the online query service of this embodiment, so that the query operators under various query sub-conditions are pre-registered in the online query service. When the query operation needs to be executed later, the corresponding query total condition can be generated through different combination logics between the query operators, and then the corresponding query of the online data set is executed.

[0068] Therefore, in this embodiment, if a data query instruction is received through the online query service, the online data set in the online query library is queried through the local memory, which may specifically include: if a data query instruction is received through the online query service, the operator configuration information adapted for the current query is parsed from the data query instruction; according to the operator configuration information, the target query operator used by the current query is searched from the query operators registered in the online query service, and the combination logic between the target query operators is determined; using the target query operator and the combination logic, each online data in the online data set is queried through the local memory to obtain the corresponding query result.

[0069] Specifically, the present embodiment can develop a corresponding operator configuration interface on the online query service. Each time the querying party executes a query operation, it will first set the target query operator to be used in the current query and the combination logic between the target query operators through the operator configuration interface, thereby generating operator configuration information adapted to the current query. At this time, if a data query instruction is received through the online query service, the online query service can use the operator configuration interface developed thereon to receive the operator configuration information adapted to the current query, and by parsing the operator configuration information, determine the target query operator used in the current query and the combination logic between the target query operators, and read the operator configuration information. The data identifier of each online data to be queried in the online query library is obtained, and then each target query operator is used to query the specific value content of the online data according to the data identifier of each online data read as the key value, and the combination logic between the target query operators is used to determine whether the online data needs to be filtered, and finally all the positive row information corresponding to the online data that passes the query condition is exported to obtain the corresponding query result, thereby realizing accurate query of the data in the online query library; at this time, in this embodiment, by dynamically configuring the query operator, the query logic to be executed can be freely combined, thereby improving the reusability and maintainability of the data query code.

[0070] The technical solution provided in this embodiment deploys an online query library on the local memory of the online query service to store the online data set to be queried, and stores the update log of the online data set through the data update queue, so that the online query service can update the online data set in the online query library accordingly by reading the update log added in the data update queue in real time, thereby ensuring the accuracy of the query data in the online query library. At the same time, when the online query service receives the data query instruction, it can directly query the online data set in the online query library through the local memory, avoiding the query performance bottleneck problem caused by network bandwidth limitation when accessing the data to be queried through the network, converting the network access query into the local memory access query, greatly reducing the access delay of the data query, thereby improving the efficiency of the data query.

[0071] Embodiment 4

[0072] Figure 4 This is a flow chart of a data query method provided by Embodiment 4 of the present invention. This embodiment is optimized based on the above embodiment. Specifically, Figure 4 As shown, this embodiment mainly explains in detail the specific updating process of the online data set in the online query library.

[0073] Optional, such as Figure 4 As shown, this embodiment may include the following steps:

[0074] S410: Load and store the online data set constructed from the offline data set at the offline data source end into the online query library deployed on the local memory.

[0075] Optionally, in order to ensure the storage performance of various resource data published on the Internet, this embodiment will specially set up an offline data source to store the various resource data that have been published, that is, the offline data set in this embodiment. At this time, the offline data set can use a key-value pair method to generate corresponding positive row data for storage.

[0076] At the same time, in order to ensure the dynamic update of the online data set stored in the online query library, this embodiment will periodically trigger the execution of a batch reading task through the offline construction process to regularly read the corresponding offline data set from the offline data set stored in the offline data source end, and convert the read offline data set into a format that meets the high query performance, so as to construct the corresponding online data set, and distribute the constructed online data set to the online query service, which will store the constructed online data set in the online query library deployed in the local memory.

[0077] Specifically, this embodiment continuously obtains the offline constructed data set file from the offline construction process through the online loading process configured on the online query service, that is, the online data set distributed by the offline construction process in this embodiment, and adopts a memory mapping file method (such as mmap method) to load and store the online data set into the online query library in the local memory.

[0078] S420, reading the corresponding update log from the data update queue in real time according to the construction timestamp of the online data set, and updating the online data pointed to by the update log in the online query library accordingly.

[0079] Specifically, since the online data sets in the online query library will be dynamically updated, it is necessary to read the update logs of each online data written after the construction timestamp from the data update queue in real time according to the construction timestamp marked when the online data set is constructed by the offline construction process through the online update process configured on the online query service. That is, continuously read the latest update logs, and update the positive row content of the online data set pointed to in the online query library according to the read update logs during the reading process. In this way, the full online data set constructed offline is combined with the incremental update performed online through the data update queue to ensure the persistence and real-time update of the online data set in the online query library.

[0080] S430: If a data query instruction is received through the online query service, query the online data set in the online query library through the local memory.

[0081] S440: Submit the query result of the online data set to the corresponding recommendation service.

[0082] Optionally, after querying the online data set in the online query library through the local memory, the query results that meet the query conditions will be obtained. At this time, if in the resource recommendation scenario, the query results can be submitted to the corresponding recommendation service, and the recommendation service will accurately push them to ensure the efficiency of data recommendation.

[0083] The technical solution provided in this embodiment adopts a combination of offline construction of the full online data set and online incremental update through a data update queue to timely update the online data set in the online query library, thereby ensuring the comprehensiveness and accuracy of the online data set in the local memory. At the same time, the online query service uses local memory for independent storage and maintains a complete online data set, replacing network access with local memory access, which greatly reduces the access delay of data query, thereby improving the efficiency of data query.

[0084] Embodiment 5

[0085] Figure 5 This is a schematic diagram of the structure of a data query device provided in Embodiment 5 of the present invention, which can be configured in a data query system provided in any embodiment of the present invention. Figure 5 As shown, the device may include:

[0086] A data update module 510, configured to update the online data set in the online query library according to the update log stored in the data update queue for the online data set to be queried, wherein the online query library is deployed in the local memory of the online query service;

[0087] The data query module 520 is configured to query the online data set in the online query library through the local memory if a data query instruction is received through the online query service.

[0088] The technical solution provided in this embodiment deploys an online query library on the local memory of the online query service to store the online data set to be queried, and stores the update log of the online data set through the data update queue, so that the online query service can update the online data set in the online query library accordingly by reading the update log added in the data update queue in real time, thereby ensuring the accuracy of the query data in the online query library. At the same time, when the online query service receives the data query instruction, it can directly query the online data set in the online query library through the local memory, avoiding the query performance bottleneck problem caused by network bandwidth limitation when accessing the data to be queried through the network, converting the network access query into the local memory access query, greatly reducing the access delay of the data query, thereby improving the efficiency of the data query.

[0089] The data query device provided in this embodiment can be applied to the data query method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0090] Embodiment 6

[0091] Figure 6 A schematic diagram of the structure of a server provided in Embodiment 6 of the present invention is shown in FIG. Figure 6 As shown, the server includes a processor 60, a storage device 61 and a communication device 62; the number of processors 60 in the server can be one or more. Figure 6 A processor 60 is taken as an example; the processor 60, the storage device 61 and the communication device 62 in the server can be connected by a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0092] A server provided in this embodiment can be used to execute the data query method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0093] Embodiment 7

[0094] Embodiment 7 of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data query method in any of the above embodiments can be implemented. The method can be applied to the data query system provided by any embodiment of the present invention, and can specifically include the following non-sequentially executed steps:

[0095] According to the update log stored in the data update queue for the online data set to be queried, the online data set in the online query library is updated accordingly, and the online query library is deployed in the local memory of the online query service;

[0096] If a data query instruction is received through the online query service, the online data set in the online query library is queried through the local memory.

[0097] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided by an embodiment of the present invention are not limited to the method operations described above, and can also execute related operations in the data query method provided by any embodiment of the present invention.

[0098] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0099] It is worth noting that in the embodiment of the above-mentioned data query device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data query system, characterized in that: include: An online query library, a data update queue and an online query service, wherein the online query library stores the online data set to be queried and is deployed in the local memory of the online query service, and the data update queue stores the update log of the online data set; wherein, The online query service updates the online data set in the online query library according to the update log in the data update queue, and queries the online data set in the online query library through the local memory when receiving the data query instruction; The online query service pre-registers query operators under various query sub-conditions, so as to implement the query of the online data set through different combination logics of the query operators; wherein the query operator only supports data query that meets a corresponding query sub-condition; The data query system further includes: an offline data source end and an offline construction process, wherein the offline data source end stores an offline data set to be queried; When the offline data set is updated, the offline data source generates a corresponding update log and writes it into the data update queue; The offline construction process periodically reads the corresponding offline data set from the offline data source to construct the online data set, and distributes the constructed online data set to the online query service to be stored in the online query library.

2. The data query system according to claim 1, characterized in that: The online query service is configured with an online loading process and an online updating process; The online loading process loads the online data set distributed by the offline building process and stores it in the online query library; The online update process reads the update log of the online dataset from the data update queue in real time according to the construction timestamp of the online dataset, and updates the online dataset in the online query library accordingly.

3. The data query system according to claim 1, characterized in that: An operator configuration interface is developed on the online query service to receive operator configuration information adapted to the current query. The operator configuration information records the target query operator used by the current query and the combination logic between the target query operators.

4. The data query system according to any one of claims 1 to 3, characterized in that: The data update queue is set using Kafka message middleware.

5. A data query method, characterized in that: The data query system applied to any one of claims 1 to 4 comprises: According to the update log stored in the data update queue for the online data set to be queried, the online data set in the online query library is updated accordingly, and the online query library is deployed in the local memory of the online query service; If a data query instruction is received through the online query service, the online data set in the online query library is queried through the local memory.

6. The method according to claim 5, characterized in that If a data query instruction is received through the online query service, querying the online data set in the online query library through the local memory includes: If a data query instruction is received through the online query service, the operator configuration information adapted for the current query is parsed from the data query instruction; According to the operator configuration information, searching for a target query operator used in the current query from the query operators registered in the online query service, and determining the combination logic between the target query operators; By using the target query operator and the combination logic, each online data in the online data set is queried through the local memory to obtain a corresponding query result.

7. The method according to claim 5, characterized in that Before updating the online dataset in the online query library according to the update log stored in the data update queue for the online dataset to be queried, the following is also included: The online dataset constructed from the offline dataset at the offline data source is loaded and stored in the online query library deployed on the local memory.

8. The method according to claim 7, characterized in that The updating of the online data set in the online query library correspondingly to the update log stored in the data update queue for the online data set to be queried includes: According to the construction timestamp of the online data set, the corresponding update log is read from the data update queue in real time, and the online data pointed to by the update log is updated in the online query library accordingly.

9. The method according to any one of claims 5 to 8, characterized in that: After querying the online data set in the online query library through the local memory, the method further includes: Submit the query result of the online data set to the corresponding recommendation service.

10. A data query device, characterized in that: The data query system configured in any one of claims 1 to 4 comprises: A data update module, used to update the online data set in the online query library according to the update log stored in the data update queue for the online data set to be queried, and the online query library is deployed in the local memory of the online query service; The data query module is configured to query the online data set in the online query library through the local memory if a data query instruction is received through the online query service.

11. A server, characterized in that: The server comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data query method as described in any one of claims 5-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data query method described in any one of claims 5 to 9 is implemented.

Images