Data push method, device, server and storage medium

Abstract

The present application discloses a data push method, device, server and storage medium, the data push method comprising: obtaining a data query request sent by a business end; obtaining multiple query handles corresponding to the data query result according to the data query request; assigning each query handle in the multiple query handles to a thread, each query handle corresponding to a different thread; each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end. The method can realize the rapid push of query results to the business end.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a data push method, device, server and storage medium. Background Art

[0002] With the rapid development of network information technology, information recommendation technology based on big data technology has also emerged. Information recommendation technology mainly recommends relevant information based on the user's tag data. In information recommendation, the business party usually queries the tag data, and it takes a long time to pull and send data from the database. Summary of the invention

[0003] In view of the above problems, the present application proposes a data push method, device, server and storage medium.

[0004] In a first aspect, an embodiment of the present application provides a data push method, the method comprising: obtaining a data query request sent by a business end; obtaining multiple query handles corresponding to the data query results based on the data query request; assigning each of the multiple query handles to a thread, each query handle corresponding to a different thread; each thread pulls the data query results according to the corresponding query handle and sends the data query results to the business end.

[0005] In the second aspect, an embodiment of the present application provides a data push device, which includes: a request acquisition module, a handle acquisition module, a handle allocation module and a data sending module, wherein the request acquisition module is used to obtain a data query request sent by a business end; the handle acquisition module is used to obtain multiple query handles corresponding to the data query results according to the data query request; the handle allocation module is used to assign each query handle in the multiple query handles to a thread, and each query handle corresponds to a different thread; the data sending module is used for each thread to pull the data query results according to the corresponding query handle and send the data query results to the business end.

[0006] In a third aspect, an embodiment of the present application provides a server, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the data push method provided in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the data push method provided in the first aspect above.

[0008] The solution provided by the present application obtains a data query request sent by a business end, obtains multiple query handles corresponding to the data query results according to the query request, and assigns each of the multiple query handles to a thread, wherein each query handle corresponds to a different thread, and then each thread pulls the data query results according to the corresponding query handle and sends the data query results to the business end at the same time, thereby realizing parallel pulling and sending of data and improving the speed of data push. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flow chart of a data push method according to an embodiment of the present application is shown.

[0011] Figure 2 A flow chart of a data push method according to another embodiment of the present application is shown.

[0012] Figure 3 A flow chart of a data push method according to yet another embodiment of the present application is shown.

[0013] Figure 4 A flow chart of a data push method according to another embodiment of the present application is shown.

[0014] Figure 5 A block diagram of a data push device according to an embodiment of the present application is shown.

[0015] Figure 6 It is a block diagram of a server for executing a data push method according to an embodiment of the present application.

[0016] Figure 7 It is a storage unit of an embodiment of the present application for storing or carrying program codes for implementing the data push method according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0018] In information push, there are usually cases where the business party queries the tag data. The data end pulls the data from the database and sends it to the business party according to the business party's data requirements. The business party can then determine the push content and push object based on the obtained data.

[0019] In the traditional data extraction and transmission scheme, all data is usually read from the database to the memory, and then sent from the memory to the business side. This scheme is simple and clear, and realizes the export of data.

[0020] After long-term research, the inventor found that in the traditional data export solution, as the number and demand of business parties continue to grow, the data required by business parties is increasing, and the data end reads a large amount of query results into memory, which is not only very time-consuming, but also easy to trigger memory overflow. For example: a business party needs to export users who have installed both application A and application B. Under such conditions, it is very easy to capture more than 200 million users. For such an export of user information of the order of 100 million, the current technical solution has a very slow sending speed and has the risk of downtime.

[0021] In response to the above problems, the inventors proposed the data push method, device, server and storage medium provided in the embodiments of the present application, which divides the multiple query handles corresponding to the data query results into different threads, and each thread pulls data in parallel according to the query handles of the shards and sends the data to the business end. The specific data push method is described in detail in the subsequent embodiments.

[0022] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a data push method provided by an embodiment of the present application. In a specific embodiment, the data push method is applied to Figure 5 The data push device 400 and the server 100 ( Figure 6 ). The following will take the server as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the server used in this embodiment can be a traditional server, a cloud server, etc., which is not limited here. Figure 1 The process shown is described in detail, and the data push method may specifically include the following steps:

[0023] Step S110: Obtain the data query request sent by the service end.

[0024] In the embodiment of the present application, when the service end needs to query data, it can send a data query request to the server. Correspondingly, the server can receive the data query request sent by the service end, respond to the data query request, and return the corresponding data to the service end.

[0025] In some embodiments, the data query request may carry the data requirements of the business party corresponding to the business end, for example, it may carry query conditions. The query conditions may include one or more conditions. For example, the business party wants to deliver advertisements to users who have installed both application 1 and application 2, but not application 3, and the query condition is set to users who have installed both application 1 and application 2, but not application 3; for another example, a game company wants to increase the retention rate of recent active game players and wants to deliver game rewards to recent active game players, and the query condition is set to users who have played the game for more than 24 hours or logged in more than 10 times in the past week.

[0026] In some implementations, the data query request sent by the service end may be a request for obtaining tag data, or a request for obtaining other types of data, such as document data, etc. The type of data obtained by the data query request may not be limited.

[0027] Step S120: According to the data query request, a plurality of query handles corresponding to the data query results are obtained.

[0028] In an embodiment of the present application, after obtaining the data query request, the server may determine the data query result according to the data query request. Among them, the data query result may include the data corresponding to the data query request. In other words, the server may determine which data meets the demand information based on the demand information carried in the data query request. For example, the data query request is a request for querying tag data, and the data query request carries users who have installed application A and application B, then the server can determine which users have installed application A and application B. For another example, the data query request is a request for querying document data, and the data query request carries keywords A and B, then the server can determine which documents include keywords A and B.

[0029] In an embodiment of the present application, after determining the data query result according to the data query request, the server can determine the query handle corresponding to the data query result according to the data query result. Among them, the query handle can be information used to read the data query result. The query handle may include a storage location, an identifier of the data, a field, etc. According to the query handle, the corresponding data query result can be read. The server may pre-store query handles corresponding to each data, and the server may determine the query handle corresponding to the data query result according to the data included in the data query result.

[0030] In some implementations, the server may divide the data query result into multiple parts according to the data query result, and determine the query handles corresponding to the data query results of each part, so as to obtain multiple query handles corresponding to the data query result. It can be understood that the multiple query handles obtained can be used to read data respectively, and the data read according to the multiple query handles can constitute a complete data query result.

[0031] As an implementation method, the server may divide the data query result into multiple parts according to the location of data distribution in the data query result, and determine the query handle corresponding to each part. For example, the data corresponding to the data query result is stored in a database cluster, so it is possible to determine multiple databases storing the data query result, and determine the query handle based on the data query results included in the multiple databases. For another example, the data corresponding to the data query result is stored in different shards of the database, so it is possible to determine the multiple shards where the data query result is located, and determine the query handle based on the data query results existing in the multiple shards.

[0032] As another implementation, the server may also divide the data query results according to different types of data in the data query results, and determine the query handles corresponding to the data query results of each part according to the division results.

[0033] Of course, the specific method of obtaining multiple query handles corresponding to the data query result may not be limited.

[0034] Step S130: assigning each query handle in the multiple query handles to a thread, wherein each query handle corresponds to a different thread.

[0035] In an embodiment of the present application, after determining multiple query handles corresponding to the data query results, the server can allocate a different thread for each query handle so as to read data from the database and send it to the service end.

[0036] In some implementations, a thread pool may be created in the server, and the server may allocate a different thread from the thread pool to each query handle, and each thread is used to read data of the data query result corresponding to the allocated query handle from the database separately.

[0037] Step S140: Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end.

[0038] In the embodiment of the present application, after the server assigns a thread to each query handle, each thread can pull the data query results from the database according to the query handle assigned to it, and send the data to the business end while pulling the data, so as to avoid pulling all the data at once and then sending it, which causes excessive memory usage and memory overflow. For example, when the business end is Kafka, each thread sends the data to Kafka after pulling the data.

[0039] The data push method provided by the embodiment of the present application obtains a data query request sent by the business end, obtains multiple query handles corresponding to the data query result according to the query request, and assigns each query handle in the multiple query handles to a thread, wherein each query handle corresponds to a different thread, and then each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end, thereby achieving parallel pulling and sending of data and improving the speed of data push. In addition, when pulling and sending data, each thread adopts a method of pulling and sending at the same time, which can avoid occupying too much memory and ensure the stable operation of the program.

[0040] See also Figure 2 , Figure 2 A schematic diagram of a data push method provided by another embodiment of the present application is shown. The data push method can be applied to the above-mentioned server. Figure 2 The process shown is described in detail, and the data push method may specifically include the following steps:

[0041] Step S210: Obtain the data query request sent by the business end.

[0042] In the embodiment of the present application, step S210 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0043] Step S220: Determine multiple shards corresponding to the data query result according to the data query request.

[0044] In an embodiment of the present application, when storing data in a database, the data can be sharded. Specifically, the sharding function of ElasticSearch (ES) can be used to shard the data. Among them, ElasticSearch (ES) is a distributed search engine, an ES cluster consists of one or more nodes, and an ES instance is a node. An index in ES is a collection of documents, each index has one or more shards, and the indexed data is allocated and stored on each shard. Shards can be understood as the smallest unit block of data, and the data written into the database is allocated and stored on the shards. Specifically, an index can be created in ES, and each index can correspond to multiple shards. Multiple shards may be in the same node or in different nodes. For example, assuming that the index IndexA has 2 shards, 10 data (10 documents) are inserted into IndexA, then these 10 data can be divided into 5 and stored in the first shard, and the remaining 5 will be stored in another shard. The data corresponding to the index can be stored in its corresponding shard. When data needs to be queried, data can be queried from the shard corresponding to the index.

[0045] In an embodiment of the present application, when determining multiple query handles corresponding to a data query result, the server can determine the shards corresponding to the data in the data query result. In some embodiments, the data query request sent by the business end may carry query conditions. The server can extract features in the query conditions based on the query conditions, and determine the data that matches the features as the data query result. After determining the data query result, the server determines the shards where the data in the data query result is located. It can be understood that the server can store information about each data and the shards where each data is located. Based on this, the server can determine the data query results that meet the query conditions based on the query conditions, and determine the shards where the data is located.

[0046] In some implementations, the server may determine the index corresponding to the data query result according to the data query request, and obtain multiple shards corresponding to the index. It is understandable that the server may search for the index corresponding to the data query result according to the determined data query result, and determine the multiple shards corresponding to the index according to the index information. For example, the data query result includes index 1 and index 2, and the IDs of the shards of index 1 and index 2 are determined.

[0047] Step S230: Determine the query handle of the partial data query result corresponding to each shard, and obtain multiple query handles.

[0048] In an embodiment of the present application, after determining the multiple shards corresponding to the data query result, the server can determine the query handle of the partial data query result corresponding to each shard, and obtain multiple query handles. The query handle can be information for reading the data query result. The query handle can include a storage location, a data identifier, a field, etc.

[0049] In some implementations, identification information of a portion of the data query result corresponding to each shard is obtained, and the identification information is used as a query handle to obtain multiple query handles. It can be understood that data can be read from the corresponding shard through the identification message. The data pulled out from the corresponding shard using each query handle can constitute a complete data query result.

[0050] Step S240: assigning each query handle in the multiple query handles to a thread, wherein each query handle corresponds to a different thread.

[0051] In the embodiment of the present application, step S240 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0052] Step S250: each thread pulls a preset amount of data according to the corresponding query handle and sends the pulled data to the service end.

[0053] In an embodiment of the present application, after the server assigns a thread to each query handle, each thread can pull data query results from the database according to its assigned query handle, and send the data to the business end while pulling data. Specifically, each thread can pull data from the database multiple times. Among them, each thread can pull data from the corresponding shard according to the query handle, that is, each query handle corresponds to a partial data query result in a shard, so when each thread pulls data according to the query handle, it pulls data from the allocation corresponding to the query handle. Specifically, the shard to be retrieved can be determined based on the preference parameter. If the parameter starts with _shards, it means that only the specified shard is queried. Therefore, each thread can limit the parameter starting with _shards to pull data from the corresponding allocation.

[0054] In some implementations, each thread can use the scroll cursor function of ElasticSearch to pull data query results and send them to the business end according to the corresponding query handle. It is understandable that if we want to take out a large amount of data from the ES cluster, taking it out at one time will cause excessive IO pressure and performance problems, and paging query is likely to cause deep paging problems. Therefore, ES's scroll query can be used to query in batches until all data is queried. For example, the queried data is a user identity, and each thread uses the query handle assigned to it to query a certain amount of user identity, which is composed of a message sent to Kafka, thereby realizing batch sending within each shard query handle.

[0055] In some implementations, each thread can pull a preset amount of data according to the corresponding query handle and send the pulled data to the service end. That is, after each thread pulls a batch of data, it stores the pulled data into the memory and then sends the pulled data from the memory.

[0056] In this implementation, the server may also determine the amount of data to be pulled each time before pulling data, that is, the above-mentioned preset data amount.

[0057] As an implementation method, the server can obtain the current remaining memory value and determine the preset data volume according to the remaining memory value. It is understandable that the method of pulling and sending data in batches is adopted, mainly considering that the server's memory may be occupied by a large amount of pulled data, resulting in operation crashes. Therefore, the preset data volume pulled each time can be determined according to the current remaining memory value of the server.

[0058] In this embodiment, the server determines the preset data amount according to the current remaining memory value, which may include: judging whether the remaining memory value is greater than the specified memory value; if the remaining memory value is greater than the specified memory value, using the first data amount as the preset data amount; if the remaining memory value is less than the specified memory value, using the second data amount as the preset data amount, and the second data amount is less than the first data amount. Among them, the specified memory value can be used as a standard for judging whether the current memory usage of the server occupies a large amount. If the remaining memory value is greater than the specified memory value, it means that the server currently has a large amount of remaining memory, so the first data amount that is larger than the second data amount can be selected as the preset data amount; if the remaining memory value is less than or equal to the specified memory value, it means that the server currently has less remaining memory and its memory is relatively tight, so the second data amount that is smaller than the first data amount can be used as the preset data amount.

[0059] Furthermore, the above first data amount and second data amount are both less than the specified memory value. When determining the first data amount and the second data amount, the server can determine the memory that can be allocated when the remaining memory value is greater than the specified memory value based on the remaining memory value, and calculate the ratio of the allocatable memory to the number of threads corresponding to multiple query handles, and use the ratio as the first data amount. The server can also determine the memory that can be allocated when the remaining memory value is less than or equal to the specified memory value based on the remaining memory value, and calculate the ratio of the allocatable memory to the number of threads corresponding to multiple query handles, and use the ratio as the first data amount. It can be understood that the memory that can be allocated when the remaining memory value is greater than the specified memory value should be greater than the memory that can be allocated when the remaining memory value is less than or equal to the specified memory value, so the determined first data amount should be larger than the second data amount.

[0060] Of course, the first data volume and the second data volume can also be pre-set in the server. In addition, the first data volume should satisfy: when the threads corresponding to the multiple query handles pull data, the total amount of data stored in the memory is less than the specified memory value; the second data volume should satisfy: when the threads corresponding to the multiple query handles pull data, the total amount of data stored in the memory is less than the remaining memory value.

[0061] In this embodiment, the server determines the preset data volume according to the current remaining memory value, which may include: obtaining the preset data volume corresponding to the remaining memory value according to the correspondence between the preset memory value and the preset data volume. Among them, the cache data volume that can be supported by different memory values ​​can be determined in advance according to experiments to establish a correspondence between the memory value and the preset data volume. It should be noted that the preset data volume determined according to the correspondence can meet the normal operation of the program in the server.

[0062] In this implementation, the preset data volume is determined by the memory value of the server. On the basis of pulling the data volume in batches to avoid the problem of excessive memory usage, it is further ensured that the server memory will not be occupied too much, thereby ensuring the normal operation of the server.

[0063] In some embodiments, the server may also obtain the ratio of the total data volume to the specified number of pulls based on the total data volume of the data query result and the specified number of pulls, and use the ratio as the preset data volume. It is understandable that the server may be provided with a specified number of pulls, which is the total number of times each thread pulls data from the shard. The setting of the specified number of pulls can meet the needs of different scenarios, and can also ensure that the number of data pulls is not too many, resulting in a long overall data export time. The number of pulls can be set by the user, or it can be determined based on the current speed of reading and sending data.

[0064] Step S260: Repeat the process of each thread pulling a preset amount of data according to the corresponding query handle, and then sending the pulled data to the business end until all the data corresponding to the data query result are sent to the business end.

[0065] In an embodiment of the present application, step S250 is repeated until all data query results are read and sent to the business end, thereby completing the pulling and sending of data in batches, achieving the effect of pulling data while sending data, and avoiding the problem of the pulled data occupying more memory.

[0066] The data push method provided in the embodiment of the present application obtains a data query request sent by the business end, determines multiple shards corresponding to the data query result according to the data query request, and then determines the query handle of the partial data query result corresponding to each shard to obtain multiple query handles, and then assigns each query handle in the multiple query handles to a thread, wherein each query handle corresponds to a different thread, and then each thread pulls data in batches according to the corresponding query handle and sends it to the business end until all data in the data query result is sent to the business end, thereby achieving parallel pulling and sending of data and improving the speed of data push. Moreover, when pulling and sending data, each thread adopts a pulling-while-sending method, which does not occupy too much memory, ensures the stable operation of the program, and thus greatly improves the overall data export speed. In actual applications, the export time of billions of data can be shortened from half an hour to 2 minutes, and all data can be sent to the business end.

[0067] See also Figure 3 , Figure 3 A schematic diagram of a data push method provided by another embodiment of the present application is shown. The data push method can be applied to the above-mentioned server. Figure 3 The process shown is described in detail, and the data push method may specifically include the following steps:

[0068] Step S310: Obtain the data query request sent by the business end.

[0069] Step S320: Determine whether the total data volume of the data query result is greater than the specified data volume.

[0070] In an embodiment of the present application, after obtaining a data query request, the server can determine the data query result according to the data query request, and then determine the total data volume of the data query result, that is, the data size of all data in the data query result. The server then compares the total data volume of the data query result with the specified data volume, and determines whether the total data volume is greater than the specified data volume based on the comparison result. Among them, the specified data volume is used as a standard for determining whether the data volume in the data query result is large. If the total data volume is greater than the specified data volume, it means that the amount of data that needs to be pushed this time is large, so steps S340 to S360 can be executed, that is, after determining multiple query handles, multiple threads are used to read data from the partition in batches according to the assigned query handles, and then send it to the business end, so as to achieve parallel data pulling, improve the data export speed, and send data while sending data, so as to avoid the problem of excessive memory usage. If the total data volume is less than or equal to the specified data volume, it means that the amount of data that needs to be pushed this time is not much, so step S330 can be executed, that is, after reading all the data query results from the database, the data is stored in the memory, and then the data is sent to the business end from the memory. The specific value of the specified data volume is not a limitation. It should be understood that the total data volume is less than or equal to the specified data volume, and all data of the data query results are stored in the memory, and there will be no excessive memory usage, which will lead to operation crashes.

[0071] Step S330: After reading all the data query results from the database, the data query results are sent to the service end.

[0072] If the total data volume is less than or equal to the specified data volume, step S330 is executed.

[0073] Step S340: According to the data query request, multiple query handles corresponding to the data query results are obtained.

[0074] If the total data volume is less than or equal to the specified data volume, step S340 to step S360 are executed.

[0075] Step S350: assigning each query handle in the multiple query handles to a thread, wherein each query handle corresponds to a different thread.

[0076] Step S360: Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end.

[0077] In the embodiment of the present application, steps S330 to S360 can refer to the contents of the aforementioned embodiment and will not be repeated here.

[0078] The data push method provided in the embodiment of the present application obtains a data query request sent by a business end, and determines whether the total data volume of the data query result is greater than the specified data volume according to the query request. If it is greater than the specified data volume, multiple query handles corresponding to the data query result are obtained, and each query handle in the multiple query handles is assigned to a thread, wherein each query handle corresponds to a different thread, and then each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end at the same time. Therefore, when the total data volume of the data query result is greater than the specified data volume, parallel pulling and sending of data can be achieved, thereby improving the speed of data push. Moreover, when pulling and sending data, each thread adopts a pulling-and-sending method, which does not occupy too much memory and ensures the stable operation of the program.

[0079] See also Figure 4 , Figure 4 A flow chart of a data push method provided by another embodiment of the present application is shown. The data push method can be applied to the above-mentioned server. Figure 4 The process shown is described in detail, and the data push method may specifically include the following steps:

[0080] Step S410: Obtain the data query request sent by the business end.

[0081] Step S420: According to the data query request, multiple query handles corresponding to the data query results are obtained.

[0082] In the embodiment of the present application, step S410 and step S420 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0083] Step S430: Obtain available threads in the thread pool.

[0084] In an embodiment of the present application, the server can use a thread pool to assign threads to query handles. Among them, the thread pool is a form of multi-threaded processing, which can assign tasks to threads, and the threads can automatically execute tasks. The server can pre-create a thread pool, or create a thread pool when data push is required. For example, the server can create a thread pool by using the ThreadPoolExecutor class, or by using the Executors class. The specific way of creating the thread pool may not be limited.

[0085] In some implementations, the server may obtain available threads in a thread pool, wherein available threads may refer to threads in an idle state, that is, threads that have no tasks to be executed. Available threads may be used to allocate query handles to execute tasks of pulling data from partitions.

[0086] Step S440: Allocate each query handle in the plurality of query handles to one thread in the available threads.

[0087] In an embodiment of the present application, after the server determines the available threads from the thread pool, it can assign a query handle to the threads in the available threads, wherein each query handle should be guaranteed to be assigned to an available thread to ensure that all data query results can be pulled out and sent.

[0088] In some embodiments, after determining the available threads, the server may also determine the number of available threads, compare the number with the number corresponding to the multiple query handles, and determine whether the number of available threads is less than the number corresponding to the multiple query handles based on the comparison result. If the number of available threads is greater than or equal to the number corresponding to the multiple query handles, it means that the current available threads can guarantee that each query handle is assigned to a different thread, so each query handle in the multiple query handles can be assigned to one of the available threads. If the number of available threads is less than the number corresponding to the multiple query handles, it means that the current available threads cannot guarantee that each query handle is assigned to a different thread. At this time, it is necessary to ensure that all query handles can be assigned to threads to complete the pulling and sending of all data query results.

[0089] As an implementation method, if the number of available threads is less than the number corresponding to the multiple query handles, a target number of threads can be created in the thread pool, wherein the sum of the target number and the number of available threads is greater than or equal to the number corresponding to the multiple query handles. The server can request to allocate available resources of the processor to create a target number of new threads in the thread pool, and the sum of the target number and the number of available threads is greater than or equal to the number corresponding to the multiple query handles, so that after the threads are created, the number of available threads can ensure that each query handle is assigned to a different thread.

[0090] In this implementation, before creating a thread in the thread pool, the server may also obtain the load rate of the processor, and then compare the load rate with the specified load rate, and determine whether the load rate is less than the specified load rate based on the comparison result. The specified load rate is used to determine whether the load of the processor is high. If the load rate is less than the specified load rate, it means that the load of the processor is not high, so only the target number of threads need to be created in the thread pool. If the load rate is greater than or equal to the specified load rate, it means that the load of the processor is high and it is not suitable to create a new thread in the thread pool.

[0091] In some implementations, if the load rate is greater than or equal to a specified load rate, the server may assign at least two query handles to the same thread to ensure that all query handles are assigned to the thread so that all data corresponding to the data query results can be fully retrieved and sent.

[0092] As an implementation method, if the load rate is greater than or equal to the specified load rate, the server can determine the data volume of the partial data query results corresponding to each query handle in multiple query handles, and then give at least two query handles to the same thread based on the data volume, and make the amount of data pulled and sent by each thread basically consistent.

[0093] For example, there are 8 query handles, namely query handle 1, query handle 2, query handle 3, query handle 4, query handle 5, query handle 6, query handle 7, and query handle 8. The data volume corresponding to query handle 1 is 100 megabytes (Mb), the data volume corresponding to query handle 2 is 200Mb, the data volume corresponding to query handle 3 is 400Mb, the data volume corresponding to query handle 4 is 300Mb, the data volume corresponding to query handle 5 is 200Mb, the data volume corresponding to query handle 6 is 150Mb, the data volume corresponding to query handle 7 is 150Mb, and the data volume corresponding to query handle 8 is 110Mb. Then query handle 1 and query handle 4 can be assigned to the same thread, query handle 2 and query handle 5 can be assigned to the same thread, query handle 4 can be assigned to the same thread, and query handle 6, query handle 7, and query handle 8 can be assigned to the same thread, so that the amount of data that each thread needs to pull is about 400Mb.

[0094] In some implementations, if the load rate is greater than or equal to the specified load rate, the server may also determine the priority of the task executed by the current non-idle thread, and may assign query handles to threads corresponding to tasks with low priority to pull and send data, and after completing the current data push, these threads execute the previous tasks. The priority may be determined based on the real-time nature of the task, for example, based on the data type.

[0095] Step S450: Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end.

[0096] In the embodiment of the present application, step S450 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0097] The data push method provided by the embodiment of the present application obtains a data query request sent by the business end, obtains multiple query handles corresponding to the data query result according to the query request, then obtains the available threads in the thread pool, and then assigns each query handle in the multiple query handles to one of the available threads, and ensures that the threads corresponding to each query handle are different, and then each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end, thereby achieving parallel pulling and sending of data and improving the speed of data push. In addition, when pulling and sending data, each thread adopts the method of pulling and sending at the same time, which can not take up too much memory and ensure the stable operation of the program.

[0098] See also Figure 5 , which shows a structural block diagram of a data push device 400 provided in an embodiment of the present application. The data push device 400 applies the above-mentioned server, and the data push device 400 includes: a request acquisition module 410, a handle acquisition module 420, a handle allocation module 430 and a data sending module 440. Among them, the request acquisition module 410 is used to obtain a data query request sent by the business end; the handle acquisition module 420 is used to obtain multiple query handles corresponding to the data query result according to the data query request; the handle allocation module 430 is used to assign each query handle in the multiple query handles to a thread, and each query handle corresponds to a different thread; the data sending module 440 is used for each thread to pull the data query result according to the corresponding query handle and send the data query result to the business end.

[0099] In some embodiments, the handle acquisition module 420 may include: an allocation determination unit and a handle determination unit. The shard determination unit is used to determine the multiple shards corresponding to the data query result according to the data query request; the handle determination unit is used to determine the query handle of the partial data query result corresponding to each shard, and obtain multiple query handles.

[0100] In this implementation, the shard determination unit may be specifically configured to: obtain, according to the data query request, the shard where the data query result that meets the query condition is located.

[0101] In this implementation, the shard determination unit may be specifically configured to: determine an index corresponding to a data query result according to the data query request; and obtain multiple shards corresponding to the index.

[0102] In this embodiment, the data push device 400 may further include: a data slicing module. The data slicing module may be used to slice the data stored in the database.

[0103] In this implementation, the handle determination unit may be specifically used to: obtain identification information of a partial data query result corresponding to each shard, use the identification information as a query handle, and obtain multiple query handles.

[0104] In some embodiments, the data sending module 440 can be specifically used for: each thread sends the pulled data to the business end after pulling a preset amount of data according to the corresponding query handle; repeating the process of each thread sending the pulled data to the business end after pulling a preset amount of data according to the corresponding query handle until all the data corresponding to the data query result are sent to the business end.

[0105] In some implementations, the data sending module 440 may also be used to: obtain a current remaining memory value; and determine the preset data volume according to the remaining memory value.

[0106] In this embodiment, the data sending module 440 determines the preset data amount based on the remaining memory value, which may include: judging whether the remaining memory value is greater than the specified memory value; if the remaining memory value is greater than the specified memory value, using the first data amount as the preset data amount; if the remaining memory value is less than the specified memory value, using the second data amount as the preset data amount, and the second data amount is less than the first data amount.

[0107] In some embodiments, the data sending module 440 can also be used to: obtain the ratio of the total data volume to the specified number of pull times based on the total data volume of the data query result and the specified number of pull times, and use the ratio as the preset data volume.

[0108] In some implementations, the data sending module 440 may be specifically used for: each thread, according to the corresponding query handle, uses the scroll cursor function of ElasticSearch to pull the data query results and send the data query results to the business end at the same time.

[0109] In some embodiments, the data push device 400 may further include a data volume determination module. The data volume determination module is used to determine whether the total data volume of the data query result is greater than a specified data volume before obtaining multiple query handles corresponding to the data query result according to the data query request. If the total data volume is greater than the specified data volume, the handle acquisition module 420 acquires multiple query handles corresponding to the data query result according to the data query request.

[0110] In some implementations, the data sending module 440 may also be configured to send the data query results to the service end after reading all the data query results from the database if the total data volume is less than or equal to the specified data volume.

[0111] In some implementations, the handle allocation module may include: a thread acquisition unit and an allocation unit. The thread acquisition unit is used to acquire available threads in a thread pool; the allocation unit is used to allocate each query handle in the plurality of query handles to one of the available threads.

[0112] In this embodiment, the handle allocation module may further include: a thread judgment unit. The thread judgment unit is used to judge whether the number of available threads is less than the number corresponding to the multiple query handles. If the number of available threads is greater than or equal to the number corresponding to the multiple query handles, the allocation unit allocates each query handle in the multiple query handles to one of the available threads.

[0113] In this embodiment, the handle allocation module may further include a thread creation unit. The thread creation unit is used to create a target number of threads in the thread pool if the number of available threads is less than the number corresponding to the multiple query handles, wherein the sum of the target number and the number of available threads is greater than or equal to the number corresponding to the multiple query handles.

[0114] In this implementation, the thread creation unit may be specifically used to: obtain the load rate of the processor; if the load rate is less than a specified load rate, create a target number of threads in the thread pool.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0116] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0117] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0118] To summarize, the solution provided by the present application obtains a data query request sent by a business end, obtains multiple query handles corresponding to the data query results according to the query request, and assigns each of the multiple query handles to a thread, wherein each query handle corresponds to a different thread, and then each thread pulls the data query results according to the corresponding query handle and sends the data query results to the business end at the same time, thereby realizing parallel pulling and sending of data and improving the speed of data push.

[0119] Please refer to Figure 6 , which shows a structural block diagram of a server provided in an embodiment of the present application. The server 100 can be a traditional server, a cloud server, or other server capable of running applications. The server 100 in the present application may include one or more of the following components: a processor 110, a memory 120, a touch screen 130, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0120] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire server 100, and executes various functions and processes data of the server 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0121] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the server 100 during use.

[0122] Please refer to Figure 7 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0123] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data push method, It is characterized in that The method comprises: Get the data query request sent by the business end; Determine, according to the data query request, a plurality of shards corresponding to the data query result; Obtain identification information of a portion of the data query results corresponding to each shard, and use the identification information as a query handle to obtain multiple query handles; Assigning each query handle of the plurality of query handles to a thread, wherein each query handle corresponds to a different thread; Each thread pulls the data query results according to the corresponding query handle and sends the data query results to the business end.

2. The method according to claim 1, It is characterized in that The data query request carries a query condition, and determining the multiple shards corresponding to the data query result according to the data query request includes: According to the data query request, the shards where the data query results that meet the query conditions are located are obtained.

3. The method according to claim 1, It is characterized in that The step of determining, according to the data query request, a plurality of shards corresponding to the data query result includes: Determine, according to the data query request, an index corresponding to the data query result; Get multiple shards corresponding to the index.

4. The method according to claim 1, It is characterized in that Before determining the plurality of shards corresponding to the data query result according to the data query request, the method further includes: Shard the data stored in the database.

5. The method according to any one of claims 1 to 4, It is characterized in that Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end, including: Each thread, according to the corresponding query handle, after pulling a preset amount of data, sends the pulled data to the business end; Repeat the process of each thread pulling a preset amount of data according to the corresponding query handle and sending the pulled data to the business end until all the data corresponding to the data query result are sent to the business end.

6. The method according to claim 5, It is characterized in that Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end, and also includes: Get the current remaining memory value; The preset data amount is determined according to the remaining memory value.

7. The method according to claim 6, It is characterized in that The step of determining the preset data amount according to the remaining memory value includes: Determine whether the remaining memory value is greater than a specified memory value; If the remaining memory value is greater than the specified memory value, taking the first data amount as the preset data amount; If the remaining memory value is less than the specified memory value, a second data amount is used as the preset data amount, and the second data amount is less than the first data amount.

8. The method according to claim 6, It is characterized in that Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end, and also includes: According to the total data volume of the data query result and the specified number of pulls, the ratio of the total data volume to the specified number of pulls is obtained, and the ratio is used as the preset data volume.

9. The method according to any one of claims 1 to 4, It is characterized in that Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end, including: Each thread pulls the data query results according to the corresponding query handle through the scroll cursor function of ElasticSearch and sends the data query results to the business end.

10. The method according to any one of claims 1 to 4, It is characterized in that Before determining the plurality of shards corresponding to the data query result according to the data query request, the method further includes: Determine whether the total data volume of the data query result is greater than the specified data volume; If the total data volume is greater than the specified data volume, multiple shards corresponding to the data query result are determined according to the data query request.

11. The method according to claim 10, It is characterized in that The method further comprises: If the total data volume is less than or equal to the specified data volume, after reading all the data query results from the database, the data query results are sent to the service end.

12. The method according to any one of claims 1 to 4, It is characterized in that The allocating each query handle of the plurality of query handles to a thread comprises: Get available threads in the thread pool; Each query handle of the plurality of query handles is assigned to one of the available threads.

13. The method according to claim 12, It is characterized in that The allocating each query handle of the plurality of query handles to one of the available threads comprises: Determine whether the number of available threads is less than the number corresponding to the multiple query handles; If the number of the available threads is greater than or equal to the number corresponding to the multiple query handles, each query handle in the multiple query handles is assigned to one of the available threads.

14. The method according to claim 13, It is characterized in that The allocating each query handle in the plurality of query handles to one of the available threads further comprises: If the number of available threads is less than the number corresponding to the multiple query handles, a target number of threads is created in the thread pool, wherein the sum of the target number and the number of available threads is greater than or equal to the number corresponding to the multiple query handles.

15. The method according to claim 14, It is characterized in that Before creating a target number of threads in the thread pool, the method further includes: Get the processor load rate; If the load rate is less than the specified load rate, a target number of threads are created in the thread pool.

16. A data push device, It is characterized in that The device comprises: a request acquisition module, a handle acquisition module, a handle allocation module and a data sending module, wherein: The request acquisition module is used to obtain the data query request sent by the service end; The handle acquisition module is used to determine the multiple shards corresponding to the data query result according to the data query request; obtain identification information of the partial data query result corresponding to each shard, use the identification information as the query handle, and obtain multiple query handles; The handle allocation module is used to allocate each query handle of the multiple query handles to a thread, and each query handle corresponds to a different thread; The data sending module is used for each thread to pull the data query result according to the corresponding query handle and send the data query result to the business end.

17. A server, It is characterized in that include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-15.

18. A computer readable storage medium, It is characterized in that The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 15.

Images