A data processing method, apparatus, device, and medium

Abstract

The application relates to a data processing method, device and equipment and medium, and the method comprises the following steps: acquiring multiple data write requests; judging whether the number of data write requests exceeds a write request number threshold; if the write request threshold is exceeded, the multiple data write requests are cached to redis; when a preset condition is met, all data write requests stored in the redis are batch exported, spliced based on all the exported data write requests to obtain a bulk request, wherein the preset condition comprises that time reaches a preset time and the number of data write requests reaches a preset number; and the exported all data write requests are written into a disk by using the bulk request. In this way, the problem that ES refuses to execute data write requests when the data write requests exceed the ES write capability is effectively solved, and the ES write capability is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and medium. Background Technology

[0002] With the rapid development of the internet and information technology, more and more businesses can conduct operations online. For example, in the express delivery industry, online claims processing is now available. Along with the booming development of the express delivery industry, the amount of data related to claims processing has also increased. Therefore, most companies use Elasticsearch (ES, a distributed search and analytics engine) indexing methods to process this data and improve its performance.

[0003] While Elasticsearch (ES) is generally efficient at data processing, and projects using ES can typically fulfill its data processing requirements, it will reject write requests when they exceed ES's write capacity, leading to write failures.

[0004] Therefore, how to improve the write capability of Elasticsearch is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a data processing method, apparatus, device, and medium to solve at least one of the above-mentioned technical problems.

[0006] The above-mentioned inventive objective of this application is achieved through the following technical solutions:

[0007] Firstly, this application provides a data processing method, which adopts the following technical solution:

[0008] A data processing method, the method comprising:

[0009] Retrieve multiple data write requests;

[0010] Determine whether the number of data write requests exceeds the write request count threshold;

[0011] If the write request threshold is exceeded, the multiple write requests will be cached in Redis.

[0012] When preset conditions are met, all data write requests stored in Redis are exported in batches, and the exported data write requests are concatenated to obtain a bulk request. The preset conditions include the time reaching a preset time and the number of data write requests reaching a preset number.

[0013] Using the bulk request, all exported data is written to the disk along with the corresponding data to be written.

[0014] By adopting the above technical solution, if the number of data write requests exceeds the write request threshold, multiple data write requests will be cached in Redis. When preset conditions are met, all data write requests stored in Redis are exported in batches and concatenated into a bulk request. Using the bulk request, the data to be written corresponding to all exported data write requests is written to disk. This method effectively solves the problem of Elasticsearch refusing to execute data write requests when the number of data write requests exceeds the Elasticsearch write capacity, thus improving the Elasticsearch write capability.

[0015] In a preferred embodiment, this application can be further configured as follows: after using the bulk request to write all the exported data to be written corresponding to the write request to the disk, it further includes:

[0016] Obtain the first feedback information after the data to be written is written to the disk, and determine whether the batch writing of the data to be written has been successfully executed based on the first feedback information;

[0017] If the execution fails, the number of failures is recorded, and it is determined whether the number of failures exceeds the failure threshold.

[0018] If the failure threshold is not exceeded, the bulk request is re-executed to write all the exported data to the disk corresponding to the write request.

[0019] If the failure threshold is exceeded, all data to be written corresponding to the data write request in the bulk request will be written to the disk in sequence.

[0020] By adopting the above technical solution, based on the obtained first feedback information, it is determined whether the batch write of data to be written has been successfully executed. If the batch write operation fails and is within the failure threshold, the operation of batch writing using bulk requests will be re-executed. If the number of failures exceeds the failure threshold, the batch write to Elasticsearch will be changed to a single data write to Elasticsearch. This approach balances improving write performance by using bulk requests for batch writing with the ability to write data, while also ensuring that correct data write requests can still be written normally even if there are errors in the data write request portion.

[0021] In a preferred embodiment, this application can be further configured as follows: the step of sequentially writing the data to be written corresponding to all data write requests in the bulk request to the disk includes:

[0022] The bulk request is split into multiple target data write requests;

[0023] Using each target data write request, the data to be written corresponding to the target data write request is written to the disk to complete the writing of all data to be written.

[0024] Obtain the second feedback information for each target data write request, and determine whether the data to be written corresponding to the target data write request has been successfully written to the disk based on the second feedback information;

[0025] If the write to disk fails, the write request for the target data that failed to write to disk is placed in the write failure queue, and the failure information is recorded in the log.

[0026] By adopting the above technical solution, when the number of failures exceeds the failure threshold, the bulk request is split into multiple target data write requests. Each target data write request then writes the corresponding data to disk, ensuring that even if an error exists in one part of the data write request, the correct data write request can still be written successfully. After execution, a second feedback message is obtained, and based on this message, the target data write requests that failed to write to disk are placed in a write failure queue. Failure information is recorded in a log. Using the failure information recorded in the log, erroneous data write requests can be accurately identified, and effective countermeasures can be taken.

[0027] In a preferred embodiment, this application may be further configured such that the data processing method further includes:

[0028] Obtain a data read request, and determine the data query conditions and the data information of the index item for the read operation based on the data read request;

[0029] The storage location of the index item is determined based on the data information of the index item, and the data to be read corresponding to the data read request is determined based on the storage location and the data query conditions.

[0030] The data to be read is sent to the display interface.

[0031] By adopting the above technical solution, the data query conditions and the data information of the index item to be read are determined according to the data read request, and the storage location of the index item is determined according to the data information of the index item. Based on the storage location and the data query conditions, the data to be read corresponding to the data read request is read and sent to the display interface. In this way, the data to be read can be accurately read from ES based on the data read request.

[0032] In a preferred embodiment, this application can be further configured such that the data information of the index item includes feature labels.

[0033] Determining the storage location of the index entry based on its data information includes:

[0034] When the feature label is a hot data label, the storage location of the index item is determined to be the first position;

[0035] When the feature label is a cold data label, the storage location of the index item is determined to be the second location;

[0036] The hot data tags are feature tags for index items whose creation time is less than a first time threshold; the cold data tags are feature tags for index items whose creation time is not less than the first time threshold.

[0037] By adopting the above technical solution, the feature labels of index items are divided into hot data labels and cold data labels according to the creation time of the index items, and different types of feature labels are stored in different locations to facilitate data reading by classification and improve the speed of data reading.

[0038] In a preferred embodiment, this application may be further configured such that the data processing method further includes:

[0039] At a preset time point, query the index item whose feature tag is a hot data tag, and record it as a hotspot index item;

[0040] Based on the hotspot index entries, hotspot data is identified and loaded into the cache.

[0041] By adopting the above technical solution, hot data in index items with hot data tags are loaded into the cache at preset time points, which greatly improves the read speed of Elasticsearch.

[0042] In a preferred embodiment, this application can be further configured such that the data information of the index item includes the number of shards.

[0043] Methods for determining the number of shards include:

[0044] The number of shards is adjusted based on the data volume, where the data volume is used to characterize the level of data that the business needs to process.

[0045] By adopting the above technical solution, the number of shards was adjusted according to the amount of data that the business needs to process, thereby improving the performance of ES data reading.

[0046] Secondly, this application provides a data processing apparatus, which adopts the following technical solution:

[0047] A data processing apparatus, comprising,

[0048] The acquisition module is used to acquire multiple data write requests.

[0049] The judgment module is used to determine whether the number of data write requests exceeds the write request number threshold.

[0050] The caching module is used to cache the multiple data write requests in Redis if the write request threshold is exceeded.

[0051] The request concatenation module is used to export all data write requests stored in Redis in batches when preset conditions are met, and concatenate all exported data write requests to obtain bulk requests. The preset conditions include the time reaching a preset time and the number of data write requests reaching a preset number.

[0052] The bulk write module is used to write all the data to be written corresponding to the exported data write request to the disk using the bulk request.

[0053] By adopting the above technical solution, if the number of data write requests exceeds the write request threshold, multiple data write requests will be cached in Redis. When preset conditions are met, all data write requests stored in Redis are exported in batches and concatenated into a bulk request. Using the bulk request, the data to be written corresponding to all exported data write requests is written to disk. This method effectively solves the problem of Elasticsearch refusing to execute data write requests when the number of data write requests exceeds the Elasticsearch write capacity, thus improving the Elasticsearch write capability.

[0054] Thirdly, this application provides an electronic device that adopts the following technical solution:

[0055] At least one processor;

[0056] Memory;

[0057] At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application being configured to: perform the method described above.

[0058] By adopting the above technical solution, if the number of data write requests exceeds the write request threshold, multiple data write requests will be cached in Redis. When preset conditions are met, all data write requests stored in Redis are exported in batches and concatenated into a bulk request. Using the bulk request, the data to be written corresponding to all exported data write requests is written to disk. This method effectively solves the problem of Elasticsearch refusing to execute data write requests when the number of data write requests exceeds the Elasticsearch write capacity, thus improving the Elasticsearch write capability.

[0059] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0060] A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described above.

[0061] By adopting the above technical solution, if the number of data write requests exceeds the write request threshold, multiple data write requests will be cached in Redis. When preset conditions are met, all data write requests stored in Redis are exported in batches and concatenated into a bulk request. Using the bulk request, the data to be written corresponding to all exported data write requests is written to disk. This method effectively solves the problem of Elasticsearch refusing to execute data write requests when the number of data write requests exceeds the Elasticsearch write capacity, thus improving the Elasticsearch write capability.

[0062] In summary, this application includes at least one of the following beneficial technical effects:

[0063] 1. If the number of write requests exceeds the write request threshold, multiple write requests will be cached in Redis. When preset conditions are met, all write requests stored in Redis are exported in batches and concatenated into a bulk request. This bulk request then writes the data to be written corresponding to all exported write requests to disk. This method effectively solves the problem of Elasticsearch refusing to execute write requests when the number of write requests exceeds its write capacity, thus improving Elasticsearch's write performance.

[0064] 2. At preset time points, hot data within index items with the feature label "hot data label" is loaded into the cache, greatly improving the reading speed of hot data. Attached Figure Description

[0065] Figure 1 This is a flowchart illustrating a data processing method according to one embodiment of this application.

[0066] Figure 2 This is a schematic diagram illustrating the process of loading a data write request into Redis according to one embodiment of this application.

[0067] Figure 3 This is a schematic diagram of the structure of a data processing device according to one embodiment of this application.

[0068] Figure 4 This is a schematic diagram of the structure of an electronic device according to one embodiment of this application. Detailed Implementation

[0069] The following combination Figures 1 to 4 This application will be described in further detail.

[0070] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.

[0071] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0073] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0074] This application provides a data processing method executed by an electronic device, which can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication. This application does not impose any limitations on this connection. Figure 1 As shown, the method includes steps S101, S102, S103, S104, and S105, wherein:

[0075] Step S101: Obtain multiple data write requests;

[0076] Step S102: Determine whether the number of data write requests exceeds the write request number threshold;

[0077] In this embodiment, the data write request is initiated by the client and is used to write index data in Elasticsearch (ES). When the volume of business data is large, the number of data write requests may exceed the write request limit in ES, meaning the data write requests exceed ES's write capacity. Therefore, by determining whether the number of data write requests exceeds the write request limit, it can be determined whether the data write requests exceed ES's write capacity. In the prior art, if the data write requests exceed ES's write capacity, ES will refuse to write data, resulting in the loss of data write requests.

[0078] Step S103: If the write request threshold is exceeded, cache multiple data write requests in Redis;

[0079] In this embodiment of the application, if the write request threshold is exceeded, it indicates that the data write request exceeds the write capacity of ES. In this case, multiple data write requests are cached in Redis to save multiple data write requests and avoid data loss due to ES refusing to execute the data write request.

[0080] If the number of data write requests does not exceed the write request quantity threshold, it indicates that the data write requests are within the Elasticsearch (ES) write capability. There are several ways to execute write requests. In one possible approach, the data to be written for each data write request is written to disk one by one. In another possible approach, multiple data write requests are aggregated into bulk requests, and the data to be written for each of these requests is written to disk in batches. Of course, there may be other ways to execute write requests, which are not limited in this embodiment.

[0081] Step S104: When the preset conditions are met, export all data write requests stored in Redis in batches, and concatenate all the exported data write requests to obtain a bulk request. The preset conditions include the time reaching a preset time and the number of data write requests reaching a preset number.

[0082] For the embodiments of this application, the preset conditions may include a preset time period and a preset number of data write requests. Specifically, if the preset condition is that the preset time period is reached, a scheduled task can be used to export all data write requests stored in Redis in batches when the preset time period is reached. The preset time can be set by the user, for example, 1 second, 2 seconds, etc. If the preset condition is that the number of data write requests reaches a preset number, a quantitative task can be used to export all data write requests stored in Redis in batches when the number of data write requests reaches a preset number. The preset number can be set by the user, for example, 300 records, 500 records, etc. Using Redis, data write requests from multiple servers can be received, similar to the function of a buffer in MySQL. Each time data write requests are retrieved in batches from Redis, a distributed lock is introduced. The machine that acquires the lock retrieves the data write requests in batches from Redis, concatenates them into bulk requests, and writes them to Elasticsearch all at once.

[0083] By concatenating all exported data write requests, a bulk request can be obtained. When there are a large number of data write requests, the performance of batch writing using bulk requests is much higher than that of writing based on each individual data write request. However, the performance of the bulk request is not necessarily improved by increasing the number of concatenated data write requests, as a large bulk request can lead to excessive memory pressure. Therefore, the number of concatenated data write requests in a bulk request is important. The optimal bulk request size is determined as follows: concatenate 100 data write requests into a bulk request and test Elasticsearch (ES) write performance using this 100-data-write-request-concatenation bulk request; then concatenate 200 data write requests and 400 data write requests into bulk requests respectively, and test ES write performance using these bulk requests; and so on, doubling the number of concatenated data write requests each time. By testing bulk requests with different numbers of concatenated data write requests one by one, the optimal bulk request is the one whose ES write performance begins to plateau.

[0084] Step S105: Using bulk requests, write all the exported data to disk, corresponding to the data to be written in the write request.

[0085] In this embodiment, bulk requests are used to achieve batch writing of data write requests, improving the write capability of Elasticsearch (ES). Bulk requests are used to write the data to be written corresponding to all exported data write requests to disk. Specifically, the bulk requests are submitted to the ES cluster, and the ES cluster determines the node that submitted the bulk request as the coordinating node, which is used to process the bulk requests. Then, the coordinating node determines the target node of the bulk request based on the target index and routing information of each data write request in the bulk request. The coordinating node forwards the bulk request to the target node once based on the routing information, and writes the corresponding data to be written to the target node on disk.

[0086] In this embodiment of the application, a schematic diagram of the process of loading a data write request into Redis is shown below. Figure 2 As shown, specifically, multiple data write requests correspond to Figure 2 Write request 1, write request 2, and write request 3, with preset conditions corresponding to... Figure 2 The system uses a scheduled job to batch export write requests, then concatenates these requests into bulk requests, and uses these bulk requests to write the requests to Elasticsearch in batches. After the write is complete, the results are returned to the client interface, which can include whether the write was successful or not. Using this method, Elasticsearch's write capabilities are significantly improved, by approximately an order of magnitude.

[0087] As can be seen, in this embodiment, if the number of data write requests exceeds the write request threshold, multiple data write requests are cached in Redis. When a preset condition is met, all data write requests stored in Redis are exported in batches and concatenated into a bulk request. Using the bulk request, the data to be written corresponding to all exported data write requests is written to disk. This method effectively solves the problem of Elasticsearch (ES) refusing to execute data write requests when the number of data write requests exceeds ES's write capacity, thus improving ES's write capability.

[0088] Furthermore, to balance write performance and ensure that correct data write requests can still be written normally even when errors exist in the data write request portion, this embodiment of the application, after using bulk requests to write the data to be written corresponding to all exported data write requests to the disk, also includes:

[0089] Obtain the first feedback information after the data to be written is written to the disk, and determine whether the batch writing of the data to be written has been successfully executed based on the first feedback information;

[0090] If the execution fails, record the number of failures and determine whether the number of failures exceeds the failure threshold.

[0091] In this embodiment of the application, after writing all exported data to disk using a bulk request, a first feedback message can be obtained. This first feedback message includes two types: "bulk request executed successfully" and "bulk request executed unsuccessfully." The meanings of the first feedback messages can be preset; for example, "true" in the first feedback message indicates successful bulk request execution, and "false" indicates unsuccessful bulk request execution. Alternatively, if feedback is obtained within a preset time, it indicates successful bulk request execution; if no feedback is obtained within the preset time, it also indicates successful bulk request execution. The specific method used to determine whether a bulk request was successfully executed using the first feedback message is not limited in this embodiment of the application.

[0092] If the bulk request is determined to have been executed successfully based on the first feedback information, the results of the batch write will be sent to the display interface for presentation; if the bulk request fails to be executed based on the first feedback information, the number of failures will be recorded and compared with the failure threshold to determine the subsequent execution operation.

[0093] If the failure threshold is not exceeded, the bulk request will be re-executed to write all the exported data to the disk corresponding to the write request.

[0094] If the failure threshold is exceeded, all data in the bulk request will be written to the disk sequentially, corresponding to the data to be written in the write request.

[0095] In this embodiment of the application, when bulk write operations fail, there are various reasons for the failure, including but not limited to one or more of the following: write timeout due to network issues, ES cluster crash, errors in the data write requests within the bulk request, etc. If the number of bulk request execution failures does not exceed the failure threshold, network and ES cluster issues are considered first, and the bulk request is re-executed to write the data to be written corresponding to all exported data write requests to disk. By using a retry mechanism, the impact of unstable factors on bulk write operations can be minimized.

[0096] If the number of failed bulk request executions exceeds the failure threshold, it indicates that at least one of the data write requests concatenated in the bulk request has an error. Continuing to use the bulk request batch writing method will not succeed. Therefore, the bulk request that concatenates multiple data write requests is split into single data write requests, and the data write requests are executed one by one, writing the data to be written corresponding to all data write requests to the disk in sequence.

[0097] As can be seen, in this embodiment, based on the obtained first feedback information, it is determined whether the batch write of data to be written has been successfully executed. If the batch write operation fails and is within the failure threshold, the operation of batch writing using bulk requests will be re-executed. If the number of failures exceeds the failure threshold, the batch write to Elasticsearch will be changed to single data write to Elasticsearch. This approach balances improving write performance by using bulk requests for batch writing with the ability to write data, while also ensuring that correct data write requests can be written normally even when there are errors in the data write request portion.

[0098] Furthermore, to ensure that correct data write requests can be written normally, and to accurately identify erroneous data write requests based on the unsuccessful information recorded in the log, in this embodiment, the data to be written corresponding to all data write requests in the bulk request is sequentially written to the disk, including:

[0099] The bulk request is split into multiple target data write requests;

[0100] Using each target data write request, the data to be written corresponding to the target data write request is written to the disk to complete the writing of all data to be written.

[0101] In this embodiment, since a bulk request is composed of multiple data write requests concatenated together, the multiple target data write requests obtained by splitting the bulk request correspond to the multiple data write requests that were used to construct the bulk request. Because bulk requests cannot be used for batch writing, the bulk request is split into multiple target data write requests, and a write operation to Elasticsearch is performed separately for each target data write request. This avoids the situation where errors in some data write requests within the bulk request prevent all data write requests from failing to write.

[0102] Obtain the second feedback information for each target data write request, and determine whether the data to be written corresponding to the target data write request has been successfully written to the disk based on the second feedback information;

[0103] If the write to disk fails, the write request for the target data that failed to write to disk is placed in the write failure queue, and the failure information is recorded in the log.

[0104] In this embodiment, after each target data write request executes the write operation to Elasticsearch (ES), a second feedback message is obtained. This second feedback message includes two types: successful execution of the target data write request and failed execution of the target data write request. The form of the second feedback message is not limited in this embodiment; users can set it according to their needs. If the second feedback message determines that a target data write request has failed, this target data write request is placed in a failure queue. A target data write request placed in the failure queue indicates that the data write request has an error and will not be written to ES again. Simultaneously, information about unsuccessful target data write requests can be recorded in a log, and the log content including the failure information can be sent to the display interface to facilitate determining that an error has occurred in the target data write request and to promptly troubleshoot the problem.

[0105] As can be seen, in this embodiment, when the number of failures exceeds the failure threshold, the bulk request is split into multiple target data write requests. Each target data write request is used to write the corresponding data to the disk, ensuring that even if an error exists in one data write request, the correct data write request can still be written normally. After execution, second feedback information is obtained, and based on this information, the target data write requests that failed to write to the disk are placed in a write failure queue. Failure information is recorded in a log. Using the failure information recorded in the log, erroneous data write requests can be accurately identified, and effective countermeasures can be taken.

[0106] Furthermore, in order to accurately read the data to be read from ES based on the data read request, in this embodiment of the application, the data processing method further includes:

[0107] Obtain the data read request, and determine the data query conditions and the data information of the index item to be read based on the data read request;

[0108] In this embodiment, the data read request is initiated by the client and is a request to read index data in Elasticsearch (ES). Based on the data read request, the index item to be read can be determined, and thus, the data information of the index item can be determined. The data information of the index item may include the number of shards allocated to the index item, the number of replica shards, the feature tags of the index item, and the data orientation configuration path, etc. The number of shards for the index item can be adjusted according to the data volume, and replica shards can handle data read requests; therefore, the number of replica shards can be increased within the capacity limits of ES.

[0109] Based on the data information of the index entries, the data characteristics of the index entries can be determined, and subsequent data reading operations can be implemented. When it is determined that the obtained data is a data reading request, the data query conditions can be determined according to the data reading request, so as to use the data query conditions to query the data to be read in the ES cluster.

[0110] The storage location of the index item is determined based on the data information of the index item, and the data to be read corresponding to the data read request is determined based on the storage location and data query conditions.

[0111] Send the data to be read to the display interface.

[0112] For the embodiments of this application, there are multiple ways to determine the storage location of an index item based on its data information. One possible approach is to determine the storage location based on feature tags in the index item's data information. Specifically, feature tags can include cold data tags, warm data tags, and hot data tags. Different types of feature tags correspond to different storage locations, thus enabling the determination of the storage location based on the feature tags. Another possible approach is to first determine the storage layer corresponding to the determined feature tag, where the storage layer refers to different partitioned storage media. After determining the storage layer corresponding to the feature tag, the storage location of the index item is determined within that storage layer.

[0113] Then, based on the defined query conditions, the system retrieves the data to be read corresponding to the data read request from the storage location and sends the data to the display interface to complete the read operation. When the amount of data to be read is large, there is a problem of large export or list pagination page counts. Using the ordinary `from size` method to read data is too inefficient and does not meet the performance requirements of Elasticsearch data reading. At the same time, `scroll` is based on a memory snapshot, which has low real-time performance and high performance consumption. In addition, if there are cluster or node outliers, it will cause problems. Preferably, `search after` is used for deep pagination. Since `search after` only maintains a temporary cursor, its performance consumption is low.

[0114] As can be seen, in this embodiment, the data query conditions and the data information of the index item to be read are determined according to the data read request, and the storage location of the index item is determined according to the data information of the index item. Based on the storage location and the data query conditions, the data to be read corresponding to the data read request is read and sent to the display interface. In this way, the data to be read can be accurately read from ES based on the data read request.

[0115] Furthermore, in order to classify and read data and improve the speed of data reading, in this embodiment of the application, the data information of the index item includes feature tags.

[0116] Determining the storage location of an index entry based on its data information includes:

[0117] When the feature label is a hot data label, the storage location of the index item is determined to be the first position;

[0118] When the feature label is a cold data label, the storage location of the index item is determined to be the second position;

[0119] Among them, hot data tags are feature tags for index items whose creation time is less than the first time threshold; cold data tags are feature tags for index items whose creation time is not less than the first time threshold.

[0120] In the embodiments of this application, in practical applications, each index entry stores data for a period of time, for example, one month's worth of data. The number of times data in an index entry is accessed also varies with the index entry's creation time. Simply put, when reading data, access to recently stored data is higher, while access to data stored for a longer period is lower. For example, in a claims business scenario, access to claims drops significantly after the reconciliation period on the 2nd of each month. Based on this business scenario, an index template is created for claims. The data for each month is written into the index for that month, and in the lifecycle management, it is set that if the index creation time is less than 45 days, the feature tag is a hot data tag; if the index creation time exceeds 45 days, the feature tag is downgraded to a cold data tag. Of course, it is also possible to set it to automatically delete the index after the creation time exceeds 730 days.

[0121] Based on the creation time of index entries, feature tags can be divided into hot data tags and cold data tags. Feature tags are set for index entries. If an index entry's feature tags are designated as hot data tags, then all data stored in that index entry is hot data; if the feature tags are designated as cold data tags, then all data stored in that index entry is cold data. Using feature tags to differentiate data in Elasticsearch and allocating different storage locations for index entries based on different feature tags facilitates categorized data retrieval and improves data retrieval speed.

[0122] As can be seen, in this embodiment of the application, the feature tags of the index items are divided into hot data tags and cold data tags according to the creation time of the index items, and different types of feature tags are stored in different locations to facilitate the classification and reading of data and improve the speed of data reading.

[0123] Furthermore, to improve the read speed of Elasticsearch, in this embodiment of the application, the data processing method further includes:

[0124] At a preset time point, the index items whose feature tags are hot data tags are queried and recorded as hot index items;

[0125] Based on the hot index items, identify hot data and load it into the cache.

[0126] In this embodiment, since the data written to Elasticsearch (ES) is actually written to disk files, during data reading, ES loads some data from the disk into the cache. Because the cache acts as a high-speed data exchange storage device, data is read from the cache much faster than from the disk. Therefore, within the limits of ES capacity, the cache space can be allocated as much as possible. The data written to ES is stored in disk files. Preferably, index entries can be allocated according to the time the data was written, and different feature tags can be assigned to different index entries. Considering reality, recently accessed data is accessed more frequently, while data stored for a longer period is accessed less frequently. Therefore, placing frequently accessed data in the cache can greatly improve the read performance of ES.

[0127] Specifically, at a preset time point, hot data from index entries labeled as "hot data" is loaded into the cache. This preset time point is preferably during off-peak hours for Elasticsearch (ES) read / write operations, such as 2 AM daily, when ES read / write activity is extremely low. Since the access volume for hot data during ES reads is significantly higher than for data in index entries labeled as "cold data," iterating through the hot data in the index entries labeled as "hot data" at the preset time point ensures that as much hot data as possible is loaded into the cache, greatly utilizing cache resources and improving ES read speed.

[0128] As can be seen, in this embodiment of the application, at a preset time point, the hot data in the index item with the feature tag as hot data tag is loaded into the cache, which greatly improves the reading speed of ES.

[0129] Furthermore, to improve the performance of ES data reading, in this embodiment, the data information of the index item includes the number of shards.

[0130] Methods for determining the number of shards include:

[0131] Adjust the number of shards based on the data volume, where the data volume is used to characterize the level of data that the business needs to process.

[0132] In the embodiments of this application, both the number of shards and the size of the shards affect ES data reading. When the shard size is too large, the speed of reading data from the shards becomes slower; when the shard size is too small, although the speed of reading data in a single shard will increase, reading data from a large number of small shards requires queuing and processing in sequence, which is not very efficient. Therefore, the size of the shards is quite important, and preferably, the shard size is controlled between 20-32G.

[0133] Furthermore, the number of shards is adjusted based on the data volume and the recommended shard size to improve Elasticsearch (ES) data read performance. Data volume characterizes the amount of data the business needs to process. The amount of data stored in ES can be determined by historical data volume, or it can be estimated based on the data volume over a short period. For example, a business might have 2000 records per month, and use data from 12 months during operation, resulting in a total data volume of 24000 records. The data volume determines the required storage space. Dividing the total storage space by the recommended shard size determines the optimal number of shards to ensure high ES data read performance.

[0134] As can be seen, in this embodiment of the application, the number of shards is adjusted according to the amount of data that the business needs to process, thereby improving the performance of ES data reading.

[0135] The above embodiments describe a data processing method from the perspective of process flow. The following embodiments describe a data processing device from the perspective of virtual module or virtual unit. For details, please refer to the following embodiments.

[0136] This application provides a data processing device 200, such as... Figure 3 As shown, the data processing device 200 may specifically include:

[0137] Module 210 is used to acquire multiple data write requests;

[0138] The judgment module 220 is used to determine whether the number of data write requests exceeds the write request number threshold;

[0139] The caching module 230 is used to cache multiple data write requests in Redis if the write request threshold is exceeded.

[0140] The request splicing module 240 is used to export all data write requests stored in Redis in batches when preset conditions are met, and splice all the exported data write requests to obtain bulk requests. The preset conditions include the time reaching a preset time and the number of data write requests reaching a preset number.

[0141] The bulk write module 250 is used to write all exported data to disk using bulk requests, corresponding to the data to be written in the write request.

[0142] In this embodiment, if the number of data write requests exceeds a threshold, multiple data write requests are cached in Redis. When a preset condition is met, all data write requests stored in Redis are exported in batches and concatenated into a bulk request. Using the bulk request, the data to be written corresponding to all exported data write requests is written to disk. This method effectively solves the problem that Elasticsearch (ES) refuses to execute data write requests when the number of data write requests exceeds its write capacity, thus improving ES's write capability.

[0143] In one possible implementation of this application embodiment, the data processing apparatus 200 further includes:

[0144] The single-write module is used to obtain the first feedback information after the data to be written is written to the disk, and to determine whether the batch writing of the data to be written has been successfully executed based on the first feedback information;

[0145] If the execution fails, record the number of failures and determine whether the number of failures exceeds the failure threshold.

[0146] If the failure threshold is not exceeded, the bulk request will be re-executed to write all the exported data to the disk corresponding to the write request.

[0147] If the failure threshold is exceeded, all data in the bulk request will be written to the disk sequentially, corresponding to the data to be written in the write request.

[0148] In one possible implementation of this application embodiment, when the single-write module executes the sequential writing of the data to be written corresponding to all data write requests in the bulk request to the disk, it is used to:

[0149] The bulk request is split into multiple target data write requests;

[0150] Using each target data write request, the data to be written corresponding to the target data write request is written to the disk to complete the writing of all data to be written.

[0151] Obtain the second feedback information for each target data write request, and determine whether the data to be written corresponding to the target data write request has been successfully written to the disk based on the second feedback information;

[0152] If the write to disk fails, the write request for the target data that failed to write to disk is placed in the write failure queue, and the failure information is recorded in the log.

[0153] In one possible implementation of this application embodiment, the data processing apparatus 200 further includes:

[0154] The data reading module is used to obtain data reading requests and determine the data query conditions and the data information of the index items to be read based on the data reading requests.

[0155] The storage location of the index item is determined based on the data information of the index item, and the data to be read corresponding to the data read request is determined based on the storage location and data query conditions.

[0156] Send the data to be read to the display interface.

[0157] In one possible implementation of this application embodiment, the data information of the index item includes a feature tag. When the data reading module determines the storage location of the index item based on the data information of the index item, it is used to:

[0158] When the feature label is a hot data label, the storage location of the index item is determined to be the first position;

[0159] When the feature label is a cold data label, the storage location of the index item is determined to be the second position;

[0160] Among them, hot data tags are feature tags for index items whose creation time is less than the first time threshold; cold data tags are feature tags for index items whose creation time is not less than the first time threshold.

[0161] In one possible implementation of this application embodiment, the data processing apparatus 200 further includes:

[0162] The data preheating module is used to query index items whose feature tags are hot data tags at preset time points, and these are denoted as hot index items.

[0163] Based on the hot index items, identify hot data and load it into the cache.

[0164] One possible implementation of this application embodiment includes index item data information including the number of fragments, and the data processing device 200 further includes:

[0165] The sharding quantity determination module is used to adjust the sharding quantity based on the data volume, where the data volume is used to characterize the amount of data that the business needs to process.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the data processing device 200 described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0167] This application provides an electronic device, such as... Figure 4 As shown, Figure 4The illustrated electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device 300 does not constitute a limitation on the embodiments of this application.

[0168] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0169] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0170] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0171] The memory 303 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0172] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0173] This application provides a computer-readable storage medium storing a computer program. When the program is run on a computer, it enables the computer to execute the corresponding content in the aforementioned method embodiments. Compared with related technologies, this application, based on the obtained first feedback information, determines whether the batch writing of data to be written has been successfully executed. If the batch writing fails and is within a failure threshold, the operation of batch writing using bulk requests will be re-executed. If the number of failures exceeds the failure threshold, the batch writing to Elasticsearch will be changed to writing single data records to Elasticsearch. This approach balances improving write performance by utilizing bulk requests for batch writing with the ability to write data correctly while ensuring that correct data write requests can still be written normally even when errors exist in the data write request portion.

[0174] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0175] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Retrieve multiple data write requests; Determine whether the number of data write requests exceeds the write request count threshold; If the write request threshold is exceeded, the multiple write requests will be cached in Redis. When preset conditions are met, all data write requests stored in Redis are exported in batches, and the exported data write requests are concatenated to obtain a bulk request. The preset conditions include the time reaching a preset time and the number of data write requests reaching a preset number. Using the bulk request, all data to be written corresponding to the exported data write request is written to the disk; wherein, the bulk request is obtained by concatenating all the exported data write requests, including: testing the bulk requests with different numbers of data write requests one by one, and when the ES write performance of the bulk request begins to level off, the concatenated bulk request is the best bulk request, and the number of requests is doubled after each time the concatenated data in the bulk request is written; After using the bulk request to write all the data to be written corresponding to the exported data write request to the disk, the process further includes: Obtain the first feedback information after the data to be written is written to the disk, and determine whether the batch writing of the data to be written has been successfully executed based on the first feedback information; If the execution fails, the number of failures is recorded, and it is determined whether the number of failures exceeds the failure threshold. If the failure threshold is not exceeded, the bulk request is re-executed to write all the exported data to the disk corresponding to the write request. If the failure threshold is exceeded, all data to be written corresponding to the data to be written in the bulk request will be written to the disk in sequence. The step of sequentially writing the data to be written corresponding to all data write requests in the bulk request to the disk includes: The bulk request is split into multiple target data write requests; Using each target data write request, the data to be written corresponding to the target data write request is written to the disk to complete the writing of all data to be written. Obtain the second feedback information for each target data write request, and determine whether the data to be written corresponding to the target data write request has been successfully written to the disk based on the second feedback information; If the write to disk fails, the write request for the target data that failed to write to disk is placed in the write failure queue, and the failure information is recorded in the log.

2. The data processing method according to claim 1, characterized in that, Also includes: Obtain a data read request, and determine the data query conditions and the data information of the index item for the read operation based on the data read request; The storage location of the index item is determined based on the data information of the index item, and the data to be read corresponding to the data read request is determined based on the storage location and the data query conditions. The data to be read is sent to the display interface.

3. The data processing method according to claim 2, characterized in that, The data information of the index item includes feature labels. Determining the storage location of the index entry based on its data information includes: When the feature label is a hot data label, the storage location of the index item is determined to be the first position; When the feature label is a cold data label, the storage location of the index item is determined to be the second location; The hot data tags are feature tags for index items whose creation time is less than a first time threshold; the cold data tags are feature tags for index items whose creation time is not less than the first time threshold.

4. The data processing method according to claim 3, characterized in that, Also includes: At a preset time point, query the index item whose feature tag is a hot data tag, and record it as a hotspot index item; Based on the hotspot index entries, hotspot data is identified and loaded into the cache.

5. The data processing method according to claim 2, characterized in that, The data information in the index entry includes the number of shards. Methods for determining the number of shards include: Adjust the number of shards based on the data volume, where the data volume is used to characterize the level of data that the business needs to process.

6. A data processing apparatus, characterized in that, include, The acquisition module is used to acquire multiple data write requests. The judgment module is used to determine whether the number of data write requests exceeds the write request number threshold. The caching module is used to cache the multiple data write requests in Redis if the write request threshold is exceeded. The request concatenation module is used to export all data write requests stored in Redis in batches when preset conditions are met, and concatenate all exported data write requests to obtain bulk requests. The preset conditions include the time reaching a preset time and the number of data write requests reaching a preset number. The batch write module is used to write the data to be written corresponding to all exported data write requests to the disk using the bulk request; wherein, the bulk request is obtained by concatenating all exported data write requests, including: testing the bulk requests with different numbers of data write requests one by one, and when the ES write performance of the bulk request begins to level off, the concatenated bulk request is the optimal bulk request, and the number of requests is doubled after each batch of data write requests is written; The single-write module is used to obtain the first feedback information after the data to be written is written to the disk, and to determine whether the batch writing of the data to be written has been successfully executed based on the first feedback information. If the execution fails, the number of failures is recorded, and it is determined whether the number of failures exceeds the failure threshold. If the failure threshold is not exceeded, the bulk request is re-executed to write all the exported data to the disk corresponding to the write request. If the failure threshold is exceeded, all data to be written corresponding to the data to be written in the bulk request will be written to the disk in sequence. The single-line write module is also used for: The bulk request is split into multiple target data write requests; Using each target data write request, the data to be written corresponding to the target data write request is written to the disk to complete the writing of all data to be written. Obtain the second feedback information for each target data write request, and determine whether the data to be written corresponding to the target data write request has been successfully written to the disk based on the second feedback information; If the write to disk fails, the write request for the target data that failed to write to disk is placed in the write failure queue, and the failure information is recorded in the log.

7. An electronic device, characterized in that, include: At least one processor; Memory; At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, said at least one application being configured to: perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1 to 5.

Images