Task processing method and device, electronic equipment and storage medium

Abstract

The application relates to a task processing method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining a data writing task, and extracting a plurality of pieces of to-be-written data carried in the data writing task; performing storage format conversion on the plurality of pieces of to-be-written data according to the storage format of an ES cluster; storing the plurality of pieces of to-be-written data subjected to the storage format conversion into a temporary storage space; obtaining the plurality of pieces of to-be-written data subjected to the storage format conversion in the temporary storage space and synchronizing the plurality of pieces of to-be-written data to the ES cluster, and updating the synchronization state of the plurality of pieces of to-be-written data subjected to the storage format conversion. The to-be-written data is synchronized to the ES cluster through the temporary storage space, and the synchronization state of the to-be-written data is updated, so that the data is written into the ES cluster. If writing fails or is interrupted, the corresponding to-be-written data is synchronized to the ES cluster again according to the synchronization state of the to-be-written data, so that the data writing task is not executed again, resource waste is avoided, and time consumption is reduced.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a task processing method, apparatus, electronic device, and... Background Technology

[0002] Currently, the abnormal data system connects to related systems to collect abnormal logs and other abnormal data, writes them to an Elasticsearch (ES) cluster for indexing, and provides query interfaces for business users to retrieve abnormal data. As the number of systems connected to the abnormal data system increases, the data writing tasks to the ES cluster also surge, impacting the query performance of the abnormal data system.

[0003] In related technologies, to avoid impacting the query performance of abnormal data systems, there are two optimization methods: Method 1, read-write separation, i.e., separating read services and write services; Method 2, adding a message middleware as a buffer to control the write frequency. However, when dealing with situations where data write tasks carry a large amount of abnormal data, if write failures or interruptions occur, the data write task needs to be re-executed, leading to resource waste and increased time consumption. Summary of the Invention

[0004] To address the technical problem mentioned above, where write tasks carrying large amounts of abnormal data require re-execution of the data write task due to write failures or interruptions, resulting in resource waste and increased time consumption, this application provides a task processing method, apparatus, electronic device, and storage medium. The specific technical solution is as follows:

[0005] Firstly, this application provides a task processing method, the method comprising:

[0006] Obtain the data writing task and extract the multiple pieces of data to be written carried in the data writing task;

[0007] According to the storage format of the ES cluster, the storage format of multiple pieces of data to be written is converted.

[0008] The multiple records to be written, after being converted to the storage format, are stored in a temporary storage space;

[0009] The system retrieves multiple pieces of data to be written from the temporary storage space after storage format conversion, synchronizes them to the ES cluster, and updates the synchronization status of the multiple pieces of data to be written after storage format conversion.

[0010] In an optional implementation, after storing the multiple records to be written in the converted storage format to a temporary storage space, the method further includes:

[0011] Determine the first storage space occupied by the multiple data entries to be written after storage format conversion;

[0012] Determine the second storage space occupied by multiple historical data entries to be written after storage format conversion;

[0013] Multiple historical data entries to be written, after storage format conversion, are derived from multiple historical data entries to be written carried in the data writing history task after storage format conversion;

[0014] Based on the first storage space and the second storage space, determine the changes in storage space occupancy;

[0015] The temporary storage space is expanded or shrunk according to the changes in storage space occupancy.

[0016] In one optional implementation, the number of temporary storage spaces is multiple, and the synchronization period of each temporary storage space is different;

[0017] The step of storing multiple records of data to be written, after storage format conversion, into temporary storage space includes:

[0018] For any of the data to be written that has undergone storage format conversion, the following processing is performed:

[0019] Determine the data source corresponding to the data to be written after storage format conversion, and determine the data query metrics for the data source;

[0020] Based on the data query indicators, a target temporary storage space is determined among multiple temporary storage spaces, and the data to be written, after being converted to a storage format, is stored in the target temporary storage space.

[0021] The step of obtaining multiple records of data to be written from the temporary storage space after storage format conversion, synchronizing them to the Elasticsearch cluster, and updating the synchronization status of the multiple records of data to be written after storage format conversion includes:

[0022] For any of the temporary storage spaces, according to the synchronization cycle of the temporary storage space, the data to be written in the temporary storage space after storage format conversion is obtained and synchronized to the ES cluster, and the synchronization status of the data to be written after storage format conversion is updated.

[0023] In one optional implementation, each of the temporary storage spaces has its own corresponding range of data query indicators;

[0024] The step of determining the target temporary storage space among multiple temporary storage spaces based on the data query indicators includes:

[0025] Locate the data query indicator range to which the data query indicator belongs, and determine the temporary storage space corresponding to the query indicator range as the target temporary storage space.

[0026] In an optional implementation, the step of obtaining multiple records of data to be written from the temporary storage space after storage format conversion and synchronizing them to the ES cluster includes:

[0027] Obtain a data query task, and determine the synchronization period of multiple data entries to be written in the temporary storage space after storage format conversion based on the data query task;

[0028] According to the synchronization cycle, multiple records of data to be written, after being converted to the storage format, are obtained from the temporary storage space and synchronized to the ES cluster.

[0029] In an optional implementation, determining the synchronization period of multiple data entries to be written in the temporary storage space after storage format conversion, based on the data query task, includes:

[0030] The query times of the data query tasks are counted, and the query time range corresponding to the data query tasks is determined based on the query times;

[0031] If the current time is within the query time range, the preset first synchronization period is determined as the synchronization period of multiple pieces of data to be written in the temporary storage space after storage format conversion;

[0032] If the current time is not within the query time range, the preset second synchronization period will be determined as the synchronization period of the multiple pieces of data to be written in the temporary storage space after storage format conversion;

[0033] The preset first synchronization period is greater than the preset second synchronization period.

[0034] In an optional implementation, determining the synchronization period of multiple data entries to be written in the temporary storage space after storage format conversion, based on the data query task, includes:

[0035] The query frequency of the data query task within a preset time period is statistically analyzed, and the synchronization period of multiple pieces of data to be written in the temporary storage space after storage format conversion is determined based on the query frequency.

[0036] or,

[0037] The query frequency of the data query task within a preset time period is counted, and the synchronization period of multiple pieces of data to be written in the temporary storage space after storage format conversion is determined based on the query frequency.

[0038] Secondly, this application provides a task processing apparatus, the apparatus comprising:

[0039] The data extraction module is used to acquire data writing tasks and extract multiple pieces of data to be written carried in the data writing tasks.

[0040] The format conversion module is used to convert the storage format of multiple pieces of data to be written according to the storage format of the ES cluster;

[0041] The data storage module is used to store multiple records of data to be written, after conversion of the storage format, into a temporary storage space;

[0042] The data synchronization module is used to acquire multiple pieces of data to be written from the temporary storage space after storage format conversion, synchronize them to the ES cluster, and update the synchronization status of the multiple pieces of data to be written after storage format conversion.

[0043] Thirdly, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0044] Memory, used to store computer programs;

[0045] When a processor executes a program stored in memory, it implements any of the task processing methods described in the first aspect above.

[0046] Fourthly, a storage medium is also provided, wherein the storage medium stores instructions that, when executed on a computer, cause the computer to perform any of the task processing methods described in the first aspect above.

[0047] Fifthly, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the task processing methods described above.

[0048] Compared with the prior art, the above-mentioned technical solution provided in this application embodiment has the following advantages: The method provided in this application embodiment obtains a data writing task, extracts multiple pieces of data to be written carried in the data writing task, performs storage format conversion on the multiple pieces of data to be written according to the storage format of the ES cluster, stores the multiple pieces of data to be written after storage format conversion in a temporary storage space, obtains the multiple pieces of data to be written after storage format conversion in the temporary storage space, synchronizes them to the ES cluster, and updates the synchronization status of the multiple pieces of data to be written after storage format conversion. By extracting multiple data entries to be written from the data write task, converting them to the Elasticsearch cluster's storage format, and storing them in temporary storage space, the converted data entries are then retrieved from the temporary storage space and synchronized to the Elasticsearch cluster. The synchronization status of these converted data entries is also updated. This process of synchronizing data to the Elasticsearch cluster via temporary storage and updating their synchronization status completes the data write operation. If a write failure or interruption occurs, the corresponding data is re-synchronized to the Elasticsearch cluster based on its synchronization status, avoiding the need to re-execute the data write task, thus preventing resource waste and reducing time consumption. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0052] Figure 1 A schematic diagram illustrating the implementation flow of a task processing method provided in this application embodiment;

[0053] Figure 2 A schematic diagram illustrating the implementation process of a capacity expansion / shrinkage processing method provided in this application embodiment;

[0054] Figure 3 A schematic diagram illustrating the implementation flow of another task processing method provided in this application embodiment;

[0055] Figure 4 A schematic diagram illustrating the implementation flow of another task processing method provided in this application embodiment;

[0056] Figure 5 This is a schematic diagram of the structure of a task processing device provided in an embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0060] like Figure 1 The diagram shown illustrates the implementation flow of a task processing method provided in this application. This method is applied to an electronic device (e.g., a server) and may specifically include the following steps:

[0061] S101, obtain the data writing task and extract the multiple pieces of data to be written carried in the data writing task.

[0062] In this embodiment of the application, the abnormal data system will connect to the relevant system to collect abnormal data such as abnormal logs of the relevant system, write them into the ES cluster for indexing, and provide a query interface for business users to query abnormal data.

[0063] Based on this, for abnormal data systems, abnormal data such as abnormal logs from relevant systems are collected to generate data writing tasks. These data writing tasks carry multiple pieces of data to be written, which are the aforementioned abnormal logs and other abnormal data.

[0064] In this embodiment of the application, a data writing task can be obtained, and multiple pieces of data to be written carried in the data writing task can be extracted. These multiple pieces of data to be written carried in the data writing task ultimately need to be written into the ES cluster.

[0065] S102 performs storage format conversion on multiple data entries to be written, according to the storage format of the ES cluster.

[0066] In this embodiment, for multiple data entries to be written carried in a data writing task, the storage format of the multiple data entries to be written can be converted according to the storage format of the ES cluster. Specific storage format conversion can refer to existing processing procedures.

[0067] Therefore, before writing to the ES cluster, the storage format of the data to be written is converted in advance, so that it can be directly synchronized from the temporary storage space to the ES cluster to complete the data writing to the ES cluster.

[0068] S103 stores multiple records to be written in a temporary storage space after the storage format has been converted.

[0069] In this embodiment of the application, for multiple pieces of data to be written after storage format conversion, they can be stored in a temporary storage space first. The temporary storage space records the synchronization status of each piece of data to be written.

[0070] It should be noted that the advantage of temporary storage air conditioning is that it unifies the storage format of the data to be written and records the synchronization status of each piece of data to be written. If the writing fails or is interrupted, it is not necessary to re-execute the entire data writing task.

[0071] S104: Retrieve multiple pieces of data to be written from the temporary storage space after storage format conversion, synchronize them to the ES cluster, and update the synchronization status of the multiple pieces of data to be written after storage format conversion.

[0072] In this embodiment of the application, multiple pieces of data to be written that have undergone storage format conversion in the temporary storage space can be obtained and synchronized to the ES cluster. In addition, the synchronization status of the multiple pieces of data to be written that have undergone storage format conversion can be updated. The synchronization status of the multiple pieces of data to be written that have undergone storage format conversion is stored in the temporary storage space.

[0073] In particular, for temporary storage space, multiple pieces of data to be written that have undergone storage format conversion in the temporary storage space can be obtained and synchronized to the ES cluster according to a certain synchronization strategy (such as timed or rate-limited), and the synchronization status of the multiple pieces of data to be written that have undergone storage format conversion can be updated. This application embodiment does not limit this.

[0074] For example, for multiple pieces of data to be written in temporary storage space that have undergone storage format conversion, 10 pieces of data to be written that have undergone storage format conversion can be retrieved and synchronized to the ES cluster every minute, and the synchronization status of the 10 pieces of data to be written that have undergone storage format conversion can be updated. In this way, through timed and rate-limited strategies, the data to be written will eventually be written to the ES cluster.

[0075] It should be noted that for multiple pieces of data to be written in the temporary storage space after storage format conversion, synchronizing them to the ES cluster through strategies such as timed and rate-limited synchronization can ensure the accuracy of data writing to the ES cluster while maintaining the stability of the abnormal data system, and avoid affecting the query performance of the abnormal data system.

[0076] Based on the above description of the technical solution provided in the embodiments of this application, the data to be written is synchronized to the ES cluster through temporary storage space, and the synchronization status of the data to be written is updated to complete the data writing to the ES cluster. If a writing failure or writing interruption occurs, the corresponding data to be written is resynchronized to the ES cluster according to the synchronization status of the data to be written, avoiding the re-execution of the data writing task, avoiding the waste of resources, and reducing the time consumption.

[0077] Furthermore, in this embodiment, the size of the temporary storage space can be adjusted in real time according to the changes in the storage space occupancy, so as to make reasonable use of the temporary storage space and avoid the waste of storage space or the insufficiency of storage space.

[0078] Based on this, such as Figure 2 The diagram shown illustrates the implementation flow of a scaling up / down processing method provided in this application. This method is applied to electronic devices (e.g., servers) and specifically includes the following steps:

[0079] S201, determine the first storage space occupied by multiple pieces of data to be written after storage format conversion.

[0080] In this embodiment of the application, after storing multiple pieces of data to be written that have undergone storage format conversion into temporary storage space, the first storage space occupied by the multiple pieces of data to be written that have undergone storage format conversion can be determined.

[0081] S202, determine the second storage space occupied by multiple historical data entries to be written after storage format conversion.

[0082] In this embodiment of the application, the data writing history task is processed similarly to the data writing task described above. The same process is to extract multiple pieces of historical data to be written carried in the data writing history task, convert the storage format of the multiple pieces of historical data to be written according to the storage format of the ES cluster, and then store the multiple pieces of historical data to be written after storage format conversion in the temporary storage space. In this way, the second storage space occupied by the multiple pieces of historical data to be written after storage format conversion can be determined.

[0083] It should be noted that there can be multiple data writing history tasks, and thus multiple second storage spaces. This application embodiment does not limit this.

[0084] S203, determine the change in storage space occupancy based on the relationship between the first storage space and the second storage space.

[0085] S204, expand / shrink the temporary storage space according to the changes in storage space occupancy.

[0086] In this embodiment of the application, for the first storage space and the second storage space, the first storage space and the second storage space can be determined, and the change in the storage space occupancy can be determined. Then, based on the change in the storage space occupancy, the temporary storage space can be expanded / shrunken.

[0087] Specifically, for the first and second storage spaces, the spatial variance of the first and second storage spaces can be calculated to characterize the change in storage space occupancy. A smaller spatial variance indicates a smaller change in storage space occupancy, while a larger spatial variance indicates a larger change in storage space occupancy.

[0088] When the spatial variance is small, the change in the amount of temporary storage space occupied is small, and there is no need to expand or shrink the temporary storage space. When the spatial variance is large, the change in the amount of temporary storage space occupied is large, and there is a need to expand or shrink the temporary storage space.

[0089] Specifically, the size of the first storage space and the second storage space can be compared. If the first storage space is larger than the second storage space, the temporary storage space is expanded. If the first storage space is smaller than the second storage space, the temporary storage space is shrunk.

[0090] In this way, the size of the temporary storage space can be adjusted in real time according to the changes in storage space usage, so as to make reasonable use of the temporary storage space and avoid the waste of storage space or the insufficiency of storage space.

[0091] Furthermore, in this embodiment, multiple temporary storage spaces can be configured, each with a different synchronization period, to meet the needs of writing different data to the ES cluster. For example, for abnormal data in system A, which users frequently query and therefore wish to write as quickly as possible to the ES cluster, the abnormal data can be stored in a temporary storage space with a shorter synchronization period. Conversely, for abnormal data in system B, which users query less frequently, there is no need to rush to write the abnormal data to the ES cluster. Therefore, the abnormal data can be stored in a temporary storage space with a longer synchronization period, allowing for a slower but more gradual writing process.

[0092] Based on this, such as Figure 3 The diagram shown illustrates the implementation flow of another task processing method provided in this application. This method is applied to an electronic device (e.g., a server) and may specifically include the following steps:

[0093] S301, obtain the data writing task and extract the multiple pieces of data to be written carried in the data writing task.

[0094] In this embodiment of the application, this step is similar to step S101 above, and this embodiment of the application does not limit it.

[0095] S302 performs storage format conversion on multiple data entries to be written, according to the storage format of the ES cluster.

[0096] In this embodiment of the application, this step is similar to step S102 above, and this embodiment of the application does not limit it.

[0097] S303, for any data to be written that has undergone storage format conversion, the following processing is performed.

[0098] S304, determine the data source corresponding to the data to be written after storage format conversion, and determine the data query metrics for the data source.

[0099] S305: Based on the data query indicators, determine the target temporary storage space among multiple temporary storage spaces, and store the data to be written after storage format conversion to the target temporary storage space.

[0100] In this embodiment of the application, after converting the storage format of multiple pieces of data to be written according to the storage format of the ES cluster, the following processing is performed on any piece of data to be written after the storage format conversion:

[0101] Identify the data source corresponding to the data to be written after storage format conversion (e.g., the system that generates the data to be written), and determine the data query metrics for the data source. Based on the data query metrics, determine the target temporary storage space among multiple temporary storage spaces, and store the data to be written after storage format conversion in the target temporary storage space.

[0102] It should be noted that the data query metrics, such as data query frequency or data query count, are not limited in this application embodiment.

[0103] For example, taking the data to be written 1 after storage format conversion as an example, the system A corresponding to the data to be written 1 after storage format conversion is determined, which means that system A generates the data to be written 1. The data query frequency for system A is determined. Based on the data query frequency, the target temporary storage space among multiple temporary storage spaces is determined, and the data to be written 1 after storage format conversion is stored in the target temporary storage space.

[0104] Furthermore, in this embodiment, each temporary storage space is configured with its own corresponding data query indicator range, which is related to the synchronization period. For example, a shorter synchronization period for a temporary storage space indicates a faster synchronization frequency, allowing for a larger data query indicator range. Conversely, a longer synchronization period indicates a slower synchronization frequency, allowing for a smaller data query indicator range.

[0105] Based on this, in this embodiment of the application, for data query indicators targeting a data source, the range of data query indicators where the data query indicator is located can be found, and the temporary storage space corresponding to the query indicator range can be determined as the target temporary storage space.

[0106] For example, as shown in Table 1 below, there are multiple temporary storage spaces with different synchronization periods and different data query frequency ranges. For the data query frequency of system A (85 times per second), find the data query frequency range 1 where the data query frequency of system A is located, and determine the temporary storage space 1 corresponding to the data query frequency range 1 as the target temporary storage space.

[0107]

[0108]

[0109] Table 1

[0110] S306: For any temporary storage space, according to the synchronization cycle of the temporary storage space, obtain the data to be written in the temporary storage space after storage format conversion and synchronize it to the ES cluster, and update the synchronization status of the data to be written after storage format conversion.

[0111] In this embodiment of the application, after the above processing, multiple pieces of data to be written after storage format conversion can be stored in different temporary storage spaces. Some temporary storage spaces may have a large amount of data to be written after storage format conversion, while others may have a small amount, or even none at all.

[0112] Therefore, for any temporary storage space, according to the synchronization cycle of the temporary storage space, the data to be written in the temporary storage space after storage format conversion is retrieved and synchronized to the Elasticsearch cluster, and the synchronization status of the data to be written after storage format conversion is updated. In this way, the data to be written can be written to the Elasticsearch cluster on demand according to the query requirements of the data source corresponding to the data to be written, so as to meet the needs of writing different data to the Elasticsearch cluster.

[0113] In addition, after storing multiple records of data to be written in the storage format after conversion to temporary storage space, in order not to affect the query performance of the abnormal data system, the synchronization period of the multiple records of data to be written in the temporary storage space after conversion to temporary storage format can be adjusted according to the query requirements of the abnormal data system.

[0114] Based on this, such as Figure 4 The diagram shown illustrates the implementation flow of another task processing method provided in this application. This method is applied to an electronic device (e.g., a server) and may specifically include the following steps:

[0115] S401: Obtain the data writing task and extract the multiple pieces of data to be written carried in the data writing task.

[0116] In this embodiment of the application, this step is similar to step S101 above, and will not be described in detail here.

[0117] S402 performs storage format conversion on multiple data entries to be written, according to the storage format of the ES cluster.

[0118] In this embodiment of the application, this step is similar to step S102 above, and will not be described in detail here.

[0119] S403 stores multiple records to be written in a temporary storage space after the storage format has been converted.

[0120] In this embodiment of the application, this step is similar to step S103 above, and will not be described in detail here.

[0121] S404: Obtain the data query task. Based on the data query task, determine the synchronization period for multiple data entries to be written in the temporary storage space after storage format conversion.

[0122] S405, according to the synchronization cycle, retrieves multiple pieces of data to be written from the temporary storage space after storage format conversion and synchronizes them to the ES cluster.

[0123] S406 updates the synchronization status of multiple data entries to be written after storage format conversion.

[0124] In this embodiment of the application, after storing multiple pieces of data to be written that have undergone storage format conversion into temporary storage space, a data query task can be obtained. Based on the data query task, the synchronization period of the multiple pieces of data to be written that have undergone storage format conversion in the temporary storage space is determined. According to the synchronization period, the multiple pieces of data to be written that have undergone storage format conversion in the temporary storage space are synchronized to the ES cluster, and the synchronization status of the multiple pieces of data to be written that have undergone storage format conversion is updated.

[0125] Specifically, the query time of the statistical data query task is determined based on the query time, and the corresponding query time range is determined accordingly. If the current time is within the query time range, the preset first synchronization period is determined as the synchronization period of multiple data to be written in the temporary storage space after storage format conversion. If the current time is not within the query time range, the preset second synchronization period is determined as the synchronization period of multiple data to be written in the temporary storage space after storage format conversion. The preset first synchronization period is greater than the preset second synchronization period.

[0126] It should be noted that, based on the query time, the query time range corresponding to the data query task can be determined. The query times can be clustered to obtain several cluster categories. The query times in each cluster category can be considered to be continuous and can form a query time range. The query time ranges corresponding to several cluster categories can form the query time range corresponding to the data query task. This application embodiment does not limit this.

[0127] For example, for a statistical data query task, the query time range is determined based on the query time: 9:00–12:00 and 14:00–18:00. If the current time falls within this query time range, a preset first synchronization period (per hour) is set as the synchronization period for multiple data entries to be written in the temporary storage space after storage format conversion. If the current time does not fall within this query time range, a preset second synchronization period (per minute) is set as the synchronization period for the multiple data entries to be written in the temporary storage space after storage format conversion. This speeds up the synchronization of the multiple data entries to be written in the temporary storage space within the query time range, and slows down the synchronization otherwise.

[0128] In addition, the query frequency of data query tasks within a preset time period is counted, and the synchronization period of multiple pieces of data to be written in the temporary storage space after storage format conversion is determined based on the query frequency; or, the query frequency of data query tasks within a preset time period is counted, and the synchronization period of multiple pieces of data to be written in the temporary storage space after storage format conversion is determined based on the query frequency.

[0129] It should be noted that the synchronization period for multiple data entries to be written in the temporary storage space after storage format conversion is determined based on the query frequency. Specifically, if the query frequency is high, a longer synchronization period can be set as the synchronization period for the multiple data entries to be written in the temporary storage space after storage format conversion; otherwise, a shorter synchronization period can be set as the synchronization period for the multiple data entries to be written in the temporary storage space after storage format conversion. In this way, when queries are concentrated, the synchronization of the multiple data entries to be written in the temporary storage space after storage format conversion can be slowed down to avoid affecting the query performance of the abnormal data system. When queries are more dispersed, the synchronization of the multiple data entries to be written in the temporary storage space after storage format conversion can be accelerated, and the impact on the query performance of the abnormal data system can be disregarded.

[0130] In addition, based on the query frequency, the synchronization period for multiple data entries to be written in the temporary storage space after storage format conversion is determined. Specifically, if the query frequency is high, a larger synchronization period can be set as the synchronization period for the multiple data entries to be written in the temporary storage space after storage format conversion; otherwise, a smaller synchronization period can be set as the synchronization period for the multiple data entries to be written in the temporary storage space after storage format conversion. In this way, when queries are concentrated, the synchronization of the multiple data entries to be written in the temporary storage space after storage format conversion can be slowed down to avoid affecting the query performance of the abnormal data system. When queries are more dispersed, the synchronization of the multiple data entries to be written in the temporary storage space after storage format conversion can be accelerated, and the impact on the query performance of the abnormal data system can be disregarded.

[0131] In this way, after storing the multiple records to be written in the storage format after conversion to the storage format into the temporary storage space, the synchronization period of the multiple records to be written in the temporary storage space after conversion to the storage format can be adjusted according to the query requirements of the abnormal data system, so as to avoid affecting the query performance of the abnormal data system.

[0132] Corresponding to the above method embodiments, this application also provides a task processing device, such as... Figure 5 As shown, the device may include: a data extraction module 510, a format conversion module 520, a data storage module 530, and a data synchronization module 540.

[0133] The data extraction module 510 is used to acquire the data writing task and extract multiple pieces of data to be written carried in the data writing task;

[0134] The format conversion module 520 is used to convert the storage format of multiple pieces of data to be written according to the storage format of the ES cluster.

[0135] Data storage module 530 is used to store multiple records of data to be written, after storage format conversion, into temporary storage space;

[0136] The data synchronization module 540 is used to acquire multiple pieces of data to be written that have undergone storage format conversion in the temporary storage space, synchronize them to the ES cluster, and update the synchronization status of the multiple pieces of data to be written that have undergone storage format conversion.

[0137] This application also provides an electronic device, such as... Figure 6 As shown, it includes a processor 61, a communication interface 62, a memory 63, and a communication bus 64, wherein the processor 61, the communication interface 62, and the memory 63 communicate with each other through the communication bus 64.

[0138] Memory 63 is used to store computer programs;

[0139] When processor 61 executes the program stored in memory 63, it performs the following steps:

[0140] Obtain a data write task and extract multiple pieces of data to be written carried in the data write task; convert the multiple pieces of data to be written to the storage format according to the storage format of the ES cluster; store the multiple pieces of data to be written after storage format conversion to a temporary storage space; obtain the multiple pieces of data to be written after storage format conversion in the temporary storage space and synchronize them to the ES cluster, and update the synchronization status of the multiple pieces of data to be written after storage format conversion.

[0141] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0142] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0143] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0144] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0145] In another embodiment provided in this application, a storage medium is also provided, which stores instructions that, when run on a computer, cause the computer to execute any of the task processing methods described in the above embodiments.

[0146] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the task processing methods described in the above embodiments.

[0147] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0148] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0150] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A task processing method, characterized in that, The method includes: Obtain the data writing task and extract the multiple pieces of data to be written carried in the data writing task; According to the storage format of the ES cluster, the storage format of multiple pieces of data to be written is converted. Multiple records of data to be written, after storage format conversion, are stored in temporary storage space. The number of temporary storage spaces is multiple, and each temporary storage space has a different synchronization period. The process of storing multiple records of data to be written, after storage format conversion, in temporary storage space includes: for any record of data to be written after storage format conversion, performing the following processing: determining the data source corresponding to the data to be written after storage format conversion, and determining a data query indicator for the data source; based on the data query indicator, determining a target temporary storage space among the multiple temporary storage spaces, and storing the data to be written after storage format conversion in the target temporary storage space. Acquiring multiple pieces of data to be written that have undergone storage format conversion in the temporary storage space and synchronizing them to the ES cluster, and updating the synchronization status of the multiple pieces of data to be written that have undergone storage format conversion, includes: for any temporary storage space, according to the synchronization period of the temporary storage space, acquiring the data to be written that has undergone storage format conversion in the temporary storage space and synchronizing it to the ES cluster, and updating the synchronization status of the data to be written that has undergone storage format conversion.

2. The method according to claim 1, characterized in that, After storing the multiple records to be written, which have undergone storage format conversion, in a temporary storage space, the method further includes: Determine the first storage space occupied by the multiple data entries to be written after storage format conversion; Determine the second storage space occupied by multiple historical data entries to be written after storage format conversion; Multiple historical data entries to be written, after storage format conversion, are derived from multiple historical data entries to be written carried in the data writing history task after storage format conversion; Based on the first storage space and the second storage space, determine the change in storage space occupancy; wherein, calculate the spatial variance of the first storage space and the second storage space, and use the spatial variance to characterize the change in storage space occupancy; The temporary storage space is expanded or shrunk according to the changes in storage space occupancy.

3. The method according to claim 1, characterized in that, Each of the aforementioned temporary storage spaces has its own corresponding range of data query metrics; The step of determining the target temporary storage space among multiple temporary storage spaces based on the data query indicators includes: Locate the data query indicator range to which the data query indicator belongs, and determine the temporary storage space corresponding to the query indicator range as the target temporary storage space.

4. The method according to claim 1, characterized in that, The step of obtaining multiple records of data to be written from the temporary storage space after storage format conversion and synchronizing them to the Elasticsearch cluster includes: Obtain a data query task, and determine the synchronization period of multiple data entries to be written in the temporary storage space after storage format conversion based on the data query task; According to the synchronization cycle, multiple records of data to be written, after being converted to the storage format, are obtained from the temporary storage space and synchronized to the ES cluster.

5. The method according to claim 4, characterized in that, The step of determining the synchronization period of multiple records of data to be written in the temporary storage space after storage format conversion, based on the data query task, includes: The query times of the data query tasks are counted, and the query time range corresponding to the data query tasks is determined based on the query times; If the current time is within the query time range, the preset first synchronization period is determined as the synchronization period of multiple pieces of data to be written in the temporary storage space after storage format conversion; If the current time is not within the query time range, the preset second synchronization period will be determined as the synchronization period of the multiple pieces of data to be written in the temporary storage space after storage format conversion; The preset first synchronization period is greater than the preset second synchronization period.

6. The method according to claim 4, characterized in that, The step of determining the synchronization period of multiple records of data to be written in the temporary storage space after storage format conversion, based on the data query task, includes: The query frequency of the data query task within a preset time period is statistically analyzed, and the synchronization period of multiple pieces of data to be written in the temporary storage space after storage format conversion is determined based on the query frequency. or, The query frequency of the data query task within a preset time period is counted, and the synchronization period of multiple pieces of data to be written in the temporary storage space after storage format conversion is determined based on the query frequency.

7. A task processing device, characterized in that, The device includes: The data extraction module is used to acquire data writing tasks and extract multiple pieces of data to be written carried in the data writing tasks. The format conversion module is used to convert the storage format of multiple pieces of data to be written according to the storage format of the ES cluster; A data storage module is used to store multiple pieces of data to be written after storage format conversion to temporary storage space; the number of temporary storage spaces is multiple, and the synchronization period of each temporary storage space is different; the process of storing multiple pieces of data to be written after storage format conversion to temporary storage space includes: for any piece of data to be written after storage format conversion, performing the following processing: determining the data source corresponding to the data to be written after storage format conversion, and determining the data query index for the data source; determining the target temporary storage space among the multiple temporary storage spaces according to the data query index, and storing the data to be written after storage format conversion to the target temporary storage space; The data synchronization module is used to acquire multiple pieces of data to be written that have undergone storage format conversion in the temporary storage space, synchronize them to the ES cluster, and update the synchronization status of the multiple pieces of data to be written that have undergone storage format conversion. The module includes: for any temporary storage space, according to the synchronization period of the temporary storage space, acquiring the data to be written that has undergone storage format conversion in the temporary storage space, synchronizing it to the ES cluster, and updating the synchronization status of the data to be written that has undergone storage format conversion.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-6.

9. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images