Data asynchronous execution method, device, controller, medium and program product

Abstract

The application discloses a data asynchronous execution method and device, a controller, a medium and a program product. The application is applied to a cache component for SQL asynchronous execution. The cache component comprises a consumer process and a cache component unit. The cache component unit comprises a component process group and a cache file set. The component process group at least comprises a first component process node. The first component process node comprises a first thread and a second thread. The cache file set is stored in a preset storage area of a disk. The method comprises the following steps: the consumer process acquires SQL data and an IP address of the first component process node, parses the SQL data to obtain a data type of the SQL data, and sends the SQL data to the first thread according to the IP address; the first thread stores the SQL data into a component cache file according to a preset data storage strategy; and the second thread reads the SQL data from the component cache file and batch-executes SQL script data corresponding to the SQL data. Thus, the SQL statement can be executed better.

Description

Technical Field

[0001] This application relates to the field of big data technology, specifically to a method, apparatus, controller, medium, and program product for asynchronous data execution. Background Technology

[0002] With the advent of the big data era, how to process data, especially how to process Structured Query Language (SQL) data, has become a hot research topic.

[0003] Currently, when processing SQL data, such as caching it, a fixed-size cache is typically used. This means the rate at which SQL data is consumed must exceed the rate at which it is stored. If the consumption rate is lower than the storage rate, a cache overflow problem can occur. Furthermore, if SQL data processing fails, the data is deleted, leading to data loss. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, controller, medium, and program product for asynchronous data execution, so as to achieve better execution results for SQL statements.

[0005] The technical solution of this application is as follows:

[0006] Firstly, an asynchronous data execution method is provided. This method is applied to a caching component for asynchronous execution of a structured query language. The caching component includes a consumer process and a caching component unit. The caching component unit includes a component process group and a cache file set. The component process group includes at least a first component process node. The first component process node includes a first thread and a second thread. The cache file set is stored in a preset storage area on the disk.

[0007] The method includes:

[0008] The consumer process retrieves the structured query language data from the message queue and the network protocol address corresponding to the first component process node;

[0009] The consumption process parses the structured query language data to obtain the data type of the structured query language data;

[0010] The consumption process sends the structured query language data to the first thread of the first component process node corresponding to the data type, based on the network protocol address.

[0011] The first thread stores the structured query language data into the corresponding component cache file in the cache file set according to a preset data storage strategy;

[0012] The second thread, corresponding to the first thread, reads the structured query language data from the component cache file;

[0013] The second thread executes in batches the structured query language script data corresponding to the structured query language data in the component cache file.

[0014] Secondly, an asynchronous data execution device is provided, which is applied to a caching component for asynchronous execution of a structured query language. The caching component includes a consumer process and a caching component unit. The caching component unit includes a component process group and a cache file set. The component process group includes at least a first component process node. The first component process node includes a first thread and a second thread. The cache file set is stored in a preset storage area on the disk.

[0015] The device includes:

[0016] The first determining module is used for the consumer process to obtain the structured query language data in the message queue and the network protocol address corresponding to the first component process node;

[0017] The second determining module is used for the consumption process to parse the structured query language data to obtain the data type of the structured query language data;

[0018] The sending module is used by the consumer process to send the structured query language data to the first thread of the first component process node corresponding to the data type, according to the network protocol address.

[0019] The storage module is used by the first thread to store the structured query language data into the corresponding component cache file in the cache file set according to a preset data storage strategy;

[0020] The first execution module is used for the second thread corresponding to the first thread to read the structured query language data from the component cache file;

[0021] The second execution module is used by the second thread to batch execute the structured query language script data corresponding to the structured query language data in the component cache file.

[0022] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of any of the asynchronous data execution methods described in embodiments of this application.

[0023] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions, which, when executed by a processor, implement the steps of any of the asynchronous data execution methods described in embodiments of this application.

[0024] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform the steps of any of the asynchronous data execution methods described in embodiments of this application.

[0025] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0026] The asynchronous data execution method provided in this application can be applied to a caching component for asynchronous execution of Structured Query Language (SCL). This caching component includes a consumer process and a caching component unit. Each caching component unit includes a component process group and a cache file set. The component process group includes at least a first component process node, which includes a first thread and a second thread. The cache file set is stored in a preset storage area on the disk. Because the cache file set is stored in a preset storage area on the disk, rather than in a fixed-size cache, the disk size is adjustable, thus avoiding data overflow and data loss. This method is well-suited for scenarios with various message generation rates. The asynchronous data execution method obtains SCL data and the first component process from the message queue through the consumer process. The network protocol address corresponding to the node is used by the consumer process to parse the structured query language data, obtain the data type, and send it to the first thread of the first component process node corresponding to the data type. Then, the first thread stores the structured query language data in the corresponding component cache file in the cache file set according to the preset data storage strategy. The second thread corresponding to the first thread reads the structured query language data from the component cache file and executes the structured query language script data corresponding to the structured query language data in the component cache file in batches. In this way, multi-threading and the structured query language cache file set are combined to realize asynchronous execution of structured query language data.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0029] Figure 1 This is a diagram of the cache component architecture for implementing the asynchronous data execution method provided in the first aspect embodiment of this application;

[0030] Figure 2 This is a flowchart illustrating an asynchronous data execution method provided in the first aspect of this application.

[0031] Figure 3 This is a schematic diagram illustrating the flow of structured query language data according to the first aspect of the embodiment of this application;

[0032] Figure 4 This is a schematic diagram of the process by which the first thread stores structured query language data into the corresponding component cache file in the cache file set, according to the first aspect embodiment of this application;

[0033] Figure 5 This is a state diagram of a first-thread write operation according to the first aspect of this application;

[0034] Figure 6 This is a schematic diagram of the process of the second thread reading structured query language data from the component cache file according to the first aspect embodiment of this application;

[0035] Figure 7 This is a state diagram of the second thread read operation according to the first aspect of the embodiment of this application;

[0036] Figure 8 This is a flowchart illustrating the structured query language script data corresponding to the structured query language data in the cache file of the second thread batch execution component according to the first aspect of the embodiment of this application;

[0037] Figure 9 This is a schematic diagram of the structure of an asynchronous data execution device provided in the second aspect of this application;

[0038] Figure 10 This is a schematic diagram of the structure of an electronic device provided in the third aspect of this application. Detailed Implementation

[0039] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0040] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples consistent with some aspects of this application as detailed in the appended claims.

[0041] As described in the background section, existing technologies suffer from data overflow and data loss problems. To address these issues, this application provides a data asynchronous execution method, apparatus, controller, medium, and program product. The data asynchronous execution method provided in this application can be applied to a caching component for asynchronous execution of Structured Query Language (SCL). This caching component includes a consumer process and a caching component unit. The caching component unit includes a component process group and a cache file set. The component process group includes at least a first component process node, which includes a first thread and a second thread. The cache file set is stored in a preset storage area on the disk. Because the cache file set is stored in a preset storage area on the disk, rather than in a fixed-size cache, the disk size is adjustable, thus avoiding data overflow and data loss. This method is well-suited for scenarios with various message generation rates. The asynchronous execution method obtains the Structured Query Language (SCL) data and the network protocol address corresponding to the first component process node from the message queue through the consumer process. The consumer process parses the SCL data to obtain its data type. Based on the network protocol address, the consumer process sends the SCL data to the first thread of the first component process node corresponding to the data type. Then, the first thread stores the SCL data in the corresponding component cache file in the cache file set according to a preset data storage strategy. The second thread corresponding to the first thread then reads the SCL data from the component cache file and executes the corresponding SCL script data in the component cache file in batches. In this way, multi-threading and the SCL cache file set are combined to achieve asynchronous execution of the SCL data.

[0042] Before introducing the technical solution of this application, let's first introduce the system architecture for implementing the asynchronous data execution method of this application. Figure 1 This is a diagram of the cache component architecture that implements the asynchronous data execution method provided in this application, such as... Figure 1As shown, the caching component may include a consumer process and a caching component unit. The caching component unit may include a component process group and a cache file set. The component process group may include at least a first component process node. The first component process node may include a first thread and a second thread. The cache file set may be stored in a preset storage area on the disk.

[0043] The first component process node may include multiple thread groups, each of which includes a first thread, a second thread, and a set of cached files.

[0044] The cache file collection can include component cache files.

[0045] exist Figure 1 It can achieve four process stages: A1-A4.

[0046] A1 Process Phase (S110): This is the consumption phase of the structured query language data. Specifically, it can be the phase where the consumption process processes the structured query language data. How the consumption process specifically processes the structured query language data will be described in detail in later embodiments.

[0047] A2 process phase (S120): This is the phase where the North thread (i.e., the first thread) writes the structured query language data. Details will be provided in later embodiments.

[0048] A3 process phase (S130): This is the phase where the southern thread (i.e., the second thread) reads the structured query language data. Details will be provided in later embodiments.

[0049] A4 Process Phase (S140): This phase involves the southern thread processing the structured query language data. Details will be provided in later embodiments.

[0050] The asynchronous data execution method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0051] Figure 2 This is a flowchart illustrating an asynchronous data execution method provided in an embodiment of this application. The method can be applied to a caching component for asynchronous execution of a structured query language. The caching component may include a consumer process and a caching component unit. The caching component unit may include a component process group and a cache file set. The component process group may include at least a first component process node, which may include a first thread and a second thread. The cache file set may be stored in a preset storage area on a disk.

[0052] like Figure 2 As shown, the asynchronous data execution method provided in this application embodiment may include steps 210-260.

[0053] Step 210: The consumer process obtains the structured query language data in the message queue and the network protocol address corresponding to the first component process node.

[0054] Step 220: The consumption process parses the structured query language data to obtain the data type of the structured query language data.

[0055] Step 230: The consumer process sends the Structured Query Language data to the first thread of the first component process node corresponding to the data type, based on the network protocol address.

[0056] Step 240: The first thread stores the structured query language data into the corresponding component cache file in the cache file collection according to the preset data storage strategy.

[0057] Step 250: The second thread, corresponding to the first thread, reads the structured query language data from the component cache file.

[0058] Step 260: The second thread executes in batches the structured query language script data corresponding to the structured query language data in the component cache file.

[0059] The asynchronous data execution method provided in this application can be applied to a caching component for asynchronous execution of Structured Query Language (SCL). This caching component includes a consumer process and a caching component unit. Each caching component unit includes a component process group and a cache file set. The component process group includes at least a first component process node, which includes a first thread and a second thread. The cache file set is stored in a preset storage area on the disk. Because the cache file set is stored in a preset storage area on the disk, rather than in a fixed-size cache, the disk size is adjustable, thus avoiding data overflow and data loss. This method is well-suited for scenarios with various message generation rates. The asynchronous data execution method obtains SCL data and the first component process from the message queue through the consumer process. The network protocol address corresponding to the node is used by the consumer process to parse the structured query language data, obtain the data type, and send it to the first thread of the first component process node corresponding to the data type. Then, the first thread stores the structured query language data in the corresponding component cache file in the cache file set according to the preset data storage strategy. The second thread corresponding to the first thread reads the structured query language data from the component cache file and executes the structured query language script data corresponding to the structured query language data in the component cache file in batches. In this way, multi-threading and the structured query language cache file set are combined to realize asynchronous execution of structured query language data.

[0060] The asynchronous data execution method provided in the embodiments of this application will be described in detail below.

[0061] First, let's introduce step 210, where the consumer process obtains the structured query language data in the message queue and the network protocol address corresponding to the first component process node.

[0062] The consumer process can be a processing process node that obtains the structured query language data in the message queue and the network protocol address corresponding to the first component process node.

[0063] A message queue can be a queue for storing structured query language data, where multiple retrieved structured query language data are stored.

[0064] In some embodiments of this application, the message queue here may be a first-in-first-out (FIFO) queue.

[0065] The first component process node can be a component process node in a component process group, such as the main process node.

[0066] In some embodiments of this application, the caching component can provide an application programming interface (API) for external applications to call, and add SQL data that needs to be asynchronously stored in the database through the component to the message queue using the SQL data data structure.

[0067] Then the consumer process retrieves SQL data from the message queue and obtains the network protocol address corresponding to the first component process node.

[0068] It should be noted that the component process described in this application embodiment is managed by HA-ROSE, which has high availability. In the ROSE configuration, two component process nodes are enabled simultaneously (a first component process node and a second component process node, wherein the first component process node can be a master node and the second component process node can be a backup node). The two component process nodes each have their own real network protocol address. ROSE is configured with a virtual network protocol address, which points to the real network protocol address of the first component process node (i.e., the master node). The consumer process can access the component process through the virtual network protocol address and ultimately access the services provided by the component process of the current first component process node (i.e., the master node).

[0069] Then, step 220 is introduced, in which the consumption process parses the structured query language data to obtain the data type of the structured query language data.

[0070] In some embodiments of this application, the consumer end (i.e., the dequeue end) of the consumption process parses the acquired SQL data. Specifically, it parses the fields of the acquired SQL data to obtain the data type of the SQL data. This part can be implemented using a parser.

[0071] In one example, the SQL data contains a data type field, where different values ​​represent different data types. For instance, the value "1" indicates that the SQL data is of type A, and the value "2" indicates that the SQL data is of type B. By analyzing the data type field of the SQL data in this way, the data type of the SQL data can be determined.

[0072] In some embodiments of this application, the data type of SQL data may correspond one-to-one with the business type of the SQL data. For example, if a certain SQL data is modified, then the business type of the SQL data is "modification", and its corresponding data type may be type A. Specifically, it may include what the SQL data was modified to and which field of the SQL data was modified.

[0073] Next, step 230 is introduced, in which the consumer process sends the structured query language data to the first thread of the first component process node corresponding to the data type, according to the network protocol address.

[0074] The first thread can be a thread in the first component process node, such as the north thread.

[0075] In some embodiments of this application, the consuming process may send the acquired SQL data to the first thread of the first component process node corresponding to the data type of the SQL data, based on the network protocol address of the acquired first component process node.

[0076] It should be noted that the data type of SQL data can be defined by the user according to their needs, and there are no restrictions here. During the initialization of the caching component, a component process group will be created according to the configured set of data types, and each component process group corresponds one-to-one with a data type.

[0077] In one example, such as Figure 3 As shown, different SQL data have different data types, and different data types correspond to different thread groups of the first component process node. Therefore, according to the different data types, the SQL data can be sent to different thread groups of the first component process node 310 corresponding to the data type. Specifically, it can be sent to the first thread of the different thread group of the first component process node 310.

[0078] Steps 210-230 above are... Figure 1 The A1 process stage in the process.

[0079] Next, we will introduce step 240, in which the first thread stores the structured query language data into the corresponding component cache file in the cache file collection according to the preset data storage strategy.

[0080] The preset data storage strategy can be a pre-defined data storage strategy. The specifics of this storage strategy will be described later.

[0081] In some embodiments of this application, the cache file set may correspond one-to-one with an external application and be associated with a component process group. Each cache file set may contain a component cache file and a preset file (error file).

[0082] The component cache file contains a data table, the structure of which is shown in Table 1 below:

[0083] Table 1

[0084] field name Field type Can it be NULL? Field Description Rectime Int Can't Data creation timestamp Data Varchar(5000) Can't SQL script data Datatype Int Can't Business Type ID Lastopertime Int Can't Last execution timestamp

[0085] In some embodiments of this application, after the first thread receives the SQL data sent by the consumer process, it can store the SQL data in the corresponding component cache file in the cache file set according to a preset data storage strategy.

[0086] To ensure the correct persistence of SQL data, the following details how to store SQL data into the corresponding component cache file in the cache file set according to a preset storage strategy. Specifically, step 240 may include:

[0087] Create a component cache sub-file in the component cache file and add a first lock tag to the component cache sub-file;

[0088] Repeat the following steps until the structured query language data in the message queue has been traversed and the target component cache file is obtained:

[0089] In response to determining that the amount of structured query language data in the component cache subfile is less than a preset threshold, the first structured query language data is read from the message queue based on the first thread;

[0090] The first structured query language data is parsed to generate structured query language script data corresponding to the first structured query language data;

[0091] Insert the Structured Query Language script data into the component cache subfile.

[0092] The component cache subfile can be a subfile created from the component cache file.

[0093] The first lock flag can be a lock flag added to a component cache sub-file. This lock flag is used to mark the component cache sub-file as the current file pointed to by the first thread, and does not allow other threads to access the component cache sub-file.

[0094] The target component cache file can be the component cache file obtained after traversing the structured query language data in the message queue and writing the structured query language data in the message queue to the component cache file.

[0095] The preset threshold can be a pre-defined threshold for the amount of structured query language data in the component cache subfiles. This threshold can be set by the user according to their needs; there are no restrictions here.

[0096] The first set of Structured Query Language (SQL) data can be the SQL data at the head of the message queue, awaiting reading. Since the message queue is a FIFO queue, the SQL data at the head of the queue is preferred to be read.

[0097] In some embodiments of this application, after creating a component cache sub-file in the component cache file and adding a first lock flag to the component cache sub-file, the above-mentioned asynchronous data execution method may further include:

[0098] In response to determining that the amount of Structured Query Language data in the component cache subfile is greater than or equal to a preset threshold, return to execute the creation of the component cache subfile in the component cache file.

[0099] In some embodiments of this application, to better understand the workflow of the first thread (i.e., step 240), refer to... Figure 4 , Figure 4 This is a flowchart illustrating the process by which the first thread stores Structured Query Language data into the corresponding component cache file in the cache file collection according to a preset data storage strategy. The details are provided below. Figure 4 Corresponding flowchart:

[0100] Step 410: When the first thread starts, a new component cache sub-file is created under the specified component cache file. The name can be "current timestamp.sqlite". A read-write lock (i.e., the first lock mark) is added to the component cache sub-file.

[0101] Step 420: Determine whether the number of data entries in the current file (i.e., the component cache sub-file) is less than a preset threshold (e.g., 5000). If not, proceed to step 430; if yes, proceed to step 440.

[0102] Step 430: Create a new component cache sub-file, mark it as the current file, add a read-write lock to the file, and remove the lock object of the original current file.

[0103] Step 440: The first thread reads a piece of SQL data (i.e., the first Structured Query Language data) published by the consumer process from the data receiving port. If there is no SQL data waiting to be read at this time, it waits for 10ms and continues to receive data for the next time. If valid SQL data is received, it parses the SQL data and generates the corresponding SQL script data.

[0104] Step 450: Insert the SQL script data into the component cache subfile.

[0105] Step 460: After executing one SQL script data insertion, return to step 420 to execute the next processing cycle.

[0106] In some embodiments of this application, a component cache sub-file is created in the component cache file, and a first lock flag is added to the component cache sub-file. Then, in response to determining that the amount of structured query language data in the component cache sub-file is less than a preset threshold, the first structured query language data is read from the message queue based on the first thread, the first structured query language data is parsed, and structured query language script data corresponding to the first structured query language data is generated. The structured query language script data is inserted into the component cache sub-file. The above steps are executed repeatedly until the structured query language data in the message queue is traversed completely, and the target component cache file is obtained. In this way, the SQL data is cached in the component cache file on the disk, ensuring data persistence and avoiding data overflow.

[0107] In one example, such as Figure 5 As shown, the first thread writes the SQL data to the component cache sub-file: 1645767635015.sqlite, and adds a read-write lock to the component cache sub-file to prevent other threads from accessing it.

[0108] Steps 410-460 above are... Figure 1 The A2 process stage in the process.

[0109] Next, we introduce step 250, where the second thread, corresponding to the first thread, reads structured query language data from the component cache file.

[0110] The second thread can be another thread in the process node of the first component besides the first thread; for example, it can be the second thread. There is a one-to-one correspondence between the first thread and the second thread.

[0111] In some embodiments of this application, the component process group is the core process of the asynchronous execution cache component of the Structured Query Language. A component process group may include multiple component process nodes, and each component process node includes a first thread and a second thread.

[0112] In some embodiments of this application, after the first thread has completed traversing the structured query language data in the message queue and written it to the component cache file, the second thread corresponding to the first thread can read the SQL data from the target component cache file.

[0113] In some embodiments of this application, in order to improve the efficiency of SQL data reading, step 250 may specifically include:

[0114] Based on the second thread corresponding to the first thread, scan each target component cache sub-file in the target component cache file and generate each memory image file corresponding to each target component cache sub-file;

[0115] Repeat the following steps until all memory image files have been traversed:

[0116] A second lock flag is added to the first memory image file. In response to determining that the target component cache file has a target component cache sub-file corresponding to the first memory image file, and the first memory image file is not the last file among all memory image files, and the generation timestamp of the first memory image file is less than the first preset timestamp, and the memory of the first memory image file is not 0, the target cache sub-file corresponding to the first memory image file is read and the second lock flag is removed.

[0117] For each target component cache sub-file, the target component cache sub-file can be the final component cache sub-file generated after the first thread inserts the SQL data into the component cache sub-file.

[0118] A memory image file can be an image file corresponding to each target component cache sub-file generated by the second thread after scanning each target component cache sub-file in the target component cache file.

[0119] In some embodiments of this application, each memory image file may be named according to its generation timestamp. The memory image files may be sorted from front to back according to their generation timestamps.

[0120] The first memory image file can be any one of the memory image files.

[0121] The second lock flag can be a lock flag added to the first memory image file. This lock flag is used to mark the first memory image file as the current file pointed to by the second thread, and other threads are not allowed to access the first memory image file.

[0122] The first preset timestamp can be a threshold value for the generation timestamp of the first memory image file. This threshold can be selected by the user and is not limited here.

[0123] In some embodiments of this application, the asynchronous data execution method described above may further include:

[0124] In response to determining that the target component cache file does not contain a target component cache sub-file corresponding to the first memory image file, the first memory image file is deleted. Then, it is determined whether the target component cache file contains a target component cache sub-file corresponding to the second memory image file. In response to determining that the target component cache file contains a target component cache sub-file corresponding to the second memory image file, the second memory image file is updated to the first memory image file, and the process returns to add a second lock flag to the first memory image file. In response to determining that the target component cache file contains a target component cache sub-file corresponding to the first memory image file, and the first memory image file is not the last file among all memory image files, and the generation timestamp of the first memory image file is less than a first preset timestamp, and the memory of the first memory image file is not 0, the first memory image file is read, and the second lock flag is removed.

[0125] The second memory image file can be the memory image file located at the next position after the first memory image file among all memory image files.

[0126] In some embodiments of this application, the asynchronous data execution method described above may further include:

[0127] In response to determining that the first memory image file is the last file among all memory image files, the execution returns to the second thread corresponding to the first thread to scan each target component cache sub-file in the target component cache file and generate each memory image file corresponding to each target component cache sub-file.

[0128] In some embodiments of this application, the asynchronous data execution method described above may further include:

[0129] In response to determining that the generation timestamp of the first memory image file is greater than or equal to a first preset timestamp, or that the memory of the first memory image file is 0, delete the target component cache sub-file corresponding to the first memory image file in the target component cache file, as well as the first memory image file.

[0130] In some embodiments of this application, to more clearly understand the workflow of the second thread reading structured query language data from the component cache file (i.e., step 250), please refer to... Figure 6 The flowchart shown below will be explained in detail. Figure 6 The flowchart shown below:

[0131] Step 601: Start the second thread. The second thread scans each target component cache sub-file in the target component cache file and generates each memory image file corresponding to each target component cache sub-file.

[0132] In some embodiments of this application, for each memory image file, the memory image file contains the key (i.e., the timestamp in the target component cache subfile name) and the address of the target component cache subfile corresponding to the memory image file. The memory image files are sorted in ascending order according to the size of the key.

[0133] Step 602: Process the target component cache sub-files in a loop. According to the generation timestamp order of each memory image file, retrieve the next memory image file and proceed to step 603.

[0134] Step 603: Locate the target component cache sub-file corresponding to the first memory image on the disk and determine whether a real target component cache sub-file exists. If it does not exist, proceed to step 604; if it exists, proceed to step 605.

[0135] Step 604: Delete the first memory image file and return to step 602.

[0136] Step 605: Determine whether the first memory image file is the last file among all memory image files. If yes, proceed to step 606; otherwise, proceed to step 607.

[0137] Step 606: Wait for a preset time period (e.g., 10 seconds), then return to step 601.

[0138] Step 607: Determine whether the generation timestamp of the first memory image file is less than the first preset timestamp. If yes, proceed to step 608; otherwise, proceed to step 609.

[0139] In some embodiments of this application, it is determined whether the generation timestamp of the first memory image file is less than a first preset timestamp, that is, whether the first memory image file has expired.

[0140] Step 608: Delete the target component cache sub-file corresponding to the first memory image file in the target component cache file, as well as the first memory image file, and return to execute step 602.

[0141] Step 609: Determine if the memory of the first memory image file is 0. If yes, proceed to step 608; otherwise, proceed to step 610.

[0142] Step 610: Read the target cache sub-file corresponding to the first memory image file and remove the second lock flag.

[0143] In the embodiments of this application, structured query language data is read from the component cache file using a second thread. Since the entire target component cache sub-file can be read during the structured query language data reading process, the large amount of data read improves data reading efficiency. Furthermore, expired target component cache sub-files are promptly cleaned up, relieving disk pressure. Additionally, a lock is added to the currently processed file, effectively preventing conflicts between the read / write operations of the first and second threads.

[0144] In one example, such as Figure 7 As shown, when the second thread reads SQL data, it first scans each target component cache sub-file in the target component cache file, generates each memory image file corresponding to each target component cache sub-file, and then reads the target component cache sub-files (164550845898.sqlite and 1645767635015.sqlite). The two target component cache sub-files 164550845898.sqlite and 1645767635015.sqlite are in a locked state, and other threads are not allowed to access these two target component cache sub-files.

[0145] Steps 601-610 above are... Figure 1 The A3 process stage in the process.

[0146] Finally, step 260 is introduced, where the second thread executes in batches the structured query language script data corresponding to the structured query language data in the component cache file.

[0147] In some embodiments of this application, after reading the target cache sub-file corresponding to the first memory image file in step 610 above, the second thread performs batch processing on the SQL data in the target cache sub-file corresponding to the first memory image file.

[0148] In some embodiments of this application, step 260 may specifically include:

[0149] For each target component cache subfile, obtain the number of structured query language script data in the target component cache subfile;

[0150] Based on quantity, the structured query language script data in the target component cache sub-files are allocated to different processing pages;

[0151] In response to the determination that the last execution timestamp of multiple structured query language script data in the target processing page is greater than a second preset timestamp, the multiple structured query language script data in the target processing page are executed in batches.

[0152] Each processing page can contain multiple structured query language scripts.

[0153] The target processing page can be any one of multiple processing pages.

[0154] The last execution timestamp can be the time information when the SQL data was last executed.

[0155] The second preset timestamp can be a pre-set threshold for the last execution timestamp of the SQL data. This threshold can be set according to user needs and is not limited here.

[0156] In some embodiments of this application, the asynchronous data execution method described above may further include:

[0157] In response to the successful execution of multiple structured query language script data in the target processing page, the multiple structured query language script data in the target processing page are submitted to the structured query language database so that the database can process the target structured query language script data;

[0158] In response to the determination that multiple structured query language script data in the target processing page have failed to execute, the multiple failed structured query language script data are moved to a preset file in the cache file set, and the last execution timestamp of the target structured query language script data is updated to the current timestamp. If it is determined that the updated last execution timestamp is greater than or equal to a second preset timestamp, the target structured query language script data is re-executed.

[0159] The target structured query language script data can be at least one of multiple structured query language script data that failed to execute.

[0160] The default file can be an error file, which is used to store SQL data that failed to execute and cached subfiles of corrupted target components.

[0161] In some embodiments of this application, the asynchronous data execution method described above may further include:

[0162] In response to determining the number of structured query language script data that were not retrieved from the target component cache subfile, the target component cache subfile is moved to a preset file in the cache file set, and the memory image file corresponding to the target component cache subfile is deleted.

[0163] In some embodiments of this application, the asynchronous data execution method described above may further include:

[0164] In response to the determination that the number of Structured Query Language statements in the target component cache subfile is 0 and the memory image file corresponding to the target component cache subfile is the last one in the memory image files, the target component cache subfile is deleted from the disk, and the memory image file corresponding to the target component cache subfile is also deleted.

[0165] In some embodiments of this application, to more clearly understand the workflow (i.e., step 260) of the structured query language script data corresponding to the structured query language data in the cache file of the second thread batch execution component, please refer to... Figure 8 The flowchart shown below will be explained in detail. Figure 8 The flowchart shown below:

[0166] Step 801: For each target component cache sub-file, determine whether the SQL data has expired based on the creation timestamp field of the SQL data in the target component cache sub-file. If it has expired, delete the expired SQL data in the target component cache sub-file in batches.

[0167] Step 802: Obtain the number of structured query language script data in the target component's cache subfile. If obtained, proceed to step 803; otherwise, proceed to step 804.

[0168] In some embodiments of this application, if the number of structured query language script data in the target component cache sub-file is not obtained, it indicates that the query operation has encountered an error, that is, the target component cache sub-file has been corrupted during processing. In this case, the target component cache sub-file is moved to a preset file in the cache file set.

[0169] Step 803: Determine whether the number of SQL data in the target component cache subfile is 0. If it is 0, proceed to step 805; otherwise, proceed to step 807.

[0170] Step 804: Move the target component cache sub-file to the preset file in the cache file set, and then execute step 806.

[0171] Step 805: Determine whether the memory image file corresponding to the target component cache sub-file is the last one among all memory image files. If yes, proceed to step 806; otherwise, proceed to step 807.

[0172] Step 806: Delete the target component cache sub-file from the disk and delete the memory image file corresponding to the target component cache sub-file.

[0173] Step 807: Based on the number of structured query language script data in the target component cache sub-file, allocate each structured query language script data in the target component cache sub-file to different processing pages.

[0174] In some embodiments of this application, each processing page may contain a preset number of SQL data entries, such as 50 SQL data entries per processing page. This results in a relatively small amount of SQL data per processing page, saving execution time and improving efficiency.

[0175] Step 808: Retrieve all SQL data for the next page in pagination order.

[0176] In some embodiments of this application, when obtaining SQL data of the target processing page, it can be determined whether the last execution timestamp of multiple structured query language script data in the target processing page is greater than a second preset timestamp. If it is not greater, each SQL data in the target processing page is discarded (that is, it is determined whether the last execution time of the SQL data is more than a certain time away from the current time. If it is not greater, no processing is performed). If it is greater, step 809 is executed.

[0177] Step 809: Batch execute multiple Structured Query Language scripts on the target processing page.

[0178] Step 810: Determine whether the execution of multiple SQL data in the target processing page was successful. If successful, proceed to step 811; if unsuccessful, proceed to step 812.

[0179] Step 811: Submit the successfully executed SQL data to the Structured Query Language database so that the database can process the target Structured Query Language script data.

[0180] Step 812: Move the failed SQL data to a preset file in the cache file set, update the last execution timestamp of the failed SQL data to the current timestamp, and if it is determined that the updated last execution timestamp is greater than or equal to the second preset timestamp, re-execute the target structured query language script data.

[0181] Step 813: Determine if there is a next page. If there is, return to step 808; otherwise, proceed to step 814.

[0182] Step 814: Batch delete the SQL data that has been successfully executed.

[0183] In the embodiments of this application, the structured query language script data corresponding to the structured query language data in the component cache file is executed in batches by a second thread. This moves failed SQL data and corrupted target component cache sub-files to a preset file. This allows failed SQL data to be re-executed after a certain period, ensuring repeated execution within the valid period and preventing data loss due to processing anomalies. Ultimately, the data will be executed successfully or saved, resulting in higher data and business reliability. Corrupted target component cache sub-files are archived for later manual investigation of the cause of corruption. Furthermore, successfully executed SQL data is ensured to be executed quickly in the SQL database, improving SQL execution efficiency. During SQL data execution, all SQL data on the target page is processed in batches, further enhancing SQL data processing efficiency.

[0184] Steps 801-814 above constitute the above. Figure 1 The A4 process stage.

[0185] In some embodiments of this application, in order to improve the high availability of the caching component, the component process group may also include a second component process node.

[0186] The second component process node can be a backup node of the first component process node.

[0187] The asynchronous data execution methods mentioned above may also include:

[0188] Obtain the runtime data, network data, and transmission control protocol port data of the first component process node;

[0189] In response to the determination of an anomaly in at least one of the runtime data, network data, and transmission control protocol port data, the first component process node is updated to the second component process node.

[0190] In some embodiments of this application, during the execution of steps 210-260 above, fault self-checking and deep detection of the ROSE process can be realized. In the event of a failure of the first component process node (master node), the process is switched to the second component process node (i.e., backup node), thereby achieving high availability of the cache component.

[0191] In some embodiments of this application, fault self-testing may include two tests: general detection and deep detection, which may be as follows:

[0192] (1) Routine testing

[0193] The ROSE monitoring process periodically checks whether the process nodes of the cache component are running normally (specifically, it can obtain the running data of the first component process node). If it finds that the ROSE master node process (i.e. the first component process node) is closed, it triggers the ROSE switching logic to switch to the second component process node.

[0194] The ROSE monitoring process periodically sends heartbeat data and obtains network data from the first component process node, i.e., it checks whether the ROSE master node server network is normal. If the server is found to be inaccessible, the ROSE switching logic is triggered to switch to the second component process node.

[0195] (2) Depth Detection

[0196] Within the component process, there is a built-in fault detection thread and a Transmission Control Protocol (TCP) port that provides access to the current state. The ROSE process periodically makes TCP requests to this port to access the current state. If the state is faulty, the ROSE failover logic is triggered.

[0197] The built-in fault detection logic is as follows:

[0198] (a) Detect the thread status of all component process groups in the component process. If one or more threads are in an abnormal state, the process is considered to have encountered an error, and the current state is updated to fault.

[0199] (b) Check if the data receiving port of the first thread in the component process group is normal. If the configured port is found to be closed, the process is considered to have an error and the current status is updated to fault.

[0200] The specific ROSE switching logic is as follows: automatically switch the ROSE master node to the backup node, simultaneously enabling read / write permissions for the backup node's component cache files and disabling write permissions for the master node's component cache files. After a successful switch, restart the component process of the original ROSE master node.

[0201] In the embodiments of this application, by acquiring the running data, network data, and transmission control protocol port data of the first component process node; in response to determining that at least one of the running data, network data, and transmission control protocol port data is abnormal, the first component process node is updated to the second component process node. In this way, when the first component process node is abnormal, the running failure of the cache can be detected in time and replaced by the second component process node, so as to achieve uninterrupted business processing, continuous business processing, and improve the high availability of the cache component.

[0202] It should be noted that the ROSE software is used as the supporting application for the dual-machine hot standby part of the caching component in this application embodiment. However, the implementation method for high availability of the database SQL asynchronous execution caching component described in this application embodiment is not limited to use with ROSE software. All dual-machine hot standby software that has master-slave switching (i.e., switching between master node and standby node) and can be deeply interactive through shell scripts can replace the ROSE software dual-machine hot standby part in this application embodiment.

[0203] In some embodiments of this application, many software systems currently based on asynchronous execution of database SQL script messages encounter problems such as memory overflow, high scalability, data loss due to faults, and single-node service anomalies, leading to the unavailability of the entire caching component when faced with massive database SQL execution requests. The crashes of the message component or data loss caused by excessively frequent database SQL execution requests can result in serious bugs that render some functions or the entire system unusable in production environments, causing irreparable economic losses.

[0204] The database SQL asynchronous execution high-availability caching component provided in this application embodiment can serve as a high-availability solution for massive database SQL execution in a software system. Using the caching component provided in this embodiment can achieve the following beneficial effects:

[0205] (1) By using the caching component provided in this application embodiment to execute the asynchronous data execution method, the asynchronous execution of database SQL scripts can be realized, which can cope with various complex message production / consumption frequency scenarios. This improves the overall horizontal scalability of the software system.

[0206] (2) Using the caching component provided in this application to execute the asynchronous data execution method can realize the asynchronous execution of database SQL scripts, and can realize the isolation of data of various businesses. During the execution of massive database SQL, various business data do not interfere with each other. This improves the overall security of the software system.

[0207] (3) Using the caching component provided in this application embodiment to execute the asynchronous data execution method can realize the asynchronous execution of database SQL scripts. Combined with ROSE (or other HA software) and deep fault detection scripts, it can realize fault self-detection of the SQL script execution component and primary / backup service switching. This improves the overall high availability of the software system.

[0208] (4) The newly developed centralized monitoring system for new energy power stations (i.e., fault self-checking system) is based on the database SQL asynchronous execution high availability cache component described in the embodiments of this application. It enables the software system to stably and quickly process massive real-time alarm data in the scenario of accessing 50GW and 100,000 new energy devices, and provides a safe and reliable database SQL execution guarantee for the entire centralized monitoring system.

[0209] It should be noted that the data asynchronous execution method provided in this application embodiment can be executed by a data asynchronous execution device or a control module in the data asynchronous execution device for executing the data asynchronous execution method.

[0210] Based on the same inventive concept as the above-described asynchronous data execution method, this application also provides an asynchronous data execution apparatus. The following is in conjunction with... Figure 9 The asynchronous data execution apparatus provided in the embodiments of this application will be described in detail.

[0211] Figure 9 This is a schematic diagram illustrating the structure of an asynchronous data execution device according to an exemplary embodiment.

[0212] like Figure 9 As shown, the asynchronous data execution device 900 can be applied to a caching component for asynchronous execution of Structured Query Language. The caching component includes a consumer process and a caching component unit. The caching component unit includes a component process group and a cache file set. The component process group includes at least a first component process node. The first component process node includes a first thread and a second thread. The cache file set is stored in a preset storage area on the disk.

[0213] The asynchronous data execution device 900 may include:

[0214] The first determining module 910 is used for the consumer process to obtain the structured query language data in the message queue and the network protocol address corresponding to the first component process node;

[0215] The second determining module 920 is used for the consumption process to parse the structured query language data to obtain the data type of the structured query language data;

[0216] The sending module 930 is used by the consumer process to send the structured query language data to the first thread of the first component process node corresponding to the data type, according to the network protocol address.

[0217] Storage module 940 is used by the first thread to store the structured query language data into the corresponding component cache file in the cache file set according to a preset data storage strategy;

[0218] The first execution module 950 is used for a second thread corresponding to the first thread to read the structured query language data from the component cache file;

[0219] The second execution module 960 is used by the second thread to batch execute the structured query language script data corresponding to the structured query language data in the component cache file.

[0220] In the embodiments of this application, the asynchronous data execution device can be applied to a caching component for asynchronous execution of Structured Query Language (SCL). This caching component includes a consumer process and a caching component unit. The caching component unit includes a component process group and a cache file set. The component process group includes at least a first component process node, which includes a first thread and a second thread. The cache file set is stored in a preset storage area on the disk. Because the cache file set is stored in the preset storage area on the disk, rather than in a fixed-size cache, the disk size is adjustable, thus avoiding data overflow and data loss. This makes it well-suited for scenarios with various message generation rates. Based on the first determining module obtaining the SCL data in the message queue and the network protocol address corresponding to the first component process node through the consumer process, and based on the second determining module... The consumption process parses the structured query language data to obtain its data type. Based on the sending module, the consumption process sends the structured query language data to the first thread of the first component process node corresponding to the data type according to the network protocol address. Then, based on the storage module, the first thread stores the structured query language data into the corresponding component cache file in the cache file set according to the preset data storage strategy. Then, based on the first execution module, the second thread corresponding to the first thread reads the structured query language data from the component cache file. Based on the second execution module, the structured query language script data corresponding to the structured query language data in the component cache file is executed in batches. In this way, multi-threading and the structured query language cache file set are combined to realize asynchronous execution of structured query language data.

[0221] In some embodiments of this application, the storage module 940 may specifically be used for:

[0222] Create a component cache sub-file in the component cache file, and add a first lock tag to the component cache sub-file;

[0223] The following steps are executed repeatedly until the structured query language data in the message queue has been traversed completely, resulting in the target component cache file:

[0224] In response to determining that the number of structured query language data in the component cache subfile is less than a preset threshold, the first thread reads the first structured query language data from the message queue; wherein, the first structured query language data is the structured query language data to be read at the head of the message queue.

[0225] The first structured query language data is parsed to generate structured query language script data corresponding to the first structured query language data;

[0226] The structured query language script data is inserted into the component cache subfile.

[0227] In some embodiments of this application, the storage module 940 may also be specifically used for:

[0228] In response to determining that the amount of structured query language data in the component cache subfile is greater than or equal to the preset threshold, the process returns to create the component cache subfile in the component cache file.

[0229] In some embodiments of this application, the first execution module 950 may specifically be used for:

[0230] Based on the second thread corresponding to the first thread, each target component cache sub-file in the target component cache file is scanned, and each memory image file corresponding to each target component cache sub-file is generated; wherein, each memory image file is named according to the generation timestamp of each memory image file; each memory image file is sorted from front to back according to the generation timestamp;

[0231] Repeat the following steps until all memory image files have been traversed:

[0232] A second lock flag is added to the first memory image file. In response to determining that the target component cache file has a target component cache sub-file corresponding to the first memory image file, and that the first memory image file is not the last file among the memory image files, and that the generation timestamp of the first memory image file is less than a first preset timestamp, and that the memory of the first memory image file is not 0, the target cache sub-file corresponding to the first memory image file is read, and the second lock flag is removed; wherein, the first memory image file is any one of the memory image files.

[0233] In some embodiments of this application, the first execution module 950 may further be used for:

[0234] In response to determining that the target component cache file does not have a target component cache sub-file corresponding to the first memory image file, the first memory image file is deleted, and it is determined whether the target component cache file has a target component cache sub-file corresponding to the second memory image file. In response to determining that the target component cache file has a target component cache sub-file corresponding to the second memory image file, the second memory image file is updated to the first memory image file, and the process returns to add a second lock flag to the first memory image file. In response to determining that the target component cache file has a target component cache sub-file corresponding to the first memory image file, and the first memory image file is not the last file among all the memory image files, and the generation timestamp of the first memory image file is less than a first preset timestamp, and the memory of the first memory image file is not 0, the first memory image file is read, and the second lock flag is removed; wherein, the second memory image file is the next memory image file after the first memory image file.

[0235] In some embodiments of this application, the first execution module 950 may further be used for:

[0236] In response to determining that the first memory image file is the last file among all the memory image files, the execution returns to the second thread corresponding to the first thread to scan each target component cache sub-file in the target component cache file and generate each memory image file corresponding to each target component cache sub-file.

[0237] In some embodiments of this application, the first execution module 950 may further be used for:

[0238] In response to determining that the generation timestamp of the first memory image file is greater than or equal to the first preset timestamp, or that the memory of the first memory image file is 0, the target component cache sub-file corresponding to the first memory image file in the target component cache file, as well as the first memory image file, are deleted.

[0239] In some embodiments of this application, the second execution module 960 may specifically be used for:

[0240] For each target component cache sub-file, obtain the number of structured query language script data in the target component cache sub-file;

[0241] Based on the stated quantity, the structured query language script data in the target component cache sub-file are allocated to different processing pages; wherein each processing page contains multiple structured query language script data.

[0242] In response to determining that the last execution timestamp of multiple structured query language script data in the target processing page is greater than a second preset timestamp, the multiple structured query language script data in the target processing page are executed in batches; wherein, the target processing page is any one of the multiple processing pages.

[0243] In some embodiments of this application, the second execution module 960 may further be used for:

[0244] In response to determining that multiple structured query language script data in the target processing page have been successfully executed, the multiple structured query language script data in the target processing page are submitted to the structured query language database so that the database can process the target structured query language script data;

[0245] In response to determining that multiple structured query language script data in the target processing page have failed to execute, the multiple failed structured query language script data are moved to a preset file in the cache file set, and the last execution timestamp of the target structured query language script data is updated to the current timestamp. If it is determined that the updated last execution timestamp is greater than or equal to the second preset timestamp, the target structured query language script data is re-executed; wherein, the target structured query language script data is at least one of the multiple failed structured query language script data.

[0246] In some embodiments of this application, the second execution module 960 may further be used for:

[0247] In response to determining the number of structured query language script data that were not obtained in the target component cache sub-file, the target component cache sub-file is moved to a preset file in the cache file set, and the memory image file corresponding to the target component cache sub-file is deleted.

[0248] In some embodiments of this application, the second execution module 960 may further be used for:

[0249] In response to determining that the number of Structured Query Language statements in the target component cache subfile is 0, and that the memory image file corresponding to the target component cache subfile is the last one in the memory image files, the target component cache subfile is deleted from the disk, and the memory image file corresponding to the target component cache subfile is also deleted.

[0250] In some embodiments of this application, the component process group further includes a second component process node; the aforementioned asynchronous data execution device may also include:

[0251] The acquisition module is used to acquire the running data, network data, and transmission control protocol port data of the first component process node;

[0252] An update module is configured to update the first component process node to the second component process node in response to determining that at least one of the running data, the network data, and the transmission control protocol port data is abnormal.

[0253] The asynchronous data execution apparatus provided in this application embodiment can be used to execute the asynchronous data execution methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and for the sake of brevity, it will not be described in detail here.

[0254] Based on the same inventive concept, embodiments of this application also provide an electronic device.

[0255] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device may include a processor 1001 and a memory 1002 storing computer programs or instructions.

[0256] Specifically, the processor 1001 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0257] Memory 1002 may include mass storage for data or instructions. For example, and not limitingly, memory 1002 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1002 may include removable or non-removable (or fixed) media. Where appropriate, memory 1002 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 1002 is non-volatile solid-state memory. Memory may include read-only memory (ROM), random-access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, a memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described in the asynchronous data execution method provided in the above embodiments.

[0258] The processor 1001 implements any of the asynchronous data execution methods described in the above embodiments by reading and executing computer program instructions stored in the memory 1002.

[0259] In one example, the electronic device may also include a communication interface 1003 and a bus 1010. For example, Figure 10 As shown, the processor 1001, memory 1002, and communication interface 1003 are connected through bus 1010 and complete communication with each other.

[0260] The communication interface 1003 is mainly used to realize communication between various modules, devices, units and / or devices in the embodiments of the present invention.

[0261] Bus 1010 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1010 may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0262] The electronic device can execute the asynchronous data execution method in the embodiments of the present invention, thereby achieving... Figure 2 Describes the asynchronous execution method for data.

[0263] Furthermore, in conjunction with the asynchronous data execution methods in the above embodiments, this invention can be implemented using a readable storage medium. This readable storage medium stores program instructions; when these program instructions are executed by a processor, they implement any of the asynchronous data execution methods described in the above embodiments.

[0264] Furthermore, in conjunction with the asynchronous data execution methods in the above embodiments, this invention can be implemented using a computer program product. When the instructions in this computer program product are executed by the processor of an electronic device, the electronic device performs any of the asynchronous data execution methods described in the above embodiments.

[0265] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0266] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0267] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0268] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0269] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A method for asynchronous execution of data, characterized by, The method is applied to a cache component of structured query language asynchronous execution, the cache component includes a consumer process and a cache component unit, the cache component unit includes a component process group and a cache file set, the component process group at least includes a first component process node, the first component process node includes a first thread and a second thread, and the cache file set is stored in a preset storage area of a disk; The method comprises: The consumer process obtains structured query language data in a message queue and a network protocol address corresponding to the first component process node; The consumer process parses the structured query language data to obtain a data type of the structured query language data; The consumer process sends the structured query language data to the first thread of the first component process node corresponding to the data type according to the network protocol address; The first thread stores the structured query language data in a corresponding component cache file in the cache file set according to a preset data storage strategy; The second thread corresponding to the first thread reads the structured query language data from the component cache file; The second thread batch-executes structured query language script data corresponding to the structured query language data in the component cache file; The first thread stores the structured query language data in a corresponding component cache file in the cache file set according to a preset data storage strategy, comprising: Creating a component cache sub-file in the component cache file and adding a first lock mark to the component cache sub-file; Cyclically executing the following steps until the structured query language data in the message queue is traversed to obtain a target component cache file: In response to determining that the number of structured query language data in the component cache sub-file is less than a preset threshold, reading first structured query language data from the message queue based on the first thread; wherein the first structured query language data is structured query language data at the head of the queue to be read in the message queue; Parsing the first structured query language data to generate structured query language script data corresponding to the first structured query language data; Inserting the structured query language script data into the component cache sub-file.

2. The method of claim 1, wherein, After creating a component cache sub-file in the component cache file and adding a first lock mark to the component cache sub-file, the method further comprises: In response to determining that the number of structured query language data in the component cache sub-file is greater than or equal to the preset threshold, returning to create a component cache sub-file in the component cache file.

3. The method of claim 1, wherein, The second thread corresponding to the first thread reads the structured query language data from the component cache file, comprising: generating each memory image file corresponding to each target component cache sub-file in the target component cache file based on a second thread corresponding to the first thread; wherein each memory image file is named according to a generation time stamp of each memory image file; and each memory image file is sorted in a forward direction according to the generation time stamp; cyclically performing the following steps until each memory image file is traversed: adding a second lock mark to a first memory image file, in response to determining that the target component cache file has a target component cache sub-file corresponding to the first memory image file, and the first memory image file is not the last file in the memory image files corresponding to each memory image file, and the generation time stamp of the first memory image file is less than a first preset time stamp, and the memory of the first memory image file is not 0, reading the target cache sub-file corresponding to the first memory image file, and removing the second lock mark; wherein the first memory image file is any one of the memory image files.

4. The method of claim 3, wherein, The method further comprises: in response to determining that the target component cache file does not have a target component cache sub-file corresponding to the first memory image file, deleting the first memory image file, and determining whether the target component cache file has a target component cache sub-file corresponding to the second memory image file, in response to determining that the target component cache file has a target component cache sub-file corresponding to the second memory image file, updating the second memory image file to the first memory image file, and returning to perform adding a second lock mark to a first memory image file, in response to determining that the target component cache file has a target component cache sub-file corresponding to the first memory image file, and the first memory image file is not the last file in the memory image files, and the generation time stamp of the first memory image file is less than a first preset time stamp, and the memory of the first memory image file is not 0, reading the first memory image file, and removing the second lock mark; wherein the second memory image file is the next memory image file of the first memory image file.

5. The method of claim 3, wherein, The method further comprises: in response to determining that the first memory image file is the last file in the memory image files, returning to perform generating each memory image file corresponding to each target component cache sub-file in the target component cache file based on a second thread corresponding to the first thread.

6. The method of claim 3, wherein, The method further comprises: in response to determining that the generation time stamp of the first memory image file is greater than or equal to the first preset time stamp, or the memory of the first memory image file is 0, deleting the target component cache sub-file corresponding to the first memory image file in the target component cache file and the first memory image file.

7. The method of claim 1, wherein, The batch execution of the structured query language script data corresponding to the structured query language data in the component cache file comprises: For each target component cache sub-file, obtain the number of structured query language script data in the target component cache sub-file; Based on the number, allocate each structured query language script data in the target component cache sub-file to different processing pages; wherein each processing page has a plurality of structured query language script data; In response to determining that the last execution timestamp of the plurality of structured query language script data in the target processing page is greater than a second preset timestamp, batch executing the plurality of structured query language script data in the target processing page; wherein the target processing page is any one of the plurality of processing pages.

8. The method of claim 7, wherein, After the batch execution of the plurality of structured query language script data in the target processing page, the method further comprises: In response to determining that the plurality of structured query language script data in the target processing page is executed successfully, submitting the plurality of structured query language script data in the target processing page to a structured query language database, so that the database processes the plurality of structured query language script data in the target processing page; In response to determining that the plurality of structured query language script data in the target processing page fails to execute, moving the plurality of structured query language script data that fails to execute to a preset file in a cache file set, updating the last execution timestamp of the target structured query language script data to a current timestamp, and re-executing the target structured query language script data if the updated last execution timestamp is greater than or equal to the second preset timestamp; wherein the target structured query language script data is at least one of the plurality of structured query language script data that fails to execute.

9. The method of claim 7, wherein, The method further comprises: In response to determining that the number of structured query language script data in the target component cache sub-file is not obtained, moving the target component cache sub-file to a preset file in a cache file set, and deleting a memory image file corresponding to the target component cache sub-file.

10. The method of claim 7, wherein, After obtaining the number of structured query language script data in the target component cache sub-file, the method further comprises: In response to determining that the number of structured query language statements in the target component cache sub-file is 0, and the memory image file corresponding to the target component cache sub-file is located at the last of each memory image file, deleting the target component cache sub-file from the disk, and deleting the memory image file corresponding to the target component cache sub-file.

11. The method of claim 1, wherein, The component process group further comprises a second component process node; The method further comprises: Obtaining running data, network data, and transmission control protocol port data of the first component process node; In response to determining that at least one of the running data, the network data, and the transmission control protocol port data is abnormal, updating the first component process node to the second component process node.

12. A data asynchronous execution apparatus, comprising: The device is applied to a cache component of structured query language asynchronous execution, the cache component comprises a consumer process and a cache component unit, the cache component unit comprises a component process group and a cache file set, the component process group at least comprises a first component process node, the first component process node comprises a first thread and a second thread, and the cache file set is stored in a preset storage area of a disk; The device comprises: A first determination module configured to acquire, by a consumer process, structured query language data in a message queue and a network protocol address corresponding to a first component process node; A second determination module configured to parse, by the consumer process, the structured query language data to obtain a data type of the structured query language data; A sending module configured to send, by the consumer process, the structured query language data to a first thread of the first component process node corresponding to the data type according to the network protocol address; A storage module configured to store, by the first thread, the structured query language data in a corresponding component cache file in a cache file set according to a preset data storage strategy; A first execution module configured to read, by a second thread corresponding to the first thread, the structured query language data from the component cache file; A second execution module configured to batch execute, by the second thread, structured query language script data corresponding to the structured query language data in the component cache file; The storage module is specifically configured to create a component cache sub-file in the component cache file and add a first lock mark to the component cache sub-file; The following steps are cyclically executed until the structured query language data in the message queue is traversed and a target component cache file is obtained: In response to determining that the number of structured query language data in the component cache sub-file is less than a preset threshold, reading first structured query language data from the message queue based on the first thread; wherein the first structured query language data is structured query language data at the head of the queue to be read in the message queue; Parsing the first structured query language data to generate structured query language script data corresponding to the first structured query language data; Inserting the structured query language script data into the component cache sub-file.

13. A data asynchronous execution controller, comprising: The data asynchronous execution controller comprises a processor and a memory storing computer program instructions; the processor implements the data asynchronous execution method of any one of claims 1-11 when executing the computer program instructions.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the data asynchronous execution method of any one of claims 1-11.

15. A computer program product, characterised in that, The instructions in the computer program product are executed by a processor of an electronic device, so that the electronic device executes the data asynchronous execution method of any one of claims 1-11.

Images