Data processing method, system and apparatus, and device and medium

By pushing down the target operator of the machine learning model to the storage node, using data processing in the target timeline, the problem of low data processing efficiency in the database is solved, efficient data processing and parallel processing of multiple storage nodes is realized, and the Data transmission pressure.

WO2024235152A1PCT designated stage expired Publication Date: 2024-11-21CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PTE LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/092589
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-18
Filing Date
2024-05-11
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

When using databases, machine learning models require a large amount of data transmission when performing data processing tasks, resulting in low data processing efficiency, consume a large amount of bandwidth and storage resources, and the existing technology is complicated.

Method used

By pushing down the target operator of the machine learning model built into the database to the storage node, the target timing data in the target timeline is used for data processing, avoiding the transmission of timing data to the computing node, and achieving parallel processing of multiple storage nodes.

Benefits of technology

It effectively reduces the amount of data transmission between nodes, improves computing efficiency and data processing capabilities, and ensures the security and processing efficiency of database data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024092589_21112024_PF_FP_ABST
    Figure CN2024092589_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present application are a data processing method, system and apparatus, and a device and a medium. The method comprises: in response to a data processing request for a target attribute, determining a target operator of a machine learning model built into a database; on the basis of a time series identifier corresponding to the target attribute, determining a target timeline corresponding to the time series identifier, and a storage node that stores the target timeline; and pushing the target operator to the storage node, such that the storage node executes the data processing request for target time series data in the target timeline by using the target operator. According to a data processing request for a target attribute, a target operator of a machine learning model built into a database is sent to a storage node which stores a timeline corresponding to the target attribute, without the need for transmitting target time series data in the storage node to a computing node, such that the amount of data transmitted between nodes can be effectively reduced, thereby improving the computing efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Data processing method, system, device, equipment and medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 18, 2023, with application number 202310574685.X and application name “Data processing method, system, device, equipment and medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to data processing methods, systems, devices, equipment, and media. Background Art

[0003] With the development of artificial intelligence technology, machine learning models will be used in more and more scenarios to achieve related learning, prediction and other tasks, thereby effectively reducing the workload of personnel.

[0004] In practical applications, when using data from a database, it's often necessary to retrieve the required data from the database. For example, when using a machine learning model to perform a task, data needs to be retrieved from the database and then processed using the model. This requires a large amount of data transmission, resulting in low data processing efficiency and significant bandwidth and storage resource consumption.

[0005] Summary of the Invention

[0006] To solve or improve the problems existing in the prior art, the embodiments of the present application provide data processing methods, systems, devices, equipment and media.

[0007] In a first aspect, in one embodiment of the present application, a data processing method is provided. Applied to a computing node, the method comprises:

[0008] In response to a data processing request for a target attribute, determining a target operator of a machine learning model built into a database;

[0009] Based on the timing identifier corresponding to the target attribute, determining a target timeline corresponding to the timing identifier and a storage node storing the target timeline;

[0010] The target operator is pushed down to the storage node, so that the storage node uses the target operator to execute the data processing request for the target time series data in the target timeline.

[0011] In a second aspect, in one embodiment of the present application, a data processing method is provided. Applied to a storage node, the method includes:

[0012] Receive a target operator in a machine learning model pushed down by a computing node for executing a data processing request; the computing node is used to determine a corresponding target timeline and a storage node for storing the target timeline based on a timing identifier of a target attribute corresponding to the data processing request;

[0013] Determining target time series data for satisfying the data processing request based on the timeline;

[0014] The target operator is used to perform data processing on the target time series data, and the data processing result is sent to the computing node.

[0015] In a third aspect, in one embodiment of the present application, a data processing device is provided, the device comprising:

[0016] In response to a data processing request for a target attribute, determining a target operator of a machine learning model built into a database;

[0017] Based on the timing identifier corresponding to the target attribute, determining a target timeline corresponding to the timing identifier and a storage node storing the target timeline;

[0018] The target operator is pushed down to the storage node, so that the storage node uses the target operator to execute the data processing request for the target time series data in the target timeline.

[0019] In a fourth aspect, an embodiment of the present application provides a data processing system, including:

[0020] A computing node, configured to execute the method according to any one of the first aspects;

[0021] A storage node, configured to execute the method described in the second aspect.

[0022] In a fifth aspect, in one embodiment of the present application, an electronic device is provided, including a memory and a processor; wherein,

[0023] The memory is used to store programs;

[0024] The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the method described in the first aspect or the method described in the second aspect.

[0025] In the sixth aspect, in one embodiment of the present application, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect, or executes the method described in the second aspect.

[0026] The technical solution provided by the embodiment of the present application is that in order to facilitate data management, some databases will store time series data according to the time line. In some database systems, machine learning models are used to meet their data processing needs. Therefore, in this solution, according to the data processing request for the target attribute, the target operator of the machine learning model built into the database is sent to the storage node storing the time line corresponding to the target attribute, without the need to transmit the target time series data in the storage node to the computing node, which can effectively reduce the amount of data transmission between nodes and improve computing efficiency. At the same time, when using a machine learning model to process time series data, it is only necessary to select the corresponding target operator according to the data processing request and push the target operator down to the corresponding storage node, without the need to transmit an overly complex machine learning model to meet the data processing needs. At the same time, by pushing the target operators in the same machine learning model to different storage nodes respectively, and having different storage nodes perform corresponding data processing tasks, multi-storage node parallel processing is achieved, effectively improving data processing efficiency and data processing capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0028] FIG1 is a flow chart of a data processing method according to an embodiment of the present application;

[0029] FIG2 is a flow chart of another data processing method provided in an embodiment of the present application;

[0030] FIG3 is a schematic diagram of a data processing device provided in an embodiment of the present application;

[0031] FIG4 is a schematic diagram of another data processing device provided in an embodiment of the present application;

[0032] FIG5 is a schematic diagram of the system structure provided in an embodiment of the present application;

[0033] FIG6 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0035] In some processes described in the specification, claims and the above-mentioned figures of this application, multiple operations appearing in a specific order are included. These operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit "first" and "second" to different types. In addition, the embodiments described below are only some of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0036] In existing databases, machine learning models are often used to perform data processing tasks to achieve better results. There are two existing implementations: using machine learning models outside the database to perform data processing tasks, and using machine learning models built into the database to perform data processing tasks. Specifically, when using machine learning models outside the database to perform data processing tasks, data must be retrieved from the database. The larger the amount of data required, the greater the bandwidth consumed, the lower the data processing efficiency, and the more complex the database system. Machine learning models can also be built into the database. Specifically, in relational databases, when performing timeline-based data processing, the data in a timeline is flattened into multiple rows. Since the time series data in a timeline is fragmented—in other words, each time series data in the timeline is stored in a different storage node—then the time series data must be transferred to the compute nodes to perform the corresponding computations. This data transfer between nodes consumes a significant amount of bandwidth.

[0037] Explanation of terms:

[0038] Time Series: A data source's metric changes over time, forming a timeline. When storing data, the database clusters and stores data from the same timeline as much as possible to improve access efficiency for time series data within the timeline. This also provides better support for time series data compression. In a time series data table, a series of data rows with the same Tag column value constitutes a timeline.

[0039] Operator: An operator that performs calculations on data in a database.

[0040] The technical solutions implemented in this application will be explained below with reference to specific embodiments.

[0041] FIG1 is a flow chart of a data processing method provided in an embodiment of the present application. The execution subject of the method may be a computing node. As can be seen from FIG1 , the method specifically includes the following steps:

[0042] 101: In response to a data processing request for a target attribute, determine a target operator of a machine learning model built into a database.

[0043] 102: Based on the time sequence identifier corresponding to the target attribute, determine a target timeline corresponding to the time sequence identifier and a storage node storing the target timeline.

[0044] 103: Push the target operator down to the storage node, so that the storage node uses the target operator to execute the data processing request for the target time series data in the target timeline.

[0045] It should be noted that in the technical solution of this application, the data stored in the database is time series data. Specifically, the data in the database is time series data stored according to a timeline. Time series data for the same target attribute is often stored in the same timeline, and time series data within a continuous period of time in a timeline is stored in the same storage node, thereby facilitating the management of time series data. When a data processing task needs to be performed for a certain target attribute, it is often not necessary to collect time series data across multiple storage nodes.

[0046] In this application scheme, the machine learning model is built into the database. When using the machine learning model, there is no need to export the data in the database to an external platform, which can effectively improve data processing efficiency and ensure the security of data in the database.

[0047] The target operators required for the same data processing request with different target attributes are different, the target operators required for different data processing requests with the same target attribute are different, and the target operators required for different data processing requests with different target attributes are different. It should be noted here that different target attributes can be understood as the attributes of different products, which can generate time-series-related data. For example, the target attribute can be the customer repurchase rate of a certain piece of clothing, or the time when peak electricity consumption occurs in a certain area. It can be seen that in actual applications, in order to make the machine learning model have more accurate processing results, different machine learning models need to be trained for different target attributes to meet the data processing requirements for different target attributes. The different data processing requests mentioned here can be, for example, time series prediction requests or time series anomaly detection requests.

[0048] In machine learning models, various operators are provided. In other words, different target operators are selected to meet different data processing requirements for different target attributes. For example, the operators required for time series prediction requests may include: DeepAR algorithm, a deep neural network algorithm based on recurrent neural networks (RNN), or Attention-based Temporal Fusion Transformer (TFT) algorithm, a deep neural network algorithm based on the Transformer mechanism. The operators required for time series anomaly detection requests may include: nsigma The principle is simple and facilitates analysis of the causes of time series data anomalies.

[0049] It should be noted here that in time series data, each time line has a unique time series identifier. Time series data storage refers to the combination of a metric and a set of tags as a time line. Under a time line, the sampling data of continuous time points are time series data. Continuous time series data is stored under the same time line, (timestamp, value). When storing, the same time line is stored in the same storage node. Of course, if the storage node is expanded, the time series data after expansion and the time series data before expansion may be stored in different storage nodes. In this case, it is necessary to determine the specific data processing method based on the distribution of the required time series data in the storage node based on the routing table (including: the computing node pushes the target operator down to the storage node, or the storage node transmits the time series data to the computing node). The specific selection process will be explained in detail in the following embodiments, and will not be repeated here.

[0050] When executing data processing tasks, machine learning models select target time series data within a certain time period and determine the storage node where the target time series data resides. In a database system, a compute node corresponds to multiple storage nodes. When processing time series data using a target operator, the time series data typically requires continuity. Therefore, the target operator is pushed down to a storage node that stores continuous timelines. The completion of the data processing request mentioned here may involve the participation of multiple different target timelines corresponding to the same target attribute. In this case, the same target operator can be sent to the corresponding storage node, and each storage node can execute the corresponding data processing task separately, enabling multiple storage nodes to execute data processing tasks in parallel. In addition, it is possible that multiple different target timelines are stored in the same storage node. In this case, the target operator can be sent to the storage node, and the storage node can execute the data processing tasks for different target timelines sequentially, or the storage node can execute data processing for multiple different target timelines in parallel. If multiple storage nodes send time series data to computing nodes respectively, then due to the limited computing power of computing nodes, it is necessary to execute the data processing tasks for the time series data provided by each storage node in sequence. The efficiency of data processing performed by a single computing node is significantly lower than the efficiency of data processing performed in parallel. It should be noted that in the database system of the present application solution, each storage node has a certain computing power that can meet the needs of the target operator to perform data processing tasks. The amount of data processing results output after executing the data processing task is significantly smaller than the amount of data of the time series data in the timeline, thereby effectively reducing the bandwidth consumed by data transmission.

[0051] Through the above solution, the machine learning model is built into the database, and when executing the data processing task, the appropriate target operator in the machine learning model is selected according to the data processing request and the target attribute. At the same time, the storage node storing the corresponding timeline is determined based on the time series identifier corresponding to the target attribute. Then, the target operator is pushed down to the corresponding storage node without the need to transfer the time series data in the storage node to the computing node, which can effectively reduce the pressure of data transmission. In addition, by processing the time series data according to the timeline, the data processing task can be sent to each storage node for parallel execution, which can effectively improve the data processing efficiency.

[0052] In one or more embodiments of the present application, determining the target timeline corresponding to the timing identifier and the storage node storing the target timeline based on the timing identifier corresponding to the target attribute includes: determining the timing identifier corresponding to the target attribute, and determining the corresponding target timeline based on the timing identifier; determining the routing table corresponding to the target timeline according to the timing identifier; and determining the storage node corresponding to the target timeline and the number of storage nodes according to the routing information stored in the routing table.

[0053] In actual applications, when accessing data, the corresponding storage node is found through the routing table and data access is performed. Specifically, the corresponding timing identifier and target timeline are determined according to the target attribute. And, the routing table corresponding to the target timeline is determined using the timing identifier. The storage node corresponding to the timeline and the number of storage nodes can be understood through the routing path in the routing table. In other words, when the number of storage nodes included in the routing table is not one, it can be seen that the timing data corresponding to the timeline is dispersed and stored in different storage nodes. When the number of storage nodes included in the routing table is one, it means that the timing data corresponding to the timeline are all stored in the same storage node.

[0054] For example, a timeline of {"metric":"cpu","tags":"site":"et2","ip":"1.1.1.1"} is obtained. The combination of the metric and the tag forms a timeline, and continuous time series data (timestamp, value) is stored under the same timeline. For example, timestamp: 12345, value: 1; timestamp: 12346, value: 2.

[0055] Since the time series data in the timeline is time-related, when the storage node is expanded, the routing table will be updated. The routing table can accurately understand the storage node corresponding to the timeline. Then, corresponding data processing methods are used for different storage methods of the timeline (storage in the same storage node or in multiple different storage nodes), so as to meet the data processing needs while maximizing data processing efficiency.

[0056] In one or more embodiments of the present application, pushing the target operator to the storage node includes: if the storage node corresponding to the target timeline is found to be the only node, pushing the target operator to the storage node.

[0057] In actual applications, if it is determined through the routing table that the time series data of the time line are all stored in the same storage node, it means that when the machine learning model is used to process the time series data, the target operator can be pushed down to the corresponding storage node, and then the data processing task can be completed without uploading the time series data to the computing node. Since the machine learning model often needs to select continuous data within a period of time when processing time series data, only when the time series data of the time line is in the same storage node can the target operator be pushed down to the storage node and the corresponding data processing task be performed. The time series data in the storage node meets the requirements of the data processing task, and there is no need to obtain data from other storage nodes. In other words, the corresponding data processing results can be achieved in the storage node, avoiding the need to obtain data from the storage node and reducing the consumption of network bandwidth resources caused by data transmission.

[0058] In one or more embodiments of the present application, if it is found that the target timeline corresponds to multiple storage nodes, the target time series data in the multiple storage nodes corresponding to the same target timeline are received.

[0059] In actual applications, if storage space is insufficient, the storage nodes may be expanded, that is, the number of storage nodes is increased, which will increase the routing paths in the routing table. In other words, multiple storage nodes can be found from the routing table based on the time series identifier. This means that the time series data required by the machine learning model is stored in different storage nodes. If the target operator is sent to multiple storage nodes, since each storage node does not store the complete time series data, it cannot meet the data processing requirements. Therefore, multiple storage nodes can send the time series data of the same timeline to the computing nodes, and then the computing nodes perform the data processing tasks.

[0060] For example, there are storage nodes A and B. Storage node A stores the time series data for timeline A1, while storage node B stores the time series data for timeline B1. Assume that storage node B has insufficient storage space and needs to add storage node C, which is used to store the time series data for timeline B1. If a data processing request for timeline B1 is received at this time, a routing table search reveals that timeline B1 is stored in both storage node B and storage node C. In this case, storage node B and storage node C can respectively send the time series data corresponding to timeline data B1 to the compute node, and the compute node performs the corresponding data processing task.

[0061] In one or more embodiments of the present application, determining a target operator of a machine learning model built into a database in response to a data processing request for a target attribute includes:

[0062] In response to a data processing request for a target attribute, a target operator type and the target operator required by the data processing request are determined; or, the target operator is determined based on a function name carried in the data processing request.

[0063] In practical applications, operators in machine learning models can be stored as database functions in a database function set. This database function set can contain a wide variety of machine learning model operators, such as time series prediction operators, time series anomaly detection operators, and so on. Furthermore, different specialized machine learning models are employed for different target attributes, with varying methods, from selecting training samples for the machine learning models and training them to the data used for data processing. Therefore, upon receiving a data processing request, a target operator type matching the data processing request type and target attribute is determined. Based on this target operator type, the corresponding target operator is then searched for from the database functions. Alternatively, when initiating a data processing request, the user can specify the target operator to be used using a function name. Upon receiving the data processing request, the computation order can parse the request and accurately locate the appropriate target operator based on the function name.

[0064] When a data processing request is a comprehensive request involving multiple different processing methods, it is necessary to split the data processing request into tasks. For example, after splitting, two data processing requests are obtained: prediction first and anomaly detection second. Because different data processing requests select different target operator types, the corresponding target operators are selected and sent to the storage node, and the corresponding data processing tasks are executed in sequence. Of course, if the two data processing tasks are related, online execution means that the data processing results can be used as input for the data processing task executed later, allowing the data processing tasks to be completed without data transmission.

[0065] In one or more embodiments of the present application, determining the storage node and the number of storage nodes corresponding to the target timeline according to the routing information stored in the routing table includes:

[0066] Determining, according to the time range specified in the data processing request, the routing information corresponding to the time range in the routing table;

[0067] The storage node corresponding to the target timeline and the number of storage nodes are determined based on the routing information.

[0068] In practical applications, when using machine learning models to perform data processing tasks on time series data, data processing is often performed on continuous time series data. Although the time series data of some timelines are stored in different storage nodes, when the machine learning model performs data processing tasks, the time series data required may be concentrated in the same storage node. In other words, there is no need to collect time series data across multiple storage nodes. Therefore, the corresponding routing information can be determined based on the time range and routing table (in other words, the routing information contains the time tag range of the time series data and the corresponding storage node information. For example, the time series data from time A1 to time A99 is stored in storage node D1, and the time series data corresponding to time B1 to time B99 is stored in storage node D2). Then, the storage node corresponding to the timeline required to execute the data processing request and the number of storage nodes are determined. When the storage node corresponding to the target timeline determined based on the routing information and the number of storage nodes are multiple, the multiple storage nodes respectively send the time series data to the computing node, and the computing node performs the corresponding data processing task. If the target timeline, as determined by the routing information, corresponds to a single storage node and the number of storage nodes is one, the compute node can send the corresponding target operator to the storage node, which then performs the corresponding data processing task. It should be noted that this time range can be either a specified start and end time in the data processing request, or a specified time duration without a specified start and end time.

[0069] Of course, the timeline to be split, as well as the time range and corresponding storage nodes of the sub-timelines split from the same timeline, can also be determined based on the routing information stored in the routing table. Furthermore, the corresponding data processing results can be obtained using the time series data in different storage nodes.

[0070] With this solution, because time series data is time-dependent, even if storage nodes are expanded, the actual analysis results within the specified time range can be used to determine whether operators can be pushed down. Although the target timelines contained in the routing table correspond to routing paths for multiple different storage nodes, if the time series data for the time range required by the machine learning model happens to be in a specific storage node, the target operator can be pushed down to that storage node, effectively improving data processing efficiency.

[0071] In one or more embodiments of the present application, the training method of the machine learning model includes:

[0072] Push the target operator in the machine learning model to be trained to the storage node;

[0073] The machine learning model to be trained is trained using the target time series data in the timeline stored in the storage node.

[0074] In practical applications, when training a machine learning model, it can also be performed in the corresponding storage nodes. Specifically, when training a machine learning model, the purpose of the model is clarified, and then the operator that meets the purpose is determined. When training a machine learning model, since the target operator ideally needs to be pushed down to the storage node when using the machine learning model, the corresponding data processing task needs to be performed using the time series data stored in the storage node. Therefore, when training a machine learning model, it is also necessary to train it using the time series data stored in the storage node. Because a large number of training samples are required to obtain relatively accurate training results, the target operator in the machine learning model is pushed down to a storage node that can provide training samples. The storage node provides historical time series data as training samples to train the machine learning model operator and obtain the target operator. Alternatively, if the training samples corresponding to the same timeline are distributed across multiple different storage nodes, these different storage nodes need to be distributed to upload the training samples to the compute node, which then performs the training. Through the above method, when the training samples of the same timeline are in the same storage node, it is possible to train the machine learning model without extracting a large number of training samples from the storage node and transmitting them to the computing node. This can effectively reduce the network bandwidth occupied by data transmission and improve training efficiency.

[0075] In addition, if the training samples provided by multiple different timelines are in different storage nodes, the operators to be trained can be sent to different storage nodes to perform corresponding training tasks, thereby achieving parallel training and improving training efficiency.

[0076] In this way, there is no need to send a large number of training samples (that is, time series data stored in storage nodes) to training nodes (for example, computing nodes). Instead, a large amount of time series data can be used as training samples in the storage nodes for training, which can effectively improve training efficiency. The trained machine learning model or target operator is sent to the computing node.

[0077] In one or more embodiments of the present application, determining a target timeline corresponding to the timing identifier based on the timing identifier corresponding to the target attribute, and a storage node storing the target timeline, includes:

[0078] Determining target timelines corresponding to the respective timing identifiers based on at least one of the target attributes;

[0079] Determining the storage nodes corresponding to the target timelines;

[0080] The pushing down the target operator to the storage node includes: pushing down the target operator to each of the storage nodes.

[0081] In practical applications, if a user initiates a data processing request for multiple different target attributes, it is necessary to determine the corresponding time series identifiers based on at least one target attribute, as well as the target timelines corresponding to each time series identifier. It should be noted that the target timelines corresponding to each time series identifier may be stored in completely different storage nodes or in the same storage node. If stored in different storage nodes, the compute node can push the corresponding target operators down to the corresponding storage node. If stored in the same storage node, due to the limited computing power of the storage node, it may be necessary to execute the data processing task in batches. In other words, the same or different target operators may be sent to the same storage node in batches for data processing. Alternatively, the compute node and storage node can each execute data processing tasks for different target attributes. For example, the timelines corresponding to target attribute 1 and target attribute 2 are both stored in the same storage node. After comparison, it is found that the amount of time series data corresponding to target attribute 1 is smaller than that corresponding to target attribute 2. Therefore, the target operator corresponding to target attribute 2 can be pushed down to the storage node, and the storage node can transmit the time series data corresponding to target attribute 1 to the compute node. Therefore, while meeting the needs of parallel computing and improving data processing efficiency, it can minimize the occupation of network bandwidth resources by data transmission.

[0082] In one or more embodiments of the present application, a timing prediction result or anomaly detection result fed back by at least one of the storage nodes based on the data processing request is received.

[0083] In practical applications, the corresponding processing results are fed back based on the type of data processing request. These processing results can be time series prediction results or anomaly detection results. When a storage node sends data processing results to a compute node, the amount of data required is significantly less than the amount of time series data involved in the data processing. This can meet data processing requirements while reducing data transmission. It should be noted that data processing methods are not limited to the time series prediction results or anomaly detection results mentioned above. Data processing requests can also involve time series data analysis, and the corresponding feedback results are time series analysis results and other corresponding results.

[0084] Based on the same idea, the present embodiment also provides a data processing method. FIG2 is a flow chart of another data processing method provided by the present embodiment. The method is applied to a storage node. As can be seen from FIG2, the method specifically includes the following steps:

[0085] 201: Receive a target operator in a machine learning model pushed down by a computing node for executing a data processing request; the computing node is used to determine a corresponding target timeline and a storage node for storing the target timeline based on a timing identifier of a target attribute corresponding to the data processing request.

[0086] 202: Determine target time series data for satisfying the data processing request based on the timeline.

[0087] 203: Utilize the target operator to perform data processing on the target time series data, and send the data processing result to the computing node.

[0088] In practical applications, one compute node corresponds to multiple storage nodes. Machine learning models are built into a database, and the time series data stored in this database is stored according to timelines. Generally speaking, time series data from the same timeline is stored in the same storage node. When a data processing task is required using a machine learning model, the corresponding timeline and the storage node storing the timeline are determined based on the data processing request. The target operator capable of executing the data processing request is then pushed down to the corresponding storage node to perform the corresponding data processing task. This approach satisfies data processing requests while avoiding the need to send large amounts of data from storage nodes to compute nodes, reducing network bandwidth usage for data transmission. Simultaneously, the operators used in the machine learning model to execute data processing tasks are sent to storage nodes in a targeted manner, effectively improving data processing efficiency. Furthermore, if data is sent to multiple storage nodes, they can execute the data processing task in parallel, resulting in higher data processing efficiency than if only compute nodes were executing multiple data processing tasks.

[0089] Based on the same idea, an embodiment of the present application also provides a data processing device. FIG3 is a schematic diagram of a data processing device provided in an embodiment of the present application. As can be seen from FIG3, the device includes:

[0090] The first determination module 31 is used to determine the target operator of the machine learning model built into the database in response to a data processing request for the target attribute.

[0091] The second determining module 32 is configured to determine, based on the time sequence identifier corresponding to the target attribute, a target timeline corresponding to the time sequence identifier and a storage node storing the target timeline.

[0092] The push module 33 is configured to push the target operator to the storage node, so that the storage node uses the target operator to execute the data processing request for the target time series data in the target timeline.

[0093] Optionally, the second determining module 32 is configured to determine a timing identifier corresponding to the target attribute, and determine a corresponding target timeline based on the timing identifier;

[0094] Determining a routing table corresponding to the target timeline according to the timing identifier;

[0095] The storage nodes and the number of storage nodes corresponding to the target timeline are determined according to the routing information stored in the routing table.

[0096] Optionally, the push module 33 is configured to push the target operator to the storage node if the storage node corresponding to the target timeline is found to be the only node.

[0097] Optionally, the system further includes a receiving module 34 for receiving the target time series data from the multiple storage nodes corresponding to the same target time line if multiple storage nodes are found to correspond to the target time line.

[0098] Optionally, the first determining module 31 is configured to determine, in response to a data processing request for a target attribute, a target operator type and the target operator required by the data processing request;

[0099] Alternatively, the target operator is determined based on a function name carried in the data processing request.

[0100] Optionally, a second determining module 32 is configured to determine, according to a time range specified in the data processing request, the routing information corresponding to the time range in the routing table;

[0101] The storage node corresponding to the target timeline and the number of storage nodes are determined based on the routing information.

[0102] Optionally, a training module 35 is further included, configured to push the target operator in the machine learning model to be trained to the storage node;

[0103] The machine learning model to be trained is trained using the target time series data in the timeline stored in the storage node.

[0104] A second determining module 32 is configured to determine a target timeline corresponding to each timing identifier based on at least one timing identifier corresponding to each target attribute;

[0105] Determining the storage nodes corresponding to the target timelines;

[0106] The pushing down the target operator to the storage node includes: pushing down the target operator to each of the storage nodes.

[0107] Optionally, the receiving module 34 is configured to receive a timing prediction result or anomaly detection result fed back by at least one of the storage nodes based on the data processing request.

[0108] Based on the same idea, the embodiment of the present application also provides another data processing device. FIG4 is a schematic diagram of another data processing device provided by the embodiment of the present application. As can be seen from FIG4, the device includes:

[0109] The receiving module 41 is used to receive the target operator in the machine learning model pushed down by the computing node for executing the data processing request; the computing node is used to determine the corresponding target timeline and the storage node that stores the target timeline according to the timing identifier of the target attribute corresponding to the data processing request.

[0110] The determination module 42 is configured to determine target time series data for satisfying the data processing request based on the timeline.

[0111] The sending module 43 is configured to use the target operator to perform data processing on the target time series data and send the data processing result to the computing node.

[0112] Based on the same idea, the embodiment of the present application also provides a data processing system. As shown in Figure 5, the system structure diagram provided by the embodiment of the present application is as follows. As can be seen from Figure 5, the system includes:

[0113] Computing node 51, which corresponds to at least one storage node 52. In the computing node 51, it can be divided into functional layers, namely the SQL layer for supporting the SQL language provided by the user and the data processing layer for performing data processing tasks. The data processing layer has built-in machine learning model-related operators (for example, time series prediction operators or anomaly detection operators). In the storage node 52, it can be divided into a data processing layer and a data storage layer according to its function; wherein, the data processing layer has built-in pushed-down target operators (for example, time series prediction operators or anomaly detection operators), and the data storage layer stores time series data according to timelines. For example, as shown in Figure 5, timeline 1 and timeline 2 are stored in storage node 1, and timeline 3 is stored in storage node 2.

[0114] For example, suppose the data processing request is for time series prediction or anomaly detection. The data processing layer is primarily responsible for accessing and processing time series data. It has the ability to pull data from the data storage layer according to the timeline dimension and perform some timeline-related processing (such as downsampling and some aggregation calculations), and supports the expansion of processing operators for the timeline dimension. The data processing layer exists in both compute nodes and storage nodes. Therefore, the time series engines of some existing databases can push some time series data processing operators from compute nodes to storage nodes when conditions permit. On the one hand, multiple storage nodes can be used to achieve parallel computing optimization, and on the other hand, it can achieve optimization close to the data computing, thereby reducing the data transmitted between nodes and improving computing efficiency.

[0115] Both time series prediction and time series anomaly detection are algorithms that process data in the timeline dimension. Time series prediction predicts a continuous window in the future based on a historical window of data on the timeline. Time series anomaly detection also requires testing newly generated data based on historical timeline data. During implementation, data is usually batch processed according to the timeline dimension to improve detection efficiency.

[0116] This solution supports the reasoning function of time series prediction / anomaly detection algorithms based on machine learning. On the one hand, it can use the characteristics of the time series engine for timeline dimension processing to achieve batch processing optimization. On the other hand, it can use the data processing push-down capability of the time series engine to push the prediction / anomaly detection algorithm down to the storage node, thereby achieving parallel and close-to-data reasoning optimization.

[0117] An embodiment of the present application also provides an electronic device. The electronic device is a master node electronic device in a computing unit. FIG6 is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device includes a memory 601, a processor 602 and a communication component 603; wherein,

[0118] The memory 601 is used to store programs;

[0119] The processor 602 is coupled to the memory and is configured to execute the program stored in the memory to:

[0120] In response to a data processing request for a target attribute, determining a target operator of a machine learning model built into a database;

[0121] Based on the timing identifier corresponding to the target attribute, determining a target timeline corresponding to the timing identifier and a storage node storing the target timeline;

[0122] The target operator is pushed down to the storage node, so that the storage node uses the target operator to execute the data processing request for the target time series data in the target timeline.

[0123] The processor 602 is configured to determine a timing identifier corresponding to the target attribute, and determine a corresponding target timeline based on the timing identifier;

[0124] Determining a routing table corresponding to the target timeline according to the timing identifier;

[0125] The storage nodes and the number of storage nodes corresponding to the target timeline are determined according to the routing information stored in the routing table.

[0126] The processor 602 is configured to push down the target operator to the storage node if the storage node corresponding to the target timeline is found to be the only node.

[0127] The processor 602 is configured to receive the target time series data from the multiple storage nodes corresponding to the same target timeline if it is found that the target timeline corresponds to multiple storage nodes.

[0128] The processor 602 is configured to determine, in response to a data processing request for a target attribute, a target operator type and the target operator required by the data processing request;

[0129] Alternatively, the target operator is determined based on a function name carried in the data processing request.

[0130] The processor 602 is configured to determine, according to the time range specified in the data processing request, the routing information corresponding to the time range in the routing table;

[0131] The storage node corresponding to the target timeline and the number of storage nodes are determined based on the routing information.

[0132] The processor 602 is configured to push the target operator in the machine learning model to be trained to the storage node;

[0133] The machine learning model to be trained is trained using the target time series data in the timeline stored in the storage node.

[0134] The processor 602 is configured to determine a target timeline corresponding to each timing identifier based on at least one timing identifier corresponding to each target attribute;

[0135] Determining the storage nodes corresponding to the target timelines;

[0136] The pushing down the target operator to the storage node includes: pushing down the target operator to each of the storage nodes.

[0137] The processor 602 is configured to receive a timing prediction result or an anomaly detection result fed back by at least one of the storage nodes based on the data processing request.

[0138] The processor 602 is configured to receive a target operator in a machine learning model pushed down by a computing node for executing a data processing request; the computing node is configured to determine a corresponding target timeline and a storage node for storing the target timeline based on a timing identifier of a target attribute corresponding to the data processing request;

[0139] Determining target time series data for satisfying the data processing request based on the timeline;

[0140] The target operator is used to perform data processing on the target time series data, and the data processing result is sent to the computing node.

[0141] The memory 601 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0142] Furthermore, the processor 602 in this embodiment may specifically be: a programmable switching processing chip, which is configured with a data replication engine and can replicate received data.

[0143] When executing the program in the memory, the processor 602 can realize other functions in addition to the above functions, and the details can be found in the description of the above embodiments. Further, as shown in FIG6 , the electronic device further includes: a power supply component 604 and other components.

[0144] An embodiment of the present application further provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the corresponding embodiment of FIG. 1 or FIG. 2 .

[0145] Based on the above embodiment, in order to facilitate data management, some databases will store time series data according to the timeline. In some database systems, machine learning models are used to meet their data processing needs. Therefore, in this solution, according to the data processing request for the target attribute, the target operator of the machine learning model built into the database is sent to the storage node storing the timeline corresponding to the target attribute, without the need to transfer the target time series data in the storage node to the computing node, which can effectively reduce the amount of data transmission between nodes and improve computing efficiency. At the same time, when using a machine learning model to process time series data, it is only necessary to select the corresponding target operator according to the data processing request and push the target operator down to the corresponding storage node, without the need to transmit an overly complex machine learning model to meet the data processing needs. At the same time, by pushing the target operators in the same machine learning model to different storage nodes respectively, and having different storage nodes perform the corresponding data processing tasks, multi-storage node parallel processing is achieved, effectively improving data processing efficiency and data processing capabilities.

[0146] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, applied to a computing node, comprising: In response to a data processing request for a target attribute, determining a target operator of a machine learning model built into a database; Based on the timing identifier corresponding to the target attribute, determining a target timeline corresponding to the timing identifier and a storage node storing the target timeline; The target operator is pushed down to the storage node, so that the storage node uses the target operator to execute the data processing request for the target time series data in the target timeline.

2. The method according to claim 1, wherein determining the target timeline corresponding to the timing identifier based on the timing identifier corresponding to the target attribute and the storage node storing the target timeline comprises: Determine a timing identifier corresponding to the target attribute, and determine a corresponding target timeline based on the timing identifier; Determine, according to the timing identifier, a routing table corresponding to the target timeline; The storage nodes and the number of storage nodes corresponding to the target timeline are determined according to the routing information stored in the routing table.

3. The method according to claim 2, wherein pushing the target operator down to the storage node comprises: If the storage node corresponding to the target timeline is found to be the only node, the target operator is pushed down to the storage node.

4. The method according to claim 2, further comprising: If it is found that the target timeline corresponds to multiple storage nodes, the target time series data in the multiple storage nodes corresponding to the same target timeline are received.

5. The method according to claim 1, wherein, in response to a data processing request for a target attribute, determining a target operator of a machine learning model built into a database comprises: In response to a data processing request for a target attribute, determining a target operator type and the target operator required by the data processing request; Alternatively, the target operator is determined based on a function name carried in the data processing request.

6. The method according to claim 2, wherein determining the storage node and the number of storage nodes corresponding to the target timeline according to the routing information stored in the routing table comprises: Determine, according to the time range specified in the data processing request, the routing information corresponding to the time range in the routing table; The storage node corresponding to the target timeline and the number of the storage nodes are determined based on the routing information.

7. According to the method of claim 1, the training method of the machine learning model comprises: Push the target operator in the machine learning model to be trained to the storage node; The target time series data in the timeline stored in the storage node is used to train the machine to be trained. Learning model for training.

8. The method according to claim 1, wherein determining the target timeline corresponding to the timing identifier based on the timing identifier corresponding to the target attribute and the storage node storing the target timeline comprises: Based on the timing identifiers respectively corresponding to at least one of the target attributes, determining the target timelines respectively corresponding to the timing identifiers; Determine the storage nodes corresponding to the target timelines respectively; The pushing down the target operator to the storage node includes: pushing down the target operator to each of the storage nodes.

9. The method according to claim 1, further comprising: Receive a timing prediction result or anomaly detection result fed back by at least one of the storage nodes based on the data processing request.

10. A data processing method, applied to a storage node, the method comprising: Receive the target operator in the machine learning model pushed down by the computing node for executing the data processing request; The computing node is used to determine the corresponding target timeline and the storage node storing the target timeline according to the timing identifier of the target attribute corresponding to the data processing request; Determine target time series data for satisfying the data processing request based on the timeline; The target operator is used to perform data processing on the target time series data, and the data processing result is sent to the computing node.

11. A data processing device, characterized in that: The device comprises: In response to a data processing request for a target attribute, determining a target operator of a machine learning model built into a database; Based on the timing identifier corresponding to the target attribute, determining a target timeline corresponding to the timing identifier and a storage node storing the target timeline; The target operator is pushed down to the storage node, so that the storage node uses the target operator to execute the data processing request for the target time series data in the target timeline.

12. A data processing system, the system comprising: A computing node, configured to execute the method according to any one of claims 1 to 9; A storage node, configured to execute the method according to claim 10.

13. An electronic device comprising a memory and a processor; wherein: The memory is used to store programs; The processor is coupled to the memory, and is used to execute the program stored in the memory, so as to implement the method of any one of claims 1 to 9 above, or to implement the method of claim 10 above.

14. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method as claimed in any one of claims 1 to 9, or the method as claimed in claim 10.

Citation Information

Patent Citations

  • Storage system

    CN114063889A

  • Time series data processing method and system, storage medium and electronic equipment

    CN114722092A

  • Data storage method and device, electronic equipment and computer readable storage medium

    CN114844911A

  • Data processing method, system and device, equipment and medium

    CN116720588A

  • Task execution method and storage device

    US20230124520A1