A data processing method, device and electronic equipment
By providing a data reference path through the central control service platform, the target server node can directly obtain and process the data, which solves the problem of computing pressure on the target server node, improves data processing efficiency, and reduces network transmission and storage management costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2021-12-30
- Publication Date
- 2026-07-14
AI Technical Summary
In existing technologies, the target server node needs to collect and process business data from various business parties in a unified manner, resulting in high computational pressure and low efficiency.
By providing data reference paths associated with target processing rules through the central control service platform, the target server node can directly obtain and process data from the central control service node, reducing the unified transmission and processing steps of data.
It reduces the computational burden on the target server nodes, improves data processing efficiency, and lowers the costs of network transmission and storage management.
Smart Images
Figure CN114327392B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, particularly to the fields of cloud computing and cloud platforms, and further to a data processing method, apparatus, and electronic device. Background Technology
[0002] With the rapid development of internet technology and business, massive amounts of business data are scattered across various business teams.
[0003] Currently, business data from these business teams is collected uniformly into the data center, and customized development supports the processing and calculation of business data. The data output from these calculations is then further processed. Summary of the Invention
[0004] This disclosure provides a data processing method, apparatus, and electronic device.
[0005] In a first aspect, one embodiment of this disclosure provides a data processing method, the method comprising:
[0006] The target server node receives the target processing rules and reference paths sent by the central control service platform, wherein the reference paths are the reference paths of the data associated with the target processing rules;
[0007] The target server node obtains the data based on the reference path, and processes the data based on the data and the target processing rules to obtain the processing result.
[0008] In this embodiment, the central control service node provides a data reference path to the target server node. After receiving the reference path and the target processing rule, the target server node can obtain the data through the reference path, and then perform data processing based on the data and the target processing rule to obtain the processing result. There is no need to transmit the business data of each business party to the target server node in a unified manner through the network in advance, nor is there a need for the target server node to uniformly process and calculate the data of each business party to obtain the data associated with the rule. In this embodiment, the target server node only needs to obtain the reference path of the data associated with the target processing rule from the central control service node. The data associated with the target processing rule can be obtained through the reference path, and then subsequent data processing can be performed to obtain the processing result. In this way, the computational pressure of the target server node can be reduced, thereby improving the data processing efficiency of the target server node.
[0009] Secondly, one embodiment of this disclosure provides a data processing method, the method comprising:
[0010] The central control service platform sends the target processing rules and reference paths to the target server node, where the reference paths are the reference paths of the data associated with the target processing rules;
[0011] The reference path is used by the target server node to obtain the data based on the reference path, and the target processing rule and the data are used by the target server node to process the data and the target processing rule to obtain the processing result.
[0012] Thirdly, one embodiment of this disclosure also provides a data processing apparatus, the apparatus comprising:
[0013] The first receiving module is used to receive the target processing rule and the reference path sent by the central control service platform, wherein the reference path is the reference path of the data associated with the target processing rule;
[0014] The data processing module is used to obtain the data based on the reference path, and to process the data based on the data and the target processing rules to obtain the processing result.
[0015] Fourthly, one embodiment of this disclosure also provides a data processing apparatus, the apparatus comprising:
[0016] The first sending module is used to send the target processing rule and the reference path to the target server node, wherein the reference path is the reference path of the data associated with the target processing rule;
[0017] The reference path is used by the target server node to obtain the data based on the reference path, and the target processing rule and the data are used by the target server node to process the data and the target processing rule to obtain the processing result.
[0018] Fifthly, one embodiment of this disclosure also provides an electronic device, including:
[0019] At least one processor; and
[0020] A memory communicatively connected to the at least one processor; wherein,
[0021] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data processing method provided in the first aspect of this disclosure or the data processing method provided in the second aspect.
[0022] In a sixth aspect, one embodiment of this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the data processing method provided in the first aspect of this disclosure or the data processing method provided in the second aspect.
[0023] In a seventh aspect, one embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method provided in the first aspect or the data processing method provided in the second aspect of this disclosure.
[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0026] Figure 1 This is one of the schematic flowcharts of a data processing method provided in this disclosure;
[0027] Figure 2 This is a second schematic flowchart of a data processing method according to an embodiment of this disclosure;
[0028] Figure 3 This is a schematic diagram of a data processing system according to an embodiment of the present disclosure;
[0029] Figure 4 This is one of the structural diagrams of a data processing apparatus according to an embodiment of the present disclosure;
[0030] Figure 5 This is a second structural diagram of a data processing apparatus according to an embodiment of the present disclosure;
[0031] Figure 6 This is a block diagram of an electronic device used to implement the data processing method of the embodiments of this disclosure. Detailed Implementation
[0032] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0033] like Figure 1 As shown, according to embodiments of this disclosure, this disclosure provides a data processing method, the method comprising:
[0034] Step S101: The target server node receives the target processing rules and reference paths sent by the central control service platform. The reference paths are the reference paths of the data associated with the target processing rules.
[0035] It should be noted that the target processing rule can be understood as a workflow input (written) by the user, which may include processing order and processing logic. The target server node needs to execute the work according to the target processing rule. In addition, it also needs to utilize the data associated with the target processing rule. In this embodiment, the central control service platform only needs to provide the data reference path for the target server node.
[0036] Additionally, it should be noted that target processing rules can be understood as rules entered by the user in the control platform. The control platform provides an input interface where the user can enter target processing rules. The control platform can then transmit the target processing rules to the data storage platform for storage. The central control service platform can monitor the data storage platform, retrieve the target processing rules from the data storage platform, and distribute them to the target server nodes.
[0037] Step S102: The target server node obtains data based on the reference path, and processes the data based on the data and the target processing rules to obtain the processing result.
[0038] After receiving the target processing rules and the reference path, the target server node can obtain the above data based on the reference path, perform data processing based on the data and the target processing rules, and obtain the processing result.
[0039] In this embodiment, the central control service node provides a data reference path to the target server node. After receiving the reference path and the target processing rule, the target server node can obtain the data through the reference path, and then perform data processing based on the data and the target processing rule to obtain the processing result. There is no need to transmit the business data of each business party to the target server node in a unified manner through the network in advance, nor is there a need for the target server node to uniformly process and calculate the data of each business party to obtain the data associated with the rule. In this embodiment, the target server node only needs to obtain the reference path of the data associated with the target processing rule from the central control service node. The data associated with the target processing rule can be obtained through the reference path, and then subsequent data processing can be performed to obtain the processing result. In this way, the computational pressure of the target server node can be reduced, thereby improving the data processing efficiency of the target server node.
[0040] In one embodiment, before the target server node receives the target processing rules and reference path sent by the central control service platform, the method further includes:
[0041] The target server node receives the function computation engine sent by the resource scheduling service platform;
[0042] The target server node establishes a connection with the central control service platform through the Function Compute engine.
[0043] In other words, the resource scheduling service platform schedules server nodes from the distributed server node cluster to perform the above data processing. The target server node can be understood as the node scheduled by the resource scheduling service platform. In one example, the resource scheduling platform can obtain the registration node information from the data storage platform, and schedule the target server node (one or more server nodes) in the distributed server node cluster based on the registration node information. It sends the function computing engine (Data Agent) to the target server node. The function computing engine is used to establish a connection with the central control service platform. After receiving the function computing engine sent by the resource scheduling service platform, the target server node can establish a connection with the central control service platform through the function computing engine. In this way, the central control service platform can know which server nodes are scheduled, i.e., the target server nodes, and can send the target processing rules and reference paths to the target server nodes. It should be noted that the above-mentioned registered node information can be understood as the information of the nodes registered in the distributed server node cluster. The registered node can represent the node to be used, and this information can include IP address, etc. This registered node information can be entered on the control platform, which stores it in the data storage platform. The resource scheduling service platform can obtain the registered node information from the data storage platform, and then schedule the node and distribute the function computing engine. It should be noted that the function computing engine provides a runtime that runs on a single machine. In this implementation, the function computing engine is deployed on the target server node, which can provide a runtime that runs on the target server node.
[0044] In this embodiment, the target server node receives the function computing engine sent by the resource scheduling service platform. In this way, a connection can be established with the central control service platform through the function computing engine to facilitate the subsequent transmission of target processing rules and reference paths.
[0045] In one embodiment, the target server node obtains data based on a reference path, and processes the data according to the data and target processing rules to obtain a processing result, including:
[0046] The target server node obtains data based on the reference path through the Function Compute engine, and then processes the data based on the data and the target processing rules to obtain the processing result.
[0047] In this embodiment, the target server node receives the function computing engine issued by the resource scheduling service platform. After establishing a connection with the central control service platform through the function computing engine, it can receive the reference path and target processing rules issued by the central control service platform. The target server node obtains data based on the reference path through the function computing engine. In this way, the function computing engine can obtain data through the reference path and process the data based on the data and the target processing rules to obtain the processing result. This can improve the data processing efficiency of the target server node.
[0048] In one embodiment, the data processing method further includes at least one of the following:
[0049] The target server node reports the reference path of the processing result to the central control service platform;
[0050] The target server node reports the processing results to the central data service platform.
[0051] After the target server node processes the data and obtains the processing results, it can store the processing results locally and report the reference path of the processing results to the central control service platform. Alternatively, it can report the processing results to the central data service platform for storage, thus improving the flexibility of processing result storage.
[0052] like Figure 2 As shown, this disclosure provides a data processing method according to an embodiment, the method including:
[0053] Step S201: The central control service platform sends the target processing rule and the reference path to the target server node. The reference path is the reference path of the data associated with the target processing rule.
[0054] The reference path is used by the target server node to obtain data based on the reference path, and the target processing rules and data are used by the target server node to process the data and the target processing rules to obtain the processing result.
[0055] In other words, after the target server node receives the target processing rules and the reference path, it can obtain the above data based on the reference path, and perform data processing based on the data and the target processing rules to obtain the processing result.
[0056] In this embodiment, the central control service node provides the target processing rules and data reference paths to the target server node. After receiving the reference path and target processing rules, the target server node can obtain the data through the reference path, and then perform data processing through the data and target processing rules to obtain the processing results. There is no need for the central control service platform to transmit the business data of each business party to the target server node in a unified manner through the network in advance. Therefore, the target server node does not need to process and calculate the data of each business party to obtain the data associated with the rules. In this embodiment, the central control service node provides the target server node with the reference path of the data associated with the target processing rules. In this way, the processing pressure of the central control service platform can be reduced, thereby improving the efficiency of data processing.
[0057] In one embodiment, before the central control service platform sends the target processing rules and reference path to the target server node, the method further includes:
[0058] The central control service platform establishes a connection with the target server node through the function computing engine, which is the resource scheduling service platform that sends data to the target server node.
[0059] In other words, the resource scheduling service platform schedules server nodes from the distributed server node cluster to perform the above data processing. The target server node can be understood as the node scheduled by the resource scheduling service platform. The resource scheduling service platform sends a function computing engine to the target server node. After receiving the function computing engine sent by the resource scheduling service platform, the target server node can establish a connection with the central control service platform through the function computing engine. In this way, the central control service platform and the target server node establish a connection through the function computing engine, and the central control service platform can send the target processing rules and reference paths to the target server node.
[0060] In this embodiment, before the central control service platform sends the target processing rules and reference paths to the target server node, the central control service platform establishes a connection with the target server node through the function computing engine to facilitate the subsequent transmission of the target processing rules and reference paths.
[0061] In one embodiment, the data processing method further includes:
[0062] The reference path for the processing results reported by the target server node received by the central control service platform;
[0063] The central control service platform sends the reference path of the processing result to the data storage platform.
[0064] The central control service platform can collect the application paths of the processing results reported by the target server nodes and report the collected application paths of the processing results to the data storage platform for storage, so that the processing results can be referenced in the future through the reference path of the processing results.
[0065] The process of the above method is described in detail below with a specific embodiment, providing a data processing system that can implement the above data processing method, such as... Figure 3 As shown, the data processing system includes: a control platform, a data storage platform, a resource scheduling service platform, a central control service platform, and a central data service platform.
[0066] The control platform (also known as the orchestration service platform) can receive metadata input from various business parties, as well as target processing rules input from users. The metadata includes at least one of the following:
[0067] Data usage description;
[0068] Data format;
[0069] Data access request methods;
[0070] Registration node information;
[0071] The control platform can transmit received metadata and target processing rules to the data storage platform for storage. These target processing rules can be orchestrated by the user on the control platform, primarily used to coordinate the execution and event observation of multiple distributed data dependencies in the topology. This simplifies development and allows for orchestration in sequential, branching, and parallel modes, forming a directed acyclic computation topology graph. Data participates in topology graph computation according to the aforementioned data format. Furthermore, the control platform can provide observable data computation processes; that is, it can acquire and display the processing results from the function computation engine, showing the bottlenecks and processes of data processing—providing a visual overview to the user. Additionally, the control platform can read data reported by the function computation engine to the central control service platform and, according to predetermined rules, send alarms and display data metrics.
[0072] For the function computation engine, in addition to providing a framework for executing core data computation code, it also manages permissions, obtains results from other data nodes as input, stores data according to a predetermined format, and synchronously outputs the results to the central control service platform. To accommodate the language habits of the various sub-business data production teams, this invention supports multi-language runtimes, providing coding and integration frameworks for commonly used languages such as Java, Python, Go, C#, and shell.
[0073] On the orchestration service side, data users can customize their processing rules by modifying function templates and then deploy them to the sub-regional node cluster under that data node for execution. This requires no intervention or manpower from the platform or data provider.
[0074] The data storage platform can store metadata, target processing rules, and reference paths to data associated with the target processing rules.
[0075] The resource scheduling service platform can obtain registration node information from the data storage platform, schedule target server nodes in the distributed service node cluster based on the registration node information, and send function computing engines to the target server nodes. The target server nodes are used by the central control service platform to implement heartbeat detection through the function computing engine. It should be noted that the resource scheduling service platform can also perform node scheduling scaling. That is, to prevent hotspot issues during node operation in the cluster, the resource scheduling service platform can subscribe to scaling policy configurations set on the control platform. According to the scaling policy, the server node cluster provided by the sub-business data producer is dynamically divided into an allocation pool and a reservation pool, where:
[0076] Allocation pool: Reserved computing nodes (Nodes). The number of Nodes N must be greater than or equal to 1 (N>=1).
[0077] Reserved pool: Used to dynamically expand and shrink when the scaling strategy is triggered concurrently. The number of nodes in the reserved pool is between (1 and N-1), where N>=2.
[0078] The central control service platform can monitor storage changes of metadata and arrangement methods (processing rules) of user data arrangement on the data storage platform through event listening, and load them into the central control service platform in real time, waiting for the Data Agent to establish a connection.
[0079] Specifically, health checks (heartbeat checks) can be performed: the health status of nodes can be checked so that users can adjust orchestration rules and disaster recovery strategies;
[0080] Collectible execution results: This function collects the reference paths to the processing results from the Data Agent and stores them on the data storage platform for display by the control platform. The execution results reported by the Data Agent are reference paths to the data results and do not include the data entities themselves.
[0081] The central data service platform is used to obtain the result processing methods corresponding to pre-configured target processing rules from the data storage platform, and to store the processing results when the result processing method representation is shared. Each sub-business area node can choose to make its data available to designated users or to make it available directly. The forms of access include two types:
[0082] Sub-business area nodes have limited storage capacity and cannot store data results with long historical periods. In such cases, the data can be reported to the central data service platform. At this point, the reported data is not a reference type, but the data itself.
[0083] Sub-business area nodes have storage capabilities and prefer to store their own data while providing a reference method for sharing with the central data service platform. Sub-business area nodes have the right to disable sharing at any time and the control over the time periods for sharing.
[0084] If the data itself is uploaded to the central data service platform, the central data service platform receives the data report from the Data Agent of the current sub-region node and performs merging, summarization, and formatting according to the rules provided by the orchestration service.
[0085] The solution disclosed herein shifts the centralized data collection method to a distributed approach, significantly reducing the cost burden of standardized data center management, collection, reporting, and handling of massive amounts of data. It also achieves good results in areas such as network traffic, link maintenance, storage, and machine computing.
[0086] Furthermore, by dynamically scaling up and down the node cluster in the sub-business area, the computing power of the nodes is guaranteed, and the data orchestration needs of various users are met in a highly efficient collaborative manner.
[0087] By writing and deploying functions, the computing logic of computing nodes can be dynamically upgraded, playing a crucial foundational role in meeting business needs in a timely manner.
[0088] like Figure 4 As shown, according to an embodiment of this disclosure, this disclosure also provides a data processing apparatus 400, the apparatus comprising:
[0089] The first receiving module 401 is used to receive the target processing rule and the reference path sent by the central control service platform. The reference path is the reference path of the data associated with the target processing rule.
[0090] The data processing module 402 is used to obtain data based on the reference path, and to process the data based on the data and the target processing rules to obtain the processing result.
[0091] In one embodiment, the apparatus further includes:
[0092] The second receiving module is used to receive the function calculation engine sent by the resource scheduling service platform before the first receiving module executes the target processing rules and reference paths sent by the central control service platform.
[0093] The first connection module is used to establish a connection with the central control service platform through the function computing engine.
[0094] In one embodiment, data is obtained based on a reference path, and data processing is performed based on the data and target processing rules to obtain a processing result, including:
[0095] Data is obtained through the Function Compute engine based on the reference path, and then processed by the Function Compute engine based on the data and target processing rules to obtain the processing result.
[0096] In one embodiment, the apparatus further includes at least one of the following:
[0097] The first reporting module is used by the target server node to report the reference path of the processing result to the central control service platform;
[0098] The second reporting module is used by the target server node to report the processing results to the central data service platform.
[0099] The data processing apparatus of the above embodiments is an apparatus for implementing the data processing methods of the above embodiments applied to the target server node. The technical features and technical effects are corresponding, and will not be repeated here.
[0100] like Figure 5 As shown, according to an embodiment of this disclosure, this disclosure also provides a data processing apparatus 500, the apparatus comprising:
[0101] The first sending module 501 is used to send the target processing rule and the reference path to the target server node. The reference path is the reference path of the data associated with the target processing rule.
[0102] The reference path is used by the target server node to obtain data based on the reference path, and the target processing rules and data are used by the target server node to process the data and the target processing rules to obtain the processing result.
[0103] In one embodiment, the apparatus further includes:
[0104] The second connection module is used to establish a connection with the target server node through the function computing engine before the first sending module executes the sending of the target processing rules and reference paths to the target server node. The function computing engine is the resource scheduling service platform that sends the data to the target server node.
[0105] In one embodiment, the apparatus further includes:
[0106] The third receiving module is used to receive the reference path of the processing result reported by the target server node;
[0107] The fourth receiving module is used to send the reference path of the processing result to the data storage platform.
[0108] The data processing apparatus of the above embodiments is an apparatus for implementing the data processing methods of the above embodiments applied to the central control service platform. The technical features and technical effects are corresponding, and will not be repeated here.
[0109] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0110] The non-transitory computer-readable storage medium of this disclosure stores computer instructions for causing a computer to perform the data processing methods provided in this disclosure.
[0111] The computer program product of this disclosure includes a computer program that causes a computer to execute the data processing methods provided in the various embodiments of this disclosure.
[0112] Figure 6 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0113] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 606 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0114] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0115] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (I) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware). Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0116] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0117] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.
[0120] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.
[0121] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A data processing method, the method comprising: The target server node receives the function computation engine sent by the resource scheduling service platform; The resource scheduling service platform obtains registration node information from the data storage platform, schedules target server nodes in the distributed service node cluster based on the registration node information, and sends function computing engines to the target server nodes. The resource scheduling service platform subscribes to the scaling policy configuration set on the control platform, and divides the distributed server node cluster provided by the resource scheduling service platform into an allocation pool and a reservation pool based on the configured scaling policy. The distributed server node cluster includes the target server node. The allocation pool is used to reserve computing nodes, and the number of reserved computing nodes is greater than or equal to 1. The reservation pool is used to dynamically expand and shrink when the scaling policy is triggered concurrently. The number of nodes in the reservation pool is between 1 and N-1, where N>=2. The target server node establishes a connection with the central control service platform through the function computing engine; The target server node receives the target processing rules and reference paths sent by the central control service platform. The reference paths are the reference paths of the data associated with the target processing rules. The central control service node monitors the storage changes of the data storage platform regarding the metadata and metadata processing rules for user data orchestration through event listening, and loads them into the central control service platform in real time. The target processing rules are rules obtained by the data users through orchestration, used to coordinate the execution and event observation of multiple distributed data dependencies in the topology, providing sequential, branching, and parallel orchestration methods to form a directed acyclic computation topology graph. Data participates in the topology graph calculation according to the data format in the metadata. The storage changes of the data storage platform regarding the metadata and metadata processing rules for user data orchestration are transmitted by the control platform, which receives the metadata and target processing rules input by each business party and transmits them to the data storage platform. The target server node obtains the data based on the reference path, and processes the data based on the data and the target processing rules to obtain the processing result.
2. The method according to claim 1, wherein, The target server node obtains the data based on the reference path, and processes the data based on the data and the target processing rules to obtain the processing result, including: The target server node obtains the data based on the reference path through the function computing engine, and processes the data based on the data and the target processing rules through the function computing engine to obtain the processing result.
3. The method according to any one of claims 1 to 2, wherein, The method further includes at least one of the following: The target server node reports the reference path of the processing result to the central control service platform; The target server node reports the processing result to the central data service platform.
4. A data processing method, the method comprising: The central control service platform establishes a connection with the target server node through the function computing engine, which is sent by the resource scheduling service platform to the target server node; The resource scheduling service platform obtains registration node information from the data storage platform, schedules target server nodes in the distributed service node cluster based on the registration node information, and sends function computing engines to the target server nodes. The resource scheduling service platform subscribes to the scaling policy configuration set on the control platform, and divides the distributed server node cluster provided by the resource scheduling service platform into an allocation pool and a reservation pool based on the configured scaling policy. The distributed server node cluster includes the target server node. The allocation pool is used to reserve computing nodes, and the number of reserved computing nodes is greater than or equal to 1. The reservation pool is used to dynamically expand and shrink when the scaling policy is triggered concurrently. The number of nodes in the reservation pool is between 1 and N-1, where N>=2. The central control service platform sends target processing rules and reference paths to the target server nodes. The reference paths are the reference paths of the data associated with the target processing rules. The central control service nodes monitor changes in the storage of metadata and metadata processing rules related to user data orchestration on the data storage platform via event listening, and load these changes into the central control service platform in real time. The target processing rules are rules obtained by the data users through orchestration, used to coordinate the execution and event observation of multiple distributed data dependencies in the topology. They provide sequential, branching, and parallel orchestration methods to form a directed acyclic computation topology graph. Data participates in the topology graph computation according to the data format in the metadata. Changes in the storage of metadata and metadata processing rules related to user data orchestration on the data storage platform are transmitted by the control platform. The control platform receives metadata and target processing rules input by each business party and transmits them to the data storage platform. The reference path is used by the target server node to obtain the data based on the reference path, and the target processing rule and the data are used by the target server node to process the data and the target processing rule to obtain the processing result.
5. The method according to claim 4, wherein, The method further includes: The central control service platform receives the reference path of the processing result reported by the target server node; The central control service platform sends the reference path of the processing result to the data storage platform.
6. A data processing apparatus applied to a target server node, the apparatus comprising: The first receiving module is used to receive the target processing rules and reference paths sent by the central control service platform. The reference paths are the reference paths of the data associated with the target processing rules. The central control service node listens to the storage changes of the metadata and metadata processing rules of the data storage platform regarding user data orchestration through event listening, and loads them into the central control service platform in real time. The target processing rules are the rules obtained by the data users through orchestration, which are used to coordinate the execution and event observation of multiple distributed data dependencies in the topology, providing sequential, branching and parallel orchestration methods to form a directed acyclic computation topology graph. The data participates in the topology graph calculation according to the data format in the metadata. The storage changes of the metadata and metadata processing rules of the data storage platform regarding user data orchestration are transmitted by the control platform. The control platform receives the metadata and target processing rules input by each business party and transmits them to the data storage platform. The data processing module is used to obtain the data based on the reference path, and to process the data based on the data and the target processing rules to obtain the processing result; The second receiving module is used to receive the function computing engine sent by the resource scheduling service platform before the first receiving module executes the target processing rules and reference paths sent by the central control service platform. The resource scheduling service platform obtains registration node information from the data storage platform, schedules the target server node in the distributed service node cluster based on the registration node information, and sends the function computing engine to the target server node. The resource scheduling service platform subscribes to the scaling policy configuration set on the control platform and divides the distributed server node cluster provided by the resource scheduling service platform into an allocation pool and a reservation pool based on the configured scaling policy. The distributed server node cluster includes the target server node. The allocation pool is used to reserve computing nodes. The number of reserved computing nodes is greater than or equal to 1. The target server node is a computing node in the allocation pool. The reservation pool is used to dynamically expand and shrink when the scaling policy is triggered concurrently. The number of nodes in the reservation pool is between 1 and N-1, where N>=2. The first connection module is used to establish a connection with the central control service platform through the function computing engine.
7. The apparatus according to claim 6, wherein, The process of obtaining the data based on the reference path and processing the data based on the data and the target processing rules to obtain the processing result includes: The function computing engine obtains the data based on the reference path, and then processes the data based on the data and the target processing rules to obtain the processing result.
8. The apparatus according to any one of claims 6 to 7, wherein, The device further includes at least one of the following: The first reporting module is used by the target server node to report the reference path of the processing result to the central control service platform; The second reporting module is used by the target server node to report the processing result to the central data service platform.
9. A data processing apparatus, the apparatus comprising: The first sending module is used to send target processing rules and reference paths to the target server node. The reference path is the reference path of the data associated with the target processing rule. The central control service node monitors the storage changes of metadata and metadata processing rules related to user data orchestration on the data storage platform through event listening, and loads them into the central control service platform in real time. The target processing rules are rules obtained by the data user through orchestration, used to coordinate the execution and event observation of multiple distributed data dependencies in the topology, providing sequential, branching, and parallel orchestration methods to form a directed acyclic computation topology graph. Data participates in the topology graph calculation according to the data format in the metadata. The storage changes of metadata and metadata processing rules related to user data orchestration on the data storage platform are transmitted by the control platform. The control platform receives metadata and target processing rules input by each business party and transmits them to the data storage platform. The reference path is used by the target server node to obtain the data based on the reference path, and the target processing rule and the data are used by the target server node to process the data and the target processing rule to obtain the processing result. The second connection module is used to establish a connection with the target server node through a function computing engine before the first sending module executes the sending of the target processing rules and reference paths to the target server node. The function computing engine is sent by the resource scheduling service platform to the target server node. The resource scheduling service platform obtains registration node information from the data storage platform, schedules the target server node in the distributed service node cluster based on the registration node information, and sends the function computing engine to the target server node. The resource scheduling service platform subscribes to the scaling policy configuration set on the control platform and divides the distributed server node cluster provided by the resource scheduling service platform into an allocation pool and a reservation pool based on the configured scaling policy. The distributed server node cluster includes the target server node. The allocation pool is used to reserve computing nodes, and the number of reserved computing nodes is greater than or equal to 1. The reservation pool is used to dynamically expand and shrink when the scaling policy is triggered concurrently. The number of nodes in the reservation pool is between 1 and N-1, where N>=2.
10. The apparatus according to claim 9, wherein, The device further includes: The third receiving module is used to receive the reference path of the processing result reported by the target server node; The fourth receiving module is used to send the reference path of the processing result to the data storage platform.
11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data processing method according to any one of claims 1-3 or the data processing method according to any one of claims 4-5.
12. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the data processing method of any one of claims 1-3 or the data processing method of any one of claims 4-5.
13. A computer program product comprising a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-3 or the data processing method according to any one of claims 4-5.