Data processing method, device, electronic device and non-volatile storage medium

Abstract

The present application discloses a method, device, electronic device and non-volatile storage medium for data processing. The method includes: obtaining first state data generated during the execution of a stream computing job, and obtaining the data volume of the first state data; when the data volume of the first state data is less than the memory size of the storage device, caching the first state data in the memory of the storage device; when the data volume of the first state data is greater than or equal to the memory size of the storage device, caching the first state data in the local disk of the storage device; transferring the first state data stored in the memory and local disk of the storage device to a distributed file system. The present application solves the technical problem of effectively solving the reliance on local disk capabilities for data storage in related technologies, especially the failure of jobs when the disk space is insufficient, and the waste of resources caused by excessive disk allocation.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular, to a method, device, electronic device and non-volatile storage medium for data processing. Background Art

[0002] In the related technologies, the deployment mode, workload mode and hardware improvement of stream computing have all changed dramatically. From the map-reduce era (the early big data processing era marked by the MapReduce programming model, the working nodes were nodes with tight computing-storage coupling) to the cloud native world (Kubernetes, abbreviated as K8s, K8s is an open source container orchestration platform, and containerized deployment on K8s has become the standard). At the same time, hardware improvements, such as a tenfold increase in network bandwidth (from just a few hundred Mbps to 25Gbps, comparable to local solid-state drives (SSDs)), provide a compelling choice for state storage. Fast intranet transmission and low-cost object storage make cloud storage a viable and economical choice. The workload scale of stream computing has also changed dramatically. The state size of a few hundred megabytes, once considered large, is insignificant compared to the terabytes of state often encountered in today's applications.

[0003] Considering these major changes in deployment, hardware, and workloads, the current state access model that relies heavily on local disks is no longer consistent with the future of stream computing in the cloud-native era. The current deployment, hardware, and workloads are not in line with the development prospects of the cloud-native era. Improving the stability of large-scale batch job computing, avoiding job failures when local disk space is insufficient, and avoiding resource waste caused by excessive disk allocation have become urgent issues to be solved.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, device, electronic device and non-volatile storage medium to at least effectively solve the technical problems in the related art that data storage depends on local disk capabilities, especially causing job failure when disk space is insufficient, and waste of resources due to excessive disk allocation.

[0006] According to one aspect of an embodiment of the present application, a method for data processing is provided, including: obtaining first state data generated during the execution of a stream computing job, and obtaining the data volume of the first state data; when the data volume of the first state data is less than the memory size of the storage device, caching the first state data in the memory of the storage device; when the data volume of the first state data is greater than or equal to the memory size of the storage device, caching the first state data in the local disk of the storage device; and transferring the first state data stored in the memory and the local disk of the storage device to a distributed file system.

[0007] In some embodiments of the present application, the method also includes: receiving status read and write requests from different operators, and caching the status read and write requests in a buffer in the memory; when the number of status read and write requests is greater than or equal to a preset number, merging the status read and write requests into a compound read and write request; and sending the compound read and write request to the distributed file system for read and write operations.

[0008] In some embodiments of the present application, a status read and write request includes a status read request and a status write request, and the status read and write requests are merged into a compound read and write request, including: when the number of status read requests is greater than or equal to a first preset number, merging the status read requests into a compound read request, wherein the compound read request is used to batch read multiple key values; sending the compound read request to a distributed file system for a read operation; when the number of status write requests is greater than or equal to a second preset number, merging the status write requests into a compound write request, wherein the compound write request is used to write a preset amount of data; and sending the compound write request to a distributed file system for a write operation.

[0009] In some embodiments of the present application, before sending a compound read request to a distributed file system for a read operation, the method also includes: checking whether the local cache contains the data read by the compound read request; if the local cache contains the data read by the compound read request, reading the data directly from the local cache; if the local cache does not contain the data read by the compound read request, reading the data from the distributed file system.

[0010] In some embodiments of the present application, the method also includes: when the amount of data in the memory reaches the storage upper limit, obtaining the call frequency of the amount of data stored in the memory within a preset time period; storing the first data in the memory whose call frequency within the preset time period is lower than the first threshold in the local disk, and deleting the first data in the memory; obtaining the second data whose call frequency is greater than the second threshold from the distributed file system; and storing the second data in the memory.

[0011] In some embodiments of the present application, the method also includes: when the stream computing job runs normally, creating checkpoints at a preset frequency; obtaining the second state data of all operators in the checkpoint; reading the second state data from the memory and the local disk, and transmitting it to the distributed file system.

[0012] In some embodiments of the present application, the method also includes: in the event of a failure in the stream computing job, reading the target state data of the most recent checkpoint from the distributed file system; reading the target state data back into the memory of the recovered computing node, and restarting the stream computing job.

[0013] According to another aspect of an embodiment of the present application, a data processing device is also provided, including: an acquisition module, used to acquire first state data generated during the execution of a stream computing job, and to acquire the data volume of the first state data; a first cache module, used to cache the first state data in the memory of a storage device when the data volume of the first state data is less than the memory size of the storage device; a second cache module, used to cache the first state data in the local disk of the storage device when the data volume of the first state data is greater than or equal to the memory size of the storage device; and a transmission module, used to transmit the first state data stored in the memory and local disk of the storage device to a distributed file system.

[0014] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a memory for storing program instructions; a processor, connected to the memory, for executing program instructions to implement the following functions: obtaining first state data generated during the execution of a stream computing job, and obtaining the data volume of the first state data; when the data volume of the first state data is less than the memory size of the storage device, caching the first state data in the memory of the storage device; when the data volume of the first state data is greater than or equal to the memory size of the storage device, caching the first state data in the local disk of the storage device; and transferring the first state data stored in the memory and the local disk of the storage device to a distributed file system.

[0015] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned data processing method by running the computer program.

[0016] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, and when the computer instructions are executed by a processor, the above-mentioned data processing method is implemented.

[0017] In an embodiment of the present application, the first state data generated during the execution of the stream computing job is obtained, and the data volume of the first state data is obtained; when the data volume of the first state data is less than the memory size of the storage device, the first state data is cached in the memory of the storage device; when the data volume of the first state data is greater than or equal to the memory size of the storage device, the first state data is cached in the local disk of the storage device; and the first state data stored in the memory of the storage device and the local disk is transferred to the distributed file system. By caching the first state data in the memory of the storage device when the data volume of the first state data is less than the memory size of the storage device, the first state data is cached in the memory of the storage device instead of the local disk, thereby avoiding dependence on the local disk for data storage. When the data volume of the first state data is greater than or equal to the memory size of the storage device, the first state data stored in the memory of the storage device and the local disk is transferred to the distributed file system, achieving the purpose of using the local disk as a supplement to avoid waste of resources due to excessive disk allocation, thereby effectively solving the technical problems of relying on the local disk capacity for data storage in the related art, especially causing job failure when the disk space is insufficient, and waste of resources due to excessive disk allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a data processing method provided in an embodiment of the present application;

[0020] Figure 2 It is a flowchart of a data processing method provided according to an embodiment of the present application;

[0021] Figure 3 A stream computing state storage model diagram provided according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a checkpoint / recovery / rescaling mechanism provided according to an embodiment of the present application;

[0023] Figure 5 It is a structural diagram of a data processing device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0027] The data processing method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a data processing method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0028] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0029] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0032] It should be noted that, in some optional embodiments, the above Figure 1The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0033] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for data processing. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] Figure 2 is a flow chart of a data processing method provided according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0035] Step S202: obtaining first state data generated during the execution of the stream computing job, and obtaining the data volume of the first state data (generated by different operators).

[0036] In order to further improve the efficiency of data processing, state read and write requests from different operators (for example, stream computing operators) can be received and cached in a buffer in the memory; when the number of state read and write requests is greater than or equal to the preset number, the state read and write requests are merged into a composite read and write request; the composite read and write request is sent to a distributed file system (for example, a distributed file system, Distributed File System, referred to as DFS) for read and write operations. By merging read and write requests, the number of accesses to the distributed file system is reduced, network latency is reduced, and data processing efficiency is improved. In scenarios that process high-concurrency data streams, such as social media analysis and real-time log processing, this method can effectively improve the throughput and stability of the system.

[0037] For remote storage, simply scaling I / O (Input / Output) parallelism is not enough. For network I / O operations, especially those involving Remote Procedure Calls (RPCs), a large amount of round-trip time (RTT) is a key factor that significantly affects performance. In contrast, the size of a single I / O operation (i.e., the amount of data transferred over the network each time) has a smaller impact on performance. Therefore, how to effectively merge I / O requests into a single network I / O operation and obtain multiple keys in a single call plays an important role in reducing the overhead of a single interaction and making more efficient use of available bandwidth. This is critical for large-state jobs, as frequent state data acquisition can cause bottlenecks. Classifying state operations and batching adjacent requests can reduce the overhead of a single RPC and utilize network efficiency. State operations can be classified in the following ways: state read and write requests include state read requests and state write requests. There are many ways to merge state read and write requests into a composite read and write request, for example: when the number of state read requests is greater than or equal to the first preset number, merge the state read requests into a composite read request, wherein the composite read request is used to batch read multiple key values; send the composite read request to the distributed file system for read operations; when the number of state write requests is greater than or equal to the second preset number, merge the state write requests into a composite write request, wherein the composite write request is used to write a preset amount of data; send the composite write request to the distributed file system for write operations. This method further reduces the I / O operations of the distributed file system through batch read and write operations, and improves the efficiency of data processing. In scenarios where a large number of small files are processed, such as IoT device data, sensor data, etc., this method can significantly improve the performance of data processing.

[0038] The following is a specific implementation example: When the stream computing job is executed, the operator will frequently send state read and write requests to the state backend. These requests will be collected and temporarily stored in a dedicated request buffer. When the number of requests reaches the preset threshold, or within a predetermined time interval, the system will trigger the generation and sending process of a compound read and write request. The compound request will contain requests from multiple operators. The compound read and write requests will be distinguished according to the request type. The read request will be optimized through a compound read request, and the write request will be processed through a compound write request. The compound read request will collect read requests for multiple key-value pairs to form a batch read request, and the compound write request will collect write requests for multiple data items to form a batch write request, thereby reducing the number of network I / O operations and improving the read and write efficiency of DFS.

[0039] For example, through the request aggregator module, the request aggregator module is used to collect and aggregate state read and write requests, and when the preset conditions are met, the requests are merged into a composite request. A request queue is implemented in the module, which adopts a first-in-first-out or priority queue strategy to temporarily store the received requests. The request aggregator triggers the generation of a composite request based on the request type (read or write) and the preset number (for example, every 50 read requests or every 20 write requests). After the composite request is generated, it is sent to the DFS through the network interface for processing. In the request aggregator module, two submodules are designed, one for the composite of read requests and the other for the composite of write requests. The read request composite module implements a key-value search algorithm for extracting and batch processing multiple key-value pairs from the received read requests. The write request composite module implements a data merging algorithm for extracting and batch processing multiple data items from the write requests. The work of these two submodules will be based on the preset read and write request number threshold to ensure that the size and frequency of the composite request are moderate and will not have a negative impact on real-time data processing.

[0040] Before sending a compound read request to the distributed file system for a read operation, it is necessary to check whether the local cache contains the data read by the compound read request; if the local cache contains the data read by the compound read request, the data is read directly from the local cache; if the local cache does not contain the data read by the compound read request, the data is read from the distributed file system. By checking the local cache first, unnecessary network transmission is reduced, further improving the efficiency of data reading. In scenarios where the same data needs to be accessed frequently, such as recommendation systems and advertising, this method can significantly reduce latency and improve user experience.

[0041] The following is a specific implementation example: In the composite read request processing flow, when a composite read request is generated, the request processor first queries the local cache to obtain the metadata of the requested data. If the data exists in the local cache, the data is read directly; if the data does not exist, the request processor reads the data from the DFS and fills it into the local cache to facilitate the rapid processing of subsequent requests. In this process, the system needs to implement an intelligent cache replacement strategy to ensure that the data in the local cache is the most commonly used and avoid a decrease in the cache hit rate.

[0042] Step S204: When the data volume of the first state data is smaller than the memory size of the storage device, cache the first state data in the memory of the storage device.

[0043] Step S206: When the data volume of the first state data is greater than or equal to the memory size of the storage device, cache the first state data in the local disk of the storage device.

[0044] Step S208: The first state data stored in the memory of the storage device and the local disk is transferred to the distributed file system.

[0045] When the amount of data in the memory reaches the storage upper limit, obtain the call frequency of the amount of data stored in the memory within a preset time period; store the first data in the memory whose call frequency within the preset time period is lower than the first threshold in the local disk, and delete the first data in the memory; obtain the second data whose call frequency is greater than the second threshold from the distributed file system; store the second data (for example, a remote SST file) in the memory. Copy the remote SST file to the local disk, and when the corresponding file resides in the local storage, it can be read directly from the local storage, thereby avoiding expensive DFS round trips. This method ensures that the most commonly used data is stored in the memory and improves the efficiency of memory use by dynamically adjusting the storage strategies of the memory and disk. In resource-constrained scenarios, such as edge computing and mobile devices, this method can effectively utilize limited storage resources and improve data processing performance.

[0046] The following is a specific embodiment:

[0047] When the amount of data cached in memory reaches the upper limit, the storage of data between memory and local disk is intelligently adjusted according to the call frequency of status data within a preset time interval (for example, 10 minutes). Data with lower call frequency will be migrated to the local disk, while data with higher call frequency will be loaded into memory to improve data access speed. In the data management module, a cache frequency analyzer is implemented to monitor and count the call frequency of status data. When the amount of data cached in memory is close to the upper limit (greater than the first threshold, for example, 90%), the frequency analyzer will generate a frequency report indicating which data has a call frequency lower than the first threshold and which data has a call frequency higher than the second threshold. Based on the frequency report, the cache policy adjuster will be responsible for migrating low-frequency data from memory to local disk and loading high-frequency data from DFS into memory. In this process, seamless data transfer is ensured while avoiding data inconsistency issues.

[0048] Another optional implementation is hybrid block cache: hybrid block cache divides data into fixed-size blocks, which can be hundreds of KB to several MB in size, depending on data characteristics and access patterns. Each block is regarded as an independent cache unit and can be stored exclusively in memory or local disk, rather than the entire file. In traditional file-based caches, files may exist in both memory and disk, resulting in data redundancy and waste of storage space. Hybrid block cache eliminates this redundancy by ensuring that each data block resides only on one storage level (memory or local disk), while allowing memory and local disk to be regarded as a unified cache level to optimize data access and storage. When cache space is insufficient, it determines which blocks should be moved from memory to disk and which blocks should be loaded from DFS to memory based on the recent use of the blocks, so as to keep the most frequently used or recently accessed data in memory and improve cache hit rate. The organization and management granularity of the cache is reduced to the block level, rather than the traditional file level, so as to more finely control the data cache and improve cache efficiency and system performance. Memory is an expensive and limited resource, while local disks provide large storage space but slow access speed. By optimizing data storage location at the block level, Hybrid Block Cache can more efficiently utilize these two resources, ensuring optimal resource usage. For I / O-intensive jobs, Hybrid Block Cache can significantly improve job execution speed and throughput by reducing remote storage access and network I / O operations.

[0049] When the stream computing job is running normally, checkpoints are created at a preset frequency; the second state data of all operators in the checkpoint is obtained; the second state data is read from the memory and local disk, and transferred to the distributed file system. This method ensures the consistency and integrity of the data and improves the efficiency of data recovery by creating checkpoints regularly. In scenarios where high data availability needs to be guaranteed, such as online transactions and real-time data analysis, this method can effectively prevent data loss and improve system reliability.

[0050] In the event of a stream computing job failure, the target state data of the most recent checkpoint is read from the distributed file system; the target state data is read back to the memory of the restored computing node, and the stream computing job is restarted. This method can quickly restore the job status when a stream computing job fails, ensuring the continuity and reliability of data processing. In scenarios where large-scale real-time data is processed, such as big data analysis and real-time monitoring, this method can effectively reduce fault recovery time and improve the overall performance of the system.

[0051] The following is a specific embodiment:

[0052] When the stream computing job is running normally, a checkpoint is created at a preset frequency (e.g., every 5 minutes) to obtain the current state data of all operators, read the data from the memory and local disk, and transfer it to DFS for persistent storage. The creation of checkpoints will ensure the integrity and consistency of the data, and provide a basis for the recovery and expansion and contraction of the stream computing job. When a stream computing job fails, the system will read the state data of the most recent checkpoint from DFS, quickly restore it to the memory of the computing node, and restart the stream computing job. The fast recovery mechanism will significantly reduce the recovery time of the stream computing job and improve the stability of the system. For example, the checkpoint manager module is used to create checkpoints periodically. The checkpoint manager will work with the state data management module to obtain the current state data of all operators. When creating a checkpoint, the manager will read the state data in the memory and local disk to form a complete state snapshot, and transfer it to DFS through the network for storage. In this process, it is ensured that when the checkpoint is created, the state data will not be accessed and modified by the operator to avoid data inconsistency problems. The recovery processing submodule in the checkpoint manager module, when a stream computing job fails, will read the state data of the most recent checkpoint from DFS, download it to the memory of the computing node through the network, and restore the integrity of the state data. In this process, ensure that the data downloaded from DFS is consistent with the data at the time of the checkpoint to avoid inconsistency problems during data recovery. The recovery processing submodule will also work with the state data management module to ensure fast data loading and fast restart of operators to minimize the impact of failures.

[0053] In addition, in distributed computing and stream processing systems, recovery time and rescaling efficiency are important indicators for measuring system elasticity and flexibility. Using DFS as the main mechanism for state storage can significantly improve these two aspects. Using DFS as the main storage of state can avoid downloading a large number of state files from remote storage to the local disk during recovery. In the traditional state management model, when the system needs to recover a job, most of the state data must be downloaded from DFS to the local disk and then loaded into the memory by the state backend. This process may lead to long waits, especially when the amount of state data is huge, the download time may become the main bottleneck for recovering the job. In this application, since the state file is already in DFS, the state data can be directly read from DFS and loaded into the memory when the job is recovered, avoiding the time-consuming download process. This means that the job can be started without the local disk state and quickly recovered using the state data in DFS. In addition, the local disk, as an optional cache, can be gradually warmed up after the job is started, that is, the frequently accessed state data is loaded from DFS to the local disk in the background to further optimize performance. This warm-up strategy reduces the remote I / O during subsequent data access, speeds up data processing, and thus improves the overall performance of the job. Re-scaling refers to dynamically adjusting the parallelism of a job at runtime to adapt to resource changes or load requirements. It allows state data to be processed and redistributed directly on DFS without being downloaded to the local disk. When rescaling, the redistribution and merging of state data can be performed directly on DFS, which means that the replication and merging operations of state data are no longer subject to the I / O limitations of the local disk. In addition, since the state data already resides in DFS, the rescaling process can read and write state data directly from DFS, reducing the time and cost of data transmission. This strategy of operating directly on DFS, combined with independent data cleaning and compression mechanisms, can significantly speed up the rescaling process and improve the elasticity and responsiveness of the system.

[0054] In order to optimize resource utilization and improve overall performance, remote compression technology can also be used to separate compression tasks from data processing nodes and delegate them to dedicated LSM (Log-Structured Merge) tree engine worker nodes. This strategy aims to optimize resource utilization and improve overall system performance. In the remote compression strategy, a portion of nodes will be allocated specifically for performing compression tasks. These nodes are called compression nodes, and they have sufficient CPU, memory, and network resources to efficiently execute compression algorithms. Compared with data processing nodes, compression nodes can have more optimized hardware configurations to meet the specific requirements of compression algorithms. In the traditional state management model, data processing nodes may perform data processing and compression tasks at the same time, resulting in resource utilization conflicts. For example, when the data processing task is heavy, the compression task may not get enough resources, thus affecting the overall performance. Remote compression separates resource requirements so that data processing nodes and compression nodes can independently optimize their resource allocation, avoiding resource contention and improving overall resource utilization. The separation of data processing and compression tasks allows data processing nodes to focus on real-time data processing without having to deal with sporadic and resource-intensive compression tasks. This ensures the continuity and efficiency of data processing, avoids performance interruptions and delays caused by compression tasks, and achieves a smoother data processing flow. Compression tasks are isolated to dedicated compression nodes, which means that the main LSM engine (i.e., data processing node) does not need to interrupt its data processing tasks to perform compression. This isolation reduces the load on the data processing nodes, allowing the main LSM engine to better focus on data processing, thereby improving the throughput and responsiveness of data processing. Since compression tasks no longer compete with data processing tasks for resources, performance interruptions of data processing nodes are minimized. Data processing nodes can maintain their peak performance while compression nodes process compression tasks in the background, ensuring efficient data processing and state storage management. Remote compression nodes can perform compression tasks asynchronously, which means that they can perform compression when the data processing nodes are idle, or dynamically adjust the priority of compression tasks according to the overall load of the system. This asynchronous processing method further improves resource utilization efficiency and ensures the stability and responsiveness of the system when processing large amounts of data.

[0055] During the execution of stream computing jobs, the amount of state data generated by each operator is collected and measured in real time or periodically. If the amount of state data is small and can be fully accommodated by the memory cache, the data will be cached directly in the memory of the computing node. However, if the amount of state data is too large and exceeds the storage limit of the memory, the state data will be cached to the local disk of the computing node. The local disk can also exceed the storage limit of the memory and cache the state data to DFS. The role of memory and local disk as cache is to act as a cache, that is, whether it is state data in memory or on the local disk, it will be continuously uploaded to the distributed file system DFS (such as S3, OSS, HDFS, etc.) through the network protocol (for example, HTTP / 2, GRPC, etc.). DFS is used as the primary storage of the state, and the state is continuously transmitted to DFS to ensure data persistence and provide a basis for data redundancy and recovery.

[0056] Through the above steps, the problem that data storage in related technologies depends on local disk capacity, especially causing job failure when disk space is insufficient, and the problem that excessive disk allocation leads to resource waste, can be effectively solved.

[0057] The embodiment of the present application provides a stream computing state storage model diagram, such as Figure 3As shown, the model is presented, where the key components include: Task Managers: responsible for task management. Compute Node: contains task managers. Task: contains operators (OP). State Backend: manages state. Primary Storage: stores primary data. Local Disk: stores state files. Periodically Snapshotting & Dumping States: Checkpoint files (CP Files) are stored through DFS (distributed file system). Stream computing operators read and write states through a component called State Backend. If the state cannot be placed in memory, the state is placed on a file on the local disk. In order to recover or scale after a failure, the state is refreshed to disk, and the state file is periodically dumped to persistent storage to create checkpoints and read back after recovery / scaling. This model relies heavily on the capabilities of the local disk. The right side is an improvement of the model on the left side by the implementation scheme of this application, showing a decentralized model, the difference is: Continuously Uploading State Files: State files are continuously uploaded through DFS (distributed file system). Optional Cache: Store state files. Independent Compaction: Independent data cleaning. Independent Data Cleanup: Independent compression and data cleaning. CP Shareablebetween operators&Queryable: Checkpoint files can be shared between operators and are queryable. In the new model, the local disk is optional as a secondary cache, and the remote DFS is used as the primary storage of the state. Therefore, the remote DFS becomes the real data source. This is a fundamental change from relying on the local disk to a complete computing and storage decomposition model. The state is continuously transferred to DFS; memory and local disk act as cache. This decentralized architecture adopts a cloud-native approach and brings the following advantages: DFS is used as the primary storage, and the local disk is no longer a limitation; checkpoint generation is fast and can be completed within 10 seconds regardless of the size of the state. Continuously transferring state to DFS significantly reduces the amount of data that needs to be written during the checkpoint process, allowing it to complete quickly. As for recovery and scaling, the state files are accessible immediately after the StateBackend is started, without the need for time-consuming local disk downloads. This separated model also helps provide a clean architecture for state sharing and querying. Checkpoint files can be shared between operators to eliminate redundant storage.For status query, a file-level API is provided to support direct query similar to normal tables without relying on backend details or status details.

[0058] A schematic diagram of a checkpoint / recovery / re-scaling mechanism provided in an embodiment of the present application is as follows: Figure 4 As shown, the left side is the mechanism of the current model, including: direct operations between LocalFileSystem (local file system) and RemoteFileSystem (remote file system), such as checkpoint (upload) (checkpoint upload), restore (download) (resuming download) and rescale (download) (re-scaling download). The right side is the mechanism of the decentralized model after the improvement of the left model in the embodiment of the present application, which has the following characteristics: LocalFileSystem (local file system) processes data through cache, such as load (cache) (loading from cache) and flush data (writing to cache). Fast copy / link operations between RemoteFileSystem (remote file system) and LocalFileSystem, such as checkpoint (fast-duplicate / link), restore (fast-duplicate / link) and rescale (fast-duplicate / link).

[0059] Figure 5 is a schematic diagram of a data processing device provided in an embodiment of the present application, such as Figure 5 As shown, including:

[0060] The acquisition module 502 is used to acquire the first state data generated during the execution of the stream computing job, and acquire the data volume of the first state data.

[0061] The first cache module 504 is configured to cache the first state data into the memory of the storage device when the amount of the first state data is smaller than the memory size of the storage device.

[0062] The first cache module 504 is also used to receive status read and write requests from different operators, and cache the status read and write requests in a buffer in the memory; when the number of status read and write requests is greater than or equal to a preset number, the status read and write requests are merged into a compound read and write request; and the compound read and write request is sent to the distributed file system for read and write operations.

[0063] The first cache module 504 is also used to merge the status read requests into a compound read request when the number of status read requests is greater than or equal to a first preset number, wherein the compound read request is used to batch read multiple key values; send the compound read request to the distributed file system for a read operation; merge the status write requests into a compound write request when the number of status write requests is greater than or equal to a second preset number, wherein the compound write request is used to write a preset amount of data; and send the compound write request to the distributed file system for a write operation.

[0064] The first cache module 504 is also used to check whether the local cache contains the data read by the compound read request; if the local cache contains the data read by the compound read request, the data is read directly from the local cache; if the local cache does not contain the data read by the compound read request, the data is read from the distributed file system.

[0065] The second cache module 506 is configured to cache the first state data in a local disk of the storage device when the data volume of the first state data is greater than or equal to the memory size of the storage device.

[0066] The transmission module 508 is used to transmit the first state data stored in the memory of the storage device and the local disk to the distributed file system.

[0067] The transmission module 508 is also used to obtain the call frequency of the data stored in the memory within a preset time period when the amount of data in the memory reaches the storage upper limit; store the first data in the memory whose call frequency within the preset time period is lower than the first threshold in the local disk, and delete the first data in the memory; obtain the second data whose call frequency is greater than the second threshold from the distributed file system; and store the second data in the memory.

[0068] The transmission module 508 is also used to create checkpoints at a preset frequency when the stream computing job runs normally; obtain the second state data of all operators in the checkpoint; read the second state data from the memory and the local disk, and transmit it to the distributed file system.

[0069] The transmission module 508 is also used to read the target state data of the most recent checkpoint from the distributed file system when a stream computing job fails; read the target state data back to the memory of the restored computing node, and restart the stream computing job.

[0070] The embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the program instructions to implement the following functions:

[0071] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following data processing method by running the computer program: obtaining first state data generated during the execution of the stream computing job, and obtaining the data volume of the first state data; when the data volume of the first state data is less than the memory size of the storage device, caching the first state data in the memory of the storage device; when the data volume of the first state data is greater than or equal to the memory size of the storage device, caching the first state data in the local disk of the storage device; and transferring the first state data stored in the memory and local disk of the storage device to a distributed file system.

[0072] An embodiment of the present application also provides a computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the data processing method in each embodiment of the present application are implemented.

[0073] The embodiments of the present application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the data processing method in each embodiment of the present application.

[0074] The embodiments of the present application also provide a computer program, which, when executed by a processor, implements the steps of the data processing method in each embodiment of the present application.

[0075] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0076] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0078] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0079] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0080] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0081] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for data processing, characterized in that: include: Acquire first state data generated during the execution of the stream computing job, and acquire the data volume of the first state data; When the amount of the first state data is smaller than the memory size of the storage device, caching the first state data in the memory of the storage device; When the data volume of the first state data is greater than or equal to the memory size of the storage device, caching the first state data in a local disk of the storage device; The first state data stored in the memory of the storage device and the local disk is transferred to a distributed file system.

2. The method according to claim 1, characterized in that The method further comprises: Receive status read and write requests from different operators, and cache the status read and write requests in a buffer in the memory; When the number of the status read / write requests is greater than or equal to a preset number, merging the status read / write requests into a composite read / write request; The composite read and write request is sent to the distributed file system for read and write operations.

3. The method according to claim 2, characterized in that The state read and write request includes a state read request and a state write request, and the state read and write requests are combined into a composite read and write request, including: When the number of the status read requests is greater than or equal to a first preset number, merging the status read requests into a composite read request, wherein the composite read request is used to batch read multiple key values; Sending the composite read request to the distributed file system for a read operation; In a case where the number of the status write requests is greater than or equal to a second preset number, merging the status write requests into a composite write request, wherein the composite write request is used to write a preset amount of data; The composite write request is sent to the distributed file system for a write operation.

4. The method according to claim 3, characterized in that Before sending the composite read request to the distributed file system for a read operation, the method further includes: Check whether the local cache contains the data read by the composite read request; In a case where the local cache contains the data read by the compound read request, directly reading the data from the local cache; When the local cache does not contain the data read by the compound read request, the data is read from the distributed file system.

5. The method according to claim 1, characterized in that The method further comprises: When the amount of data in the memory reaches a storage upper limit, obtaining a call frequency of the amount of data stored in the memory within a preset time period; Store first data in the memory whose calling frequency within the preset time period is lower than a first threshold in the local disk, and delete the first data in the memory; Acquire, from the distributed file system, second data whose calling frequency is greater than a second threshold; The second data is stored in the memory.

6. The method according to claim 1, characterized in that The method further comprises: When the stream computing job is running normally, creating checkpoints at a preset frequency; Acquire second state data of all operators in the checkpoint; The second state data is read out from the memory and the local disk, and transmitted to the distributed file system.

7. The method according to claim 6, characterized in that The method further comprises: In the event of a failure in the stream computing job, reading the target state data of the most recent checkpoint from the distributed file system; The target state data is read back into the memory of the restored computing node, and the stream computing job is restarted.

8. A data processing device, characterized in that: include: An acquisition module, used to acquire first state data generated during the execution of a stream computing job, and to acquire the data volume of the first state data; A first cache module, configured to cache the first state data into a memory of the storage device when the amount of the first state data is smaller than the memory size of the storage device; A second cache module, configured to cache the first state data in a local disk of the storage device when the amount of the first state data is greater than or equal to the memory size of the storage device; The transmission module is used to transmit the first state data stored in the memory of the storage device and the local disk to a distributed file system.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor, connected to the memory, and configured to execute program instructions that implement the following functions: obtaining first state data generated during the execution of a stream computing job, and obtaining the data volume of the first state data; when the data volume of the first state data is less than the memory size of the storage device, caching the first state data in the memory of the storage device; when the data volume of the first state data is greater than or equal to the memory size of the storage device, caching the first state data in the local disk of the storage device; The first state data stored in the memory of the storage device and the local disk is transferred to a distributed file system.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the data processing method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the data processing method described in any one of claims 1 to 7 is implemented.

Images