A method and device for data processing

Abstract

The present invention discloses a method and device for data processing, and relates to the field of computer technology. A specific implementation of the method includes: obtaining a data stream to be stored, the data stream including state data and metadata of the state data; saving the state data into a pre-established storage table, the storage table being located outside a data stream processing platform; and in response to an instruction to take a data snapshot of the data stream processing platform, taking a snapshot of the metadata of the state data based on a snapshot mechanism. This implementation realizes the stripping of state data from data stream management, reduces the amount of snapshot data, shortens the snapshot time, and solves the snapshot jamming problem during real-time calculation of massive data, thereby effectively enhancing the robustness, stability, and fault tolerance of real-time data calculation.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data processing. Background Art

[0002] With the rapid development of the big data industry, stateful data computing on data streams is widely used in big data real-time computing scenarios. Take the Flink framework as an example. As an open source computing framework for distributed data stream processing and batch data processing, the Flink framework can be deployed in various cluster environments to achieve fast computing of various data scales. Flink starts the snapshot (Check Point) mechanism to achieve persistence of state data to prevent data loss.

[0003] In the process of implementing the present invention, the inventors found that the prior art has the following problems:

[0004] During real-time computing, state data is stored using existing technology. In real-time computing of massive data, the amount of data processed and the number of states increase sharply, the snapshot time of the data stream processing platform becomes longer, and even tasks become stuck, affecting the performance of the real-time computing program. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method for data processing, which strips state data from the data stream and manages the external storage for persistence of state data, so that the data stream processing platform only needs to perform snapshot processing on the metadata of the state data, thereby reducing the data volume of the data stream processing platform snapshot, shortening the snapshot time, and solving the jamming problem caused by snapshot processing during real-time calculation of massive data, thereby effectively enhancing the robustness, stability and fault tolerance of real-time data calculation.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0007] Acquire a data stream to be stored, the data stream comprising state data and metadata of the state data;

[0008] Saving the state data in a pre-established storage table, where the storage table is located outside the data stream processing platform;

[0009] In response to an instruction to perform a data snapshot on the data stream processing platform, snapshot processing is performed on the metadata of the state data based on a snapshot mechanism.

[0010] Optionally, before saving the status data into a pre-established storage table, it also includes: creating a storage table and configuring the storage table, the configuration including configuring the correspondence between the storage table and the status data, and configuring structured processing rules for the status data; and saving the status data into a pre-established storage table includes: determining the storage table corresponding to the status data based on the correspondence between the storage table and the status data; performing structured processing on the status data based on the structured processing rules, and saving the structured processed status data into the corresponding storage table.

[0011] Optionally, the configuration also includes configuring the data structure of a storage table; saving the structured state data to the corresponding storage table includes: saving the structured state data to a cache; and saving the state data in the cache to the corresponding storage table according to the data structure of the storage table.

[0012] Optionally, the configuration also includes configuring inverse structure processing rules for the status data; and the method also includes: when it is necessary to read the status data, reading the structured status data from the storage table; performing inverse structure processing on the structured status data according to the inverse structure processing rules to obtain the status data.

[0013] Optionally, the structured processing rule and the inverse structured processing rule are implemented based on a data structure of a storage table.

[0014] Optionally, the configuration further includes: configuring the number of copies of the state data storage.

[0015] According to a second aspect of an embodiment of the present invention, there is provided a data processing device, including:

[0016] A data stream acquisition module, used to acquire a data stream to be stored, wherein the data stream includes state data and metadata of the state data;

[0017] A data storage module, used for saving the state data into a pre-established storage table, wherein the storage table is located outside the data stream processing platform;

[0018] The snapshot module is used to respond to an instruction to perform a data snapshot on the data stream processing platform and perform snapshot processing on the metadata of the state data based on a snapshot mechanism.

[0019] Optionally, it also includes a storage table configuration module, which is used to: create a storage table and configure the storage table before saving the status data in a pre-established storage table, wherein the configuration includes configuring the correspondence between the storage table and the status data, and configuring the structured processing rules of the status data; and the data storage module is also used to: determine the storage table corresponding to the status data according to the correspondence between the storage table and the status data; perform structured processing on the status data according to the structured processing rules, and save the structured processed status data to the corresponding storage table.

[0020] According to a third aspect of an embodiment of the present invention, there is provided an electronic device for data processing, characterized in that it includes:

[0021] one or more processors;

[0022] a storage device for storing one or more programs,

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiment of the present invention.

[0024] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0025] An embodiment of the invention has the following advantages or beneficial effects: by obtaining a data stream to be stored, the data stream includes state data and metadata of the state data; saving the state data in a pre-established storage table, the storage table is located outside the data stream processing platform; in response to an instruction to take a data snapshot of the data stream processing platform, a technical solution of performing snapshot processing on the metadata of the state data based on the snapshot mechanism is implemented, thereby realizing the external storage management of stripping the state data from the data stream and persisting the state data, and only snapshot processing is performed on the metadata of the state data, thereby reducing the data volume of the snapshot of the data stream processing platform, shortening the snapshot time, and solving the jamming problem caused by snapshot processing during real-time calculation of massive data, thereby effectively enhancing the robustness, stability and fault tolerance of real-time data calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0027] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;

[0028] Figure 2This is a schematic diagram of the main process of writing status data into a storage table according to an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the main process of reading the status data of the storage table according to an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of the main configuration modules of an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the state data access principle based on Flink in an embodiment of the present invention;

[0032] Figure 6 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0033] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0034] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0036] At present, for the scenario of real-time computing of massive data, the existing technology is used to store state data. Due to the large amount of data processed and the large number of states, the amount of state data stored increases sharply, the snapshot time of the data stream processing platform becomes longer, and even tasks become stuck, which affects the performance of the real-time computing program and cannot meet the actual application requirements well.

[0037] In order to solve the above problems existing in the prior art, the present invention proposes a data processing method, which realizes that the data stream processing platform only needs to perform snapshot processing on the metadata of the state data by stripping the state data from the data stream and the external storage management of the state data persistence, thereby reducing the data volume of the data stream processing platform snapshot, shortening the snapshot time, and solving the jamming problem caused by snapshot processing during real-time calculation of massive data, thereby effectively enhancing the robustness, stability and fault tolerance of real-time data calculation.

[0038] In the introduction to the embodiments of the present invention, the terms and their meanings are as follows:

[0039] Flink: An open source stream processing framework, the core of which is a distributed stream processing engine written in Java and Scala. Flink executes arbitrary data stream processing programs in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute batch and stream processing programs. In addition, the Flink runtime itself also supports the execution of iterative algorithms;

[0040] Hbase - a distributed, column-oriented open source database.

[0041] Job Manager: Job manager, responsible for the scheduling of tasks and resource management of the entire Flink cluster;

[0042] Task Manager: Task manager, responsible for specific task execution and resource application and management of corresponding tasks on each node;

[0043] Check Point mechanism: snapshot mechanism, a lightweight fault-tolerant mechanism.

[0044] Figure 1 is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention, such as Figure 1 As shown, the data processing method of the embodiment of the present invention includes the following steps S101 to S103.

[0045] Step S101: Acquire a data stream to be stored, wherein the data stream includes state data and metadata of the state data.

[0046] Specifically, in the Flink real-time computing scenario, Job Manager coordinates resource allocation according to the front-end tasks and submits the tasks to Task Manager. After Task Manager receives the tasks, the data flow enters the thread of Task Manager for data processing. The data metadata, as data describing the data, coexists with the data. For stateful computing performed by the data stream processing platform, the data flow includes state data and metadata information corresponding to the state data.

[0047] Step S102: Save the state data to a pre-established storage table, where the storage table is located outside the data stream processing platform.

[0048] According to one embodiment of the present invention, before saving the status data into a pre-established storage table, it also includes: creating a storage table, and configuring the storage table, the configuration including configuring the correspondence between the storage table and the status data, and configuring the structured processing rules of the status data; and saving the status data into a pre-established storage table includes: determining the storage table corresponding to the status data according to the correspondence between the storage table and the status data; performing structured processing on the status data according to the structured processing rules, and saving the structured processed status data into the corresponding storage table.

[0049] Specifically, before the state data is saved in a pre-established storage table, it also includes: creating a storage table and configuring the storage table. According to business needs, a database storage table for persistent storage of state data is created outside the data stream processing platform. This storage table can be one table or multiple tables, and the name of the created storage table is defined. Exemplarily, HBase is created outside the data stream processing platform for persistent storage of the state data, and the table name is defined as table_name_1.

[0050] Configure the correspondence between the storage table and the status data based on the created storage table. According to the storage table created above, set the correspondence between the status data and the storage table where it is located. If multiple storage tables are created, it is necessary to set the correspondence between the status data of each storage table and the storage table where it is located. Exemplarily, if all status data are stored in one HBase, the configuration file can set the correspondence between the storage table and the status data to all table_name_1; if the status data is stored in multiple HBases, such as: the order data Order is stored in HBase1 with a table name of Table_order, the amount data Amount is stored in HBase2 with a table name of Table_amount, and the detail data Detail is stored in HBase3 with a table name of Table_detail, then the configuration file can set the correspondence between the storage table and the status data to: Order table_order, Amount table_amount and Detail Table_detail.

[0051] According to another embodiment of the present invention, the configuration also includes configuring the data structure of a storage table; saving the structured state data to the corresponding storage table includes: saving the structured state data to a cache; and saving the state data in the cache to the corresponding storage table according to the data structure of the storage table.

[0052] According to yet another embodiment of the present invention, the structured processing rules are implemented based on a data structure of a storage table.

[0053] Specifically, the data structure of the status data is configured based on the created storage table, such as a data structure for storing Key-Value pairs of string type. According to the data structure in the configured storage table, the structured processing rules of the status data are configured so that the status data is stored in accordance with the data structure of the storage table. Exemplarily, taking the storage of order status data as an example, an order record includes: order number, product ID, order amount, order user ID, and order time. According to the corresponding relationship between the storage table and the status data, the storage table to be stored is determined, and the data structure of the storage table is defined as the value of the order number is the Key, the product ID value, the order amount value, the order user ID value, and the order time value are the Values ​​corresponding to the above Keys. At the same time, the fields are serialized, and these field values ​​are concatenated into a string with commas and converted into a byte array type, that is, the structured processing of the status data according to the structured processing rules is completed.

[0054] According to the structured processing of the state data, the state data after structured processing is saved in the cache, and then the state data in the cache is saved in the corresponding storage table according to the data structure of the storage table to realize the storage of the state data in the storage table. By saving the state data after structured processing in the cache of the data flow processing platform, and then saving the state data in the cache to the storage table after a certain amount of data is met, the number of data exports between the data flow processing platform and the storage table can be reduced, thereby avoiding the jamming problem caused by frequent data exports.

[0055] Figure 2 It is a main flow chart of writing status data into a storage table in an embodiment of the present invention. After receiving a write request for status data, the storage table in which the status data is to be stored is determined based on the correspondence between the storage table and the status data, and then the status data is structured. The data that conforms to the data structure of the storage table after the structured processing is written into the storage table to realize the writing of the status data.

[0056] According to another embodiment of the present invention, the configuration also includes configuring inverse structure processing rules for the status data; and the method also includes: when it is necessary to read the status data, reading the structured status data from the storage table; performing inverse structure processing on the structured status data according to the inverse structure processing rules to obtain the status data.

[0057] According to yet another embodiment of the present invention, the inverse structured processing rule is implemented based on the data structure of the storage table.

[0058] Specifically, the data structure of the state data is configured based on the created storage table, such as a Key-Value pair data structure of a string type. According to the data structure in the configured storage table, the inverse structure processing rules of the state data are configured so that the data stored in the storage table that conforms to the data structure of the storage table is restored to the state data of the data stream for use by the data stream processing platform. Exemplarily, taking the storage of order status data as an example, an order record includes: order number, product ID, order amount, order user ID, order time, and according to the corresponding relationship between the storage table and the state data, the storage table corresponding to the state data is determined, and according to the order number value Key, the corresponding byte array is read, and the byte array is deserialized, and the byte array type is converted to a string type, and the string array is split with commas to restore it to the state data, that is, the inverse structure processing of the structured state data according to the inverse structure processing rules is completed to obtain the state data.

[0059] Figure 3 It is a schematic diagram of the main process of reading the status data of the storage table in an embodiment of the present invention. After receiving a read request for the status data, the storage table is first determined to perform a table read operation, and then the data that conforms to the data structure of the storage table is reversely structured to realize the reading of the status data.

[0060] According to yet another embodiment of the present invention, the configuration further includes: configuring the number of copies of the state data storage.

[0061] In order to provide reliable data management, multiple copies of storage tables outside the data stream processing platform are supported.

[0062] Figure 4 It is a schematic diagram of the main configuration modules of an embodiment of the present invention. Based on the configuration of the correspondence between data and storage tables and the configuration of the copy strategy, combined with the configuration of the data structure of the storage table, the configuration of structured processing rules and inverse structured processing rules is implemented, providing basic support for external storage management of state data.

[0063] Figure 5 Schematic diagram of the state data access principle based on Flink in an embodiment of the present invention. Figure 5As shown, in an embodiment of the present invention, the task manager Task Manage internally constructs a self-operation and maintenance state machine to export and save the state data to a storage table outside the data flow processing platform, and imports the state data in the external storage table into the data flow processing platform. After receiving the input data flow (including the state data and the metadata of the state data), the task manager Task Manage performs operation processing on the data flow through the calculation operator; then, the self-operation and maintenance state machine is called to perform structured processing on the state data in the data flow after the operation processing and save it to an external data table; or the self-operation and maintenance state machine is called to read the state data in the external data table and perform inverse structure processing for the operation processing of the calculation operator. Finally, the metadata of the state data in the data flow can be saved as the output flow of the task manager, and when a data snapshot is required, only a snapshot of the output data flow is required, and there is no need to take a snapshot of the data in the external storage table, thereby greatly reducing the amount of data that needs to be snapshotted. In the specific implementation process, it can be set to save part of the state data in the external storage table according to the needs of business processing, and the system state data that is not paid much attention to is still saved in the data flow processing platform.

[0064] Step S103: in response to an instruction to perform a data snapshot on the data stream processing platform, snapshot processing is performed on the metadata of the state data based on a snapshot mechanism.

[0065] When the data stream processing platform receives the instruction of data stream snapshot, the data stream processing platform only needs to perform snapshot processing on the metadata corresponding to the state data. In the stateful real-time calculation of massive data, the amount of data in the data stream processing platform snapshot is reduced and the snapshot time is shortened.

[0066] Figure 6 Schematic diagram of the main modules of a data processing device according to an embodiment of the present invention. Figure 6 As shown, the data processing device 600 mainly includes a data flow acquisition module 601 , a data storage module 602 and a snapshot module 603 .

[0067] A data stream acquisition module 601 is used to acquire a data stream to be stored, wherein the data stream includes state data and metadata of the state data;

[0068] A data storage module 602, used to save the state data into a pre-established storage table, where the storage table is located outside the data stream processing platform;

[0069] The snapshot module 603 is used to respond to an instruction to perform a data snapshot on the data stream processing platform and perform snapshot processing on the metadata of the state data based on a snapshot mechanism.

[0070] According to an embodiment of the present invention, the data processing device 600 of the embodiment of the present invention may further include a storage table configuration module (not shown in the figure) for:

[0071] Before saving the state data into a pre-established storage table, creating a storage table and configuring the storage table, wherein the configuration includes configuring a corresponding relationship between the storage table and the state data and configuring a structured processing rule for the state data;

[0072] In addition, the data storage module 602 may also be used for:

[0073] According to the correspondence between the storage table and the state data, the storage table corresponding to the state data is determined; the state data is structured according to the structured processing rule, and the structured state data is saved in the corresponding storage table.

[0074] According to another embodiment of the present invention, the configuration also includes a data structure of a configuration storage table; the data storage module 602 can also be used to: save the structured status data into the cache; save the status data in the cache into the corresponding storage table according to the data structure of the storage table.

[0075] According to another embodiment of the present invention, the configuration further includes configuring a reverse structural processing rule of the state data; and the data processing device 600 of the embodiment of the present invention may further include a data reading module (not shown in the figure) for:

[0076] When the state data needs to be read, the structured state data is read from the storage table; and the structured state data is subjected to inverse structure processing according to the inverse structure processing rule to obtain the state data.

[0077] According to yet another embodiment of the present invention, the structured processing rules and the inverse structured processing rules are implemented based on the data structure of the storage table.

[0078] According to yet another embodiment of the present invention, the configuration further includes: configuring the number of copies of the state data storage.

[0079] Figure 7 An exemplary system architecture 700 is shown to which the data processing method or data processing device according to the embodiment of the present invention can be applied.

[0080] like Figure 7As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0081] Users can use terminal devices 701, 702, 703 to interact with server 705 through network 704 to receive or send messages, etc. Terminal devices 701, 702, 703 can be installed with various communication client applications, such as data processing applications, data computing applications, etc. (only examples).

[0082] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0083] The server 705 may be a server that provides various services, such as a background management server (only an example) that supports data processing performed by users using the terminal devices 701, 702, and 703. The background management server may obtain a data stream to be stored, the data stream including state data and metadata of the state data; save the state data to a pre-established storage table, the storage table being located outside the data stream processing platform; in response to an instruction to perform a data snapshot on the data stream processing platform, perform snapshot processing on the metadata of the state data based on a snapshot mechanism, and feed back the processing result (such as snapshot data, etc. - only an example) to the terminal device.

[0084] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the data processing device is generally set in the server 705 .

[0085] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0086] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0087] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0088] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0089] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the functions defined in the system of the present invention are executed.

[0090] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0091] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] The units involved in the embodiments of the present invention may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes: a data stream acquisition module, a data storage module and a snapshot module.

[0093] The names of these modules do not, in certain cases, limit the modules themselves. For example, a data stream acquisition module may also be described as a “module for acquiring a data stream to be stored”.

[0094] On the other hand, the present invention also provides a computer-readable medium, which may be included in the device described in the embodiment; or may exist independently without being assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by a device, the device includes: obtaining a data stream to be stored, the data stream including state data and metadata of the state data; saving the state data to a pre-established storage table, the storage table being located outside the data stream processing platform; and in response to an instruction to take a data snapshot of the data stream processing platform, taking a snapshot of the metadata of the state data based on a snapshot mechanism.

[0095] The technical solution according to the embodiment of the present invention has the following advantages or beneficial effects: by obtaining the data stream to be stored, the data stream includes state data and metadata of the state data; the state data is saved in a pre-established storage table, and the storage table is located outside the data stream processing platform; in response to an instruction to take a data snapshot of the data stream processing platform, a technical solution of performing snapshot processing on the metadata of the state data based on the snapshot mechanism is implemented, which realizes the external storage management of stripping the state data from the data stream and persisting the state data, reduces the amount of data that the data stream processing platform needs to snapshot, shortens the snapshot time, and solves the jamming problem caused by snapshot processing during real-time calculation of massive data, thereby effectively enhancing the robustness, stability and fault tolerance of real-time data calculation.

[0096] The specific implementation methods described herein do not constitute limitations on the scope of protection of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for data processing, characterized in that: include: Acquire a data stream to be stored, the data stream comprising state data and metadata of the state data; Creating a storage table and configuring the storage table, wherein the configuration includes configuring a corresponding relationship between the storage table and the state data and configuring a structured processing rule for the state data; Saving the state data in a pre-established storage table, where the storage table is located outside the data stream processing platform; Wherein, saving the state data into a pre-established storage table comprises: determining the storage table corresponding to the state data according to the corresponding relationship between the storage table and the state data; performing structured processing on the state data according to the structured processing rule, and saving the structured state data into the corresponding storage table; In response to an instruction to perform a data snapshot on the data stream processing platform, snapshot processing is performed on the metadata of the state data based on a snapshot mechanism.

2. The method according to claim 1, characterized in that The configuration also includes a data structure of a configuration storage table; Saving the structured state data to the corresponding storage table includes: Save the structured state data into the cache; The state data in the cache is saved in the corresponding storage table according to the data structure of the storage table.

3. The method according to claim 1, characterized in that The configuration further includes configuring a destructuring processing rule for the state data; Furthermore, the method further comprises: When it is necessary to read the state data, read the structured state data from the storage table; The structured state data is subjected to inverse structure processing according to the inverse structure processing rule to obtain state data.

4. The method according to claim 3, characterized in that The structured processing rules and the inverse structured processing rules are implemented based on the data structure of the storage table.

5. The method according to claim 1, characterized in that The configuration also includes: configuring the number of copies of the state data storage.

6. A data processing device, characterized in that: include: A data stream acquisition module, used to acquire a data stream to be stored, wherein the data stream includes state data and metadata of the state data; A storage table configuration module, used to create a storage table and configure the storage table, wherein the configuration includes configuring a corresponding relationship between the storage table and the state data and configuring a structured processing rule for the state data; A data storage module, used for saving the state data into a pre-established storage table, wherein the storage table is located outside the data stream processing platform; The data storage module is further used to determine the storage table corresponding to the state data according to the corresponding relationship between the storage table and the state data; perform structured processing on the state data according to the structured processing rule, and save the structured state data into the corresponding storage table; The snapshot module is used to respond to an instruction to perform a data snapshot on the data stream processing platform and perform snapshot processing on the metadata of the state data based on a snapshot mechanism.

7. A mobile electronic device terminal, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images