Data processing method and device, computer equipment and computer readable storage medium
By processing data in the state backend of the computing engine and saving it to an external file, the data consistency problem during abnormal restarts of the computing engine is solved, and complete data sending and receiving is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SENSOR NETWORKS TECH BEIJING CO LTD
- Filing Date
- 2022-10-10
- Publication Date
- 2026-07-21
AI Technical Summary
The existing computing engine cannot guarantee data consistency with external systems when the service restarts abnormally, which may lead to problems such as duplicate or missing data transmission.
The system obtains the data to be processed from the state backend of the computing engine, updates the data state after processing and saves it to an external file, and restores the unsent data state and identifier from the external file in case of an anomaly, thus ensuring data consistency.
This avoids data resending and missing data after the computing engine restarts, ensuring the integrity and consistency of data processing.
Smart Images
Figure CN115658745B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of communication technology, specifically to a data processing method, apparatus, computer equipment, and computer-readable storage medium. Background Technology
[0002] In the field of big data, computing engines such as Flink, MapReduce, and Spark can be used to process data and interact with external systems. Some computing engines have mechanisms in place to ensure data consistency, automatically handling internal data consistency. However, they cannot guarantee consistency for data interacting with external systems. When a computing engine experiences a service failure and restarts, the data state reverts to the state indicated by the checkpoint event. The computing engine will resend the processed data obtained after the data was saved, but will not resend the processed data obtained before the data was saved, leading to problems such as duplicate or missing data sent to external systems. Summary of the Invention
[0003] This application provides a data processing method, apparatus, computer device, and computer-readable storage medium. When the computing engine restarts due to a service anomaly, the data state of the computing engine will be restored to the state of the save point indicated by the checkpoint event. The save data state in the data backend can be updated according to the data state stored in the external file. The target processed data that was obtained before the updated data state and has not been sent to the data receiving end can be determined according to the data identifier stored in the external file. The target processed data can then be sent to the data receiving end to avoid data retransmission and omission after the computing engine restarts.
[0004] This application provides a data processing method, including:
[0005] The data to be processed is obtained through the computing engine, and the data to be processed is processed according to the data status in the state backend of the computing engine to obtain the processed data.
[0006] The data state of the computing engine's state backend is updated based on the processed data to obtain the updated data state.
[0007] Save the processed data and the updated data status to an external file;
[0008] The processed data is sent to the data receiving end that requested the data, and the processed data that has been sent to the data receiving end is identified in the external file;
[0009] When the computing engine experiences a service failure and the data state is restored to the disk data state indicated by the checkpoint event, the stored unidentified target processed data and the target data state corresponding to the target processed data are obtained from the external file.
[0010] The status of the stored data at the back end of the status is updated according to the target data status, and the target processed data is sent to the data receiving end.
[0011] Accordingly, this application also provides a data processing apparatus, including:
[0012] The processing unit is used to obtain the data to be processed through the computing engine, process the data to be processed according to the data status in the status backend of the computing engine, and obtain the processed data.
[0013] An update unit is used to update the data state of the state backend of the computing engine based on the processed data to obtain the updated data state.
[0014] A storage unit is used to save the processed data and the updated data status to an external file;
[0015] The identification unit is used to send the processed data to the data receiving end that requests data, and to identify the processed data that has been sent to the data receiving end in the external file;
[0016] The acquisition unit is used to acquire, from the external file, the stored unidentified target processed data and the target data state corresponding to the target processed data when the computing engine experiences a service exception and the data state is restored to the disk data state indicated by the checkpoint event.
[0017] The sending unit is used to update the stored data status of the status backend according to the target data status, and send the target processed data to the data receiving end.
[0018] In one embodiment, the identification unit includes:
[0019] The first identifier acquisition subunit is used to acquire the data identifier corresponding to the processed data that has been sent to the data receiving terminal;
[0020] The storage subunit is used to store the data identifier in a preset index file in the external file.
[0021] In one embodiment, the identification unit includes:
[0022] The submission subunit is used to submit the processed data to a message queue so that the processed data can be sent to the data receiving end through the message queue;
[0023] The data identification subunit is used to identify the processed data submitted to the message queue in the external file.
[0024] In one embodiment, the transmitting unit includes:
[0025] The second identifier acquisition subunit is used to acquire the data identifier corresponding to the processed data in the message queue.
[0026] The data sending subunit is used to send the target processed data to the message queue if the data identifier in the message queue does not match the data identifier corresponding to the target processed data.
[0027] In one embodiment, the processing unit includes:
[0028] A receiving subunit is used to receive data to be processed and store the data to be processed in a data receiving queue;
[0029] The data acquisition subunit is used to acquire data to be processed from the data receiving queue through the computing engine.
[0030] In one embodiment, the data processing apparatus further includes:
[0031] A location storage unit is used to store the data to be processed at the data read position in the data receiving queue to the external file;
[0032] An information acquisition unit is used to acquire the currently pending data and the current data reading position corresponding to the currently pending data from the data receiving queue.
[0033] A location acquisition unit is used to acquire the target data reading location from the external file;
[0034] The data processing unit is used to process the current data to be processed to obtain processed data if the positional relationship between the data reading position and the target data reading position meets a preset condition.
[0035] In one embodiment, the processing unit includes:
[0036] The state subunit is used to obtain the data state corresponding to the data to be processed from the state backend of the computing engine. The data state includes the processed data obtained by processing the historical data to be processed.
[0037] The data processing subunit is used to perform data processing based on the data status and the data to be processed to obtain processed data.
[0038] Accordingly, this application also provides a computer device including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute any of the data processing methods provided in this application.
[0039] Accordingly, embodiments of this application also provide a computer-readable storage medium for storing a computer program, which is loaded by a processor to execute any of the data processing methods provided in embodiments of this application.
[0040] This application embodiment obtains the data to be processed through a computing engine, processes the data to be processed according to the data state in the state backend of the computing engine, and obtains the processed data; updates the data state in the state backend of the computing engine according to the processed data, and obtains the updated data state; saves the processed data and the updated data state to an external file; sends the processed data to the data receiving end that requests data, and marks the processed data sent to the data receiving end in the external file; when the computing engine experiences a service exception and the data state is restored to the disk data state indicated by the checkpoint event, retrieves the stored unmarked target processed data and the target data state corresponding to the target processed data from the external file; updates the disk data state in the state backend according to the target data state, and sends the target processed data to the data receiving end.
[0041] When the computing engine restarts due to a service anomaly, the computing engine's data state will be restored to the save point state indicated by the checkpoint event. Based on the data state stored in the external file, the save data state in the data backend can be updated. Based on the data identifier stored in the external file, the target processed data that was obtained before the updated data state and has not been sent to the data receiver can be determined and sent to the data receiver to avoid data resending and missing after the computing engine restarts. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 This is a flowchart of the data processing method provided in the embodiments of this application;
[0044] Figure 2 This is an interactive schematic diagram of the computing engine provided in an embodiment of this application;
[0045] Figure 3 This is a schematic diagram of the data processing apparatus provided in an embodiment of this application;
[0046] Figure 4 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation
[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0048] This application provides a data processing method, apparatus, computer device, and computer-readable storage medium. The data processing apparatus can be integrated into a computer device, which may be a server or a terminal, etc.
[0049] The terminal may include mobile phones, wearable smart devices, tablets, laptops, personal computers (PCs), and in-vehicle computers, etc.
[0050] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0051] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.
[0052] This embodiment will be described from the perspective of a data processing device, which can be integrated into a computer device, such as a server or a terminal.
[0053] This application provides a data processing method, such as... Figure 1 As shown, the specific process of this data processing method can be summarized as follows:
[0054] 101. Obtain the data to be processed through the computing engine, and process the data to be processed according to the data status in the backend of the computing engine to obtain the processed data.
[0055] Among them, the computing engine can be a data processing engine used for data processing, such as Flink. Flink is a framework for stateful computation processing of unbounded and bounded data streams. It is a relatively efficient data stream processing engine that believes that all data exists in the form of a stream.
[0056] The state backend is a component provided by Flink for managing state. Flink provides three types of state backends: MemoryStateBackend, RocksDBStateBackend, and FsStateBackend. Data state is used in Flink to store intermediate computation results or cached data. Some data processing needs to rely on the intermediate data stored in the state. Such computations that depend on previous or subsequent events are called stateful computations.
[0057] For example, the data to be processed is input into the computing engine. The computing engine obtains the data status in the data backend according to the pre-set program, and processes the data to be processed according to the data status to obtain the processed data corresponding to the data to be processed.
[0058] It is understandable that the state stores the processed data obtained by processing the historical data to be processed. The data state is the intermediate calculation result saved in the state. Processing the data to be processed according to the data state means processing the data to be processed according to the intermediate calculation result saved in the state.
[0059] The data to be processed can be data that needs to be computed in a stateful manner by the computing engine, and the processed data is the data obtained by computing the data to be processed, such as the processing result or intermediate data.
[0060] The step "processing the data to be processed based on the data status in the computing engine's backend to obtain processed data" can specifically include:
[0061] Retrieve the data status corresponding to the data to be processed from the state backend of the computing engine. The data status includes the processed data obtained by processing the historical data to be processed.
[0062] Data processing is performed based on the data status and the data to be processed to obtain the processed data.
[0063] For example, it could be a computer program that processes the data to be processed according to a preset schedule, which obtains the data status that the data to be processed depends on from the state backend of the computing engine, processes the data to be processed according to the data status, and obtains the processed data.
[0064] In some application scenarios, a sudden surge in data volume may occur. This massive influx of data into the computing engine could cause it to crash. To prevent this crash, a message queue can be used to cache the data to be processed. In one embodiment, the step "obtaining the data to be processed through the computing engine" can specifically include:
[0065] Receive the data to be processed and store the data to be processed in the data receiving queue;
[0066] The computing engine retrieves data to be processed from the data receiving queue.
[0067] For example, specifically it can be like this Figure 2 As shown, a message queue (which can be called a data receiving queue for distinction) can be set up between the computing engine and the upstream service. The upstream service is the system that generates the data to be processed, such as the terminal and server that generate the data to be processed. The data to be processed generated by the upstream service is stored in the message queue, and the computing engine retrieves the data to be processed from the message queue.
[0068] 102. Update the data status of the computing engine's status backend based on the processed data to obtain the updated data status.
[0069] For example, specifically, the data state in the state backend of the computing engine can be updated based on the processed data to obtain the updated data state.
[0070] Suppose the task corresponding to processing the data to be processed is to count the total number of times each address is accessed in real time. Then the data to be processed can be a request carrying the access address. The data status in the data backend is the number of times each address is accessed. By analyzing the data to be processed, the accessed address is determined. Based on the data status and the currently accessed address, the processed data, i.e. the number of times each address is accessed, is obtained, and the data status is updated.
[0071] Updating the data status can be done by adding a record of the updated data status in the data backend.
[0072] 103. Save the processed data and the updated data status to an external file.
[0073] External files can be storage files independent of the computing engine, such as the Hadoop Distributed File System (HDFS).
[0074] For example, save the updated data status to an external file.
[0075] 104. Send the processed data to the data receiving end that requested the data, and mark the processed data sent to the data receiving end in an external file.
[0076] The data receiving end is either a terminal or a server that needs to process the data. For example, taking the above example of counting the total number of times each address is accessed, the terminal can obtain the processed data from the computing engine, render it, and display the processed data to visualize the real-time access status of each address.
[0077] Optional, such as Figure 2 As shown, a message queue can be added between the computing engine and downstream services to cache the processed data sent by the computing engine to the data receiver, and to facilitate asynchronous processing of the processed data. Specifically, the step "send the processed data to the data receiver requesting the data, and identify the processed data sent to the data receiver in an external file" includes:
[0078] The processed data is submitted to a message queue so that it can be sent to the data receiving end via the message queue;
[0079] The processed data submitted to the message queue is identified in an external file.
[0080] The processed data is submitted to the message queue. The data receiving end can obtain the processed data by subscribing to the message queue. Once the processed data is submitted to the message queue, it can be sent to the data receiving end that requested the data through the message queue.
[0081] When processed data is sent to a data receiver or submitted to a message queue, it indicates that the processed data has been sent to the device that needs it. If an exception occurs, the processed data does not need to be sent again. Therefore, it is necessary to identify the processed data that has been sent to the data receiver or submitted to the message queue. For example, an identifier can be set for processed data stored in an external file. Assuming that the processed data is stored in the form of a list, there can be a field in the list to indicate whether the processed data has been sent. The processed data can be identified by writing a value or symbol to this field to indicate that it has been sent.
[0082] For example, the processed data can be stored in the form of key-value1-value2, where the key is the data identifier of the processed data, value1 is the processed data, and value2 can indicate whether the processed data has been sent. For example, 0 means it has not been sent and 1 means it has been sent. When the processed data is stored in an external file, the value is 0.
[0083] When the processed data is sent to the data receiving terminal or submitted to the message queue, the computing engine outputs the data identifier of the processed data, and updates the corresponding value2 to 1 according to the data identifier.
[0084] Optionally, data identifiers can be stored. Based on these stored identifiers, processed data that has not yet been sent to the external file can be identified. This is the step "identifying processed data sent to the data receiver in the external file," which can specifically include:
[0085] Obtain the data identifier corresponding to the processed data that has been sent to the data receiving terminal;
[0086] The data identifier is stored in a pre-defined index file in an external file.
[0087] The data identifier can be an identifier corresponding to the data to be processed. The corresponding data to be processed can be uniquely identified based on the data identifier, or the processed data corresponding to the data to be processed can be uniquely identified based on the data identifier.
[0088] The index file can be a pre-set file used to store the data identifiers corresponding to the processed data that has been sent. The processed data can be stored in a different file than the index file.
[0089] For example, if the processed data is sent to the data terminal via a message queue, the processed data is submitted to the message queue, meaning it has not been sent to the data receiving terminal. The data identifier corresponding to the processed data that has been sent to the data receiving terminal can be obtained and stored in a preset index file of an external file.
[0090] 105. When the computing engine experiences a service failure and the data state is restored to the saved data state indicated by the checkpoint event, retrieve the stored unidentified target processed data and the target data state corresponding to the target processed data from the external file.
[0091] Among them, the data state saved on disk can be the data state saved by the checkpoint event. The checkpoint event is a fault tolerance mechanism provided by the computing engine. The computing engine can periodically trigger the checkpoint event. When the checkpoint event is triggered, a snapshot is generated based on the current state of each task in the computing engine and saved. In addition to saving the current state of the task, the checkpoint event can also save the offset of the pending data obtained from the message queue for the current state.
[0092] In Flink's Checkpoint mechanism, when the computing engine sends a service failure and restarts, it can restore the data state in the data backend based on the disk data state saved by the checkpoint event, as well as restore the offset of the data receiving queue. The computing engine obtains the data to be processed based on the restored offset. After the checkpoint event but before the failure, the data to be processed by the computing engine needs to be reprocessed and resent, resulting in data retransmission.
[0093] Processed data is sent to the data receiving terminal via a message queue. The message queue can send processed data in batches. When the CheckPoint saves the state, there may be processed data corresponding to the historical state that has not yet been sent to the data receiving terminal. If a failure occurs at this time, when restoring to the checkpoint state, this part of the processed data will be missed.
[0094] In this embodiment, the external file identifies the processed data that has already been sent. The unsent target processed data and its corresponding data status can be obtained from the external file. Submitting the unsent processed data to a message queue for transmission to the data receiver, or sending it directly to the data receiver, can prevent data retransmission and missed transmission.
[0095] Understandably, in some scenarios, the target data processing and data state are the same.
[0096] 106. Update the status of the stored data in the status backend according to the status of the target data, and send the processed target data to the data receiving end.
[0097] For example, updating the data state that has been restored to the disk data state in the state backend to the target data state can be understood as updating to the latest target data state (the most recently acquired target data state), and submitting the unsent processed data to the message queue to send it to the data receiver, or sending it directly to the data receiver, to ensure data consistency and avoid data retransmission and missing transmission.
[0098] Optionally, when sending the target processed data, the data identifier of the processed data in the message queue can be checked to see if it is the same as the target processed data. If they are the same, the target processed data is not sent to avoid retransmission. That is, in one embodiment, the step "sending the target processed data to the data receiving end" may specifically include:
[0099] Retrieve the data identifier corresponding to the processed data in the message queue;
[0100] If the data identifier in the message queue does not match the data identifier corresponding to the target processed data, the target processed data will be sent to the message queue so that the processed data can be sent to the data receiving end through the message queue.
[0101] For example, the system retrieves the data identifier of the processed data cached in the message queue, compares it with the data identifier of the target processed data, and if they are different, submits the processed data to the message queue for transmission to downstream services. If they are the same, the target processed data is not sent.
[0102] Optionally, the data to be processed can be a single data item or a batch of data. That is, the computing engine receives multiple data items at once and processes the batch of data as a task. The data identifier can be a transaction identifier (transaction ID) used to identify a batch of data received by the computing engine. Each data item in the batch of data has a corresponding sub-identifier, which is used to identify a single data item in the batch of data.
[0103] The computing engine processes each piece of data in the batch data, and the resulting processed data is the batch data. Since the transaction identifier is stored in an external file after the multiple pieces of data in the processed data are submitted to the message queue, if a failure occurs before the batch data is completely submitted to the message queue, the target processed result in the external file contains the processed data of each piece of data with the transaction ID. The sub-identifier corresponding to each piece of data in the message queue can be obtained. If the sub-identifier is different from the sub-identifier in the target processed result, the processed data is sent.
[0104] In one embodiment, after obtaining the data to be processed in step 101, the target data reading position of the data to be processed in the data receiving queue can be stored. When the computing engine sends a fault restart, the data obtained from the data receiving queue can be filtered according to the stored data reading position. That is, after the step "obtaining the data to be processed from the data receiving queue through the computing engine", the data processing method provided in this application embodiment may further include:
[0105] Store the data to be processed at the data receiving queue position in the data receiving queue to an external file;
[0106] After the step of "updating the stored data status of the status backend according to the target data status", the data processing method provided in this application embodiment may further include:
[0107] Retrieve the currently pending data from the data receiving queue, as well as the current data read position corresponding to the currently pending data;
[0108] Obtain the target data read location from an external file;
[0109] If the positional relationship between the data reading position and the target data reading position meets the preset conditions, the current data to be processed will be processed to obtain the processed data.
[0110] The data read position can represent the position of the data to be processed in the data receiving queue. For example, the data read position can be an offset.
[0111] When retrieving data to be processed from the data receiving queue through the computing engine, the data read location of the data to be processed is stored in an external file.
[0112] When the computing engine restarts due to a service failure, it begins retrieving data from the data receiving queue based on the data read position obtained from the checkpoint event recovery. It then retrieves the currently pending data and its corresponding current data read position from the data receiving queue.
[0113] Obtain the target data read position from the external file and use this target data read position as a filter condition. If the current data read position is before the target data read position, it means that the data to be processed has been processed. If the current data read position is after the target data read position, it means that the data to be processed has not been processed.
[0114] If the positional relationship between the data reading position and the target data reading position meets the preset condition, that is, the current data reading position is after the target data reading position, then the current data to be processed is processed to obtain the processed data.
[0115] As can be seen from the above, the embodiments of this application obtain the data to be processed through the computing engine, process the data to be processed according to the data state in the state backend of the computing engine to obtain the processed data; update the data state in the state backend of the computing engine according to the processed data to obtain the updated data state; save the processed data and the updated data state to an external file; send the processed data to the data receiving end that requests data, and mark the processed data sent to the data receiving end in the external file; when the computing engine experiences a service exception and the data state is restored to the disk data state indicated by the checkpoint event, retrieve the stored unmarked target processed data and the target data state corresponding to the target processed data from the external file; update the disk data state in the state backend according to the target data state, and send the target processed data to the data receiving end.
[0116] When the computing engine restarts due to a service anomaly, the computing engine's data state will be restored to the save point state indicated by the checkpoint event. Based on the data state stored in the external file, the save data state in the data backend can be updated. Based on the data identifier stored in the external file, the target processed data that was obtained before the updated data state and has not been sent to the data receiver can be determined and sent to the data receiver to avoid data resending and missing after the computing engine restarts.
[0117] To facilitate better implementation of the data processing method provided in the embodiments of this application, a data processing apparatus is also provided in one embodiment. The meanings of the terms used are the same as in the data processing method described above, and specific implementation details can be found in the description of the method embodiments.
[0118] This data processing device can be integrated into a computer device, such as... Figure 3 As shown, the data processing device may include: a processing unit 301, an updating unit 302, a saving unit 303, an identification unit 304, an acquisition unit 305, and a sending unit 306, as detailed below:
[0119] (1) Processing unit 301: used to obtain the data to be processed through the computing engine, process the data to be processed according to the data status in the backend of the computing engine, and obtain the processed data.
[0120] In one embodiment, the processing unit 301 may include a receiving subunit and a data acquisition subunit, specifically:
[0121] Receiving subunit: Used to receive data to be processed and store the data to be processed in the data receiving queue;
[0122] Data acquisition subunit: Used to acquire data to be processed from the data receiving queue through the computing engine.
[0123] In one embodiment, the processing unit 301 may include a state subunit and a data processing subunit, specifically:
[0124] State sub-unit: Used to obtain the data state corresponding to the data to be processed from the state backend of the computing engine. The data state includes the processed data obtained by processing the historical data to be processed.
[0125] Data processing subunit: Used to process data based on data status and data to be processed, and obtain processed data.
[0126] (2) Update unit 302: used to update the data status of the state backend of the computing engine based on the processed data, and obtain the updated data status.
[0127] (3) Storage unit 303: used to save the processed data and the updated data status to an external file.
[0128] (4) Identification unit 304: used to send processed data to the data receiving end that requests data, and to identify the processed data that has been sent to the data receiving end in an external file.
[0129] In one embodiment, the identification unit 304 may include a first identification acquisition subunit and a storage subunit, specifically:
[0130] First identifier acquisition subunit: used to acquire the data identifier corresponding to the processed data that has been sent to the data receiving terminal;
[0131] Storage sub-unit: Used to store data identifiers in a preset index file in an external file.
[0132] In one embodiment, the identification unit 304 may include a submission subunit and a data identification subunit, specifically:
[0133] Submission sub-unit: Used to submit the processed data to the message queue so that the processed data can be sent to the data receiving end through the message queue;
[0134] Data Identifier Subunit: Used to identify processed data submitted to the message queue in an external file.
[0135] (5) Acquisition unit 305: When the computing engine experiences a service exception and the data state is restored to the disk data state indicated by the checkpoint event, it acquires the stored unidentified target processed data and the target data state corresponding to the target processed data from an external file.
[0136] (6) Sending unit 306: used to update the status of the stored data at the back end of the status according to the status of the target data, and send the target processed data to the data receiving end.
[0137] In one embodiment, the sending unit 306 may include a second identifier acquisition subunit and a data sending subunit, specifically:
[0138] The second identifier acquisition subunit is used to obtain the data identifier corresponding to the processed data in the message queue.
[0139] Data sending subunit: If the data identifier in the message queue does not match the data identifier corresponding to the target processed data, the target processed data will be sent to the message queue.
[0140] In one embodiment, the data processing apparatus may further include a location storage unit, an information acquisition unit, a location acquisition unit, and a data processing unit, specifically:
[0141] Location storage unit: used to store the data read position of the data to be processed in the data receiving queue to an external file;
[0142] Information acquisition unit: used to acquire the currently pending data and the current data reading position corresponding to the currently pending data from the data receiving queue;
[0143] Location acquisition unit: used to obtain the target data read position from an external file;
[0144] Data processing unit: If the positional relationship between the data reading position and the target data reading position meets the preset conditions, the current data to be processed is processed to obtain the processed data.
[0145] As can be seen from the above, in the data processing device of this application embodiment, the processing unit 301 obtains the data to be processed through the computing engine, processes the data to be processed according to the data state in the state backend of the computing engine, and obtains the processed data; the updating unit 302 updates the data state in the state backend of the computing engine according to the processed data, and obtains the updated data state; the saving unit 303 saves the processed data and the updated data state to an external file; the marking unit 304 sends the processed data to the data receiving end that requests data, and marks the processed data sent to the data receiving end in the external file; when the computing engine experiences a service abnormality and the data state is restored to the disk data state indicated by the checkpoint event, the acquisition unit 305 acquires the stored unmarked target processed data and the target data state corresponding to the target processed data from the external file; the sending unit 306 updates the disk data state in the state backend according to the target data state, and sends the target processed data to the data receiving end.
[0146] When the computing engine restarts due to a service anomaly, the computing engine's data state will be restored to the save point state indicated by the checkpoint event. Based on the data state stored in the external file, the save data state in the data backend can be updated. Based on the data identifier stored in the external file, the target processed data that was obtained before the updated data state and has not been sent to the data receiver can be determined and sent to the data receiver to avoid data resending and missing after the computing engine restarts.
[0147] This application also provides a computer device, which can be a terminal or a server, such as... Figure 4 As shown, it illustrates a structural schematic diagram of the computer device involved in the embodiments of this application, specifically:
[0148] The computer device may include components such as a processor 1001 with one or more processing cores, a memory 1002 with one or more computer-readable storage media, a power supply 1003, and an input unit 1004. Those skilled in the art will understand that... Figure 4 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:
[0149] The processor 1001 is the control center of the computer device. It connects various parts of the computer device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1002, and by calling data stored in the memory 1002, thereby providing overall monitoring of the computer device. Optionally, the processor 1001 may include one or more processing cores; preferably, the processor 1001 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and computer programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1001.
[0150] The memory 1002 can be used to store software programs and modules. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, computer programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.
[0151] The computer equipment also includes a power supply 1003 that supplies power to the various components. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 1003 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0152] The computer device may also include an input unit 1004, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0153] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 1001 in the computer device loads the executable files corresponding to the processes of one or more computer programs into the memory 1002 according to the following instructions, and the processor 1001 runs the computer programs stored in the memory 1002 to realize various functions, as follows:
[0154] The data to be processed is obtained through the computing engine, and the data to be processed is processed according to the data status in the backend of the computing engine to obtain the processed data.
[0155] The data status of the computing engine's state backend is updated based on the processed data to obtain the updated data status.
[0156] Save the processed data and the updated data status to an external file;
[0157] Send the processed data to the data receiving end that requested the data, and mark the processed data that has been sent to the data receiving end in an external file;
[0158] When the computing engine experiences a service failure and the data state is restored to the disk data state indicated by the checkpoint event, retrieve the stored unidentified target processed data and the target data state corresponding to the target processed data from the external file.
[0159] Update the stored data status on the backend of the status based on the target data status, and send the processed target data to the data receiving end.
[0160] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0161] As can be seen from the above, the computer device in this embodiment can obtain the data to be processed through the computing engine, process the data to be processed according to the data state in the state backend of the computing engine, and obtain the processed data; update the data state in the state backend of the computing engine according to the processed data to obtain the updated data state; save the processed data and the updated data state to an external file; send the processed data to the data receiving end that requests data, and mark the processed data sent to the data receiving end in the external file; when the computing engine experiences a service exception and the data state is restored to the disk data state indicated by the checkpoint event, retrieve the stored unmarked target processed data and the target data state corresponding to the target processed data from the external file; update the disk data state in the state backend according to the target data state, and send the target processed data to the data receiving end.
[0162] When the computing engine restarts due to a service anomaly, the computing engine's data state will be restored to the save point state indicated by the checkpoint event. Based on the data state stored in the external file, the save data state in the data backend can be updated. Based on the data identifier stored in the external file, the target processed data that was obtained before the updated data state and has not been sent to the data receiver can be determined and sent to the data receiver to avoid data resending and missing after the computing engine restarts.
[0163] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.
[0164] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.
[0165] Therefore, embodiments of this application provide a computer-readable storage medium storing a computer program that can be loaded by a processor to execute any of the data processing methods provided in embodiments of this application.
[0166] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0167] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0168] Since the computer program stored in the computer-readable storage medium can execute any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.
[0169] The data processing method, apparatus, computer device, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A data processing method, characterized in that, include: The data to be processed is obtained through the computing engine, and the data to be processed is processed according to the data status in the state backend of the computing engine to obtain the processed data. The data state of the computing engine's state backend is updated based on the processed data to obtain the updated data state. Save the processed data and the updated data status to an external file; Sending the processed data to the data receiving end that requested the data, and identifying the processed data sent to the data receiving end in the external file, includes: submitting the processed data to a message queue so that the processed data can be sent to the data receiving end through the message queue; and identifying the processed data submitted to the message queue in the external file. When the computing engine experiences a service failure and the data state is restored to the disk data state indicated by the checkpoint event, the stored unidentified target processed data and the target data state corresponding to the target processed data are obtained from the external file. The status of the stored data at the backend of the status is updated according to the target data status, and the target processed data is sent to the data receiving end. The step of sending the target processed data to the data receiving end includes: obtaining the data identifier corresponding to the processed data in the message queue; if the data identifier in the message queue does not match the data identifier corresponding to the target processed data, the target processed data is sent to the message queue so that the processed data can be sent to the data receiving end through the message queue.
2. The method according to claim 1, characterized in that, The step of identifying the processed data sent to the data receiving end in the external file includes: Obtain the data identifier corresponding to the processed data that has been sent to the data receiving terminal; The data identifier is stored in a preset index file in the external file.
3. The method according to claim 1, characterized in that, The process of obtaining the data to be processed through the computing engine includes: Receive the data to be processed and store the data to be processed in the data receiving queue; The computing engine retrieves the data to be processed from the data receiving queue.
4. The method according to claim 3, characterized in that, After obtaining the data to be processed from the data receiving queue through the computing engine, the method further includes: The data to be processed is stored in the external file at the data reading position in the data receiving queue; After updating the stored data status of the status backend according to the target data status, the method further includes: Obtain the currently pending data and the current data reading position corresponding to the currently pending data from the data receiving queue; Obtain the target data read location from the external file; If the positional relationship between the data reading position and the target data reading position meets the preset conditions, then the current data to be processed is processed to obtain the processed data.
5. The method according to any one of claims 1-4, characterized in that, The step of processing the data to be processed based on the data status in the state backend of the computing engine to obtain processed data includes: The data status corresponding to the data to be processed is obtained from the state backend of the computing engine. The data status includes the processed data obtained by processing the historical data to be processed. Based on the data status and the data to be processed, data processing is performed to obtain processed data.
6. A data processing apparatus, characterized in that, include: The processing unit is used to obtain the data to be processed through the computing engine, process the data to be processed according to the data status in the status backend of the computing engine, and obtain the processed data. An update unit is used to update the data state of the state backend of the computing engine based on the processed data to obtain the updated data state. A storage unit is used to save the processed data and the updated data status to an external file; An identification unit is used to send the processed data to the data receiving end requesting data, and to identify the processed data sent to the data receiving end in the external file. This includes: submitting the processed data to a message queue so that the processed data can be sent to the data receiving end through the message queue; and identifying the processed data submitted to the message queue in the external file. The acquisition unit is used to acquire, from the external file, the stored unidentified target processed data and the target data state corresponding to the target processed data when the computing engine experiences a service exception and the data state is restored to the disk data state indicated by the checkpoint event. The sending unit is configured to update the stored data status of the status backend according to the target data status, and send the target processed data to the data receiving end. The step of sending the target processed data to the data receiving end includes: obtaining the data identifier corresponding to the processed data in the message queue; if the data identifier in the message queue does not match the data identifier corresponding to the target processed data, then sending the target processed data to the message queue so that the processed data can be sent to the data receiving end through the message queue.
7. A computer device, characterized in that, It includes a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to perform the data processing method according to any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which is loaded by a processor to perform the data processing method according to any one of claims 1 to 5.