Data processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention relates to a data processing method and device, computer equipment and a storage medium. The method comprises the following steps: configuring a data acquisition module, a data backup module and a data rollback module in Nifi according to a configuration instruction; controlling a data backup module to receive the first data sent by the data acquisition module for backup; and when monitoring that the data transmission state has an error, controlling a data rollback module to perform rollback operation on the first data according to the backed-up second data. Therefore, data in the transmission process can be backed up in real time through the Nifi data stream processing capacity based on a module configured in Nifi, errors or data damage occurring in the data transmission process can be monitored in real time, rollback can be carried out through the backed-up data, the real-time performance of the data transmission and processing process is guaranteed, and the data transmission efficiency is improved. Data asynchronization or errors caused by untimely data processing are avoided, and the consistency of the data processing process can be improved through real-time data backup and rollback operation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of big data technology, data transmission, backup and rollback have become an important part of big data processing systems. The efficiency and reliability of data transmission directly affect the efficiency and accuracy of data processing, while data backup and rollback are key technologies to ensure data security and consistency. Apache Nifi is a powerful data stream processing tool with high efficiency and scalability in data transmission, but in terms of data backup and rollback, existing solutions have not yet realized their potential.

[0003] In the existing data processing process, scheduled backup is often used for data backup, which makes it difficult to adjust the backup frequency in real time according to the real-time changes of data. When rolling back data, manual operation is mostly required, which has many steps, is prone to errors and is inefficient. Therefore, in the process of data transmission and processing, improper control of the timing of backup and rollback can easily lead to data inconsistency. How to improve the consistency of data in the data processing process has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, in order to solve the above technical problems or part of the technical problems, the embodiments of the present invention provide a data processing method, apparatus, computer equipment and storage medium.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0006] When receiving the configuration instruction, configure the data acquisition module, data backup module and data rollback module in Nifi according to the configuration instruction;

[0007] Controlling the data backup module to receive the first data sent by the data acquisition module for backup;

[0008] During the data backup process, controlling the data backup module to monitor the data transmission status of the first data;

[0009] When an error is detected in the data transmission state, the data rollback module is controlled to roll back the first data according to the backed-up second data.

[0010] In a possible implementation, the Nifi is further configured with a data verification module and a data conversion module, and the control of the data backup module to receive the first data sent by the data acquisition module for backup includes:

[0011] Controlling the data conversion module to receive the initial data sent by the data acquisition module, and converting the initial data into the first data in a target format;

[0012] Controlling the data verification module to perform a verification operation on the first data;

[0013] The data backup module is controlled to receive the first data that has been successfully verified, and to back up and store the first data in a data storage module.

[0014] In a possible implementation, when an error occurs in the data transmission state, controlling the data rollback module to roll back the first data according to the backed-up second data includes:

[0015] Obtaining the error type of the data transmission error;

[0016] When the error type is associated with the first data, the data rollback module is controlled to obtain second data corresponding to the first data from the data storage module, so as to perform the rollback operation through the second data.

[0017] In a possible implementation manner, controlling the data rollback module to obtain second data corresponding to the first data from the data storage module includes:

[0018] Control the data rollback module to determine the error level corresponding to the error type, and determine the data version of the first data that needs to be rolled back according to the error level;

[0019] The data rollback module is controlled to determine second data corresponding to the data version from the data storage module.

[0020] In a possible implementation manner, after controlling the data rollback module to roll back the first data according to the backed-up second data, the method further includes:

[0021] When the rollback operation fails, or when an error still occurs in the data transmission state after the rollback operation is performed, determining a previous data version corresponding to the current data version of the second data;

[0022] Determine new second data corresponding to the previous data version;

[0023] The data rollback module is controlled to perform a rollback operation on the first data according to the new second data.

[0024] In a possible implementation, the method further includes:

[0025] When the number of times the rollback operation fails to be executed is greater than a first number threshold, or the number of times the data transmission state still has errors after the rollback operation is executed is greater than a second number threshold, controlling the data verification module to switch the data verification mode, and re-performing a verification operation on each first data according to the switched verification mode;

[0026] The data backup module is controlled to receive new first data that has been successfully verified, and the data storage module is updated according to the new first data.

[0027] In a second aspect, an embodiment of the present invention provides a data processing system, including: a data acquisition module, a data backup module, a data rollback module, a data verification module and a data conversion module configured in Nifi;

[0028] The data acquisition module is used to obtain initial data and send it to the data conversion module;

[0029] The data conversion module is used to convert the initial data into first data in a target format;

[0030] The data verification module is used to verify the first data and send the first data that has been successfully verified to the data backup module;

[0031] The data backup module is used to receive the first data for backing up, and to monitor the data transmission status of the first data;

[0032] The data rollback module is used to roll back the first data according to the backed-up second data when an error is detected in the data transmission state.

[0033] In a third aspect, an embodiment of the present invention provides a data processing device, including:

[0034] A configuration module, configured to configure a data acquisition module, a data backup module, and a data rollback module in Nifi according to the configuration instruction when receiving the configuration instruction;

[0035] A first control module, used for controlling the data backup module to receive the first data sent by the data acquisition module for backup, and monitoring the data transmission status of the first data;

[0036] The second control module is used to control the data rollback module to roll back the first data according to the backed-up second data when an error is detected in the data transmission state.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer device, comprising: a processor and a memory, wherein the processor is used to execute a data processing program stored in the memory to implement the data processing method described in any one of the first aspects above.

[0038] In a fifth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method described in any one of the first aspects above.

[0039] The data processing scheme provided by the embodiment of the present invention, when receiving a configuration instruction, configures a data acquisition module, a data backup module and a data rollback module in Nifi according to the configuration instruction; controls the data backup module to receive the first data sent by the data acquisition module for backup, and monitors the data transmission status of the first data; when an error is detected in the data transmission status, controls the data rollback module to roll back the first data according to the backed-up second data. Thus, based on the module configured in Nifi, the data in the transmission process can be backed up in real time through the Nifi data stream processing capability. During the data transmission process, the errors or data damage that occur can be monitored in real time, and the backed-up data can be rolled back to ensure the real-time performance of the data transmission and processing process, avoid data asynchrony or errors due to untimely data processing, and improve the consistency of the data processing process through real-time data backup and rollback operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a flow chart of another data processing method provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the structure of a data processing system provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention;

[0045] Figure 6 A flowchart of another data processing method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0048] Figure 1 A flow chart of a data processing method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method specifically includes:

[0049] S11. When a configuration instruction is received, a data acquisition module, a data backup module and a data rollback module are configured in Nifi according to the configuration instruction.

[0050] The data processing method provided by the embodiment of the present invention is applied to computer equipment, which may include but is not limited to: servers, desktop computers, smart home devices that need to process data (for example, smart TVs, smart air conditioners, etc.), tablet computers, etc. The processed data may include data used in various business processing flows (for example, working parameters generated during the operation of smart home devices, etc.). Specifically, a module for data processing is configured in Nifi, and the data in the transmission process is backed up in real time through the Nifi data stream processing capability. During the data transmission process, errors or data corruption are monitored in real time, and rollback operations are performed based on the backed-up data.

[0051] In this embodiment, a configuration instruction for configuring Nifi sent by a client or a developer is received, and a data acquisition module, a data backup module and a data rollback module are configured in the processor of Nifi according to the configuration instruction, wherein the data acquisition module is used to collect first data in real time from a preset database through a data collector, and the first data may be data that needs to be processed by conventional business. The data acquisition module may send the first data to the processor for business processing flow, and may also send the first data to the data backup module, and the data backup module is used to back up and store the first data, and the data rollback module is used to roll back the first data when an error occurs during the transmission and processing of the first data.

[0052] S12. Control the data backup module to receive the first data sent by the data acquisition module for backup. During the data backup process, control the data backup module to monitor the data transmission status of the first data.

[0053] In this embodiment, the data backup module is responsible for receiving the first data sent from the data acquisition module and storing it in the data storage module for storing backup data. The purpose of backup is to ensure the persistence and recovery capability of data to prevent data loss or damage.

[0054] When the first data is received, the data backup module will perform a storage operation to copy the data to the data storage module. The backup method may include but is not limited to: full backup or incremental backup, etc., to improve storage efficiency.

[0055] At the same time, during the data backup process, the data backup module will continuously monitor the data transmission status of the first data. The monitoring content may include but is not limited to: Transmission progress: Real-time acquisition of the transmission progress of the backup data to ensure the smooth progress of the backup process. Whether an error occurs during the transmission process: Monitor whether the data is successfully transmitted to the backup storage. If not, the system will promptly trigger an alarm or retry mechanism. If an abnormality such as network interruption or storage error occurs during the data transmission process, the system will automatically record the abnormality log and take corresponding recovery or retry measures.

[0056] S13: When an error is detected in the data transmission state, the data rollback module is controlled to roll back the first data according to the backed-up second data.

[0057] In this embodiment, through the error handling mechanism of Nifi, errors or data corruption in data transmission are detected in real time, and when errors or data corruption occur, the data transmission state is determined to be an error in data processing.

[0058] Specifically, when the data conversion processor cannot process the data, the data is sent to the failure relationship to trigger a retry. Retry Mechanism: configure the number of retries. If the retry still fails, the error handling is triggered, triggering the defined script, such as issuing an early warning or sending an email to the system administrator. At the same time, configure a custom script or processor: use ExecuteScript or a custom processor to capture and handle errors. When the error handling is triggered, the data rollback module is controlled to obtain from the data storage module that the first data with the current error corresponds to the second data stored in the backup. Since the second data is the system state that was successfully backed up before, the first data is restored to a state consistent with the second data, that is, the content in the second data is overwritten or restored to the first data. During the recovery process, the data rollback module will check the difference between the first data and the second data, and only restore the lost or damaged data to ensure that the impact of the rollback on the system is minimized. After the rollback operation is completed, the system will verify to ensure that the first data has been restored to a normal state. If the rollback operation is successful, the system will continue to perform subsequent operations; if the rollback fails, the system will trigger further recovery mechanisms or notify the administrator to intervene manually. Each rollback operation will be recorded in the log, including the time of the rollback, the operator, the data versions before and after the operation, and the reason for the rollback. The log provides a basis for subsequent audits and analysis. This ensures that the system can quickly recover to a stable state when data anomalies occur, minimizes data loss and operational risks, and thus ensures data consistency and system reliability.

[0059] The data processing method provided by the embodiment of the present invention, when receiving a configuration instruction, configures a data acquisition module, a data backup module and a data rollback module in Nifi according to the configuration instruction; controls the data backup module to receive the first data sent by the data acquisition module for backup, and monitors the data transmission status of the first data; when an error is detected in the data transmission status, controls the data rollback module to roll back the first data according to the backed-up second data. Thus, based on the module configured in Nifi, the data in the transmission process can be backed up in real time through the Nifi data stream processing capability. During the data transmission process, the errors or data damage that occur can be monitored in real time, and the backed-up data can be rolled back to ensure the real-time performance of the data transmission and processing process, avoid data asynchrony or errors due to untimely data processing, and improve the consistency of the data processing process through real-time data backup and rollback operations.

[0060] Figure 2 A flow chart of another data processing method provided by an embodiment of the present invention is as follows: Figure 2 As shown, the method specifically includes:

[0061] S21. When a configuration instruction is received, a data acquisition module, a data backup module and a data rollback module are configured in Nifi according to the configuration instruction.

[0062] In this embodiment, similar to step S11, the specific Figure 1 For the sake of brevity, the relevant content will not be elaborated here.

[0063] S22, control the data conversion module to receive the initial data sent by the data acquisition module, and convert the initial data into first data in a target format; control the data verification module to perform verification operations on the first data; control the data backup module to receive the first data that has been successfully verified, and back up the first data and store it in the data storage module.

[0064] In this embodiment, Nifi is also configured with a data verification module and a data conversion module, and the data acquisition module is used to obtain initial data from an external device, a sensor or a database. These data may be original, not in the target format, and need further processing. The initial data is sent to the data conversion module through the data acquisition module. The data conversion module receives the initial data from the data acquisition module and converts it into the target format required by the system (for example, JSON, CSV, SQL, etc.) to obtain the first data, wherein the data conversion may include but is not limited to: data type conversion, format conversion, unit conversion, field mapping and other operations to ensure that the data meets the processing requirements of the target system.

[0065] Furthermore, after receiving the first data, the data verification module performs a verification operation on it. The verification operation may include, but is not limited to, data integrity check, range verification, format verification, etc., to ensure that the first data complies with predetermined rules and no errors or exceptions occur. For example, a processor (ExecuteScript or CustomProcessor, etc.) can be used in Nifi to perform data verification, a script can be written or an external system can be used to verify existing data, and the data consistency between systems can be verified according to the configuration rules customized by the data monitoring platform, such as fluctuation range and data range, and an early warning is issued if the data does not comply with the rules.

[0066] Further, after the data verification module determines that the first data has passed the verification, the data backup module receives the first data that has been successfully verified and backs it up to the data storage module. The backup operation ensures that the system can restore data when needed in the future to avoid data loss or damage. The data storage module can be a storage service such as Minio, Amazon S3, or Google Cloud Storage. In this embodiment, Minio is used for backup storage, and the data storage module can interact with Nifi through Minio's API. Among them, a hash algorithm is used in the backup process to generate a hash value for comparison. Version management is also required when backing up data to save multiple versions of data. Each version is identified by metadata, such as timestamp, version number, unique identifier, etc., which is convenient for screening the corresponding version when data is rolled back, and convenient for data administrators to operate and trace back problems.

[0067] Each module records logs during the data collection, conversion, verification and backup process. The log contains detailed information about the relevant operations, such as data source, conversion rules, verification results, backup status, etc. In addition, if the verification or backup fails, the system will trigger an alarm to notify relevant personnel to intervene manually. This ensures the accuracy, reliability and recoverability of the data, thereby ensuring that the system can run efficiently and stably.

[0068] S23. When an error is detected in the data transmission status, the error type of the data transmission status error is obtained; when the error type is associated with the first data, the data rollback module is controlled to obtain the second data corresponding to the first data from the data storage module, so as to perform a rollback operation through the second data.

[0069] In this embodiment, the data transmission status and data processing status are monitored during the processing and transmission of data. When an error occurs in data transmission or data processing, it is determined whether the error type is associated with the first data, that is, whether a rollback operation is required. If so, a rollback operation is required, wherein the error type may include but is not limited to: Network error: a network interruption or timeout occurs during data transmission. Data format error: the data format does not meet expectations, such as a JSON format error or incorrect business party data. Data integrity error: the checksum or hash value of the data does not match. Storage error: data may fail to be written during the storage process. Custom error: generally, a custom verification data range or script occurs. During the development process, data cleaning personnel may double the data or exceed the data threshold due to code errors, or abnormal fluctuation ranges may occur, causing warning errors.

[0070] Specifically, when detecting errors, you can use Nifi's various processors and custom scripts to detect errors during data transmission and processing. For example:

[0071] - Use the InvokeHTTP handler to detect network errors.

[0072] -Use the ValidateRecord processor to detect data format errors.

[0073] -Use the ExecuteScript processor to calculate the checksum or hash value of the data and detect data integrity errors.

[0074] -Use InvokeHTTP to call the data monitoring system to verify the data using custom rules.

[0075] When an error is detected, the error type and related information are recorded. You can use Nifi's PutDatabaseRecord processor to store the error information in the database, or use the PutFile processor to write the error information to a log file, or use RabbitMQ to send messages to consume custom error handling logic.

[0076] The control data rollback module obtains second data corresponding to the first data from the data storage module, including:

[0077] The data rollback control module determines the error level corresponding to the error type, and determines the data version of the first data that needs to be rolled back according to the error level; the data rollback control module determines the second data corresponding to the data version from the data storage module.

[0078] Specifically, when it is determined that an error occurs, a data rollback operation is performed. If the data monitoring system detects data anomalies and triggers an alarm, it can be selected whether to automatically roll back to the previous version according to the abnormal level, or the data administrator can manually select the version to trigger the rollback logic. Since multiple different versions of the second data can be backed up at different times and different stages of data processing execution during backup, the error levels corresponding to different error types can be pre-set, and the data versions of the second data corresponding to different error levels can be set. The second data of the corresponding data version can be read from the data storage module according to the current error level, and the read second data can be rolled back to the system. Among them, the FetchS3Object processor of Nifi can be used to read the second data from the data storage module, and then the PutDatabaseRecord processor can be used to write the read second data back to the database, or the PutFile processor can be used to write the data to the file system. To realize the rollback operation of the first data according to the read second data. Thus, the function of selecting the corresponding version of the second data for rollback according to the error type and error level can be realized.

[0079] S24. When the rollback operation fails, or an error still occurs in the data transmission status after the rollback operation is performed, determine the previous data version corresponding to the current data version of the second data; determine the new second data corresponding to the previous data version; and control the data rollback module to roll back the first data according to the new second data.

[0080] In this embodiment, after executing the rollback operation, the system will check the data transmission status to ensure that the first data has been successfully restored. If the system detects that there is still an error in the data transmission status or the rollback operation fails (for example, data inconsistency, verification failure, etc.), the previous data version corresponding to the current data version of the second data is determined from the data storage module. This previous data version refers to a valid version before the current second data backup.

[0081] Furthermore, the new second data corresponding to the previous data version is determined. The new second data is a candidate data source for rollback. It is the valid data version that the system should restore to when performing a rollback. The data rollback module will use the new second data to roll back the first data. The new second data will overwrite or restore the first data to ensure that the system is restored to a stable and consistent state. During the rollback process, the system will perform another check to ensure that the data is correct. After the rollback is completed, the data verification module will verify the restored first data to confirm the integrity and correctness of the data. If the data verification is successful, it means that the rollback operation has been successfully completed. If there is still a problem, the system will record the error message again and notify the administrator for further inspection.

[0082] Each rollback operation, including new rollback operations, needs to record detailed information in the log, including the version rolled back, the operator, the reason for failure, and the success / failure status. At the same time, if the rollback operation fails, the system will trigger an alarm mechanism to notify relevant personnel in a timely manner.

[0083] In this embodiment, when the rollback operation fails or the data transmission state still has errors, the system will obtain the data of the previous version from the backup storage and use the data to roll back again to ensure that the system can be restored to a stable state. This process enhances the flexibility of data recovery and ensures data consistency and system stability.

[0084] In one possible implementation, when the number of failed rollback operations is greater than a first count threshold, or the number of errors in the data transmission status after the rollback operation is performed is greater than a second count threshold, the data verification module is controlled to switch the data verification mode, and re-verify each first data according to the switched verification mode; the data backup module is controlled to receive new first data that has been successfully verified, and the data storage module is updated according to the new first data.

[0085] Specifically, when the rollback operation using the new second data of the previous data version fails, the rollback operation can be performed using the second data of other data versions, and the number of failed rollback operations is recorded. When the number of failures is greater than the first number threshold, or, after the rollback operation is successfully executed, when the number of errors in the data transmission status during the data processing process is still greater than the second number threshold, it means that there may be errors in the original first data, and the errors were not verified during the data verification process. Therefore, it is necessary to switch the currently used verification method, and use the newly switched verification method to re-verify the current first data or each first data. After the verification is completed, the new first data that has successfully been verified will be backed up again to the data storage module to update the historically backed-up second data according to the new first data.

[0086] As an example, Figure 6 The flowchart of another data processing method provided by an embodiment of the present invention is shown as follows: 1. Data collection: a data collector is configured in Nifi to collect data from a database in real time, and the collected data is sent to a processor (GetFile processor) for processing. 2. Data format conversion: a data conversion module is configured in the processor (ConvertRecord processor) to convert the collected data into a specified format (such as JSON, CSV, SQL, etc.). 3. Data backup: a data backup module is configured in the processor (PutS3Object processor) to back up the converted data to the Minio object storage. The storage module can interact with Nifi through the Minio API to achieve rapid data storage. 4. Backup verification: in the data backup module, a data verification mechanism is configured to verify the backed-up data to ensure the integrity and correctness of the data. 5. Backup failure reminder: in the backup module, a message reminder module is configured. When the data backup fails, an email, SMS or other message notification method is sent through the processor (PutEmail processor) to promptly notify relevant personnel to handle it. 6. Error detection and rollback: During the data transmission process, Nifi's error handling mechanism is used to detect errors or data corruption in data transmission in real time. When an error or data corruption is detected, a rollback operation is triggered. 7. Version selection and data recovery: In the rollback operation, the corresponding version of data is selected for rollback based on the error type and data corruption. The backup data of the corresponding version is read from Minio and restored to the specified state. 8. Data retransmission: The recovered data is retransmitted to the target system through Nifi's processor. Automated management: Through Nifi's process control capabilities, the data backup and rollback process is automatically managed, the log of each operation is recorded, and the relevant personnel are notified in time for processing through the alarm notification mechanism.

[0087] The data processing method provided in the embodiment of the present invention, by configuring the data backup mechanism of Nifi, uses the data stream processing capability of Nifi to dynamically back up the data in the transmission process in real time, thereby ensuring the security, real-time and integrity of the data. Through Nifi's process control, the data backup and rollback process is automatically managed, manual intervention is reduced, one-key rollback operation is achieved, the rollback process is simplified, and the reliability and stability of the system are improved. Use Minio distributed storage to back up data to ensure that the backup data can be quickly migrated and stored at low cost. During the data transmission process, if an error or data corruption occurs, the pre-backed up data can be rolled back, restored to the specified state and an early warning can be sent to the data administrator. Through Nifi's data flow control mechanism, the timing of the backup and rollback operations is ensured to be accurate, and the consistency and integrity of the data are guaranteed.

[0088] Figure 3 A schematic diagram of a data processing system provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the system specifically includes:

[0089] The data acquisition module 31, data backup module 32, data rollback module 33, data verification module 34 and data conversion module 35 configured in Nifi;

[0090] The data acquisition module is used to obtain initial data and send it to the data conversion module;

[0091] The data conversion module is used to convert the initial data into first data in a target format;

[0092] The data verification module is used to verify the first data and send the first data that has been successfully verified to the data backup module;

[0093] The data backup module is used to receive the first data for backing up, and to monitor the data transmission status of the first data;

[0094] The data rollback module is used to roll back the first data according to the backed-up second data when an error is detected in the data transmission state.

[0095] The data processing system provided in this embodiment can be as follows Figure 3 The system shown in FIG. 1 may execute Figure 1-2 All steps of the data processing method in Figure 1-2 For details on the technical effects of the data processing method shown, please refer to Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0096] Figure 4A structural diagram of a data processing device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the device specifically includes:

[0097] Configuration module 41, configured to configure a data acquisition module, a data backup module and a data rollback module in Nifi according to the configuration instruction when receiving the configuration instruction;

[0098] A first control module 42, used for controlling the data backup module to receive the first data sent by the data acquisition module for backup;

[0099] The first control module is further used to control the data backup module to monitor the data transmission status of the first data during the data backup process;

[0100] The second control module 43 is used to control the data rollback module to roll back the first data according to the backed-up second data when an error is detected in the data transmission state.

[0101] In a possible implementation manner, the first control module is specifically configured to control the data conversion module to receive the initial data sent by the data acquisition module, and convert the initial data into the first data in a target format;

[0102] Controlling the data verification module to perform a verification operation on the first data;

[0103] The data backup module is controlled to receive the first data that has been successfully verified, and to back up and store the first data in a data storage module.

[0104] In a possible implementation manner, the second control module is specifically configured to obtain an error type of an error in the data transmission state;

[0105] When the error type is associated with the first data, the data rollback module is controlled to obtain second data corresponding to the first data from the data storage module, so as to perform the rollback operation through the second data.

[0106] In a possible implementation manner, the second control module is specifically configured to control the data rollback module to determine an error level corresponding to the error type, and determine a data version of the first data that needs to be rolled back according to the error level;

[0107] The data rollback module is controlled to determine second data corresponding to the data version from the data storage module.

[0108] In a possible implementation manner, the second control module is further configured to determine a previous data version corresponding to the current data version of the second data when the rollback operation fails to be executed or when an error still occurs in the data transmission state after the rollback operation is executed;

[0109] Determine new second data corresponding to the previous data version;

[0110] The data rollback module is controlled to perform a rollback operation on the first data according to the new second data.

[0111] In a possible implementation manner, the third control module 44 is used to control the data verification module to switch the data verification mode, and re-perform the verification operation on each first data according to the switched verification mode when the number of times the rollback operation fails to be executed is greater than the first number threshold, or the number of times the data transmission state still has errors after the rollback operation is executed is greater than the second number threshold;

[0112] The data backup module is controlled to receive new first data that has been successfully verified, and the data storage module is updated according to the new first data.

[0113] The data processing device provided in this embodiment may be as follows Figure 4 The device shown in FIG. 1 can perform the following steps: Figure 1-2 All steps of the data processing method in Figure 1-2 For details on the technical effects of the data processing method shown, please refer to Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0114] Figure 5 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention, Figure 5 The computer device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504 and other user interfaces 503. The various components in the computer device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 505 is not described in detail. Figure 5 Various buses are labeled as bus system 505 .

[0115] The user interface 503 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0116] It can be understood that the memory 502 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0117] In some implementations, the memory 502 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 5021 and an application program 5022 .

[0118] The operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 5022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 5022.

[0119] In the embodiment of the present invention, by calling the program or instruction stored in the memory 502, specifically, the program or instruction stored in the application 5022, the processor 501 is used to execute the method steps provided by each method embodiment, for example, including:

[0120] When receiving the configuration instruction, configure the data acquisition module, data backup module and data rollback module in Nifi according to the configuration instruction;

[0121] Controlling the data backup module to receive the first data sent by the data acquisition module for backup;

[0122] During the data backup process, controlling the data backup module to monitor the data transmission status of the first data;

[0123] When an error is detected in the data transmission state, the data rollback module is controlled to roll back the first data according to the backed-up second data.

[0124] In a possible implementation, the data conversion module is controlled to receive the initial data sent by the data acquisition module, and to convert the initial data into the first data in a target format;

[0125] Controlling the data verification module to perform a verification operation on the first data;

[0126] The data backup module is controlled to receive the first data that has been successfully verified, and to back up and store the first data in a data storage module.

[0127] In a possible implementation, obtaining an error type of an error in the data transmission state;

[0128] When the error type is associated with the first data, the data rollback module is controlled to obtain second data corresponding to the first data from the data storage module, so as to perform the rollback operation through the second data.

[0129] In a possible implementation, the data rollback module is controlled to determine an error level corresponding to the error type, and a data version of the first data to be rolled back is determined according to the error level;

[0130] The data rollback module is controlled to determine second data corresponding to the data version from the data storage module.

[0131] In a possible implementation, when the rollback operation fails to be executed, or when an error still occurs in the data transmission state after the rollback operation is executed, determining a previous data version corresponding to the current data version of the second data;

[0132] Determine new second data corresponding to the previous data version;

[0133] The data rollback module is controlled to perform a rollback operation on the first data according to the new second data.

[0134] In a possible implementation, when the number of times the rollback operation fails to execute is greater than a first number threshold, or the number of times the data transmission state still has errors after the rollback operation is executed is greater than a second number threshold, the data verification module is controlled to switch the data verification mode, and the verification operation is re-performed on each first data according to the switched verification mode;

[0135] The data backup module is controlled to receive new first data that has been successfully verified, and the data storage module is updated according to the new first data.

[0136] The method disclosed in the above embodiment of the present invention can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 501 or the instruction in the form of software. The above processor 501 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware.

[0137] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0138] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0139] The computer device provided in this embodiment may be Figure 5 The computer device shown in , can execute Figure 1-2 All steps of the data processing method in Figure 1-2 For details on the technical effects of the data processing method shown, please refer to Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0140] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0141] When one or more programs in the storage medium can be executed by one or more processors, the data processing method executed on the device side can be implemented.

[0142] The processor is used to execute the data processing program stored in the memory to implement the following steps of the data processing method performed on the device side:

[0143] When receiving the configuration instruction, configure the data acquisition module, data backup module and data rollback module in Nifi according to the configuration instruction;

[0144] Controlling the data backup module to receive the first data sent by the data acquisition module for backup;

[0145] During the data backup process, controlling the data backup module to monitor the data transmission status of the first data;

[0146] When an error is detected in the data transmission state, the data rollback module is controlled to roll back the first data according to the backed-up second data.

[0147] In a possible implementation, the data conversion module is controlled to receive the initial data sent by the data acquisition module, and to convert the initial data into the first data in a target format;

[0148] Controlling the data verification module to perform a verification operation on the first data;

[0149] The data backup module is controlled to receive the first data that has been successfully verified, and to back up and store the first data in a data storage module.

[0150] In a possible implementation, obtaining an error type of an error in the data transmission state;

[0151] When the error type is associated with the first data, the data rollback module is controlled to obtain second data corresponding to the first data from the data storage module, so as to perform the rollback operation through the second data.

[0152] In a possible implementation, the data rollback module is controlled to determine an error level corresponding to the error type, and a data version of the first data to be rolled back is determined according to the error level;

[0153] The data rollback module is controlled to determine second data corresponding to the data version from the data storage module.

[0154] In a possible implementation, when the rollback operation fails to be executed, or when an error still occurs in the data transmission state after the rollback operation is executed, determining a previous data version corresponding to the current data version of the second data;

[0155] Determine new second data corresponding to the previous data version;

[0156] The data rollback module is controlled to perform a rollback operation on the first data according to the new second data.

[0157] In a possible implementation, when the number of times the rollback operation fails to execute is greater than a first number threshold, or the number of times the data transmission state still has errors after the rollback operation is executed is greater than a second number threshold, the data verification module is controlled to switch the data verification mode, and the verification operation is re-performed on each first data according to the switched verification mode;

[0158] The data backup module is controlled to receive new first data that has been successfully verified, and the data storage module is updated according to the new first data.

[0159] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0160] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0161] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: When receiving the configuration instruction, configure the data acquisition module, data backup module and data rollback module in Nifi according to the configuration instruction; Controlling the data backup module to receive the first data sent by the data acquisition module for backup; During the data backup process, controlling the data backup module to monitor the data transmission status of the first data; When an error is detected in the data transmission state, the data rollback module is controlled to roll back the first data according to the backed-up second data.

2. The method according to claim 1, characterized in that The Nifi is also configured with a data verification module and a data conversion module, and the control of the data backup module to receive the first data sent by the data acquisition module for backup includes: Controlling the data conversion module to receive the initial data sent by the data acquisition module, and converting the initial data into the first data in a target format; Controlling the data verification module to perform a verification operation on the first data; The data backup module is controlled to receive the first data that has been successfully verified, and to back up and store the first data in a data storage module.

3. The method according to claim 2, characterized in that When an error occurs in the data transmission state, controlling the data rollback module to roll back the first data according to the backed-up second data comprises: Obtaining the error type of the data transmission error; When the error type is associated with the first data, the data rollback module is controlled to obtain second data corresponding to the first data from the data storage module, so as to perform the rollback operation through the second data.

4. The method according to claim 3, characterized in that The controlling the data rollback module to obtain second data corresponding to the first data from the data storage module includes: Control the data rollback module to determine the error level corresponding to the error type, and determine the data version of the first data that needs to be rolled back according to the error level; The data rollback module is controlled to determine second data corresponding to the data version from the data storage module.

5. The method according to claim 3, characterized in that: After controlling the data rollback module to roll back the first data according to the backed-up second data, the method further includes: When the rollback operation fails, or when an error still occurs in the data transmission state after the rollback operation is performed, determining a previous data version corresponding to the current data version of the second data; Determine new second data corresponding to the previous data version; The data rollback module is controlled to perform a rollback operation on the first data according to the new second data.

6. The method according to claim 5, characterized in that The method further comprises: When the number of times the rollback operation fails to be executed is greater than a first number threshold, or the number of times the data transmission state still has errors after the rollback operation is executed is greater than a second number threshold, controlling the data verification module to switch the data verification mode, and re-performing a verification operation on each first data according to the switched verification mode; The data backup module is controlled to receive new first data that has been successfully verified, and the data storage module is updated according to the new first data.

7. A data processing system, characterized in that: include: Configure data acquisition module, data backup module, data rollback module, data verification module and data conversion module in Nifi; The data acquisition module is used to obtain initial data and send it to the data conversion module; The data conversion module is used to convert the initial data into first data in a target format; The data verification module is used to verify the first data and send the first data that has been successfully verified to the data backup module; The data backup module is used to receive the first data for backing up, and to monitor the data transmission status of the first data; The data rollback module is used to roll back the first data according to the backed-up second data when an error is detected in the data transmission state.

8. A data processing device, characterized in that: include: A configuration module, configured to configure a data acquisition module, a data backup module, and a data rollback module in Nifi according to the configuration instruction when receiving the configuration instruction; A first control module, used for controlling the data backup module to receive the first data sent by the data acquisition module for backup, and monitoring the data transmission status of the first data; The second control module is used to control the data rollback module to roll back the first data according to the backed-up second data when an error is detected in the data transmission state.

9. A computer device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a data processing program stored in the memory to implement the data processing method according to any one of claims 1 to 6.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method according to any one of claims 1 to 6.

Images