Data processing method and system based on cloud platform

Abstract

The invention discloses a data processing method and system based on a cloud platform, and the method comprises the following steps: S1, receiving data from a client through the cloud platform, and carrying out the recognition processing of the data; s2, inputting the identified data into a virtual environment for operation simulation, and judging whether the data is damaged or not; s2.1, if the data is not damaged, classifying the data, and creating a storage partition to store the data according to a classification result; according to the data processing method designed by the invention, the data can be identified and processed while the data is accessed, and then whether the data is missing and damaged is judged according to the result of running simulation of the data in the virtual environment, so that the damaged data is prevented from being transmitted to the cloud platform and occupying the storage space of the cloud platform; and meanwhile, loss of the user due to the fact that useless data are stored in the cloud platform can be avoided, and missing and damaged data can be repaired or new data are requested to be generated by the client again, so that the integrity of the data stored in the cloud platform is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud platform data processing, and in particular to a data processing method and system based on a cloud platform. Background Art

[0002] With the advent of the Internet big data era, cloud computing, as a new computing model, delivers computing, storage and other resources to users through cloud platforms in the form of services, allowing users to access the computing and storage resources of the cloud platform on demand through the Internet to run applications. However, due to network latency and other issues, data stored in the cloud platform may be damaged.

[0003] For example, a data processing method and system based on a cloud platform with an authorization announcement number of CN202211300036.2 and a publication date of 20230131, including: classifying and identifying cloud platform data; conducting cloud platform data information analysis and decision-making processing; processing and controlling risk monitoring results, wherein the classification and identification processing of cloud platform data includes: classifying the data services stored on the cloud platform into three categories; after completing the classification of the data services stored on the cloud platform, further identifying the classified business data information to obtain the risk information contained therein; by constructing a hierarchical structure model, evaluating the risk information level of the data service, wherein the cloud platform data information analysis and decision-making processing includes: dividing the risk severity according to the results of the risk assessment; conducting real-time risk monitoring of cloud platform data, and determining the operation and storage status of the cloud platform according to the generated log report. The present invention has the characteristics of high security and strong practicality.

[0004] In order to ensure the security of data in the cloud platform, data security protection and other technologies are used to protect and assess the risks of the data. Although this method can ensure the security of the data, it is unable to detect whether the data is missing or damaged. As a result, although the cloud platform stores the data, the data has become useless data, which will cause great trouble to users. Therefore, it is urgent to design a data processing method and system based on the cloud platform to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a data processing method and system based on a cloud platform to solve the above-mentioned deficiencies in the prior art.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A data processing method based on a cloud platform comprises the following steps:

[0008] S1. The cloud platform receives data from the client and identifies and processes the data;

[0009] S2. Input the identified data into the virtual environment to run the simulation and determine whether the data is damaged;

[0010] S2.1. The data is not damaged, the data is classified and processed, and storage partitions are created based on the classification results to store the data;

[0011] S2.2. If the data is damaged, the damaged data is repaired. After the repair is completed, the data is re-entered into step S2 for simulation processing. If the repair fails, a data request command is sent to the client, and the client resends the data to the cloud platform;

[0012] S3. Classify the data and create storage partitions to store the data based on the classification results;

[0013] S4. Encrypt the stored data and track the flow of data;

[0014] S5. Regularly manage the data in the storage partition, send commands to the client, and delete infrequently used data based on the client feedback results.

[0015] A data processing system based on a cloud platform, comprising an access unit, an identification unit, a virtual unit, a repair unit, a request unit, a security unit, a feedback unit, a storage unit, a tracking unit and a client;

[0016] The output end of the client is communicatively connected to the input end of the access unit, the output end of the access unit is communicatively connected to the input end of the identification unit, the output end of the identification unit is communicatively connected to the input end of the virtual unit, the output end of the virtual unit is communicatively connected to the input end of the repair unit and the input end of the storage unit, the output end of the repair unit is communicatively connected to the input end of the virtual unit and the input end of the request unit, the output end of the feedback unit is communicatively connected to the input end of the client, and the output end of the security unit is communicatively connected to the input end of the storage unit;

[0017] The access unit is connected to the client through a network, and the access unit can receive data transmitted by the client;

[0018] The identification unit is used to identify the data accessed by the access unit, so as to determine the size of the data, and after identification, the identification unit will split and compress the data;

[0019] The data identified by the identification unit is input into the simulation operation environment constructed by the virtual unit, and the virtual unit determines whether the data is damaged according to the simulation operation of the data;

[0020] The repair unit is used to repair the data judged by the virtual unit;

[0021] The request unit sends a request command to the client according to the result of the operation of the virtual unit;

[0022] The security unit protects the data stored in the storage unit by using big data security technology;

[0023] The feedback unit feeds back information to the client according to the results of the operation of the tracking unit and the storage unit;

[0024] The storage unit is used to classify and store the data judged by the virtual unit;

[0025] The tracking unit is used to track the data call flow in the storage unit.

[0026] Furthermore, the access unit is a universal network interface of the cloud platform, which is used to connect the client to the cloud platform data processing system.

[0027] Furthermore, the identification unit includes a judgment module and a processing module. The judgment module is used to judge the data, and the processing module splits and compresses the data according to the judgment result of the judgment module.

[0028] Furthermore, the judgment module judges the size of the data by a calculation formula, and the calculation formula is as follows:

[0029] , where S is the ratio between the value of the incoming data and the set value. When S≤1, the processing module does not split the data. When S>1, the processing module splits the data.

[0030] Furthermore, the virtual unit includes a simulation module and a diagnosis module. The simulation module constructs a data virtual operating environment through simulation technology, and the diagnosis module can diagnose the operation of the simulation module to determine whether the data is damaged.

[0031] Furthermore, the repair unit repairs the data diagnosed as having problems by the virtual unit through a data repair technology.

[0032] Furthermore, the storage unit includes a resource pool module and a periodic cleaning module. The periodic cleaning module periodically cleans the data stored in the resource pool module according to the feedback result of the feedback unit. The resource pool module has an automatic partitioning function and can store data according to date classification.

[0033] Furthermore, the security unit includes an encryption module and a protection wall module. The encryption module encrypts the data stored in the storage unit in an encrypted manner, and the protection wall module can build a firewall in the processing system.

[0034] Furthermore, the tracking unit tracks and monitors the calling flow of data in the storage unit through data tracking technology.

[0035] In the above technical solution, the present invention provides a data processing method and system based on a cloud platform, which has the following beneficial effects:

[0036] (1) The data processing method designed by the present invention can identify and process the data while accessing it, and then determine whether the data is missing or damaged based on the results of the data simulation running in the virtual environment, thereby preventing damaged data from being transmitted to the cloud platform and occupying the storage space of the cloud platform. It can also prevent useless data from being stored in the cloud platform and causing losses to users. In addition, the method can also repair missing or damaged data or re-request new data from the customer to ensure the integrity of the data stored in the cloud platform.

[0037] (2) The processing system designed by the present invention can not only store data, but also regularly clean up the stored data through a regular cleaning module in the storage unit to avoid useless data occupying the storage space of the cloud platform. At the same time, it will feedback information to the user during cleaning, and cleaning will only be performed based on the user's feedback to avoid cleaning up more important data.

[0038] (3) The tracking unit designed by the present invention can track the calling flow of data in the storage unit through data tracking technology to realize data monitoring. In conjunction with the security unit, it can improve the security of data stored in the storage unit and prevent the data from being stolen by criminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0040] Figure 1 A method flow chart of a data processing method and system embodiment based on a cloud platform provided by the present invention.

[0041] Figure 2 A system flow diagram of a data processing method and system embodiment based on a cloud platform provided by the present invention.

[0042] Figure 3 A schematic diagram of the structure of an identification unit provided for a data processing method and system embodiment based on a cloud platform of the present invention.

[0043] Figure 4A schematic diagram of a virtual unit structure provided for a data processing method and system embodiment based on a cloud platform of the present invention.

[0044] Figure 5 A schematic diagram of a storage unit structure provided for a data processing method and system embodiment based on a cloud platform of the present invention.

[0045] Figure 6 A schematic diagram of a security unit structure provided for a data processing method and system embodiment based on a cloud platform of the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0047] like Figure 1 As shown, a data processing method based on a cloud platform provided by an embodiment of the present invention includes the following steps:

[0048] S1. The cloud platform receives data from the client and identifies and processes the data;

[0049] S2. Input the identified data into the virtual environment to run the simulation and determine whether the data is damaged;

[0050] S2.1. The data is not damaged, the data is classified and processed, and storage partitions are created based on the classification results to store the data;

[0051] S2.2. If the data is damaged, the damaged data is repaired. After the repair is completed, the data is re-entered into step S2 for simulation processing. If the repair fails, a data request command is sent to the client, and the client resends the data to the cloud platform;

[0052] S3. Classify the data and create storage partitions to store the data based on the classification results;

[0053] S4. Encrypt the stored data and track the flow of data;

[0054] S5. Regularly manage the data in the storage partition, send commands to the client, and delete infrequently used data based on the client feedback results.

[0055] A data processing method based on a cloud platform provided by the present invention can identify and process data while accessing the data, and then determine whether the data is missing or damaged through the result of running a simulation on the data in a virtual environment, thereby avoiding the transmission of damaged data to the cloud platform to occupy the storage space of the cloud platform, and also avoiding the loss of users caused by useless data stored in the cloud platform. Moreover, the method can also repair the missing or damaged data or re-request new data from the customer to ensure the integrity of the data stored in the cloud platform.

[0056] A data processing system based on a cloud platform, such as Figures 2 to 6 As shown, it includes an access unit, an identification unit, a virtual unit, a repair unit, a request unit, a security unit, a feedback unit, a storage unit, a tracking unit and a client;

[0057] The output end of the client is communicatively connected to the input end of the access unit, the output end of the access unit is communicatively connected to the input end of the identification unit, the output end of the identification unit is communicatively connected to the input end of the virtual unit, the output end of the virtual unit is communicatively connected to the input end of the repair unit and the input end of the storage unit, the output end of the repair unit is communicatively connected to the input end of the virtual unit and the input end of the request unit, the output end of the feedback unit is communicatively connected to the input end of the client, and the output end of the security unit is communicatively connected to the input end of the storage unit;

[0058] The access unit is connected to the client through a network, and the access unit can receive data transmitted by the client. The access unit is a universal network interface of the cloud platform, which is used to connect the client to the cloud platform data processing system;

[0059] The identification unit is used to identify the data accessed by the access unit, so as to determine the size of the data, and after identification, the identification unit will split and compress the data. The identification unit includes a judgment module and a processing module. The judgment module is used to judge the data. The processing module splits and compresses the data according to the result of the judgment of the judgment module. The judgment module judges the size of the data through a calculation formula, and the calculation formula is as follows:

[0060] ,

[0061] Wherein, S is the ratio between the value of the incoming data and the set value. When S≤1, the processing module does not split the data. When S>1, the processing module splits the data.

[0062] The data identified by the identification unit will be input into the simulation operation environment constructed by the virtual unit, and the virtual unit will judge whether the data is damaged according to the simulation operation of the data. The virtual unit includes a simulation module and a diagnosis module. The simulation module constructs a data virtual operation environment through simulation technology, and the diagnosis module can diagnose the operation of the simulation module to determine whether the data is damaged;

[0063] The repair unit is used to repair the data judged by the virtual unit, and the repair unit repairs the data diagnosed as problematic by the virtual unit through data repair technology;

[0064] It should be noted that there are several data repair methods:

[0065] Data backup and recovery: Back up data regularly and restore the backup data when needed. You can use cloud storage services, such as cloud hard drives or object storage, to back up data to the cloud and restore the backup data as needed.

[0066] Snapshot technology: The cloud server provides a snapshot function that can save the server's disk status as a snapshot and then restore it to the snapshot status when needed. Snapshots can be used for data recovery, upgrade testing, and other operations.

[0067] RAID technology: Using RAID (Redundant Independent Disk Array) technology can provide data redundancy and fault tolerance. When a disk is damaged, it can be repaired using the data on other disks.

[0068] Data synchronization and replication: Data synchronization and replication technology can be used to replicate data to multiple geographical locations or multiple storage devices to improve data availability and reliability. When a server fails, data can be obtained from other servers for repair.

[0069] Database backup and recovery: For database servers, you can use the database backup and recovery function to back up the database regularly and restore the backup data when needed.

[0070] Data repair tools: There are specialized data repair tools that can help repair corrupted or lost data. These tools can scan the disk and attempt to recover the corrupted data.

[0071] The request unit sends a request command to the client according to the result of the operation of the virtual unit;

[0072] The security unit protects the data stored in the storage unit by using big data security technology. The security unit includes an encryption module and a protection wall module. The encryption module encrypts the data stored in the storage unit by encryption. The protection wall module can build a firewall in the processing system.

[0073] It should be noted that the encryption module uses the following two encryption technologies, which are:

[0074] 1. Symmetric encryption technology. In symmetric encryption technology, the same key is used for encryption and decryption; it is also called a secret key or single-key encryption method. Such algorithms include DES and IDEA.

[0075] 2. Asymmetric encryption technology. Asymmetric encryption means that the encryption and decryption keys are not the same. Usually there are two keys, called "public key" and "private key". They must be used in pairs, otherwise the encrypted file cannot be opened. The "public key" can be made public, but the "private key" cannot. It can only be known by the holder. This is its superiority. Because if the symmetric encryption method is used to transmit encrypted files on the Internet, it is difficult to tell the key to the other party, and it may be stolen by others no matter what method is used. The asymmetric encryption method has two keys, and the "public key" can be made public, so there is no need to worry about others knowing it; the recipient only needs to use his own private key to decrypt, which avoids the key transmission security problem. This type of cryptographic algorithm includes RSA, backpack cipher, McEliece cipher, etc.

[0076] The feedback unit feeds back information to the client according to the results of the operation of the tracking unit and the storage unit;

[0077] The storage unit is used to classify and store the data judged by the virtual unit. The storage unit includes a resource pool module and a regular cleaning module. The regular cleaning module regularly cleans up the data stored in the resource pool module according to the feedback result of the feedback unit. The resource pool module has an automatic partitioning function and can classify and store data according to date;

[0078] It should be noted that the regular cleanup module is selected by NineData, which is committed to simplifying the DBA's data cleanup work, improving the compliance of database operations, and providing strong support for enterprises to reduce costs and increase efficiency.

[0079] The tracking unit is used to track the data call flow in the storage unit. The tracking unit tracks and monitors the data call flow in the storage unit through data tracking technology.

[0080] It should be noted that data tracking refers to the process of collecting, monitoring and analyzing specific data through technical means. In the online environment, data tracking usually refers to tracking user activities on websites or applications, collecting data related to user behavior, and analyzing them in order to better understand user needs and interests and optimize product design and marketing strategies. It includes the following technologies:

[0081] Cookie Tracking

[0082] Use cookie technology to track user activities, including visit records, behaviors, preferences, etc. This method is suitable for online platforms such as websites and applications.

[0083] Pixel Tracking

[0084] Embed pixel codes on web pages to track user behaviors and activities, such as ad clicks, purchases, etc.

[0085] IP Address Tracking

[0086] By recording the user's IP address, the user's activities and behaviors can be tracked, such as access time, access path, etc. This method is suitable for online platforms such as websites and applications.

[0087] User ID Tracking

[0088] Tracking user activities and behaviors through user identifiers such as mobile phone numbers, emails, etc. This method is suitable for platforms such as mobile applications.

[0089] Data analysis tools

[0090] Use data analysis tools to collect and analyze user behavior and activity data. These tools can help companies better understand user needs and behaviors and optimize products and marketing strategies.

[0091] Working principle: The access unit of the cloud platform can receive data from the client, and then the identification unit will identify and process the data. In this process, the judgment module judges the size of the data through a calculation formula. According to the judgment result, the processing module will split and compress the data. After that, the identified data will be transmitted to the virtual environment composed of the virtual unit simulation module, and then run for simulation. The diagnosis module diagnoses the operation of the simulation module to determine whether the data is damaged; if the data is damaged, the data will be transmitted to the repair unit, and then the repair unit will repair the damaged data through data repair technology. After the repair is completed, the data will be re-entered into the above steps for simulation processing. If the repair fails, the repair unit A data request command will be sent to the client, causing the client to resend the data to the cloud platform. If the data is not damaged or has been repaired, the data will be directly entered into the storage unit, and then the resource pool module will automatically partition the data so that the data is stored in the corresponding resource pool module partition, thereby realizing classified storage of the data; and after being stored in the resource pool module, the encryption module in the security unit will encrypt the stored data, and when the data in the resource pool module is called, the tracking unit will track the flow of the data; and the periodic cleaning module in the storage unit will regularly manage the data in the storage partition, and will first send a command to the client during management, and the periodic cleaning module will delete infrequently used data based on the feedback from the client.

[0092] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A data processing method based on a cloud platform, characterized in that: The following steps are involved: S1. The cloud platform receives data from the client and identifies and processes the data; S2. Input the identified data into the virtual environment to run the simulation and determine whether the data is damaged; S2.

1. The data is not damaged, the data is classified and processed, and storage partitions are created based on the classification results to store the data; S2.

2. If the data is damaged, the damaged data is repaired. After the repair is completed, the data is re-entered into step S2 for simulation processing. If the repair fails, a data request command is sent to the client, and the client resends the data to the cloud platform; S3. Classify the data and create storage partitions to store the data based on the classification results; S4. Encrypt the stored data and track the flow of data; S5. Regularly manage the data in the storage partition, send commands to the client, and delete infrequently used data based on the client feedback results.

2. A data processing system based on a cloud platform, characterized in that: It includes an access unit, an identification unit, a virtual unit, a repair unit, a request unit, a security unit, a feedback unit, a storage unit, a tracking unit and a client; The output end of the client is communicatively connected to the input end of the access unit, the output end of the access unit is communicatively connected to the input end of the identification unit, the output end of the identification unit is communicatively connected to the input end of the virtual unit, the output end of the virtual unit is communicatively connected to the input end of the repair unit and the input end of the storage unit, the output end of the repair unit is communicatively connected to the input end of the virtual unit and the input end of the request unit, the output end of the feedback unit is communicatively connected to the input end of the client, and the output end of the security unit is communicatively connected to the input end of the storage unit; The access unit is connected to the client through a network, and the access unit can receive data transmitted by the client; The identification unit is used to identify the data accessed by the access unit, so as to determine the size of the data, and after identification, the identification unit will split and compress the data; The data identified by the identification unit is input into the simulation operation environment constructed by the virtual unit, and the virtual unit determines whether the data is damaged according to the simulation operation of the data; The repair unit is used to repair the data judged by the virtual unit; The request unit sends a request command to the client according to the result of the operation of the virtual unit; The security unit protects the data stored in the storage unit by using big data security technology; The feedback unit feeds back information to the client according to the results of the operation of the tracking unit and the storage unit; The storage unit is used to classify and store the data judged by the virtual unit; The tracking unit is used to track the data call flow in the storage unit.

3. A cloud platform-based data processing system according to claim 2, characterized in that: The access unit is a universal network interface of the cloud platform, which is used to connect the client to the cloud platform data processing system.

4. A cloud platform-based data processing method and system according to claim 2, characterized in that: The identification unit includes a judgment module and a processing module. The judgment module is used to judge the data, and the processing module splits and compresses the data according to the judgment result of the judgment module.

5. A cloud platform-based data processing method and system according to claim 4, characterized in that: The judgment module judges the size of the data through a calculation formula, and the calculation formula is as follows: ， Among them, S is the ratio between the numerical value of the access data and the set value. When S≤1, the processing module does not split the data. When S>1, the processing module will split the data.

6. A cloud platform-based data processing method and system according to claim 2, characterized in that: The virtual unit includes a simulation module and a diagnosis module. The simulation module constructs a data virtual operation environment through simulation technology. The diagnosis module can diagnose the operation of the simulation module to determine whether the data is damaged.

7. A cloud platform-based data processing method and system according to claim 2, characterized in that: The repair unit repairs the data diagnosed as having problems by the virtual unit through a data repair technology.

8. A cloud platform-based data processing method and system according to claim 2, characterized in that: The storage unit includes a resource pool module and a periodic cleaning module. The periodic cleaning module periodically cleans up the data stored in the resource pool module according to the feedback result of the feedback unit. The resource pool module has an automatic partitioning function and can store data according to date classification.

9. A cloud platform-based data processing method and system according to claim 2, characterized in that: The security unit includes an encryption module and a protection wall module. The encryption module encrypts the data stored in the storage unit in an encryption manner, and the protection wall module can build a firewall in the processing system.

10. A cloud platform-based data processing method and system according to claim 2, characterized in that: The tracking unit tracks and monitors the calling flow of data in the storage unit through data tracking technology.

Images