A data processing method based on cloud computing

Abstract

The present invention discloses a data processing method based on cloud computing, which relates to the field of data processing technology and solves the problem that the original abnormal data identification method is relatively slow and the similarity between data is not taken into account during the storage process. The present invention analyzes and confirms the distribution of data, confirms the data with large distribution deviation, and marks the confirmed data as abnormal data. Subsequently, the abnormal data is replaced and corrected according to the specific data law or distribution of the distribution diagram, and in the correction process, it is preferentially determined whether there are abnormal situations in the previous and next data at the same time. According to different situations, the data is determined by using equations or directly determining the correction value, so as to lock the corresponding correction value, and the data is processed by correcting as soon as possible, thereby improving the timeliness of abnormal data processing and quickly locking the abnormal data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method based on cloud computing. Background Art

[0002] Cloud computing is a type of distributed computing, which means breaking down huge data computing programs into countless small programs through the network "cloud", and then processing and analyzing these small programs through a system composed of multiple servers to obtain results and return them to users.

[0003] The big data processing method based on cloud computing and the cloud computing platform disclosed in the patent application with the patent application number CN113487193A can weaken the mutual influence between each adaptation indicator data in terms of timing. Therefore, when performing adaptability detection on each indicator adaptation data, the integrity of the obtained detection results can be ensured. When determining whether the service adaptation information meets the set conditions based on multiple detection results, since the multiple detection results are not affected by the timing asynchrony, this can avoid misjudgment of the adaptability between the data service initiator and the data service processor.

[0004] When processing and analyzing the data generated by cloud computing, a specific program will be used to identify abnormal data after data collection is completed. This identification method requires that the data be identified one by one according to the original data logic in order to determine the corresponding abnormal data. This method is too slow and not timely. At the same time, the subsequent data storage is also stored according to time, and the similarity between the data is not high. When the data is extracted later, a good data comparison effect cannot be achieved. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a data processing method based on cloud computing, which solves the problem that the original abnormal data identification method is relatively slow and the similarity between data is not taken into account during the storage process.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A data processing method based on cloud computing, comprising the following steps:

[0007] S1. Prioritize the collection of processing data generated in the cloud computing process, and classify and bundle the processing data collected from different target objects to generate data bundles for different target objects;

[0008] S2. Dynamically distribute the data in different data bundles and construct a distribution diagram belonging to the corresponding data bundle. Subsequently, according to the specific embodiment of the distribution diagram, lock the abnormal data. The specific method is as follows:

[0009] S21, sorting the data of the single data bundle package forward and backward according to the original sorting, and constructing a set of two-dimensional coordinate systems according to different data corresponding to different sorting rankings K, wherein the horizontal coordinate axis of the two-dimensional coordinate system is the sorting ranking, and the vertical coordinate axis is the specific parameter of the data, determining the corresponding points of different data corresponding to different sorting names K within the single data bundle package in the two-dimensional coordinate system, and connecting several groups of points to confirm the distribution map belonging to this single data bundle package;

[0010] S22, confirm the equations belonging to the distribution graph, and analyze whether there is a regularity among the confirmed equations. If so, confirm the line segments of the same equation and mark them as the same area. If not, do not process them and directly transmit the single set of data bundles to the external display terminal, and have external personnel verify, identify and modify the abnormal data.

[0011] S23, marking points in the distribution map that do not belong to the same area as abnormal points, and marking the data corresponding to the abnormal points as abnormal data;

[0012] S3. After the abnormal data is locked, confirm from the corresponding data bundle whether there is any abnormality in the data before and after the abnormal data. If so, make corrections using the equation corresponding to the same area. If not, make corrections to the abnormal data by confirming the slope of the data before and after. The specific method is as follows:

[0013] S31, determine the location j of the abnormal data, confirm whether the data corresponding to j+1 and j-1 are abnormal data, if they are abnormal data, determine the equation corresponding to the same area from the distribution map, input the location j into the equation, directly replace the abnormal data with the data output by the equation, and complete the data correction work of the corresponding data bundle, where j∈K;

[0014] S32, confirm the location j of the abnormal data, then confirm the slope K1 of the data segment between j+1 and j+2, then confirm the slope K2 of the data segment between j-1 and j-2, analyze and confirm whether K1 and K2 are equal, if they are equal, directly confirm the correction value of the abnormal data by geometric ratio, the correction value = data corresponding to j-1 × K1 or the correction value = data corresponding to j+1 ÷ K2, where K1 = K2, this case is generally a first-order equation;

[0015] If K1 and K2 are not equal, analyze whether K3÷K2 is equal to K2÷K1. If they are equal, directly perform geometric confirmation of the correction value of the abnormal data, where the correction value = data corresponding to j-1 × K2 2, where K3 is the slope of the data segment between j-2 and j-3. If K3÷K2 is not equal to K2÷K1, input the position j into this equation and directly replace the abnormal data with the data output by the equation;

[0016] S4. After the different abnormal data in the data bundle are corrected in sequence, the data in the different data bundles are compressed. The specific method is as follows:

[0017] S41, confirm each data in the data bundle, after confirmation, take three data as a group, and take the middle data as the standard data, and the data before and after the middle data as hidden data, respectively hidden in the front and back subscripts of the standard data, for example: the three data are 123, of which 2 is the middle data, and after the hidden subscript processing, the expression becomes: 123;

[0018] S42, processing the three data one by one in sequence, and after the processing is completed, re-sorting the data bundle according to the original sorting method, thereby completing the compression processing of the data bundle;

[0019] S5. Confirm the label data in the compressed data bundle. Subsequently, confirm the characteristic value of the data bundle according to the confirmed plurality of label data. Subsequently, classify different data bundles according to different characteristic values, and store the data bundles belonging to the same classification in the same storage location. The specific method is as follows:

[0020] S51, confirm the label data corresponding to the bundled data packet, perform mean processing on the confirmed groups of label data, confirm a group of feature means, and mark it as TZ t , where t represents different bundled data packets;

[0021] S52, then confirm the difference between adjacent labeled data in order of arrangement, and mark it as the standard deviation value, then average several groups of standard deviation values, confirm the standard mean, and mark it as BJ t , where t represents different bundled data packets;

[0022] S53, using ZZ t =TZ t ×C1+BJ t ×C2 obtains the eigenvalue ZZ belonging to the corresponding data bundle t , where C1 and C2 are preset fixed coefficient factors;

[0023] S54, establish several segmentation intervals according to the segmentation value X1, and the segmentation intervals are arranged from front to back in the form of: (0, X1], (X1, 2X1], ..., according to several groups of characteristic values ​​ZZ tThe different values ​​of are divided into different segmentation intervals, and the corresponding bundled data packets belonging to the same segmentation interval are classified as the same category and stored in the same storage location.

[0024] Beneficial Effects

[0025] The present invention provides a data processing method based on cloud computing. Compared with the prior art, it has the following beneficial effects:

[0026] The present invention analyzes and confirms the distribution of data, confirms data with large distribution deviation, and marks the confirmed data as abnormal data. Subsequently, according to the specific data law or distribution of the distribution diagram, the abnormal data is replaced and corrected. In the correction process, it is first determined whether there are abnormalities in the previous and next data at the same time. According to different situations, the data is determined by using equations or directly determining the correction value, so as to lock the corresponding correction value. The data is processed in a way that can be corrected as soon as possible, so as to improve the timeliness of abnormal data processing and quickly lock the abnormal data.

[0027] After data collection is completed, compression can be performed to fully reduce the overall capacity of stored data. After compression is completed, the characteristics between the data can be confirmed and the bundled data packets with similar characteristics can be classified as the same category and stored in the same location. This can ensure that the data stored in the same storage partition are similar data. At the same time, it is convenient for subsequent operators to review and check the data, improve the efficiency of data storage, reduce storage time, and improve the overall effect of data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] See also Figure 1 , the present application provides a data processing method based on cloud computing, comprising the following steps:

[0031] S1. Prioritize the collection of processing data generated in the cloud computing process, and classify and bundle the processing data collected from different target objects to generate data bundles for different target objects;

[0032] S2. Dynamically distribute the data in different data bundles and construct a distribution diagram belonging to the corresponding data bundle. Subsequently, according to the specific embodiment of the distribution diagram, lock the abnormal data. The specific method of locking the abnormal data is:

[0033] S21, sorting the data of the single data bundle package forward and backward according to the original sorting, and constructing a set of two-dimensional coordinate systems according to different data corresponding to different sorting rankings K, wherein the horizontal coordinate axis of the two-dimensional coordinate system is the sorting ranking, and the vertical coordinate axis is the specific parameter of the data, determining the corresponding points of different data corresponding to different sorting names K within the single data bundle package in the two-dimensional coordinate system, and connecting several groups of points to confirm the distribution map belonging to this single data bundle package;

[0034] S22. Confirm the equations belonging to the distribution graph, and analyze whether there is a pattern among the confirmed equations. If there is, confirm the line segments of the same equation and mark them as the same area. If not, do not process them and directly transmit the single set of data bundles to the external display terminal. External personnel will verify, identify and modify the abnormal data. Specifically, the specific method of analyzing whether there is a pattern is as follows:

[0035] S221, using: proportion value = number of identical equations ÷ total number of equations, confirming the proportion value of identical equations, and then confirming whether the proportion value satisfies: proportion value ≥ 90%. If so, it means that the corresponding distribution graph has a regularity;

[0036] S222. If not satisfied, it means that there is no regularity in the corresponding distribution graph;

[0037] S23, marking points in the distribution map that do not belong to the same area as abnormal points, and marking the data corresponding to the abnormal points as abnormal data;

[0038] Specifically, during the data collection process, it does not mean that all the collected data are accurate. At the same time, there will be erroneous and abnormal data. For such data, they should either be deleted or corrected. The correction method is better and can better ensure the integrity of the collected data.

[0039] S3. After the abnormal data is locked, confirm from the corresponding data bundle whether there is any abnormality in the data before and after the abnormal data. If so, make corrections using the equation corresponding to the same area. If not, make corrections to the abnormal data by confirming the slope of the data before and after. The specific method of making corrections is:

[0040] S31, determine the location j of the abnormal data, confirm whether the data corresponding to j+1 and j-1 are abnormal data, if they are abnormal data, determine the equation corresponding to the same area from the distribution map, input the location j into the equation, directly replace the abnormal data with the data output by the equation, and complete the data correction work of the corresponding data bundle, where j∈K;

[0041] S32, confirm the location j of the abnormal data, then confirm the slope K1 of the data segment between j+1 and j+2, then confirm the slope K2 of the data segment between j-1 and j-2, analyze and confirm whether K1 and K2 are equal, if they are equal, directly confirm the correction value of the abnormal data by geometric ratio, the correction value = data corresponding to j-1 × K1 or the correction value = data corresponding to j+1 ÷ K2, where K1 = K2, this case is generally a first-order equation;

[0042] If K1 and K2 are not equal, analyze whether K3÷K2 is equal to K2÷K1. If they are equal, directly perform geometric confirmation of the correction value of the abnormal data, where the correction value = data corresponding to j-1 × K2 2 , where K3 is the slope of the data segment between j-2 and j-3. If K3÷K2 is not equal to K2÷K1, input the position j into this equation and directly replace the abnormal data with the data output by the equation;

[0043] Specifically, in the process of confirming abnormal data, correction can be made by the slope corresponding to the previous and next data. Generally, when there is a slope rule, if the slope is consistent, the corresponding correction value can be directly confirmed. For example, if a string of data is 1, 2, 4, 8, then the slope is 2, which is a first-order equation.

[0044] If it is a quadratic equation, the corresponding data is 1, 2, 8, 64, and the corresponding slope is generally 2 j , where j represents the position, but the ratio between the slopes is the same, so it can also be derived directly, which is more efficient;

[0045] It is easy to replace abnormal data. Compared with directly inputting into the equation, this method is faster and can also shorten the calculation process. Because some calculation equations are more complicated and require a certain amount of computing power, the method of correcting as soon as possible is adopted to process data and improve the timeliness of abnormal data processing;

[0046] S4. After the different abnormal data in the data bundles are corrected in sequence, the data in the different data bundles are compressed. The specific method of the compression is as follows:

[0047] S41, confirm each data in the data bundle, after confirmation, take three data as a group, and take the middle data as the standard data, and the data before and after the middle data as hidden data, respectively hidden in the front and back subscripts of the standard data, for example: the three data are 123, of which 2 is the middle data, and after the hidden subscript processing, the expression becomes: 123;

[0048] S42, processing the three data one by one in sequence, and after the processing is completed, re-sorting the data bundle according to the original sorting method, thereby completing the compression processing of the data bundle;

[0049] S5. Confirm the label data in the compressed data bundle. Subsequently, confirm the characteristic value of the data bundle according to the confirmed plurality of label data. Subsequently, classify different data bundles according to different characteristic values, and store the data bundles belonging to the same classification in the same storage location. The specific method of classification is as follows:

[0050] S51, confirm the label data corresponding to the bundled data packet, perform mean processing on the confirmed groups of label data, confirm a group of feature means, and mark it as TZ t , where t represents different bundled data packets;

[0051] S52, then confirm the difference between adjacent labeled data in order of arrangement, and mark it as the standard deviation value, then average several groups of standard deviation values, confirm the standard mean, and mark it as BJ t , where t represents different bundled data packets;

[0052] S53, using ZZ t =TZ t ×C1+BJ t ×C2 obtains the eigenvalue ZZ belonging to the corresponding data bundle t , where C1 and C2 are preset fixed coefficient factors, and their specific values ​​are determined by the operator based on experience;

[0053] S54, establish several segmentation intervals according to the segmentation value X1, and the segmentation intervals are arranged from front to back in the form of: (0, X1], (X1, 2X1], ..., according to several groups of characteristic values ​​ZZ t The different values ​​of are divided into different segmentation intervals, and the corresponding bundled data packets belonging to the same segmentation interval are classified as the same category and stored in the same storage location.

[0054] Specifically, in the cloud computing process, the data generated needs to be collected first, and then compressed after the collection is completed, which can fully reduce the overall capacity of the stored data. After compression is completed, the bundled data packets with similar characteristics can be classified as the same category and stored in the same location by confirming the characteristics between the data. This can ensure that the data stored in the same storage partition are similar data. At the same time, it is convenient for subsequent operators to review and check the data, improve the efficiency of data storage, reduce storage time, and improve the overall effect of data storage.

[0055] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0056] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A data processing method based on cloud computing, characterized in that: The following steps are involved: S1. Prioritize the collection of processing data generated in the cloud computing process, and classify and bundle the processing data collected from different target objects to generate data bundles for different target objects; S2. Dynamically distribute the data in different data bundles and construct a distribution diagram belonging to the corresponding data bundle. Subsequently, according to the specific embodiment of the distribution diagram, lock the abnormal data; S3. After the abnormal data is locked, confirm from the corresponding data bundle whether there is any abnormality in the data before and after the abnormal data. If so, make corrections using the equation corresponding to the same area. If not, make corrections to the abnormal data by confirming the slope of the data before and after. S4, after different abnormal data in the data bundle are corrected in sequence, compressing the data in different data bundles; S5, confirming the label data in the compressed data bundle, and then confirming the characteristic value of the data bundle according to the confirmed plurality of label data, and then classifying different data bundles according to different characteristic values, and storing the data bundles belonging to the same classification in the same storage location; In step S2, the specific method of locking abnormal data is: S21, sorting the data of the single data bundle package forward and backward according to the original sorting, and constructing a set of two-dimensional coordinate systems according to different data corresponding to different sorting rankings K, wherein the horizontal coordinate axis of the two-dimensional coordinate system is the sorting ranking, and the vertical coordinate axis is the specific parameter of the data, determining the corresponding points of different data corresponding to different sorting rankings K within the single data bundle package in the two-dimensional coordinate system, and connecting several groups of points to confirm the distribution map belonging to this single data bundle package; S22, confirm the equations belonging to the distribution graph, and analyze whether there is a regularity among the confirmed equations. If so, confirm the line segments of the same equation and mark them as the same area. If not, do not process them and directly transmit the single set of data bundles to the external display terminal, and have external personnel verify, identify and modify the abnormal data. S23, marking points in the distribution map that do not belong to the same area as abnormal points, and marking the data corresponding to the abnormal points as abnormal data; In step S22, the specific method of analyzing whether there is a regularity is: S221, using: proportion value = number of identical equations ÷ total number of equations, confirming the proportion value of identical equations, and then confirming whether the proportion value satisfies: proportion value ≥ 90%. If so, it means that the corresponding distribution graph has a regularity; S222. If not satisfied, it means that there is no regularity in the corresponding distribution graph; In step S3, the specific method of performing correction is: S31, determine the location j of the abnormal data, confirm whether the data corresponding to j+1 and j-1 are abnormal data, if they are abnormal data, determine the equation corresponding to the same area from the distribution map, input the location j into the equation, directly replace the abnormal data with the data output by the equation, and complete the data correction work of the corresponding data bundle, where j∈K; S32, confirm the location j of the abnormal data, then confirm the slope K1 of the data segment between j+1 and j+2, then confirm the slope K2 of the data segment between j-1 and j-2, analyze and confirm whether K1 and K2 are equal, if they are equal, directly confirm the correction value of the abnormal data by geometric ratio, the correction value = data corresponding to j-1 × K1 or the correction value = data corresponding to j+1 ÷ K2, where K1 = K2; If K1 and K2 are not equal, analyze whether K3÷K2 is equal to K2÷K1. If they are equal, directly perform geometric confirmation of the correction value of the abnormal data, where the correction value = data corresponding to j-1 × K2 2 , where K3 is the slope of the data segment between j-2 and j-3. If K3÷K2 is not equal to K2÷K1, input the position j into the equation corresponding to the same area and directly replace the abnormal data with the data output by the equation.

2. The data processing method based on cloud computing according to claim 1, characterized in that: In step S4, the specific method of performing compression processing is: S41, confirm each data in the data bundle, after confirmation, take three data as a group, and take the middle data as the standard data, and the data before and after the middle data as hidden data, which are hidden in the front and back subscripts of the standard data respectively; S42, processing the three data one by one in sequence, and after the processing is completed, re-sorting the data bundle according to the original sorting method to complete the compression processing of the data bundle.

3. The data processing method based on cloud computing according to claim 2, characterized in that: In step S5, the specific method of classifying different data bundles is as follows: S51, confirm the label data corresponding to the bundled data packet, perform mean processing on the confirmed groups of label data, confirm a group of feature means, and mark it as TZ t , where t represents different bundled data packets; S52, then confirm the difference between adjacent labeled data in order of arrangement, and mark it as the standard deviation value, then average several groups of standard deviation values, confirm the standard mean, and mark it as BJ t , where t represents different bundled data packets; S53, using ZZ t =TZ t ×C1+BJ t ×C2 obtains the eigenvalue ZZ belonging to the corresponding data bundle t , where C1 and C2 are preset fixed coefficient factors; S54, according to the segmentation value X1, a number of segmentation intervals are established, and the segmentation intervals are arranged from front to back in the form of: (0, X1], (X1, 2X1], ..., according to a number of groups of characteristic values ​​ZZ t The different values ​​of are divided into different segmentation intervals, and the corresponding bundled data packets belonging to the same segmentation interval are classified as the same category and stored in the same storage location.

Images