A cloud platform operation data management method based on SaaS

Abstract

The present invention discloses a cloud platform operation data management method based on SaaS. The present invention relates to the field of data management technology and solves the technical problems of being unable to meet the storage requirements of different types of data, lacking effective partitioning and sharding strategies, resulting in low efficiency of data query and analysis, and difficulty in quickly locating and processing required data. The present invention uses innovative time span partitioning and multi-dimensional sharding strategies to finely divide cloud platform operation data according to time, data type, and text and non-text information in the data content, thereby improving the precision of data management and query and analysis efficiency. Partitioning according to time span can quickly locate data within a specific time period, and comprehensively considers the storage capacity and response delay of storage nodes, as well as the data capacity and response requirements of sharded data, thereby achieving accurate matching of storage nodes and sharded data, reducing storage costs, and improving data storage and access performance.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a cloud platform operation data management method based on SaaS. Background Art

[0002] The SaaS-based cloud platform operation data management method involves multiple links such as data collection, storage, analysis, and security assurance. With the rapid development of cloud computing technology, cloud platforms have been widely used in various fields.

[0003] Traditional data management methods are unable to meet the storage and analysis needs of large-scale, highly concurrent, and diversified data on cloud platforms. How to efficiently manage and utilize these massive amounts of operational data has become a key challenge in cloud platform operations and optimization. The scale of cloud platform operational data is huge and the types are complex. Traditional storage methods are difficult to cope with. There are problems such as insufficient storage capacity and high response latency. They cannot meet the storage needs of different types of data. Faced with a large amount of data of different time, type, and capacity, there is a lack of effective partitioning and sharding strategies, resulting in low efficiency in data query and analysis, and difficulty in quickly locating and processing the required data. Due to differences in data structure and capacity, it is difficult to select a suitable database based on the characteristics of the data, which may result in resource waste or poor data processing performance. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a cloud platform operation data management method based on SaaS, which solves the problems of being unable to meet the storage needs of different types of data, lacking effective partitioning and sharding strategies, resulting in low efficiency in data query and analysis, and difficulty in quickly locating and processing the required data.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a SaaS-based cloud platform operation data management method, the method specifically comprising the following steps:

[0006] Obtain data information of cloud platform operation data, including data time, data type and data capacity;

[0007] Based on the determined time span, the cloud platform data is partitioned according to the data time to generate data partition information. At the same time, the data partition information is partitioned according to the data type and data content to generate data partition information.

[0008] According to the data capacity corresponding to the data sharding information, combined with the storage capacity and response delay of the storage node, the corresponding storage node is selected, and the cloud platform data is stored to generate storage information;

[0009] Select the corresponding database based on the nature of the cloud platform data in the storage information, and generate management information based on the storage information.

[0010] As a further solution of the present invention, the specific method of partitioning the cloud platform operation data to generate data partition information is:

[0011] All cloud platform operation data are labeled as i, and i = 1, 2, ..., j, where j represents the number of cloud platform operation data. Then, the data time Ti corresponding to the cloud platform operation data i is obtained. At the same time, taking the time span Tc as the standard and the initial data time Ti as the standard, all cloud platform operation data i that meet the time span Tc are obtained, and the obtained cloud platform operation data i are partitioned. The obtained partitions are labeled as n, and n = 1, 2, ..., m, and data partition information is generated at the same time.

[0012] As a further solution of the present invention, the specific method of generating data fragmentation information is:

[0013] Then, all partitions n are obtained, and a group of partitions is randomly selected and marked as target objects. The cloud platform operation data in the target objects is obtained and recorded as the data to be analyzed. Then, the data type of the data to be analyzed is obtained, and the cloud platform operation data is classified according to the data type to generate data of the same type, and the first shard label is generated at the same time.

[0014] The data to be analyzed in the first sharding tag is obtained, and then the data content of the data to be analyzed is obtained, and the corresponding text information and non-text information in the data content are used for secondary sharding processing to generate a second sharding tag and generate data sharding information at the same time.

[0015] As a further solution of the present invention, the specific method of performing secondary fragmentation processing based on the corresponding text information and non-text information in the data content is:

[0016] If there is text information and non-text information in the data content, the corresponding number of text information and non-text information is obtained, and the two numbers are compared. If the number of text information is greater than the number of non-text information, it is classified as text information. Conversely, if the number of text information is less than the number of non-text information, it is classified as non-text information, and data segmentation information is generated at the same time.

[0017] As a further solution of the present invention, the specific method of generating the storage information is:

[0018] All storage nodes are labeled as o, and o = 1, 2, ..., p, where p represents the number of storage nodes. At the same time, the storage capacity and response delay corresponding to the storage node o are obtained, and the shard data in the data shard information is obtained. At the same time, the data capacity corresponding to the shard data is obtained. Then, the storage node o is screened based on the data capacity to obtain pre-selected storage nodes, and the pre-selected storage nodes are subjected to secondary analysis and processing to generate storage information.

[0019] As a further solution of the present invention, the specific method of performing secondary analysis on the pre-selected storage nodes to generate storage information is:

[0020] Obtain the response requirements corresponding to the sharded data, then select the pre-selected storage node that meets the response requirements as the marked storage node, store the sharded data, and generate data storage information.

[0021] As a further solution of the present invention, the specific method of screening the storage nodes based on the data capacity to obtain the pre-selected storage nodes is:

[0022] Obtain the data capacity of the shard data and the data capacity of the storage node at the same time, compare the data capacity of the shard data with the data capacity of the storage node, and record the storage node whose data capacity is greater than the data capacity of the shard data as the pre-selected storage node.

[0023] As a further solution of the present invention, the specific method of generating data management information is:

[0024] Obtain all sharded data, obtain the data structure of the sharded data, and classify them to obtain structured data and unstructured data. Then obtain the data capacity corresponding to the structured data and unstructured data. At the same time, perform secondary classification of the structured data and unstructured data according to the data capacity to obtain lightweight data and heavyweight data. Select different databases based on the classified data to generate database information. Then combine the storage information and database information to generate management information.

[0025] The present invention provides a SaaS-based cloud platform operation data management method. Compared with the existing technology, it has the following advantages:

[0026] This invention uses innovative time span partitioning and multi-dimensional sharding strategies to meticulously divide cloud platform operation data according to time, data type, and text and non-text information in the data content, thereby improving the precision of data management and the efficiency of query analysis. Partitioning by time span can quickly locate data within a specific time period, and sharding based on data type and content facilitates targeted processing of different types of data.

[0027] By comprehensively considering the storage capacity and response latency of storage nodes, as well as the data capacity and response requirements of sharded data, we achieve precise matching of storage nodes and sharded data, reducing storage costs and improving data storage and access performance. We accurately select the appropriate storage node based on the capacity and response requirements of different sharded data, avoiding resource waste and performance bottlenecks.

[0028] According to the data structure and data capacity, different types of databases are scientifically selected to give full play to the advantages of various databases. For example, SQLite is selected for structured lightweight data, and Redis is selected for structured heavyweight data. This improves the overall efficiency of data processing and optimizes the utilization of database resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a diagram of the steps of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] For example 1, please refer to Figure 1 , this application provides a SaaS-based cloud platform operation data management method, the method specifically includes the following steps:

[0032] Step 1: Acquire the cloud platform operation data and obtain the data information of the cloud platform operation data, and the data information includes data time, data type and data capacity.

[0033] Step 2: Partition and shard the acquired cloud platform operation data. The specific method of partition processing is as follows:

[0034] All cloud platform operation data are obtained and labeled as i, where i=1, 2, ..., j, where j represents the number of cloud platform operation data. Then, the data time Ti corresponding to the cloud platform operation data i is obtained, where the data time here represents the generation time of the cloud platform operation data. At the same time, the cloud platform operation data i is partitioned according to the data time Ti based on the time span Tc. Specifically, all cloud platform operation data i that meet the time span Tc are obtained based on the initial data time Ti, and the obtained cloud platform operation data i are partitioned. The obtained partitions are labeled as n, where n=1, 2, ..., m, and data partition information is generated at the same time.

[0035] In a specific example, all cloud platform operation data i are obtained, where i=4. Specifically, the data time corresponding to i=1 is 09:00, the data time corresponding to i=2 is 09:30, the data time corresponding to i=3 is 10:15, and the data time corresponding to i=4 is 10:45. The time span Tc is set to 1 hour (i.e. 60 minutes). The data time of the initial data time T1 is 09:00, so the time range of the first partition is from 09:00 to 10:00, so the labels i=1 and i=2 are partitioned, and the second partition time is 10:00 to 11:00, so the labels i=3 and i=4 are partitioned.

[0036] Then, all partitions n are obtained, and a group of partitions are randomly selected and marked as target objects. The cloud platform operation data in the target object is obtained and recorded as the data to be analyzed. Then, the data type of the data to be analyzed is obtained, and the cloud platform operation data is classified according to the data type to generate data of the same type. At the same time, a first sharding label is generated, and the first sharding label here represents the sharding information obtained by sharding according to the data type, and any group of first sharding labels is obtained; for each piece of data to be analyzed, its data type is accurately identified and recorded. Common data types include text, numerical, image, log, etc. The data to be analyzed is classified according to the data type, and the data of the same data type is grouped into a set, thereby generating a set of data of the same type. A corresponding first sharding label is generated for each set of data of the same type. The first sharding label covers the data type of the set and the unique sharding identifier in the target partition, which is used to clearly distinguish different types of data shards.

[0037] At the same time, the data to be analyzed in the first sharding tag is obtained, and then the data content of the data to be analyzed is obtained, and secondary sharding is performed with the corresponding text information and non-text information in the data content to generate a second sharding tag. The non-text information here is represented as the remaining information after removing the text information, such as letters, characters and numbers, which are collectively referred to as non-text information, and data sharding information is generated at the same time; and the data content is defaulted here to be a single type of content, specifically referring to the existence of only text information or only non-text information.

[0038] If there is text information and non-text information in the data content, the corresponding number of text information and non-text information is obtained, and the numbers of the two are compared. At the same time, the data content with the largest value is selected as the standard for classification processing. Specifically, if the number of text information is greater than the number of non-text information, it is classified as text information. Conversely, if the number of text information is less than the number of non-text information, it is classified as non-text information. At the same time, in the case where the two numbers are the same, they are separately fragmented to generate a third fragmentation label and generate data fragmentation information at the same time.

[0039] In a specific example, suppose we select the first shard label 3 - user login log category. The data to be analyzed corresponding to this label is data number 3 "Username: user123, Login Time: 2025-02-13 10:00". This data is analyzed to count the amount of text information and non-text information:

[0040] Text information: "User name" and "Login time", a total of 6 characters.

[0041] Non-text information: ":" "user123" "2025-02-13 10:00", a total of 17 non-text information. Since the number of non-text information is greater than the number of text information, this data is classified as non-text information.

[0042] Similarly, all cloud platform operation data are processed in the same way and the corresponding data sharding information is obtained.

[0043] Step 3: Select appropriate storage nodes for storage based on the obtained data sharding information and generate corresponding storage information. The specific selection method is as follows:

[0044] Obtain all storage nodes and label them as o, and o = 1, 2, ..., p, where p represents the number of storage nodes. At the same time, obtain the storage capacity and response delay corresponding to the storage node o, and obtain the shard data in the data shard information, and the shard data here represents the cloud platform data after sharding processing, and at the same time obtain the data capacity corresponding to the shard data. Then, use the data capacity as the standard to filter the storage node o to obtain a pre-selected storage node, and the pre-selected storage node here represents the storage requirement that can meet the data capacity corresponding to the shard data. At the same time, perform secondary analysis on the obtained pre-selected storage nodes to obtain the response requirement corresponding to the shard data, and the response requirement here represents the lowest delay requirement of the current shard data. Then, select the pre-selected storage node that meets the response requirement as the marked storage node, store the shard data, and generate data storage information;

[0045] In a specific example, shard data 1 (S1=60GB): the pre-selected storage nodes that meet the storage capacity requirements are 1 (C1=100GB), 3 (C3=200GB), and 4 (C4=80GB).

[0046] Shard data 2 (S2=20GB): The pre-selected storage nodes that meet the storage capacity requirements are 1 (C1=100GB), 2 (C2=50GB), 3 (C3=200GB), 4 (C4=80GB), and 5 (C5=30GB).

[0047] Shard data 3 (S3=120GB): The pre-selected storage node that meets the storage capacity requirements is 3 (C3=200GB).

[0048] The response requirement of shard data 1 is R1=22ms, the response requirement of shard data 2 is R2=12ms, and the response requirement of shard data 3 is R3=28ms. Shard data 1 (R1=22ms): Among the pre-selected storage nodes 1 (D1=20ms), 3 (D3=30ms), and 4 (D4=25ms), the marked storage node that meets the response requirements is 1.

[0049] Shard data 2 (R2=12ms): Among the pre-selected storage nodes 1 (D1=20ms), 2 (D2=15ms), 3 (D3=30ms), 4 (D4=25ms), and 5 (D5=10ms), the marked storage node that meets the response requirements is 5.

[0050] Shard data 3 (R3=28ms): Among the pre-selected storage nodes 3 (D3=30ms), there is no marked storage node that meets the response requirements.

[0051] Similarly, the above analysis is performed on all storage nodes and shard data, and storage information is generated.

[0052] Step 4: Analyze the shard data based on the obtained storage information, select a standard database based on the shard data itself, and generate data management information. The specific processing method is as follows:

[0053] All sharded data are obtained, and the data structure of the sharded data is obtained, and classified to obtain structured data and unstructured data. Specifically, structured data is represented as having a fixed format and a clear data type, organized in the form of rows and columns, while unstructured data has no fixed format or predefined structure, and is diverse and free in form. Then the data capacity corresponding to the structured data and the unstructured data is obtained, and the structured data and the unstructured data are secondary classified according to the data capacity to obtain lightweight data and heavyweight data. Here, the data capacity is compared with the capacity threshold, and the specific value of the capacity threshold is set by the operator. Different databases are selected based on the classified data to generate database information, and then the storage information and database information are combined to generate management information.

[0054] In a specific example, the following sharded data is obtained, where structured data 1 is an employee information table (employee ID, name, age, position), structured data 2 is a user login log table (user ID, login time, login IP), unstructured data 3 is a customer feedback audio clip, and unstructured data 4 is a product image. The capacity threshold is set to 10MB. After secondary classification, structured data 1 is lightweight data, structured data 2 is heavyweight data, unstructured data 3 is heavyweight data, and unstructured data 4 is lightweight data.

[0055] After analysis, it is found that the database selected for structured data 1 is SQLite, the database file path is: / data / sqlite / employee.db, the database selected for structured data 2 is Red is, address: 192.168.1.101, port: 6379, the database selected for unstructured data 3 is MongoDB, address: 192.168.1.102, port: 27017, database name: customer_feedback, the database selected for unstructured data 4 is Red is, address: 192.168.1.101, port: 6379.

[0056] Example 2: This example is implemented on the basis of Example 1, and differs from Example 1 in the following aspects:

[0057] When partitioning cloud platform data, you can also calculate the hash value of the cloud platform data to perform partitioning. Specifically, by performing hash calculation on the value of a column, the data is evenly distributed into multiple partitions. For a user table, hash partitioning can be performed based on user ID. Specify the hash partition column and the number of partitions when creating the table. For example, in MySQL:

[0058]

[0059]

[0060] The above code is used to partition the cloud platform data.

[0061] Embodiment 3, as the third embodiment of the present invention, focuses on combining the implementation processes of embodiment 1 and embodiment 2.

[0062] Some of the data in the above formulas are calculated based on their numerical values ​​and are not substituted into parameter units for calculation. At the same time, the contents not described in detail in this specification belong to the existing technology known to those skilled in the art.

[0063] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A cloud platform data management method based on SaaS, characterized in that: The method specifically comprises the following steps: Obtain data information of cloud platform data, including data time, data type and data capacity; Based on the determined time span, the cloud platform data is partitioned according to the data time to generate data partition information. Then, all partitions n are obtained, and a group of partitions is randomly marked as the target object. The cloud platform operation data in the target object is obtained and recorded as the data to be analyzed. Then, the data type of the data to be analyzed is obtained, and the cloud platform operation data is classified according to the data type to generate data of the same type, and the first shard label is generated at the same time. At the same time, the data to be analyzed in the first sharding tag is obtained, and then the data content of the data to be analyzed is obtained, and a secondary sharding process is performed based on the corresponding text information and non-text information in the data content to generate a second sharding tag and generate data sharding information; According to the data capacity corresponding to the data sharding information, combined with the storage capacity and response delay of the storage node, the corresponding storage node is selected, and the cloud platform data is stored to generate storage information; Select the corresponding database based on the nature of the cloud platform data in the storage information, and generate data management information based on the storage information.

2. A SaaS-based cloud platform data management method according to claim 1, characterized in that: The specific method of partitioning the cloud platform data to generate data partition information is as follows: All cloud platform data are labeled as i, and i=1, 2, ..., j, where j represents the number of cloud platform data. Then, the data time Ti corresponding to the cloud platform data i is obtained. At the same time, based on the time span Tc and the initial data time Ti, all cloud platform data i that meet the time span Tc are obtained, and the obtained cloud platform data i are partitioned. The obtained partitions are labeled as n, and n=1, 2, ..., m, and data partition information is generated at the same time.

3. The SaaS-based cloud platform data management method according to claim 1, characterized in that: The specific method of performing secondary fragmentation processing based on the corresponding text information and non-text information in the data content is: If there is text information and non-text information in the data content, the corresponding number of text information and non-text information is obtained, and the two numbers are compared. If the number of text information is greater than the number of non-text information, it is classified as text information. Conversely, if the number of text information is less than the number of non-text information, it is classified as non-text information, and data segmentation information is generated at the same time.

4. The SaaS-based cloud platform data management method according to claim 1, characterized in that: The specific method of generating the storage information is as follows: All storage nodes are labeled as o, and o=1, 2, ..., p, where p represents the number of storage nodes. At the same time, the storage capacity and response delay corresponding to the storage node o are obtained, and the shard data in the data shard information is obtained. At the same time, the data capacity corresponding to the shard data is obtained. Then, the storage node o is screened based on the data capacity to obtain pre-selected storage nodes. At the same time, the pre-selected storage nodes are subjected to secondary analysis and processing to generate storage information.

5. A SaaS-based cloud platform data management method according to claim 4, characterized in that: The specific method of performing secondary analysis on the pre-selected storage nodes to generate storage information is as follows: Obtain the response requirements corresponding to the sharded data, then select the pre-selected storage node that meets the response requirements as the marked storage node, store the sharded data, and generate data storage information.

6. A SaaS-based cloud platform data management method according to claim 4, characterized in that: The specific method of screening the storage nodes based on the data capacity to obtain the pre-selected storage nodes is as follows: Obtain the data capacity of the shard data and the data capacity of the storage node at the same time, compare the data capacity of the shard data with the data capacity of the storage node, and record the storage node whose data capacity is greater than the data capacity of the shard data as the pre-selected storage node.

7. The SaaS-based cloud platform data management method according to claim 1, characterized in that: The specific method of generating data management information is: Obtain all sharded data, obtain the data structure of the sharded data, and classify it to obtain structured data and unstructured data. Then obtain the data capacity corresponding to the structured data and unstructured data. At the same time, perform secondary classification of the structured data and unstructured data according to the data capacity to obtain lightweight data and heavyweight data. Select different databases based on the classified data to generate database information. Then combine the storage information and database information to generate data management information.

Images