Database data processing method
By creating data warehouses in the database and generating spider web topology, identifying and processing the security and read and write frequency of data content, the problem of existing databases' poor adaptability when processing large and complex data is solved, and data security is achieved and governance effects.
Patent Information
- Application Number
- CN202411295227.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-16
- Publication Date
- 2024-12-20
AI Technical Summary
Existing databases have poor adaptability when processing large and complex data storage, and when applying MySQL database to CRM systems, they require major changes, resulting in higher costs and risks.
A database data processing method is adopted to ensure data reading and writing by determining the data source, creating a data warehouse, identifying the security of the data content, generating a spider web topology, binding database nodes, and distinguishing storage and security detection of data content. Frequency and similarity to achieve effective data governance.
Through data transit and spider web topology, we ensure the security and effective governance of data content, adapt to larger and more complex data cluster storage, reducing the cost and risk of database changes.
Smart Images
Figure CN119166856A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a method for processing database data. Background Art
[0002] A database is a system used to store, manage, and retrieve data. It is like a large data warehouse that can efficiently organize various types of information. Users and applications can easily access and operate the data in it. Common types include relational databases and non-relational databases. Relational databases have a rigorous structure and are stored in tables; non-relational databases are more flexible and suitable for different scenarios.
[0003] The invention patent with application number 201210418832.6 discloses a method for processing data in a database, which is characterized in that there is only one master file in the physical layer of the database, and there is only one index table corresponding to the master file in the logical layer of the database; a partitioning strategy with time units as granularity is configured; the method also includes: according to the configured partitioning strategy, multiple sub-files are created in the physical layer, and correspond to the index table of the logical layer: wherein the association between the master file and the index table is replaced by the association between the multiple sub-files and the index table. According to the configured partitioning strategy, the execution data already stored in the master file of the physical layer is stored in the multiple sub-files created respectively.
[0004] The application aims to solve the problem that "because the execution data in the CRM system needs to be called by multiple user systems, if the MySQL database solution is applied to the SQL Server database that stores the execution data of the CRM system, the index table in the logical layer needs to be divided into multiple index tables, and the master file in the physical layer needs to be divided into multiple master files. In this case, both the logical layer and the physical layer of the SQL Server database will undergo major changes, and the method of querying the execution data will also need to be changed accordingly, which will bring about high costs and usage risks."
[0005] However, when currently used for data storage, databases are often differentiated based on certain characteristics of the data, and data differentiation storage intervals are further created to differentiate and store and manage different types of data. This approach has poor adaptability when faced with large-volume and more complex data storage.
[0006] To this end, we propose a data processing method for the database. Summary of the invention
[0007] In view of the above shortcomings of the prior art, the present invention provides a method for processing database data, which solves the technical problems raised in the above background technology.
[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0009] A method for processing data in a database, comprising:
[0010] Determine the data source, create a data warehouse, use the data warehouse to receive the data content fed back in real time by the data source, identify the data domain, attribute and security of the data content, divide and identify the data domain according to the data produced by the subject in the organizational structure, decide whether to retain the data based on the data content security identification result, and delete the data with a negative decision result in the data warehouse; create several groups of databases, generate a spider web topology, bind each node on the spider web topology to the database, and use each group of databases to receive the data content retained in the data warehouse to store the data content; generate a group of virtual buttons at each node position on the spider web state topology, synchronously create a hyperlink to access the database bound to the node position, set the click operation of the virtual button at the node position, that is, the operation of accessing the database based on the hyperlink corresponding to the database bound to the node position; record the number and time of data content being read and written in the database, calculate the data content read and write frequency based on the number and time of data content read and write, and arrange the data content stored in the database based on the data content read and write frequency calculation result to obtain a data queue.
[0011] Furthermore, the data source includes: a computer, a web page, a mobile hard disk, a database, and an existing data warehouse, and the identification result of the data content attribute is: the format of the data, the type of the data, and the size of the uncompressed data, wherein the uncompressed data refers to the data that has not been compressed or decompressed;
[0012] The identification logic of the data content security is expressed as:
[0013] DSI=AC×ω1+ES×ω2+DB×ω3+SA×ω4;
[0014] Where: DSI is the data content security performance value; AC is the historical access control effectiveness of the data content; ES is the historical encryption strength of the data content; DB is the backup integrity of the data content in the data source; SA is the efficiency of repairing vulnerabilities in the encryption algorithm used for the data content; ω1, ω2, ω3, ω4 are weights;
[0015] Among them, the data content history access control effectiveness AC is the ratio of the number of times the data content history is accessed and passed in the data source to the total number of accesses. The data content history encryption strength ES is customized by the user end. The data content history encryption strength ES takes values of 0.5, 1 or 1.5. The sum of the weights ω1, ω2, ω3, and ω4 is 1, and ω1<ω2<ω3<ω4.
[0016] Furthermore, at the stage of whether the data content is retained in the data warehouse, the user terminal manually sets a decision threshold, and based on the comparison between the decision threshold and the data content security performance value DSI, decides whether the data content corresponding to the data content security performance value DSI is retained in the data warehouse;
[0017] During the decision-making stage of whether to retain the data content in the data warehouse, after the data content is determined to be retained based on the data content security performance value DSI, the data content is synchronously traversed, the application in the data content is picked up, and run on the virtual computer, and anti-virus software is used to detect whether the running application carries a virus. If the detection result is yes, the data content is deleted from the data warehouse, otherwise, the data content is retained and stored.
[0018] Furthermore, the data content deleted in the data warehouse is retransmitted to the data source in real time, received by the data source, and the data source synchronously obtains the identification result of the data content security;
[0019] When the data warehouse receives data content feedback from the data source, the data source performs a data feedback operation based on the data feedback logic, and the data warehouse continuously receives the data content feedback from the data source. The data feedback logic indicates:
[0020] Logic1: Set the number of data sources that can simultaneously perform data content feedback operations on the data warehouse to no more than two groups;
[0021] Logic 2: When there are more than two data sources sending data content feedback requests to the data warehouse, identify the size of the data content fed back by each group of data sources, and select the data sources corresponding to the group with the largest data content and the group with the smallest data content as the targets for feeding back data content to the data warehouse;
[0022] Logic3: After completing the data content feedback to the data warehouse, jump to the ogic2 stage and execute again.
[0023] Furthermore, the number of node positions on the spider web topology is always greater than the number of newly created databases, and the number of newly created databases is always an integer multiple of the number of data types in the identification result of the attribute of the data content;
[0024] The data content stored in the database corresponding to the central node of the spider web topology is all the data content retained in the data warehouse, and the data content stored in the database corresponding to all the adjacent nodes outside the central node of the spider web topology is all the data content retained in the data warehouse. By analogy, each link node of the spider web topology corresponds to the data content stored in the database, which is always all the data content retained in the data warehouse;
[0025] Each node in the spider web topology corresponds to a database that stores data content based on different data types.
[0026] Among them, the data types include: data domain of data content, data content feedback period, area where the data source of feedback data content is located, personal perspective of data content, data content language, and data content.
[0027] The data domain classification method for data content is: classify the data content into first-level data domain, second-level data domain, third-level data domain, and fourth-level data domain in sequence according to the data produced by the enterprise's business activities, the data produced by the departments in the enterprise, the data produced by the positions under the departments, and the data produced by individual employees in each position. The data domain of the parent organization has a one-to-many subordinate relationship with the data domain of the subordinate entity, and the fourth-level data domain has a one-to-many subordinate relationship with the physical files of each data content.
[0028] Furthermore, the data content stored in the corresponding database of each node in the same ring of the spider web topology performs data migration based on the similarity of the data content, and always follows that the data with smaller content is migrated to the database where the data with larger content is located.
[0029] Furthermore, the similarity between the two sets of data contents is calculated by the following formula:
[0030]
[0031] Where: SIMM is the similarity between the two sets of data; n is the total number of characters in the data content; ω Ai is the weight of the i-th character in the data content A; ω Bi is the weight of the i-th character in the data content B; ε1 and ε2 are corrections; are the semantic vectors of data content A and data content B.
[0032] Furthermore, the data contents stored in the database are all text data contents;
[0033] When the sum of the modified ε1 and ε2 is 1, and ε1>ε2, and the similarity SIMM of the two sets of data contents is greater than 80%, the data migration operation is performed.
[0034] Furthermore, when arranging the data contents stored in the database, the data contents with high reading and writing frequencies are arranged in front, and the data contents with low reading and writing frequencies are arranged in the back;
[0035] The database bound to each node position on the spider web topology executes the data queue acquisition operation, and after acquiring the data queue, rearranges each data content stored in the database based on the order in the data queue.
[0036] Compared with the known public technology, the technical solution provided by the present invention has the following beneficial effects:
[0037] The present invention provides a method for processing data in a database. During the execution of the method, the security of the stored data content is ensured by means of data transfer. Furthermore, when the data content is stored, a local spider web topology and a plurality of groups of binding configurations are used to implement conditions for distinguishing the storage of the data content based on different attributes, thereby ultimately achieving differentiated storage of data based on different distinguishing attributes, so that the data can be effectively managed, thereby further adapting to the storage management of data groups with larger volumes and more complex types. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0039] Figure 1 A schematic diagram of a flow chart of a method for processing data in a database;
[0040] Figure 2 It is a schematic diagram of the structure of the spider web topology applied in the present invention. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] The present invention will be further described below in conjunction with the embodiments.
[0043] Embodiment 1:
[0044] A data processing method of a database in this embodiment, such as Figure 1 As shown, including:
[0045] Determine the data source, create a data warehouse, use the data warehouse to receive real-time data content feedback from the data source, identify the data domain, attributes and security of the data content, divide and identify the data domain according to the data produced by the entities in the organizational structure, and decide whether to retain the data based on the data content security identification results, so that the data with a negative decision result will be deleted from the data warehouse;
[0046] Create several groups of databases to generate a spider web topology, bind each node on the spider web topology to the database, and use each group of databases to receive the data content retained in the data warehouse and store the data content;
[0047] A set of virtual keys is generated at each node position on the spider web state topology, a hyperlink for accessing the node position binding database is created synchronously, and a click operation of the node position virtual key is set, that is, an operation of accessing the database based on the hyperlink corresponding to the node position binding database;
[0048] Record the number of times and time when the data content is read and written in the database, calculate the data content read and write frequency based on the number of times and time when the data content is read and written, and arrange the data content stored in the database based on the calculation result of the data content read and write frequency to obtain a data queue;
[0049] The data content deleted in the data warehouse is retransmitted to the data source in real time, and the data source completes the reception, and the data source synchronously obtains the identification result of the data content security;
[0050] When the data warehouse receives data content feedback from the data source, the data source performs data feedback operations based on the data feedback logic. The data warehouse continuously receives the data content feedback from the data source. The data feedback logic represents:
[0051] Logi c1: Set the number of data sources that can perform data content feedback operations to the data warehouse to no more than two groups at the same time;
[0052] Logi c2: When there are more than two groups of data sources sending data content feedback requests to the data warehouse, identify the size of the data content fed back by each group of data sources, and select the data source corresponding to the group with the largest data content and the group with the smallest data content as the target for feeding back data content to the data warehouse;
[0053] Logi c3: After completing the data content feedback to the data warehouse, jump to the ogi c2 stage and execute again;
[0054] The data content stored in the corresponding database of each node in the same ring of the spider web topology is migrated based on the similarity of the data content, and the data with smaller content is always migrated to the database with larger data content;
[0055] The similarity between the two sets of data is calculated using the following formula:
[0056]
[0057] Where: SIMM is the similarity between the two sets of data; n is the total number of characters in the data content; ω Ai is the weight of the i-th character in the data content A; ω Bi is the weight of the i-th character in the data content B; ε1 and ε2 are corrections; are the semantic vector of data content A and the semantic vector of data content B;
[0058] The data contents stored in the database are all text data contents;
[0059] When the sum of ε1 and ε2 is corrected to 1 and ε1>ε2 and the similarity SIMM of the two sets of data contents is greater than 80%, the data migration operation is performed.
[0060] In this embodiment, through the execution of the method in the above embodiment, a more stable and effective management effect is brought about for data storage, so that large-volume and complex-type data can be more properly stored and managed through the database.
[0061] Embodiment 2:
[0062] In terms of specific implementation, based on Example 1, this example refers to Figure 1 The data processing method of a database in Example 1 is further described in detail:
[0063] Data sources include: computers, web pages, mobile hard disks, databases, and existing data warehouses. The identification results of data content attributes are: data format, data type, and size of uncompressed data. Uncompressed data refers to data that has not been compressed or decompressed.
[0064] The identification logic of data content security is expressed as:
[0065] DSI=AC×ω1+ES×ω2+DB×ω3+SA×ω4;
[0066] Where: DSI is the data content security performance value; AC is the historical access control effectiveness of the data content; ES is the historical encryption strength of the data content; DB is the backup integrity of the data content in the data source; SA is the efficiency of repairing vulnerabilities in the encryption algorithm used for the data content; ω1, ω2, ω3, ω4 are weights;
[0067] Among them, the data content history access control effectiveness AC is the ratio of the number of times the data content history is accessed and passed in the data source to the total number of accesses. The data content history encryption strength ES is customized by the user end. The data content history encryption strength ES takes a value of 0.5, 1 or 1.5. The sum of the weights ω1, ω2, ω3, and ω4 is 1, and ω1<ω2<ω3<ω4;
[0068] At the stage of whether the data content is retained in the data warehouse, the user manually sets the decision threshold, and compares the decision threshold with the data content security performance value DSI to decide whether the data content corresponding to the data content security performance value DSI is retained in the data warehouse;
[0069] During the decision-making stage of whether to retain the data content in the data warehouse, after the data content is determined to be retained based on the data content security performance value DSI, the data content is synchronously traversed, the application in the data content is picked up, and run on the virtual computer, and anti-virus software is used to detect whether the running application carries a virus. If the detection result is yes, the data content is deleted from the data warehouse, otherwise, the data content is retained and stored.
[0070] In this embodiment, the above-mentioned settings and identification logic of data content security are used to provide further step execution logic support for the execution of the steps of the method in Embodiment 1.
[0071] Embodiment 3:
[0072] In terms of specific implementation, based on Example 1, this example refers to Figure 1 The data processing method of a database in Example 1 is further described in detail:
[0073] The number of node positions on the spider web topology is always greater than the number of newly created databases, and the number of newly created databases is always an integer multiple of the number of data types in the recognition result of the attribute of the data content;
[0074] The central node of the spider web topology corresponds to the data content stored in the database, which is all the data content retained in the data warehouse. All the adjacent nodes outside the central node of the spider web topology correspond to the data content stored in the database, which is all the data content retained in the data warehouse. By analogy, each link node of the spider web topology corresponds to the data content stored in the database, which is always all the data content retained in the data warehouse.
[0075] Each node in the spider web topology corresponds to a database that stores data content based on different data types.
[0076] The data types include: data content feedback period, data source region of the feedback data content, data content person perspective, data content language, and data content;
[0077] When arranging the data contents stored in the database, the data contents with high reading and writing frequency are arranged in front, and the data contents with low reading and writing frequency are arranged in the back;
[0078] The database bound to each node position on the spider web topology executes the data queue acquisition operation, and after acquiring the data queue, rearranges the data contents stored in the database based on the order in the data queue.
[0079] In this embodiment, through the above settings, the configuration of the spider web topology and the database is further limited to ensure that the spider web topology is associated with and interacts with the database, thereby jointly completing the data storage operation.
[0080] In summary, during the execution of the method in the above embodiment, the security of the stored data content is ensured by means of data transfer. Furthermore, when the data content is stored, the local spider web topology and the binding configuration of several groups realize the conditions for distinguishing the storage of data content based on different attributes, thereby ultimately realizing the differentiated storage of data according to different distinguishing attributes, so that the data can be effectively managed, thereby further adapting to the storage management of data groups with larger volume and more complex types.
[0081] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for processing data in a database, characterized in that: include: Determine the data source, create a data warehouse, use the data warehouse to receive real-time data content feedback from the data source, identify the data domain, attributes and security of the data content, divide and identify the data domain according to the data produced by the entities in the organizational structure, and decide whether to retain the data based on the data content security identification results, so that the data with a negative decision result will be deleted from the data warehouse; Create several groups of databases to generate a spider web topology, bind each node on the spider web topology to the database, use each group of databases to receive the data content retained in the data warehouse, and store the data content according to the attributes of the data content; A set of virtual keys is generated at each node position on the spider web state topology, a hyperlink for accessing the node position binding database is created synchronously, and a click operation of the node position virtual key is set, that is, an operation of accessing the database based on the hyperlink corresponding to the node position binding database; Record the number of times and time when the data content is read and written in the database, calculate the data content read and write frequency based on the number of times and time when the data content is read and written, and arrange the data content stored in the database based on the data content read and write frequency calculation result to obtain a data queue.
2. A method for processing database data according to claim 1, characterized in that: The data sources include: computers, web pages, mobile hard disks, databases, and existing data warehouses. The identification results of the data content attributes include: data format, data type, and size of uncompressed data. The uncompressed data refers to data that has not been compressed or decompressed. The identification logic of the data content security is expressed as: DSI=AC×ω1+ES×ω2+DB×ω3+SA×ω4; Where: DSI is the data content security performance value; AC is the historical access control effectiveness of the data content; ES is the historical encryption strength of the data content; DB is the backup integrity of the data content in the data source; SA is the efficiency of repairing vulnerabilities in the encryption algorithm used for the data content; ω1, ω2, ω3, ω4 are weights; Among them, the data content history access control effectiveness AC is the ratio of the number of times the data content history is accessed and passed in the data source to the total number of accesses. The data content history encryption strength ES is customized by the user end. The data content history encryption strength ES takes values of 0.5, 1 or 1.
5. The sum of the weights ω1, ω2, ω3, and ω4 is 1, and ω1<ω2<ω3<ω4.
3. A method for processing database data according to claim 2, characterized in that: At the stage of whether the data content is retained in the data warehouse, the user terminal manually sets a decision threshold, and based on the comparison between the decision threshold and the data content security performance value DSI, decides whether the data content corresponding to the data content security performance value DSI is retained in the data warehouse; During the decision-making stage of whether to retain the data content in the data warehouse, after the data content is determined to be retained based on the data content security performance value DSI, the data content is synchronously traversed, the application in the data content is picked up, and run on the virtual computer, and anti-virus software is used to detect whether the running application carries a virus. If the detection result is yes, the data content is deleted from the data warehouse, otherwise, the data content is retained and stored.
4. The method for processing database data according to claim 1, characterized in that: The data content deleted in the data warehouse is retransmitted to the data source in real time, and the data source completes the reception, and the data source synchronously obtains the identification result of the data content security; When the data warehouse receives data content feedback from the data source, the data source performs a data feedback operation based on the data feedback logic, and the data warehouse continuously receives the data content feedback from the data source. The data feedback logic indicates: Logic1: Set the number of data sources that can simultaneously perform data content feedback operations on the data warehouse to no more than two groups; Logic 2: When there are more than two data sources sending data content feedback requests to the data warehouse, identify the size of the data content fed back by each group of data sources, and select the data sources corresponding to the group with the largest data content and the group with the smallest data content as the targets for feeding back data content to the data warehouse; Logic3: After completing the data content feedback to the data warehouse, jump to the ogic2 stage and execute again.
5. The method for processing database data according to claim 1, characterized in that: The number of node positions on the spider web topology is always greater than the number of newly created databases, and the number of newly created databases is always an integer multiple of the number of data types in the identification result of the attribute of the data content; The data content stored in the database corresponding to the central node of the spider web topology is all the data content retained in the data warehouse, and the data content stored in the database corresponding to all the adjacent nodes outside the central node of the spider web topology is all the data content retained in the data warehouse. By analogy, each link node of the spider web topology corresponds to the data content stored in the database, which is always all the data content retained in the data warehouse; Each node in the spider web topology corresponds to a database that stores data content based on different data types. Among them, the data types include: data domain of data content, data content feedback period, area where the data source of feedback data content is located, personal perspective of data content, data content language, and data content. The data domain classification method for data content is: classify the data content into first-level data domain, second-level data domain, third-level data domain, and fourth-level data domain in sequence according to the data produced by the enterprise's business activities, the data produced by the departments in the enterprise, the data produced by the positions under the departments, and the data produced by individual employees in each position. The data domain of the parent organization has a one-to-many subordinate relationship with the data domain of the subordinate entity, and the fourth-level data domain has a one-to-many subordinate relationship with the physical files of each data content.
6. The method for processing database data according to claim 1, characterized in that: The data content stored in the corresponding database of each node in the same ring of the spider web topology executes data migration based on the similarity of the data content, and always follows that the data with smaller content is migrated to the database where the data with larger content is located.
7. A method for processing database data according to claim 6, characterized in that: The similarity between the two sets of data is calculated using the following formula: Where: SIMM is the similarity between the two sets of data; n is the total number of characters in the data content; ω Ai is the weight of the i-th character in the data content A; ω Bi is the weight of the i-th character in the data content B; ε1 and ε2 are corrections; are the semantic vectors of data content A and data content B.
8. A method for processing database data according to claim 7, characterized in that: The data contents stored in the database are all text data contents; When the sum of the modified ε1 and ε2 is 1, and ε1>ε2, and the similarity SIMM of the two sets of data contents is greater than 80%, the data migration operation is performed.
9. The method for processing database data according to claim 1, characterized in that: When arranging the data contents stored in the database, the data contents with high reading and writing frequency are arranged in front, and the data contents with low reading and writing frequency are arranged in the back; The database bound to each node position on the spider web topology executes the data queue acquisition operation, and after acquiring the data queue, rearranges each data content stored in the database based on the order in the data queue.