A multi-service data management and acquisition method

Abstract

The present invention relates to a multi-business data management and acquisition method, which belongs to the technical field of data storage encryption management and analysis, and solves the technical problem that multi-business multi-source heterogeneous source data cannot be fuzzily queried and data training cannot be performed after encryption. It includes extracting multi-source heterogeneous source data from multiple business systems to form a data collection, pre-processing the data collection and loading it into a data warehouse; obtaining a three-dimensional coordinate set W from the data in the data warehouse and encrypting it to obtain feature encrypted data K, directly encrypting the data in the data warehouse to obtain direct encrypted data V, and aggregating K and V to obtain encrypted data KEY; receiving a query keyword submitted by a business system user or a database administrator, generating query feature encrypted data corresponding to the query keyword, performing a fuzzy query on the encrypted data, and obtaining a matching result. The establishment and fuzzy query of a heterogeneous source data encryption database are realized.

Description

Technical Field

[0001] This invention relates to the field of data storage encryption management and analysis technology, and in particular to a method for managing and acquiring multi-service data. Background Technology

[0002] With the rapid development of information technology, enterprises face ever-increasing data management needs. In multi-business systems, data storage, encryption, management, analysis, and data mining have become particularly important. Currently, enterprises typically use Relational Database Management Systems (RDBMS) to store and manage business data. These systems provide structured data storage and powerful query capabilities. To integrate data from different sources, enterprises employ ETL (Extract, Transform, Load) tools to extract data from data sources across multiple business application systems, perform necessary transformations, and then load the data into a data warehouse.

[0003] Most data warehouses employ batch updates, updating data at predetermined time intervals. While existing database management systems offer basic security measures such as ACLs (Access Control Lists) and user authentication, these are often insufficient to address complex security threats. Traditional query techniques rely on indexes and optimized query algorithms to improve search efficiency, but these methods can encounter performance bottlenecks when handling large-scale datasets, and the inability to perform fuzzy queries after data encryption poses challenges to database management and application.

[0004] Furthermore, unstructured data increasingly constitutes a large portion of enterprise organizational data, and its volume continues to grow rapidly. Traditional data management techniques typically focus on effectively querying and ensuring security within large-scale structured datasets.

[0005] Moreover, if data is encrypted, it cannot be used for data learning and training: when enterprises introduce big data platforms or AI (Artificial Intelligence) intelligent platforms, encrypted data in the database will affect the accuracy of data governance or AI training. Data training and processing on big data platforms or AI intelligent platforms can only be carried out by decrypting the encrypted data, which can easily lead to data leakage. Summary of the Invention

[0006] Based on the above analysis, the embodiments of the present invention aim to provide a multi-service data management and acquisition method to solve the technical problems of existing databases being unable to perform fuzzy queries on encrypted data and unable to perform data training.

[0007] This invention provides a method for managing and acquiring multi-service data, comprising the following steps:

[0008] Data from multiple heterogeneous sources is extracted from multiple business systems to form a data collection, which is then preprocessed and loaded into a data warehouse.

[0009] The three-dimensional coordinate set W of the data in the data warehouse is obtained and encrypted to obtain the feature encrypted data K. The data in the data warehouse is directly encrypted to obtain the direct encrypted data V. K and V are aggregated to obtain the encrypted data KEY.

[0010] The system receives query keywords submitted by business system users or database administrators, generates encrypted query feature data corresponding to the query keywords, performs fuzzy search on the encrypted data, and obtains matching results.

[0011] Furthermore, through the heterogeneous database driving engine corresponding to the data warehouse, data is extracted from the corresponding multi-source heterogeneous source data to form a data collection;

[0012] Preprocessing is performed on the data aggregation corresponding to the multi-source heterogeneous source data, including:

[0013] Based on the data aggregation, heterogeneous source data are fused, and data cleaning is performed to obtain cleaned data, which is then loaded into the data warehouse.

[0014] Furthermore, obtaining the encrypted data KEY includes:

[0015] The corresponding three-dimensional coordinate set W is obtained by performing secondary processing on the data in the data warehouse.

[0016] Encrypt each dimension of the data elements in the three-dimensional coordinate set W to form feature-encrypted data K; encrypt the data in the data warehouse to obtain directly encrypted data V;

[0017] The feature-encrypted data K is aggregated with the direct-encrypted data V to generate complete encrypted data KEY.

[0018] Furthermore, based on the three-dimensional coordinate set W' of the query keywords submitted by the business system user or database administrator, encrypted query feature data K' is generated, including:

[0019] Obtain the query keywords and split them to form a character array collection;

[0020] The two-dimensional coordinate point set A' is obtained by decomposing the character according to its zone code or ASCII code;

[0021] Calculate the distance l' and angle r' of each coordinate in the two-dimensional coordinate set A' to the origin O;

[0022] Calculate the adjacent angle value ε' between adjacent coordinates in A';

[0023] The distance value l', angle value r', and adjacent angle value ε' are combined to form a three-dimensional feature coordinate set W';

[0024] The three-dimensional feature coordinate set W' is encrypted to form the query feature encrypted data K'.

[0025] Perform a fuzzy query K' on the encrypted data to obtain the matching results.

[0026] Further, the data elements of each dimension of the three-dimensional coordinate set W include distance values, angle values, and adjacent angle values; the corresponding three-dimensional coordinate set W is obtained by secondary processing based on the data in the data warehouse, including:

[0027] Perform character segmentation on the data in the data warehouse;

[0028] Obtain the distance and angle values ​​of the two-dimensional coordinates of each character from the origin O, as well as the adjacent angle values ​​with adjacent coordinate points; wherein, the origin O is generated based on random numbers;

[0029] Encrypting each dimension of the data elements in the three-dimensional coordinate set W to form feature-encrypted data K includes:

[0030] The distance and angle values ​​in W are concatenated and encrypted using SHA2 to obtain the SHA2 hash values ​​of the distance and angle values.

[0031] The adjacent angle values ​​in W are encrypted using MD5 hashing to obtain the MD5 hash values ​​of the adjacent angle values.

[0032] Concatenate the SHA2 hash value and the MD5 hash value to obtain the encrypted data k. i ,as follows:

[0033] k i =(sha2(l i r i )+md5(ε i ))

[0034] Obtain the encrypted data set {(k1), (k2), (k3)..., k i , ..., (k n After random shuffling and rearranging, the feature-encrypted data K is obtained.

[0035] Further, based on the two-dimensional vectorization of characters obtained from the data segmentation in the data warehouse, a two-dimensional coordinate point set A is obtained; the distance value and angle value of the two-dimensional coordinates corresponding to each character from the origin coordinates, as well as the adjacent angle value with adjacent coordinate points, are obtained, including:

[0036] Initialize the origin coordinates (0) and the common difference value (z);

[0037] Calculate the distance and angle from each coordinate point in the two-dimensional coordinate point set A to the origin coordinate 0 after subtracting the common difference z;

[0038] Calculate the adjacent angle values ​​between adjacent coordinate points in the two-dimensional coordinate point set A;

[0039] The distance value, angle value, and adjacent angle values ​​of each two-dimensional coordinate point are combined to obtain a three-dimensional coordinate set W.

[0040] Furthermore, based on the two-dimensional vectorization of characters obtained from data segmentation in the data warehouse, a two-dimensional coordinate point set A is obtained, including:

[0041] The data in the data warehouse is divided into a set of character arrays, and the set of character arrays is further decomposed according to the character's location code or ASCII code to obtain the two-dimensional vector coordinates (a, b) corresponding to each character. i b i );

[0042] Based on the initial origin coordinates 0(x, y) and the common difference z, calculate the coordinates A of each point in the two-dimensional coordinate point set A. i (a i b i The distance l from the origin 0(x, y) to the origin 0(x, y) i and angle value r i ,as follows:

[0043]

[0044] r i =sin -1 ((a i -zy) / l)×(180 / π)

[0045] Obtain the set of relationships H{(l1, r1), (l2, r2), (l3, r3)..., (l...) from the set of two-dimensional coordinate points A to the origin coordinates O. n r n )}.

[0046] Furthermore, in the set of two-dimensional coordinate points A, the adjacent coordinates A of any two points are... i (a i b i) and A i+1 (a i+1 b i+1 Calculate the adjacent angle values, and calculate the adjacent angle values ​​between the last coordinate point and the first coordinate point;

[0047] Calculate the coordinates of point A i+1 (a i+1 b i+1 Relative to coordinate point A i (a i b i The location of the coordinate system formed with the origin as the origin;

[0048] When A i+1 (a i+1 b i+1 When the location is in the first or third quadrant, calculate the coordinates of point A. i+1 (a i+1 b i+1 ) and coordinate point A i (a i b i The adjacent angle values ​​ε between )

[0049] When coordinate point A i+1 (a i+1 b i+1 When the object is located in the first quadrant, the calculation is as follows:

[0050]

[0051] ε i =sin -1 ((a i -a i+1 ) / m i )×(180 / π)

[0052] Where, m i ε i The coordinates of point A are respectively i+1 (a i+1 b i+1 ) and coordinate point A i (a i b i The distance and angle values ​​between coordinate point A; i+1 (a i+1 b i+1 When it is in the third quadrant, the last ε obtained is i Add 180 to the value;

[0053] When A i+1 (a i+1 b i+1When the location is in the second or fourth quadrant, calculate the coordinates of point A. i+1 (a i+1 b i+1 ) and coordinate point A i (a i b i The adjacent angle values ​​ε between )

[0054] When coordinate point A i+1 (a i+1 b i+1 When it is located in the second quadrant, the calculation is as follows:

[0055]

[0056] ε i =sin -1 ((a i -a i+1 ) / m i )×(180 / π)+90

[0057] When coordinate point A i+1 (a i+1 b i+1 When the value is located in the fourth quadrant, the last ε obtained is... i Add 180 to the value;

[0058] By calculating the adjacent angle values ​​between all adjacent pairs of points, we obtain the set of adjacent angle values ​​ε{(ε1), (ε2), (ε3)..., (ε... n )};

[0059] Let the relation set H{(l1, r1), (l2, r2), (l3, r3)..., (l... n r n )} and the set of adjacent angle values ​​ε{(ε1), (ε2), (ε3)..., (ε n The data is then integrated to obtain a set of three-dimensional feature vectors containing the features of the data in the data warehouse.

[0060] W{(l1, r1, ε1), (l2, r2, ε3), (l3, r3, ε3)…, (l n r n , ε n )}.

[0061] Furthermore, the character array set is decomposed according to the character's zone code or ASCII code to obtain the two-dimensional vector coordinates (a, b) corresponding to each character. i b i ),include:

[0062] If the character is a Chinese character, according to the GB2312 encoding standard, the zone code of each Chinese character is converted into two-dimensional coordinates, with the zone code as the first dimension and the position code as the second dimension, to obtain the two-dimensional vector coordinates corresponding to the Chinese character;

[0063] Otherwise, if the characters are English characters and Arabic numerals, according to the ASCII encoding, divide the ASCII code value by 16 to obtain the first dimension coordinate; take the remainder of the ASCII code value divided by 16 to obtain the second dimension coordinate.

[0064] Furthermore, based on the aforementioned data aggregation, heterogeneous source data fusion is performed, and data cleaning is carried out, including:

[0065] Based on the aforementioned data aggregation, source data with different structures and formats are mapped, transformed, and merged to obtain unified and integrated data;

[0066] Based on data fusion, further data cleaning is carried out, including removing duplicate data records, correcting erroneous data, filling in missing values, correcting erroneous data, and standardizing data formats.

[0067] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0068] 1. This invention provides a higher level of security protection for structured and unstructured data in a data warehouse by merging and encrypting a three-dimensional dataset obtained from the data warehouse, and by directly encrypting the data in the data warehouse and then aggregating the two methods. This effectively prevents data leakage and unauthorized access, strengthens the protection of data in storage and management, effectively resists external security threats and improper internal access, and improves data security.

[0069] 2. Supports fuzzy search of encrypted feature data. This invention allows fuzzy search even when data is encrypted. By generating encrypted query feature data K' of query keywords, it enables effective fuzzy retrieval of data even when the data is encrypted, solving the problem that traditional encryption methods cannot perform fuzzy searches. It overcomes the shortcomings of traditional encrypted data that cannot be fuzzily searched, improves the flexibility and efficiency of data retrieval, and enhances the efficiency of data management and application.

[0070] 3. Optimize data management and governance. By establishing a three-dimensional coordinate set and generating feature-encrypted data, this invention reduces the complexity of data management, enabling DBAs (Database Administrators) to effectively manage data without decryption, thus improving data governance efficiency.

[0071] 4. By integrating and cleaning multi-source heterogeneous data from multiple business systems, the consistency and accuracy of the data were ensured, providing a high-quality data foundation for subsequent data analysis and mining;

[0072] 5. This invention reduces the risk of data leakage during the training process of big data and AI platforms by allowing the direct use of encrypted feature data, while improving data processing efficiency and the accuracy of AI models. It supports seamless integration with big data and AI platforms.

[0073] 6. This invention provides a scalable data management strategy through a method of setting three-dimensional coordinates of data in a data warehouse. This strategy can adapt to the ever-increasing volume of unstructured data and diverse data sources, enhancing the flexibility and scalability of data management.

[0074] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0075] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0076] Figure 1 A flowchart illustrating a multi-service data management and acquisition method;

[0077] Figure 2 A diagram illustrating the architecture of 3D feature extraction and encryption algorithms for encrypted data;

[0078] Figure 3 This is a schematic diagram of the GB2312 zone code. Detailed Implementation

[0079] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0080] Existing enterprises often have multiple business systems with fragmented heterogeneous data sources, resulting in duplicate data storage and inconsistencies. With the advent of the big data era, the rapid growth of structured and unstructured data has presented unprecedented challenges to data management and security. This invention proposes a multi-business data management and acquisition method. The basic idea is to preprocess multi-source heterogeneous data from multiple business systems and load it into a data warehouse. The data in the data warehouse is then segmented, and the segmented data is modeled in a three-dimensional coordinate system, converted into three-dimensional graphics, and features are extracted and encrypted to obtain encrypted data containing these features. The source data is then encrypted using RSA encryption. Finally, the two encrypted data sets are aggregated to obtain the final encrypted data. Fuzzy queries are performed on the encrypted features of the encrypted data to obtain matching results.

[0081] A specific embodiment of the present invention discloses a method for managing and acquiring multi-service data, such as... Figure 1 , Figure 2 As shown, it includes:

[0082] Step S1: Extract heterogeneous source data from multiple business systems, preprocess it, and then load it into the data warehouse;

[0083] Step S2: Obtain the three-dimensional coordinate set W from the data in the data warehouse and encrypt it to obtain feature encrypted data K. Directly encrypt the data in the data warehouse to obtain direct encrypted data V. Aggregate K and V to obtain encrypted data KEY.

[0084] Step S3: Receive query keywords submitted by business system users or database administrators, generate encrypted query feature data corresponding to the query keywords, perform fuzzy search on the encrypted data, and obtain matching results.

[0085] Step S1 includes steps S11-S13.

[0086] Step S1: Extract heterogeneous source data from multiple business systems to form a data collection.

[0087] Within an enterprise, multiple business systems support the normal operation of its business. Due to various reasons, these business systems are often fragmented, and may even use application software from different vendors. Each business system has a different database and table structure; the same business data is repeatedly stored in different business systems, resulting in significant redundancy and making subsequent updates and maintenance extremely challenging.

[0088] With the development of business operations and the rapid advancement of IT technology, the arrival of the big data era has made the integration and collaborative operation of multiple business systems crucial for ensuring the normal operation of enterprises. Integrating multi-source, heterogeneous data from multiple business systems and effectively consolidating this data improves the efficiency of data governance and management.

[0089] For example, the multiple business systems within an enterprise include: Human Resource Management System (HRMS), Financial Management System, Customer Relationship Management System (CRM), Enterprise Resource Planning System (ERP), Project Management System (PMS), Product Lifecycle Management System (PLM), Office Automation System (OA), Knowledge Management System, Asset and Equipment Management System, Master Data Management System, Contract Management System, and Business Intelligence Data Analysis Platform, etc.

[0090] Data can also be obtained from external data sources, IoT sensor data, web crawler data, manually entered data, or in real time.

[0091] Multi-source heterogeneous data includes structured data and unstructured data.

[0092] Structured data refers to data stored in a relational database that has a fixed format or schema. For example, tabular data, where each row represents a record and each column represents a field. Data is organized in rows and columns. It is easy to sort and search. Queries are typically performed using SQL statements. Data types and formats are predefined. Examples include customer information tables, order records, and inventory lists.

[0093] Structured data is typically stored in relational database management systems such as MySQL, MS SQL Server, and Oracle. Querying and analysis are relatively simple.

[0094] Unstructured data lacks a fixed format or pattern, making it difficult to store using database tables. Examples include text files, images, videos, and audio files. The data formats are diverse, lacking a unified structure. It is difficult to store and query using traditional databases. It occupies a large amount of storage space and grows rapidly. Examples include image, video, and audio files.

[0095] They are typically stored in file systems, object storage services, or NoSQL databases, such as the Hadoop Distributed File System (HDFS) and MongoDB.

[0096] For multiple heterogeneous data sources corresponding to multiple business systems, data is extracted and aggregated through the heterogeneous database driving engine of the data warehouse.

[0097] The heterogeneous database driver engine corresponding to the data warehouse can establish connections with different types of databases, including but not limited to MySQL, Oracle, MS SQL Server, MongoDB, etc.

[0098] During the data extraction process, the engine can extract data from the source format into the format required by the data warehouse to ensure data consistency and compatibility.

[0099] The engine supports multiple data access protocols, such as ODBC (Open Database Connectivity), JDBC (Java Database Connectivity), or other proprietary protocols.

[0100] For each type of database, the engine may contain one or more drivers that act as a medium between multi-source heterogeneous source data and the data warehouse.

[0101] Step S12: After preprocessing the collected data, it is loaded into the data warehouse.

[0102] The heterogeneous database driving engine corresponding to the data warehouse is used to extract data from the corresponding multi-source heterogeneous source data to form a data collection.

[0103] Preprocessing is performed on the data aggregation corresponding to the multi-source heterogeneous source data, including:

[0104] Based on the data aggregation, heterogeneous source data are fused, and data cleaning is performed to obtain cleaned data, which is then loaded into the data warehouse.

[0105] Data fusion and cleaning of heterogeneous source data are performed as follows:

[0106] Based on the aforementioned data aggregation, heterogeneous source data fusion is performed, followed by data cleaning, including:

[0107] Based on the aforementioned data aggregation, source data with different structures and formats are mapped, transformed, and merged to obtain unified and integrated data;

[0108] Based on data fusion, further data cleaning is carried out, including removing duplicate data records, correcting erroneous data, filling in missing values, correcting erroneous data, and standardizing data formats.

[0109] Step S1 establishes the data foundation, providing support for subsequent data analysis, encryption, and fuzzy queries.

[0110] Step S2 includes S21-S24.

[0111] The three-dimensional coordinate set W of the data in the data warehouse is obtained and encrypted to obtain the feature encrypted data K. The data in the data warehouse is directly encrypted to obtain the direct encrypted data V. K and V are aggregated to obtain the encrypted data KEY.

[0112] The process of obtaining the encrypted data KEY includes:

[0113] The corresponding three-dimensional coordinate set W is obtained by performing secondary processing on the data in the data warehouse.

[0114] Encrypt each dimension of the data elements in the three-dimensional coordinate set W to form feature-encrypted data K; encrypt the data in the data warehouse to obtain directly encrypted data V;

[0115] The feature-encrypted data K is aggregated with the direct-encrypted data V to generate complete encrypted data KEY.

[0116] Step S21: Preprocess the multi-source heterogeneous source data to obtain the three-dimensional coordinate set W corresponding to the source data.

[0117] Based on the two-dimensional vectorization of characters obtained from the data segmentation in the data warehouse, a two-dimensional coordinate point set A is obtained; the distance value and angle value of the two-dimensional coordinates corresponding to each character from the origin coordinates, as well as the adjacent angle values ​​with adjacent coordinate points, are obtained, including:

[0118] Initialize the origin coordinates O and the common difference value z;

[0119] Calculate the distance and angle from each coordinate point in the two-dimensional coordinate point set A to the origin coordinate O after subtracting the common difference z;

[0120] Calculate the adjacent angle values ​​between adjacent coordinate points in the two-dimensional coordinate point set A;

[0121] The distance value, angle value, and adjacent angle values ​​of each two-dimensional coordinate point are combined to obtain a three-dimensional coordinate set W.

[0122] The data elements of each dimension of the three-dimensional coordinate set W include distance values, angle values, and adjacent angle values; the corresponding three-dimensional coordinate set W is obtained by secondary processing based on the data in the data warehouse, including:

[0123] Perform character segmentation on the data in the data warehouse;

[0124] Obtain the distance and angle values ​​of the two-dimensional coordinates of each character from the origin coordinate O, as well as the adjacent angle values ​​with adjacent coordinate points; wherein, the origin coordinate O is generated based on random numbers.

[0125] In one optional implementation, step S21 specifically includes the following steps S211-S215.

[0126] Step S211: Initialize the origin coordinates O and the common difference z.

[0127] In this invention, the origin coordinates O and the common difference z are automatically and randomly generated by the algorithm during the initialization phase.

[0128] The origin coordinates O and the common difference z are generated based on random numbers and will not be changed after generation.

[0129] For example, the method for generating the origin coordinates O is as follows:

[0130] Use a secure random number generator to generate two independent random numbers, which will be used as the coordinate values ​​for the x-axis and y-axis, respectively.

[0131] Assume the ranges of the x-axis and y-axis are x min x max and y min y max The coordinates of the origin O generated are shown in formula (1):

[0132]

[0133] Among them, O x For the range (x) min x max A random number generated within ) ; O y For the range (y) min y max A random number generated within ).

[0134] For example, the method for generating the common difference z is as follows:

[0135] Initialize using a secure random number generator to generate a random number within a predefined range;

[0136] Assume the range of the common difference z is z min , z max The resulting common difference z is shown in formula (2):

[0137] z = rand(z) min , z max ) Formula (2)

[0138] Where z is the range (z min , z max A random number generated.

[0139] The common difference z is used to establish an irregular coordinate system and reduce the possibility of the coordinate system origin being exposed.

[0140] Step S212: Based on the two-dimensional vectorization of characters obtained from data segmentation in the data warehouse, a two-dimensional coordinate point set A is obtained, including:

[0141] The data in the data warehouse is divided into a set of character arrays, and the set of character arrays is further decomposed according to the character's location code or ASCII code to obtain the two-dimensional vector coordinates (a, b) corresponding to each character. i ,b i ).

[0142] The set of two-dimensional vector coordinates of all characters in the data in the data warehouse is represented as shown in formula (3):

[0143] A{(a1,b1),(a2,b2),(a3,b3)…,(a i ,b i ),(a n ,b n )} Formula (3)

[0144] Where n is the length of the data set.

[0145] The length n of the dataset corresponds to the length of the data in the data warehouse. For a source dataset of length n, the length of its dataset is also n.

[0146] The character array set is decomposed according to the character's zone code or ASCII code to obtain the two-dimensional vector coordinates (a, b) corresponding to each character. i ,b i ),include:

[0147] If the character is a Chinese character, according to the GB2312 encoding standard, the zone code of each Chinese character is converted into two-dimensional coordinates, with the zone code as the first dimension and the position code as the second dimension, to obtain the two-dimensional vector coordinates corresponding to the Chinese character;

[0148] Otherwise, if the characters are English characters and Arabic numerals, according to the ASCII encoding, divide the ASCII code value by 16 to obtain the first dimension coordinate; take the remainder of the ASCII code value divided by 16 to obtain the second dimension coordinate.

[0149] For example, the zone code of Chinese characters is the zone code corresponding to GB2312.

[0150] GB2312 is an encoding standard for Chinese characters. For English characters and Arabic numerals, GB2312 does not have a corresponding zone code encoding. Instead, it uses ASCII code to represent English characters and Arabic numerals. GB2312 is compatible with ASCII encoding.

[0151] Each character of the data in the data warehouse (including Chinese characters, English characters, and Arabic numerals) is separately converted into a corresponding two-dimensional vector coordinate. Exemplarily, each character in the data "Li Si is a teacher" in the data warehouse will obtain a two-dimensional vector coordinate, forming a set containing multiple coordinates.

[0152] The features of each character are processed separately and then used for feature extraction and encryption.

[0153] One data value represents one coordinate.

[0154] The coordinate system here is a plane coordinate system.

[0155] Split the data "Li Si is a teacher" in the data warehouse into a set of character arrays: ['Li', 'Si', 'is', 'a', 'teacher'].

[0156] As Figure 3 shown, 32 in the upper left corner represents the area code, in decimal representation, the horizontal and vertical "78" of the character "Li" represents the "bit code", in decimal representation, the two-dimensional vector coordinate (a i , b i ) of the character "Li" = (32, 78).

[0157] And so on for other characters.

[0158] It can also be considered to subtract 0xA1 (here in hexadecimal, corresponding to 160 in binary) from the area code and the bit code at the same time. The effect of subtracting 0xA1 is to map the area-bit code encoded by GB2312 to the range of 0 to 63, simplifying data calculation.

[0159] The ASCII code value conversion step is applicable to English uppercase and lowercase letters and Arabic numerals.

[0160] Step S213, based on the initialized origin coordinate O(x, y) and the common difference z, calculate, for each coordinate A in the two-dimensional coordinate point set A i (a i , b i ), the distance l i to the origin 0(x, y) and the angle value r i , as shown in formulas (4)-(5):

[0161]

[0162] r i = sin -1 ((a i - z - y) / l) × (180 / π) Formula (5)

[0163] Obtain the set of relationships H{(l1, r1), (l2, r2), (l3, r3)..., (l...) from the set of two-dimensional coordinate points A to the origin coordinates O. n r n )}.

[0164] In set A, after subtracting the common difference z from each coordinate, we get the distance 1 from the origin and the angle r.

[0165] Subtracting the common difference z serves to establish an irregular coordinate system and reduce the likelihood of the coordinate system origin being exposed.

[0166] Step S214: Calculate the adjacent angle value ε between adjacent coordinates in set A.

[0167] In the set of two-dimensional coordinate points A, the adjacent coordinates A' of any two points are A''. i (a i b i ) and A i+1 (a i+1 b i+1 Calculate the adjacent angle values, and calculate the adjacent angle values ​​between the last coordinate point and the first coordinate point;

[0168] Calculate the coordinates of point A i+1 (a i+1 b i+1 Relative to coordinate point A i (a i b i The location of the coordinate system formed with the origin as the origin;

[0169] When A i+1 (a i+1 b i+1 When the location is in the first or third quadrant, calculate the coordinates of point A. i+1 (a i+1 b i+1 ) and coordinate point A i (a i b i The adjacent angle values ​​ε between )

[0170] When coordinate point A i+1 (a i+1 b i+1 When the y-threshold is in the first quadrant, the calculation is as shown in formulas (6)-(7):

[0171]

[0172] ε i =sin -1 ((a i -a i+1 ) / m i)×(180 / π) Formula (7)

[0173] Where, m i ε i The coordinates of point A are respectively i+1 (a i+1 b i+1 ) and coordinate point A i (a i b i The distance and angle values ​​between coordinate point A; i+1 (a i+1 b i+1 When it is in the third quadrant, the last ε obtained is i Add 180 to the value;

[0174] When A i+1 (a i+1 b i+1 When the location is in the second or fourth quadrant, calculate the coordinates of point A. i+1 (a i+1 b i+1 ) and coordinate point A i (a i b i The adjacent angle values ​​ε between )

[0175] When coordinate point A i+1 (a i+1 b i+1 When the quadrant is in the second quadrant, the calculation is as shown in formulas (8)-(9):

[0176]

[0177] ε i =sin -1 ((a i -a i+1 ) / m i )×(180 / π)+90 formula (9)

[0178] When coordinate point A i+1 (a i+1 b i+1 When the value is located in the fourth quadrant, the last ε obtained is... i Add 180 to the value;

[0179] By calculating the adjacent angle values ​​between all adjacent pairs of points, we obtain the set of adjacent angle values ​​ε{(ε1), (ε2), (ε3)..., (ε... n )};

[0180] Let the relation set H{(l1, r1), (l2, r2), (l3, r3)..., (l... nr n )} and the set of adjacent angle values ​​ε{(ε1), (ε2), (ε3)..., (ε n The data is integrated to obtain a three-dimensional feature vector set W{(l1, r1, ε1), (l2, r2, ε3), (l3, r3, ε3)..., (l...)} containing the features of the data in the data warehouse. n r n , ε n )}.

[0181] Take any two adjacent coordinates A from set A. i (a i b i ) and A i+1 (a i+1 b i+1 ), calculate the coordinates of point A i+1 (a i+1 b i+1 Relative to coordinate point A i (a i b i The location of the coordinate system formed with the origin as the origin;

[0182] Based on whether the data in the segmented data warehouse are adjacent, it can be determined whether two coordinates are adjacent coordinates.

[0183] Among them, coordinate point A i (a i b i A coordinate system is formed with the origin 0 as the origin. The directions of the x-axis and y-axis of the coordinate system are based on the initial origin 0 and are consistent with the directions of the x-axis and y-axis of the initial coordinate system.

[0184] The last coordinate point is compared with the first coordinate point to calculate the adjacent angle value; therefore, the number of adjacent angle values ​​corresponding to n coordinate points is still n.

[0185] Step S215: Combine the distance 1, angle value r, and adjacent angle values ​​ε to obtain the three-dimensional coordinate set W.

[0186] Let the relation set H{(l1, r1), (l2, r2), (l3, r3)..., (l... n r n )} and the set of adjacent angle values ​​ε{(ε1), (ε2), (ε3)..., (ε n The data in the data warehouse is integrated to obtain a three-dimensional feature vector set W{(l1, r1, ε1), (l2, r2, ε3), (l3, r3, ε3)..., (l... n r n , ε n )}.

[0187] Based on the set H{(l1, r1), (l2, r2), (l3, r3)..., (l... n r n )} and the set ε{(ε1), (ε2), (ε3)…, (ε n The data in the data warehouse is integrated to obtain a three-dimensional feature vector set W{(l1, r1, ε1), (l2, r2, ε3), (l3, r3, ε3)..., (l... n r n , ε n )}.

[0188] Step S22: Encrypt the data elements of each dimension in the three-dimensional coordinate set W to form feature encrypted data K.

[0189] Encrypting each dimension of the data elements in the three-dimensional coordinate set W to form feature-encrypted data K includes:

[0190] The distance and angle values ​​in W are concatenated and encrypted using SHA2 to obtain the SHA2 hash values ​​of the distance and angle values.

[0191] The adjacent angle values ​​in W are encrypted using MD5 hashing to obtain the MD5 hash values ​​of the adjacent angle values.

[0192] Concatenate the SHA2 hash value and the MD5 hash value to obtain the encrypted data k. i As shown in formula (10):

[0193] k i =(sha2(l i r i )+md5(ε i )) Formula (10)

[0194] Obtain the encrypted data set {(k1), (k2), (k3)..., k i , ..., (k n After random shuffling and rearranging, the feature-encrypted data K is obtained.

[0195] Join data elements (l) in set W i r i ), perform SHA2 encryption to obtain the SHA2 hash value of the distance value and angle value;

[0196] Calculate ε i The MD5 hash value; concatenate the SHA2 hash value and the MD5 hash value to get k. i , {(k1), (k2), (k3)..., ki , ..., (k n The obtained encrypted data set k i After random shuffling and rearranging, the feature-encrypted data K is obtained.

[0197] Step S23: Encrypt the data in the data warehouse to obtain directly encrypted data V.

[0198] For example, RSA encryption is applied to data in a data warehouse to obtain directly encrypted data V.

[0199] In practice, other data encryption methods can also be selected according to business needs.

[0200] Step S24: Aggregate the feature-encrypted data K with the direct-encrypted data V to generate complete encrypted data KEY and store it in the database.

[0201] The data in the data warehouse is encrypted using the RSA encryption algorithm to obtain encrypted data V; the encrypted data K and V are concatenated together.

[0202] The encrypted data V is aggregated with the feature-encrypted data K to obtain the complete encrypted data KEY, which is then stored in the database.

[0203] For example, the database supports all structured and unstructured databases.

[0204] Step S3, specifically.

[0205] Users of multiple business systems, as well as database administrators, can perform fuzzy queries on data.

[0206] Based on the three-dimensional coordinate set W' of the query keywords submitted by the user or database administrator of the business system, encrypted query feature data K' is generated, including:

[0207] Obtain the query keywords and split them to form a character array collection;

[0208] The two-dimensional coordinate point set A' is obtained by decomposing the character according to its zone code or ASCII code;

[0209] Calculate the distance value 1' and angle value r' of each coordinate in the two-dimensional coordinate set A' to the origin coordinate 0;

[0210] Calculate the adjacent angle value ε' between adjacent coordinates in A';

[0211] The distance value l', angle value r', and adjacent angle value ε' are combined to form a three-dimensional feature coordinate set W';

[0212] The three-dimensional feature coordinate set W' is encrypted to form the query feature encrypted data K'.

[0213] Perform a fuzzy query K' on the encrypted data to obtain the matching results.

[0214] For example, the query keywords are defined by each business system user or database administrator.

[0215] The result of the fuzzy query is either empty or encrypted feature data. The obtained fuzzy query result is encrypted feature data.

[0216] Each business system user has different query permissions; and database administrators are also divided into different levels with different query permissions.

[0217] If the query requires plaintext data, or if the user of the business system performing the query or the database administrator has decryption privileges, the query results can be reverse-engineered to obtain the plaintext data.

[0218] In summary, the multi-service data management and acquisition method of this invention has the following beneficial effects:

[0219] 1. This invention provides a higher level of security protection for structured and unstructured data in a data warehouse by merging and encrypting a three-dimensional dataset obtained from the data warehouse, and by directly encrypting the data in the data warehouse and then aggregating the two methods. This effectively prevents data leakage and unauthorized access, strengthens the protection of data in storage and management, effectively resists external security threats and improper internal access, and improves data security.

[0220] 2. Supports fuzzy search of encrypted feature data. This invention allows fuzzy search even when data is encrypted. By generating encrypted query feature data K' of query keywords, it enables effective fuzzy retrieval of data even when the data is encrypted, solving the problem that traditional encryption methods cannot perform fuzzy searches. It overcomes the shortcomings of traditional encrypted data that cannot be fuzzily searched, improves the flexibility and efficiency of data retrieval, and enhances the efficiency of data management and application.

[0221] 3. Optimize data management and governance. By establishing a three-dimensional coordinate set and generating feature-encrypted data, this invention reduces the complexity of data management, enabling DBAs (Database Administrators) to effectively manage data without decryption, thus improving data governance efficiency.

[0222] 4. By integrating and cleaning multi-source heterogeneous data from multiple business systems, the consistency and accuracy of the data were ensured, providing a high-quality data foundation for subsequent data analysis and mining;

[0223] 5. This invention reduces the risk of data leakage during the training process of big data and AI platforms by allowing the direct use of encrypted feature data, while improving data processing efficiency and the accuracy of AI models. It supports seamless integration with big data and AI platforms.

[0224] 6. This invention provides a scalable data management strategy through a method of setting three-dimensional coordinates of data in a data warehouse. This strategy can adapt to the ever-increasing volume of unstructured data and diverse data sources, enhancing the flexibility and scalability of data management.

[0225] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-service data management and acquisition method, characterized in that: The steps include: Extracting multi-source heterogeneous source data from multiple business systems to form data collection, pre-processing the data collection and then loading it into the data warehouse; Obtain a three-dimensional coordinate set W from the data in the data warehouse and encrypt it to obtain feature encrypted data K, directly encrypt the data in the data warehouse to obtain direct encrypted data V, and aggregate K and V to obtain encrypted data KEY; Receive a query keyword submitted by a business system user or a database administrator, generate encrypted query feature data corresponding to the query keyword, perform fuzzy query on the encrypted data, and obtain matching results; The step of obtaining the encrypted data KEY includes: Perform secondary processing based on the data in the data warehouse to obtain a corresponding three-dimensional coordinate set W; Encrypting each dimensional data element in the three-dimensional coordinate set W to form characteristic encrypted data K; encrypting the data in the data warehouse to obtain direct encrypted data V; Aggregate the characteristic encrypted data K and the direct encrypted data V to generate complete encrypted data KEY; Based on the three-dimensional coordinate set W' of the query keyword submitted by the business system user or database administrator, the query feature encrypted data K' is encrypted and generated, including: Get the query keyword and split it to form a character array set; Decompose the character according to its area code or ASCII code to obtain a two-dimensional coordinate point set A'; Calculate the distance value l' and the angle value r' from each coordinate in the two-dimensional coordinate set A' to the origin coordinate O; Calculate the adjacent angle values ​​ε' between adjacent coordinates in A'; The distance value l', the angle value r' and the adjacent angle value ε' are combined to form a three-dimensional feature coordinate set W'; Encrypt the three-dimensional feature coordinate set W' to form query feature encrypted data K'; A fuzzy query of K' is performed on the encrypted data to obtain a matching result.

2. The method according to claim 1, characterized in that: By driving the heterogeneous database engine corresponding to the data warehouse, data extraction is performed on the corresponding multi-source heterogeneous source data to form data collection; Preprocessing the data collection corresponding to the multi-source heterogeneous source data includes: Based on the data collection, heterogeneous source data is fused and data cleaning is performed to obtain cleaned data, which is then loaded into the data warehouse.

3. The method according to claim 1, characterized in that Each dimensional data element of the three-dimensional coordinate set W includes a distance value, an angle value and an adjacent angle value; The corresponding three-dimensional coordinate set W is obtained by performing secondary processing based on the data in the data warehouse, including: Performing character segmentation on the data in the data warehouse; Obtain the distance value and angle value of the two-dimensional coordinates corresponding to each character from the origin coordinate O, as well as the adjacent angle values ​​with adjacent coordinate points; wherein the origin coordinate O is generated based on a random number; Encrypting each dimensional data element in the three-dimensional coordinate set W to form feature encrypted data K, including: Connect the distance value and angle value in W, perform sha2 encryption, and obtain the sha2 hash value of the distance value and angle value; Perform md5 hash encryption on the adjacent angle values ​​in W to obtain the md5 hash values ​​of the adjacent angle values; The sha2 hash value and the md5 hash value are connected to obtain the encrypted data k i ,as follows: <h2 style=";text-align:left;direction:ltr">k<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (sha2(l<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> ,r<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> )+md5(ε<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> )) Among them, l i 、r i are the distances between the two-dimensional coordinates of each character in the three-dimensional coordinate set W and the origin coordinate O, ε i is the adjacent angle value of adjacent coordinate points in the three-dimensional coordinate set W; 1≤i≤n, n is the length of the two-dimensional vector coordinate set of all characters in the data warehouse; Get the encrypted data set {(k1),(k2),(k3)…,k i ,…,(k n )}, after random reordering, integrate to obtain the characteristic encrypted data K.

4. The method according to claim 3, characterized in that Based on the two-dimensional vectorization of the characters obtained by segmentation in the data in the data warehouse, a two-dimensional coordinate point set A is obtained; the distance value and angle value of the two-dimensional coordinate corresponding to each character from the origin coordinate, as well as the adjacent angle value with the adjacent coordinate point are obtained, including: Initialize the origin coordinates O and the common difference z; Calculate the distance and angle from each coordinate point in the two-dimensional coordinate point set A to the origin coordinate O after subtracting the common difference z; Calculate adjacent angle values ​​between adjacent coordinate points in the two-dimensional coordinate point set A; The distance value, angle value, and adjacent angle value of each two-dimensional coordinate point are combined to obtain a three-dimensional coordinate set W.

5. The method according to claim 4, characterized in that: The two-dimensional coordinate point set A is obtained by two-dimensional vectorization of characters obtained by data segmentation in the data warehouse, including: The data in the data warehouse is divided into a character array set, and the character array set is disassembled according to the character location code or ASCII code to obtain the two-dimensional vector coordinates (a i ,b i ); Based on the initialization origin coordinate O(x, y) and the common difference z, calculate each coordinate A in the two-dimensional coordinate point set A. i (a i ,b i ) to the origin O(x,y) i and the angle r i ,as follows: r i =without -1 ((to i -zy) / l)×(180 / π) Where l is the distance from each coordinate in the two-dimensional coordinate point set A to the origin after subtracting the common difference z; Obtain the relationship set H {(l1,r1),(l2,r2),(l3,r3)…,(l n ,r n )}.

6. The method according to claim 5, characterized in that: The adjacent coordinates A of any two points in the two-dimensional coordinate point set A i (a i ,b i ) and A i+1 (a i+1 ,b i+1 ) Calculate the adjacent angle values, the last coordinate point and the first coordinate point calculate the adjacent angle values; Calculate the coordinates of point A i+1 (a i+1 ,b i+1 ) relative to the coordinate point A i (a i ,b i ) as the origin; When A i+1 (a i+1 ,b i+1 ) is in the first or third quadrant, calculate the coordinate point A i+1 (a i+1 ,b i+1 ) and coordinate point A i (a i ,b i ) between adjacent angle values ​​ε; When coordinate point A i+1 (a i+1 ,b i+1 ) is in the first quadrant, the calculation is as follows: e i =sin -1 ((a i -a i+1 ) / m i )×(180 / π) Among them, m i , ε i The coordinates of point A are i+1 (a i+1 ,b i+1 ) and coordinate point A i (a i ,b i ) between the distance and angle values; when coordinate point A i+1 (a i+1 ,b i+1 ) is in the third quadrant, the last obtained ε i Add 180 to the value; When A i+1 (a i+1 ,b i+1 ) is in the second or fourth quadrant, calculate the coordinate point A i+1 (a i+1 ,b i+1 ) and coordinate point A i (a i ,b i ) between adjacent angle values ​​ε; When coordinate point A i+1 (a i+1 ,b i+1 ) is in the second quadrant, the calculation is as follows: e i =sin -1 ((a i -a i+1 ) / m i )×(180 / π)+90 When coordinate point A i+1 (a i+1 ,b i+1 ) is in the fourth quadrant, the last obtained εi value plus 180; By calculating the adjacent angle values ​​between all adjacent points, we can obtain the adjacent angle value set ε{(ε1),(ε2),(ε3)…,(ε n )}; The relationship set H{(l1,r1),(l2,r2),(l3,r3)…,(l n ,r n )} and the adjacent angle value set ε{(ε1),(ε2),(ε3)…,(ε n )}, and integrate to obtain a three-dimensional feature vector set containing the features of the data in the data warehouse W{(l1,r1,ε1),(l2,r2,ε3),(l3,r3,ε3)…,(l n ,r n ,he n )}。 7. The method according to claim 6, characterized in that: The character array set is disassembled according to the character location code or ASCII code to obtain the two-dimensional vector coordinates (a i ,b i ),include: If the character is a Chinese character, according to the GB2312 coding standard, the area code of each Chinese character is converted into a two-dimensional coordinate, the area code is the first-dimensional coordinate, and the bit code is the second-dimensional coordinate, to obtain the two-dimensional vector coordinate corresponding to the Chinese character; Otherwise, if the character is an English character or Arabic numeral, according to the ASCII encoding, the ASCII code value is divided by 16 to obtain the first-dimensional coordinate; the ASCII code value modulo 16 is obtained to obtain the second-dimensional coordinate.

8. The method according to any one of claims 1 to 7, characterized in that: Based on the data collection, heterogeneous source data is fused and data cleaning is performed, including: Based on the data collection, source data of different structures and formats are mapped, converted and merged to obtain integrated and unified data; On the basis of data fusion, further data cleaning is carried out, including removing duplicate data records, correcting erroneous data, filling missing values, correcting erroneous data and standardizing data formats.

Images