A database management system, method, device and medium based on multiple data sources

Abstract

The present invention discloses a database management system, method, device and medium based on multiple data sources, including a data request module, a data control module, a data comparison module, a data classification module, a data storage module and a security management module that are communicatively connected. The data request module responds to input requests from any of the multiple data sources, performs format conversion and preliminary deduplication on the input data to generate a deduplication file, encrypts and transmits the deduplication file to the data control module according to an agreed secret key, and extracts the initial information of the deduplication file after decryption by the data control module. The data comparison module uses written data to dedupe the initial information to generate target information and update the overlap database. The data classification module uses information keywords of the target information to determine the classification result based on the target classification model. The data storage module imports the target information into the database according to the classification result. The security management module performs security management and control on the system to improve the overall management reliability of the database.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a database management system, method, device and medium based on multiple data sources. Background Art

[0002] In the era of big data, the storage of massive data has become a focus of increasing attention. With the rapid development of the Internet, network databases, as a management device that can centrally store various data, have entered a development boom.

[0003] In the existing management technology of network databases, when collecting and storing data from different sources, the data must first be converted into a unified format, which makes system construction more complicated. At the same time, due to the wide range of data sources, it may be duplicated with data in the network database. Storing duplicate data will cause additional memory usage and low efficiency in subsequent data retrieval, affecting the overall management reliability of the network database. Summary of the Invention

[0004] The present invention provides a database management system, method, device and medium based on multiple data sources, which solves the technical problem of low overall management reliability of network databases in the prior art.

[0005] A first aspect of the present invention provides a database management system based on multiple data sources, comprising a data request module, a data control module, a data comparison module, a data classification module, a data storage module and a security management module connected in communication;

[0006] The data request module is used to respond to input requests from any of the multiple data sources, perform format conversion on the input data carried in the input request, perform deduplication operations using a preset coincidence database, and construct a corresponding encrypted file from the generated deduplication file according to the agreed secret key;

[0007] The data control module is configured to decrypt the encrypted file based on the agreed key to output the deduplication file, and extract initial information of the deduplication file;

[0008] The data comparison module is used to screen the initial information using the written data of the database based on the coincidence comparison algorithm, generate target information and update the coincidence database;

[0009] The data classification module is used to crawl information keywords of the target information using crawler technology, input the information keywords into a preset target classification model and output corresponding classification results;

[0010] The data storage module is configured to import the target information into the database according to the classification result;

[0011] The security management module is used to perform security monitoring on the data request module, the data control module, the data comparison module, the data classification module and the data storage module, and execute a corresponding alarm mechanism according to the monitoring results.

[0012] Optionally, the data request module is specifically configured to:

[0013] Responding to input requests from any of the multiple data sources, parsing the input requests to obtain corresponding input data;

[0014] Convert the input data into an XML file, and use crawler technology to crawl data keywords of the data information in the XML file;

[0015] The data keyword is matched with the written coincidence keyword in the preset coincidence database to determine the target data keyword and the target coincidence keyword that are successfully matched;

[0016] Performing an overlap operation on the data information of the target data keyword and the written overlap information of the target overlap keyword based on an overlap comparison algorithm, and outputting duplicate information;

[0017] Screening out duplicate information from the data information to generate a deduplication file;

[0018] The deduplication file is encrypted using an agreed key to construct an encrypted file.

[0019] Optionally, the data comparison module is specifically used to:

[0020] Receive the initial information and obtain the write data of the database;

[0021] Based on the coincidence comparison algorithm, the initial information and the written data are compared for coincidence and the coincidence information is output;

[0022] performing a deduplication operation on the initial information according to the overlap information to generate target information;

[0023] Using crawler technology to crawl the overlapping information for keywords and construct corresponding overlapping keywords;

[0024] The overlap keyword and the overlap information are imported into the overlap database as new write-in overlap keyword and new write-in overlap information.

[0025] Optionally, the data classification module is further configured to:

[0026] Obtaining category sample data, and dividing the category sample data into training data and test data according to a preset ratio; wherein the category sample data includes the main classification categories and subordinate classification categories of the database and corresponding classification keywords;

[0027] Based on the Monte Carlo algorithm, an initial classification model is constructed using the training data;

[0028] Using the test data to verify the initial classification model, and outputting a verification result;

[0029] If the verification result is standard, the initial classification model is used as the target classification model

[0030] If the verification result is non-standard, the initial classification model is trained and optimized using the training data, and the process jumps to the step of verifying the initial classification model using the test data and outputting the verification result.

[0031] Optionally, the security management module includes a security monitoring unit and an alarm feedback unit;

[0032] The security monitoring unit is used to perform real-time data security monitoring on the data request module, the data control module, the data comparison module, the data classification module and the data storage module and generate monitoring results;

[0033] The alarm feedback unit is used to execute a corresponding alarm mechanism according to the monitoring result.

[0034] Optionally, the alarm feedback unit is specifically configured to:

[0035] Parsing the monitoring result to obtain the type of the monitoring result;

[0036] If the type is normal, maintain normal working mode;

[0037] If the type is abnormal, the protection working mode is executed.

[0038] A second aspect of the present invention provides a database management method based on multiple data sources, which is applied to a database management system based on multiple data sources, wherein the database management system includes a data request module, a data control module, a data comparison module, a data classification module, a data storage module, and a security management module that are communicatively connected; the method includes:

[0039] Responding to an input request through the data request module, performing format conversion on the input data carried in the input request and performing a deduplication operation using a preset coincidence database, and constructing a corresponding encrypted file based on the generated deduplication file according to the agreed secret key;

[0040] Decrypting the encrypted file based on the agreed key through the data control module to output the deduplication file, and extracting initial information of the deduplication file;

[0041] The data comparison module uses the written data of the database to screen the initial information based on the coincidence comparison algorithm, generates target information and updates the coincidence database;

[0042] The data classification module uses crawler technology to crawl information keywords of the target information, inputs the information keywords into a preset target classification model, and outputs corresponding classification results;

[0043] Importing the target information into the database according to the classification result and the information keywords through the data storage module;

[0044] The security management module performs security monitoring on the data request module, the data control module, the data comparison module, the data classification module and the data storage module, and executes a corresponding alarm mechanism according to the monitoring results.

[0045] Optionally, the step of responding to an input request through the data request module, performing format conversion on the input data carried in the input request and performing a deduplication operation using a preset coincidence database, and constructing a corresponding encrypted file based on an agreed secret key for the generated deduplication file includes:

[0046] Responding to input requests from any of the multiple data sources through the data request module, parsing the input requests to obtain corresponding input data;

[0047] The data request module converts the input data into an XML file, and crawls the data keywords of the data information of the XML file using crawler technology;

[0048] The data request module uses the data keyword to match the written coincidence keyword in the preset coincidence database to determine the target data keyword and the target coincidence keyword that have been successfully matched;

[0049] Performing an overlap operation on the data information of the target data keyword and the written overlap information of the target overlap keyword by the data request module based on an overlap comparison algorithm, and outputting duplicate information;

[0050] The data request module filters out duplicate information from the data information to generate a deduplication file;

[0051] The deduplication file is encrypted by the data request module using an agreed key to construct an encrypted file.

[0052] The third aspect of the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the database management method based on multiple data sources as described in any one of the second aspects of the present invention.

[0053] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the database management method based on multiple data sources as described in any one of the second aspects of the present invention.

[0054] It can be seen from the above technical solutions that the present invention has the following advantages:

[0055] The present invention provides a database management system based on multiple data sources, including a data request module, a data control module, a data comparison module, a data classification module, a data storage module and a security management module that are communicatively connected. The data request module responds to input requests from any of the multiple data sources, performs format conversion on the received input data and uses the overlapped database for preliminary deduplication to generate a deduplication file, and encrypts and transmits the deduplication file to the data control module according to an agreed secret key, so that the initial information of the deduplication file is extracted after decryption by the data control module. The data comparison module uses the written data of the database to perform deduplication screening on the initial information to generate target information and update the overlapped database. The data classification module uses the information keywords of the target information to determine the classification result based on the target classification model. The target information is imported into the storage location corresponding to the database according to the classification result through the data storage module, and the system is securely managed and controlled by the security management module. Through the database management system based on multiple data sources of the present invention, unified collection of multi-source data can be achieved, and there is no need to convert data before data collection, thereby reducing its complexity in system construction and avoiding the problem of data entry permission security risks to a certain extent; when storing data, the dual deduplication operations of the data request module and the data comparison module make the storage content of the database more streamlined, which also helps to improve the reference efficiency when searching for information. At the same time, the target classification model is used to intelligently classify data keywords into categories, thereby improving the overall management reliability of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1A structural block diagram of a database management system based on multiple data sources provided in the first embodiment of the present invention;

[0058] Figure 2 A flowchart of the steps of a database management method based on multiple data sources provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0059] The embodiments of the present invention provide a database management system, method, device and medium based on multiple data sources, which are used to solve the technical problem of low overall management reliability of network databases in the prior art.

[0060] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0061] See also Figure 1 , Figure 1 This is a structural block diagram of a database management system based on multiple data sources provided in Example 1 of the present invention.

[0062] The present invention provides a database management system based on multiple data sources, comprising a data request module 101, a data control module 102, a data comparison module 103, a data classification module 104, a data storage module 105 and a security management module 106 connected in communication;

[0063] The data request module 101 is used to respond to input requests from any multiple data sources, convert the format of the input data carried in the input request, and use a preset overlap database to perform deduplication operations, and construct a corresponding encrypted file for the generated deduplication file according to the agreed secret key.

[0064] In one example of the present invention, the data request module 101 is specifically configured to:

[0065] Respond to input requests from any of the multiple data sources, parse the input requests and obtain the corresponding input data;

[0066] Convert the input data into an XML file, and use crawler technology to crawl the data keywords of the XML file;

[0067] The data keyword is matched with the written coincidence keyword in the preset coincidence database to determine the target data keyword and the target coincidence keyword that are successfully matched;

[0068] Based on the overlap comparison algorithm, the data information of the target data keyword and the written overlap information of the target overlap keyword are overlapped and the duplicate information is output;

[0069] Filter out duplicate information from data and generate deduplication files;

[0070] The deduplication file is encrypted using the agreed secret key to construct an encrypted file.

[0071] In an embodiment of the present invention, the port of the data request module can support multiple data source entry methods, including but not limited to page requests, Excel imports, other database data imports, external device imports, and imports from different platform languages. The data request module responds to input requests issued by any supported data source terminal, parses the input request to obtain the input data carried by the input request, converts the input data into an XML file according to a set format template, reads and writes the XML file to obtain the corresponding data information, and crawls the data information using crawler technology. The data request module also deploys an overlap database, which contains written overlap information that was previously screened out and not stored in the database, as well as the overlap keywords corresponding to the written overlap information. The data keywords are matched with the overlap keywords to determine the target data keywords and target overlap keywords that have successfully matched. The overlap comparison algorithm is used to perform an overlap operation on the data information associated with the two and the written overlap information, and outputs the data information that is substantially the same as the written overlap information as duplicate information. The duplicate information is screened out from the data information in the XML file to generate a corresponding deduplication file. The deduplication file is encrypted using a pre-determined agreed-upon key to create an encrypted file, which is then transmitted to the data control module. Preliminary overlap comparisons are performed to eliminate redundant data, reducing the amount of data processing required by the data comparison module.

[0072] Optionally, the data request module adopts the DAO mode to perform data reception management, and performs data mapping through JAVAV, and finally performs data logic verification after data mapping through JAVAV BEAN.

[0073] Optionally, the data request module uses an agreed key to construct an encrypted file to achieve data encryption transmission, and can specifically use one or a combination of MD5 encryption algorithm, PGP encryption software, RSA encryption software, DES encryption algorithm or link encryption algorithm.

[0074] The data control module 102 is used to decrypt the encrypted file based on the agreed secret key to output a deduplicated file, and extract the initial information of the deduplicated file.

[0075] In an embodiment of the present invention, the data control module encodes and decrypts the received encrypted file using the agreed key. After the encrypted file becomes plain text, that is, the deduplicated file before encryption is obtained, the data content of the deduplicated file is extracted as initial information, and the initial information is sent to the data comparison module.

[0076] Optionally, when performing initial information extraction of deduplication files in the data control module, one or a combination of MID function, Web data information extraction technology, crawler extraction technology, and Tempo data mining technology can be specifically adopted.

[0077] The data comparison module 103 is used to screen the initial information based on the coincidence comparison algorithm using the written data of the database, generate target information and update the coincidence database.

[0078] In one example of the present invention, the data comparison module 102 is specifically configured to:

[0079] Receive initial information and obtain data written into the database;

[0080] Based on the coincidence comparison algorithm, the initial information and the written data are compared and the coincidence information is output;

[0081] De-duplication is performed on the initial information based on the overlapping information to generate target information;

[0082] Use crawler technology to crawl keywords from overlapping information and construct corresponding overlapping keywords;

[0083] The overlapping keywords and overlapping information are imported into the overlap database as new write-in overlapping keywords and new write-in overlapping information.

[0084] In an embodiment of the present invention, after receiving the initial information of the deduplication file, the data comparison module retrieves the written data that has been written and stored in the database, and performs a large-scale overlap comparison between the initial information and the written data through an overlap comparison algorithm, and determines that there is initial information that is substantially the same as the written data as the overlap information. After filtering out the overlap information from the initial information, the target information to be written into the database is output, and the target information is transmitted to the data classification module. The overlap information that has been filtered out is crawled with keywords through crawler technology, and its corresponding overlap keywords are constructed as new overlap keywords. The new overlap keywords and the associated overlap information are used as new write overlap information and imported into the preset overlap database for updating.

[0085] Optionally, the coincidence comparison algorithm adopted by the data request module and the data comparison module may be specifically one or a combination of TF-IDF algorithm, cosine similarity, vector space model algorithm, SVD algorithm, LSI algorithm or LDA algorithm.

[0086] The data classification module 104 is used to crawl information keywords of the target information using crawler technology, input the information keywords into a preset target classification model, and output corresponding classification results.

[0087] In this embodiment of the present invention, after receiving target information, the data classification module uses crawler technology to extract keywords from the target information and generate corresponding information keywords. The data classification module also deploys a target classification model, which contains a correlation pattern between classification keywords and classification categories. By inputting information keywords into the target classification model, the target classification model outputs the corresponding classification category as the classification result based on the correlation pattern.

[0088] Optionally, the crawler technology used by the data request module, the data comparison module and the data classification module can be specifically one or a combination of python crawler technology, focused crawler technology, general crawler technology, incremental crawler technology or deep web crawler technology.

[0089] In one example of the present invention, the data classification module 104 is further configured to:

[0090] Obtaining category sample data and dividing the category sample data into training data and test data according to a preset ratio; wherein the category sample data includes the main classification categories and subordinate classification categories of the database and the corresponding classification keywords;

[0091] Based on the Monte Carlo algorithm, the initial classification model is constructed using training data;

[0092] Use test data to verify the initial classification model and output the verification results;

[0093] If the verification result is standard, the initial classification model is used as the target classification model;

[0094] If the verification result is non-standard, the training data is used to train and optimize the initial classification model, and the execution jumps to the step of using the test data to verify the initial classification model and output the verification result.

[0095] In an embodiment of the present invention, the data classification module can construct category sample data based on the written data of the database, wherein the category sample data includes the classification categories of the database and the corresponding classification keywords, and the classification categories of the database include main classification categories and subordinate classification categories. The category sample data is divided into training data and test data according to a certain ratio, and the initial classification model is constructed using the learning mechanism of the Monte Carlo algorithm based on the training data, and then the model is verified by inputting the test data into the initial classification model. When the output verification result meets the expectation, that is, it is the standard, the initial classification model is used as the target classification model; when the output verification result is non-standard, the training data is continued to be used to train and optimize the initial classification model based on the Monte Carlo algorithm and perform model verification until the output verification result is standard to obtain the target classification model.

[0096] The data storage module 105 is used to import the target information into the database according to the classification results.

[0097] In an embodiment of the present invention, the classification categories and classification keywords of the target classification model correspond one-to-one with the classification categories and classification keywords set in the database. Therefore, in the database, multiple primary folders and multiple secondary folders are created based on the classification categories, including primary classification categories and secondary classification categories. The primary folders and secondary folders can be sorted according to the first letter of the folder name. At the same time, the classification results output by the target classification model include the primary classification category results and the secondary classification category results to which the information keyword belongs. Based on the classification results, the data storage module can import the target information into the corresponding primary folders and secondary folders in the database step by step.

[0098] The security management module 106 is used to perform security monitoring on the data request module, the data control module, the data comparison module, the data classification module and the data storage module, and to execute corresponding alarm mechanisms according to the monitoring results.

[0099] In one example of the present invention, the security management module 106 includes a security monitoring unit and an alarm feedback unit;

[0100] A security monitoring unit is used to perform real-time data security monitoring on the data request module, data control module, data comparison module, data classification module, and data storage module and generate monitoring results;

[0101] The alarm feedback unit is used to execute the corresponding alarm mechanism according to the monitoring results.

[0102] The alarm feedback unit is specifically used to:

[0103] Parse the monitoring results and obtain the type of monitoring results;

[0104] If the type is normal, maintain normal working mode;

[0105] If the type is abnormal, the protection working mode is executed.

[0106] In an embodiment of the present invention, the security management module includes a security monitoring unit and an alarm feedback unit. During the process of data processing by the data request module, the data control module, the data comparison module, the data classification module and the data storage module, the security monitoring unit performs real-time security detection of the data to generate corresponding monitoring results, such as detecting whether there is potential virus link data. The alarm feedback unit determines the type of the monitoring result by analyzing the monitoring result. If the type is normal, it means that the data is safe, and the current working mode is maintained. If the type is abnormal, it means that there is security abnormal data, such as potential virus link data, and the security abnormal data is located, eliminated and protected from attack on the system.

[0107] Optionally, the security monitoring unit may specifically adopt a data security protection detection technology that integrates hacking technology, firewall technology, intrusion detection technology, virus protection technology, and information hiding technology.

[0108] In an embodiment of the present invention, a database management system based on multiple data sources is provided, including a data request module, a data control module, a data comparison module, a data classification module, a data storage module and a security management module that are communicatively connected. The data request module responds to input requests from any of the multiple data sources, performs format conversion on the received input data and uses the overlapped database for preliminary deduplication to generate a deduplication file, and encrypts the deduplication file according to the agreed secret key and transmits it to the data control module, so that the initial information of the deduplication file is extracted after decryption by the data control module. The data comparison module uses the written data of the database to perform deduplication screening on the initial information to generate target information and update the overlapped database. The data classification module uses the information keywords of the target information to determine the classification result based on the target classification model, imports the target information into the corresponding storage location of the database according to the classification result through the data storage module, and performs security management and control on the system through the security management module. Through the database management system based on multiple data sources of the present invention, unified collection of multi-source data can be achieved, and there is no need to convert data before data collection, thereby reducing its complexity in system construction and avoiding the problem of data entry permission security risks to a certain extent; when storing data, the dual deduplication operations of the data request module and the data comparison module make the storage content of the database more streamlined, which also helps to improve the reference efficiency when searching for information. At the same time, the target classification model is used to intelligently classify data keywords into categories, thereby improving the overall management reliability of the database.

[0109] See also Figure 2 , Figure 2 A flowchart of the steps of a database management method based on multiple data sources provided in the second embodiment of the present invention.

[0110] An embodiment of the present invention provides a database management method based on multiple data sources, which is applied to a database management system based on multiple data sources. The database management system includes a data request module, a data control module, a data comparison module, a data classification module, a data storage module, and a security management module that are communicatively connected. The method includes:

[0111] Step 201: Respond to an input request through a data request module, convert the format of the input data carried in the input request, perform deduplication operations using a preset coincidence database, and construct a corresponding encrypted file based on the generated deduplication file according to the agreed secret key;

[0112] Step 202: The data control module decrypts the encrypted file based on the agreed key to output a deduplicated file, and extracts initial information of the deduplicated file;

[0113] Step 203: Using the data comparison module based on the coincidence comparison algorithm, the initial information is screened using the written data in the database to generate target information and update the coincidence database;

[0114] Step 204: crawl the target information using crawler technology through the data classification module, input the information keywords into the preset target classification model, and output the corresponding classification results;

[0115] Step 205: Import the target information into the database according to the classification results and information keywords through the data storage module;

[0116] Step 206: The security management module performs security monitoring on the data request module, the data control module, the data comparison module, the data classification module and the data storage module, and executes a corresponding alarm mechanism according to the monitoring results.

[0117] Optionally, step 201 includes the following sub-steps:

[0118] Responding to input requests from any of the multiple data sources, parsing the input requests to obtain corresponding input data;

[0119] Convert the input data into an XML file, and use crawler technology to crawl data keywords of the data information in the XML file;

[0120] The data keyword is matched with the written coincidence keyword in the preset coincidence database to determine the target data keyword and the target coincidence keyword that are successfully matched;

[0121] Performing an overlap operation on the data information of the target data keyword and the written overlap information of the target overlap keyword based on an overlap comparison algorithm, and outputting duplicate information;

[0122] Screening out duplicate information from the data information to generate a deduplication file;

[0123] The deduplication file is encrypted using an agreed key to construct an encrypted file.

[0124] Optionally, step 203 includes the following sub-steps:

[0125] Receiving the initial information through a data comparison module and obtaining the written data of the database;

[0126] Based on the coincidence comparison algorithm, the data comparison module performs a coincidence comparison on the initial information and the written data and outputs coincidence information;

[0127] Performing a deduplication operation on the initial information according to the overlap information by a data comparison module to generate target information;

[0128] Using crawler technology to crawl the overlapping information through the data comparison module to construct corresponding overlapping keywords;

[0129] The overlapping keywords and the overlapping information are used as new written overlapping keywords and new written overlapping information through a data comparison module and imported into the overlap database.

[0130] Optionally, it also includes:

[0131] Obtaining category sample data through a data classification module, and dividing the category sample data into training data and test data according to a preset ratio; wherein the category sample data includes the main classification categories and subordinate classification categories of the database and corresponding classification keywords;

[0132] Based on the Monte Carlo algorithm, an initial classification model is constructed using the training data through a data classification module;

[0133] Verifying the initial classification model using the test data through a data classification module and outputting a verification result;

[0134] If the verification result is standard, the initial classification model is used as the target classification model through the data classification module.

[0135] If the verification result is non-standard, the data classification module uses the training data to train and optimize the initial classification model, and jumps to the step of using the test data to verify the initial classification model and outputting the verification result.

[0136] Optionally, step 206 includes the following sub-steps:

[0137] Performing real-time data security monitoring on the data request module, the data control module, the data comparison module, the data classification module, and the data storage module through a security monitoring unit and generating monitoring results;

[0138] The corresponding alarm mechanism is executed according to the monitoring results through the alarm feedback unit.

[0139] Optionally, step 206 specifically includes:

[0140] parsing the monitoring result by an alarm feedback unit to obtain a type of the monitoring result;

[0141] If the type is normal, maintaining the normal working mode through the alarm feedback unit;

[0142] If the type is abnormal, the protection working mode is executed through the alarm feedback unit.

[0143] An embodiment of the present invention also provides an electronic device, characterized in that it includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the database management method based on multiple data sources as in the method embodiment of the present invention.

[0144] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed, the database management method based on multiple data sources according to the method embodiment of the present invention is implemented.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the method described above can refer to the corresponding process in the aforementioned system embodiment and will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0148] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0150] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A database management system based on multiple data sources, characterized in that: It includes a data request module, a data control module, a data comparison module, a data classification module, a data storage module and a security management module connected to the communication; The data request module is used to respond to input requests from any of the multiple data sources, perform format conversion on the input data carried in the input request, perform deduplication operations using a preset coincidence database, and construct a corresponding encrypted file from the generated deduplication file according to the agreed secret key; The data control module is configured to decrypt the encrypted file based on the agreed key to output the deduplication file, and extract initial information of the deduplication file; The data comparison module is used to screen the initial information using the written data of the database based on the coincidence comparison algorithm, generate target information and update the coincidence database; The data classification module is used to crawl information keywords of the target information using crawler technology, input the information keywords into a preset target classification model and output corresponding classification results; The data storage module is configured to import the target information into the database according to the classification result; The security management module is used to perform security monitoring on the data request module, the data control module, the data comparison module, the data classification module and the data storage module, and execute a corresponding alarm mechanism according to the monitoring results; The data comparison module is specifically used to: Receive the initial information and obtain the write data of the database; Based on the coincidence comparison algorithm, the initial information and the written data are compared for coincidence and the coincidence information is output; performing a deduplication operation on the initial information according to the overlap information to generate target information; Using crawler technology to crawl the overlapping information for keywords and construct corresponding overlapping keywords; The overlap keyword and the overlap information are imported into the overlap database as new write-in overlap keyword and new write-in overlap information.

2. The database management system based on multiple data sources according to claim 1, characterized in that: The data request module is specifically used to: Responding to input requests from any of the multiple data sources, parsing the input requests to obtain corresponding input data; Convert the input data into an XML file, and use crawler technology to crawl data keywords of the data information in the XML file; The data keyword is matched with the written coincidence keyword in the preset coincidence database to determine the target data keyword and the target coincidence keyword that are successfully matched; Performing an overlap operation on the data information of the target data keyword and the written overlap information of the target overlap keyword based on an overlap comparison algorithm, and outputting duplicate information; Screening out duplicate information from the data information to generate a deduplication file; The deduplication file is encrypted using an agreed key to construct an encrypted file.

3. The database management system based on multiple data sources according to claim 1, characterized in that: The data classification module is further used to: Obtaining category sample data, and dividing the category sample data into training data and test data according to a preset ratio; wherein the category sample data includes the main classification categories and subordinate classification categories of the database and corresponding classification keywords; Based on the Monte Carlo algorithm, an initial classification model is constructed using the training data; Using the test data to verify the initial classification model, and outputting a verification result; If the verification result is standard, the initial classification model is used as the target classification model; If the verification result is non-standard, the initial classification model is trained and optimized using the training data, and the process jumps to the step of verifying the initial classification model using the test data and outputting the verification result.

4. The database management system based on multiple data sources according to claim 1, characterized in that: The security management module includes a security monitoring unit and an alarm feedback unit; The security monitoring unit is used to perform real-time data security monitoring on the data request module, the data control module, the data comparison module, the data classification module and the data storage module and generate monitoring results; The alarm feedback unit is used to execute a corresponding alarm mechanism according to the monitoring result.

5. The database management system based on multiple data sources according to claim 4 is characterized in that: The alarm feedback unit is specifically used to: Parsing the monitoring result to obtain the type of the monitoring result; If the type is normal, maintain normal working mode; If the type is abnormal, the protection working mode is executed.

6. A database management method based on multiple data sources, characterized in that: The method is applied to a database management system based on multiple data sources, the database management system including a data request module, a data control module, a data comparison module, a data classification module, a data storage module and a security management module connected in communication; the method includes: Responding to an input request through the data request module, performing format conversion on the input data carried in the input request and performing a deduplication operation using a preset coincidence database, and constructing a corresponding encrypted file based on the generated deduplication file according to the agreed secret key; Decrypting the encrypted file based on the agreed key through the data control module to output the deduplication file, and extracting initial information of the deduplication file; The data comparison module uses the written data of the database to screen the initial information based on the coincidence comparison algorithm, generates target information and updates the coincidence database; The data classification module uses crawler technology to crawl information keywords of the target information, inputs the information keywords into a preset target classification model, and outputs corresponding classification results; Importing the target information into the database according to the classification result and the information keywords through the data storage module; Performing security monitoring on the data request module, the data control module, the data comparison module, the data classification module and the data storage module through the security management module, and executing a corresponding alarm mechanism according to the monitoring results; The data comparison module uses the written data of the database to screen the initial information based on the coincidence comparison algorithm, generates target information and updates the coincidence database, including: Receiving the initial information through a data comparison module and obtaining the written data of the database; Based on the coincidence comparison algorithm, the data comparison module performs a coincidence comparison on the initial information and the written data and outputs coincidence information; Performing a deduplication operation on the initial information according to the overlap information by a data comparison module to generate target information; Using crawler technology to crawl the overlapping information through the data comparison module to construct corresponding overlapping keywords; The overlapping keywords and the overlapping information are used as new written overlapping keywords and new written overlapping information through a data comparison module and imported into the overlap database.

7. The database management method based on multiple data sources according to claim 6, characterized in that: The step of responding to an input request through the data request module, performing format conversion on the input data carried by the input request and performing a deduplication operation using a preset coincidence database, and constructing a corresponding encrypted file based on the generated deduplication file according to the agreed secret key includes: Responding to input requests from any of the multiple data sources through the data request module, parsing the input requests to obtain corresponding input data; The data request module converts the input data into an XML file, and crawls the data keywords of the data information of the XML file using crawler technology; The data request module uses the data keyword to match the written coincidence keyword in the preset coincidence database to determine the target data keyword and the target coincidence keyword that have been successfully matched; Performing an overlap operation on the data information of the target data keyword and the written overlap information of the target overlap keyword by the data request module based on an overlap comparison algorithm, and outputting duplicate information; The data request module filters out duplicate information from the data information to generate a deduplication file; The deduplication file is encrypted by the data request module using an agreed key to construct an encrypted file.

8. An electronic device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the database management method based on multiple data sources as described in any one of claims 6 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the method for managing a database based on multiple data sources as described in any one of claims 6 to 7 is implemented.

Images