Intelligent risk-aware data security storage method and system

Abstract

The invention relates to the technical field of data security storage, in particular to an intelligent risk-aware data security storage method and system, which realizes data monitoring in the whole data storage process through original data acquisition, data information extraction, data verification, data deduplication, data storage, data backup and data access. Various data transmission, storage and access errors including data content sending errors and the like can be found, the data source data sending error problem is difficult to find through a traditional method, meanwhile, for different modal data such as pictures, audios and texts, fuzzy search can be conducted according to the data content, and the fuzzy search efficiency is improved. And the validity of data access is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data security storage, and in particular to a method and system for intelligent risk-aware data security storage. Background Art

[0002] Data security verification is often involved in data storage. Its main function is to ensure the accuracy and reliability of data. Through data verification, the data security storage system can prevent illegal or unsafe data from entering the system, improve system security and stability. At the same time, by reducing the entry of invalid data into the business processing link, it can reduce the system burden, improve response speed and optimize system performance.

[0003] In practical applications, traditional data security verification mostly only considers the risks in the data transmission process and ignores the errors at the source of the data. At the same time, traditional data transmission usually has a single data type, so the data protocol is fixed. On the one hand, when the data is illegally intercepted, the difficulty coefficient of data cracking is low. On the other hand, when the transmission data type is diverse and the transmission timing is uncertain, traditional data security storage systems find it difficult to parse the data and store it securely. Summary of the invention

[0004] In order to solve the problems existing in the background technology, the present invention provides a data security storage method with intelligent risk perception, including: S101. Original data acquisition: obtaining source data from each data generating unit. The source data includes a data header and data content. The data header records brief information about the data. The brief information includes at least: data protocol code, data size, data checksum, data subject, and the data content format includes at least: numbers, text, images, and audio. S102. Data information extraction, extracting the source data header according to the universal data protocol and obtaining the local data protocol corresponding to the data content according to the data protocol code, parsing the data content, and issuing a data content error warning when the data content cannot be parsed according to the data protocol specified in the data header; S103. Data verification: Calculate the amount of data in the data content and compare it with the amount of data in the information header. The absolute value of the difference is recorded as Δd1. Use the data verification method of the protocol to calculate the data content check code and compare it with the check code in the information header. The absolute value of the difference is recorded as Δd2. When Δd1+Δd2>0, issue a data transmission error warning; S104. Data deduplication: the verified data is compared with the data with the same subject stored in the database for content similarity. When the content similarity is greater than the threshold, a data duplication warning is issued. After the data is manually confirmed to be duplicate data, the data is deleted; S105. Data storage: record the data header after verification and deduplication as K, record the data content as D, save the data content D to the security database, generate a unique address value V1 for data access, and save the data header K and the data content access address V1 in the form of a key-value pair to the search database; S106. Data backup: when there are fewer computing tasks, the data is automatically backed up and the backed-up data access address value V2 is mapped to V1, and the mapping relationship table is saved in the search database.

[0005] Specifically, in S102 data information extraction, the image data uses a deep learning neural network to obtain the summary information of the image and converts it into text, and the audio data uses a natural language processing model to obtain the main information of the audio and convert it into text. When proceeding to step S103, the data verification also calculates the similarity value Tt between the extracted image summary text, audio content text or original data text and the data subject text in the data header. When Tt is less than the set threshold, a data content error warning is issued; when the original data content is a number, no similarity calculation is performed.

[0006] Furthermore, the data security storage method also includes iterative optimization and updating of the deep learning neural network model for image summary generation and the natural language processing model for audio content recognition, wherein the data source for iterative optimization is the data content stored in S105, the data is obtained by addressing address V1, and the corresponding data label is the data body text in the data header information K. When a set amount of data is added to the database, the data security storage method automatically performs fine-tuning training on the basis of the original model and updates the model parameters.

[0007] Furthermore, the data security storage method also includes data security search. When a user needs to access data, the data security storage method verifies the identity of the user. When the identity verification fails, the data is not accessed and an illegal access warning is issued. When the identity verification passes, the data access task can be executed.

[0008] In particular, users who have passed the verification can perform fuzzy searches through the subject information in the data information header K, thereby achieving extensive searches of different types of data.

[0009] In particular, when the accessed data cannot be correctly parsed according to the protocol, a data corruption warning is issued. At this time, the data security storage method automatically restores the data according to the mapping between V1 and V2.

[0010] In particular, the present invention also provides an intelligent risk-aware data security storage system, including: a data acquisition module, a data risk perception module, a data storage module, and a data access module. The data acquisition module implements S101 original data acquisition; the data risk perception module implements S102-104 data information extraction, data verification and data deduplication. When corresponding problems in S102-104 occur, the system issues corresponding warnings; the data storage module saves and backs up data; the data access module implements user verification, and when the user verification passes, the matching of user access data and the addressing of data are realized.

[0011] Specifically, the data storage module includes a security database sub-library, a search database sub-library, a model database sub-library and a backup database sub-library, wherein the security database sub-library stores the original data content D after verification and deduplication, the search database sub-library stores K, V1 key-value pair data, V1 / V2 mapping table data and different data protocols, the model database sub-library stores the deep learning neural network model for image summary generation and the natural language processing model for audio content recognition, and the backup database sub-library stores the backup data.

[0012] Beneficial effects of the present invention:

[0013] 1. The present invention focuses on the risk issues in the entire process of data generation, transmission, storage and access, including: the risk of errors in the data content itself, the risk of data transmission errors, data duplication, illegal access risks, and data corruption risks. Among them, the risk of errors in the data content itself comes from errors when the data is generated, and such errors are generally difficult to detect.

[0014] 2. The present invention can identify the inconsistency between the transmitted data and the subject determined during transmission through local data protocol analysis, image summary generation, and audio content recognition. At the same time, since the local protocol code is transmitted simultaneously during the data transmission process, the present invention can achieve unified transmission of data of different modalities and ensure the security of data storage.

[0015] 3. The deep learning neural network for generating image summary information and the natural language processing large model for audio recognition of the present invention have the ability of automatic iterative training, which can realize continuous learning of the model and enhance the ability to perceive data risks.

[0016] 4. In addition to user identity verification, the data access of the present invention also provides fuzzy search for data access, which ensures the security of data storage and can provide effective search for data of different modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the logical structure of the data security storage method for intelligent risk perception of the present invention; Figure 2A schematic diagram of database classification of the intelligent risk-aware data security storage system of the present invention. DETAILED DESCRIPTION

[0018] In order to make the technical methods, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further described below in conjunction with specific implementation methods. Embodiment 1

[0019] like Figure 1 As shown, the basic steps of the intelligent risk-aware data security storage method of the present invention are: S101 to obtain data; S102 to extract data; S103 to verify data; S104 to remove data duplication; S105 to store data; S106 to back up data. In particular, since the present invention adds an image summary generation network and a speech recognition network, the present invention can receive not only common digital and text data, but also image and audio data.

[0020] When applied, the data sent by the data generating unit is packaged into two parts: the data header and the data content. The data header records the basic information of the data content for verification, including but not limited to the protocol code for data content decoding, the data volume, the data check code, and the data subject. The data protocol is selected according to the type of data (such as numbers, audio, pictures, etc.) that the sender wants to send, and the data subject is marked according to the data content that the sender wants to send. After receiving the data, the data receiver first decodes the data header according to the general protocol to obtain the main information of the data, and then matches the correct decoder in the receiving end server according to the data protocol code to decode the data content. When the corresponding decoder cannot correctly decode the data, it means that the data that the data generating unit really wants to send does not match the data actually sent. For example, the original intention is to send a picture for storage, but in fact an audio file is sent due to an erroneous operation. At this time, the data security storage method of the present invention can capture this type of error and issue a data content error warning. On the other hand, the method of the present invention further compares the content of the sent data. The comparison first determines the data content type according to the protocol. When the data is digital data, the semantic information of the digital data is difficult to obtain at this time, so the data content is not checked. When the data is text, image, audio, the present invention first obtains the data subject text information in the data header, and then generates a summary of the image data through the image summary generation neural network model, recognizes the audio data through the speech recognition network and outputs the speech text, and finally compares the similarity of the data content text with the subject text in the data header through NLP natural language processing. One embodiment of NLP processing is to first segment the words, then remove the stop words, Word2Vec word vectorization and finally use cosine similarity to calculate the similarity between the texts. The calculated text similarity is recorded as Tt. When Tt is less than the set threshold, it is considered that the data sent to the storage end is inconsistent with the actual data to be saved. For example, the original intention is to send a picture of a person, but a beautiful landscape is sent, and the original intention is to send an experimental report, but a paper is sent. At this time, the data security storage method of the present invention will capture this type of error and issue a data content error warning.

[0021] Of course, whether it is a data type mismatch error or a data content mismatch, the source of the error may also be due to incorrect data type selection or data subject description when the data is sent. Therefore, when a data content error warning is received, the operator needs to further confirm to clarify whether the current data should modify the data header information to continue saving or delete the data.

[0022] It should be noted that the reason why the data type mismatch is not automatically matched according to the file suffix at the data sending port or the data type verification is performed at the sending end is that some data will not retain the suffix due to confidentiality requirements, and the data itself is in encryption mode, so the sending end cannot perform the actual data type verification locally.

[0023] It should be noted that if video data is received, the video data can be framed at intervals, and image summaries can be generated for the acquired frame data. Finally, the basic NLP natural language processing method can be used to summarize the content of all the summaries obtained and generate the theme content of the video to obtain the video theme text.

[0024] It should be noted that as the amount of stored data increases, models such as the image summary generation neural network and speech recognition neural network will automatically perform iterative learning (fine-tuning based on the original model), where the data source is the data content stored in the database, and the image summary and speech corresponding text labels are the data subject text in the data header stored in the database. Embodiment 2

[0025] like Figure 1 As shown, the data after extracting the information needs to be verified by S103 and deduplicated by S104, wherein S103 data verification calculates the data volume in the data content and compares it with the data volume in the information header, and the absolute value of the comparison difference is recorded as Δd1. The data verification method of the protocol is used to calculate the data content check code and compare it with the check code in the information header, and the absolute value of the comparison difference is recorded as Δd2. When Δd1+Δd2>0, that is, when any one of the data volume and data verification cannot pass the verification, it is considered that data sticking, packet loss, data error and other problems may have occurred during the data transmission process, which will cause the data volume at the receiving end to be inconsistent with the sending end or the data check code to be inconsistent. The data security storage method of the present invention will issue a data transmission error warning. The data verification method can be a parity check, a longitudinal redundancy check (LRC), a checksum check, a cyclic redundancy check (CRC), etc.

[0026] The verified data needs to be deduplicated. The method of the present invention obtains the data subject content in the data header. When the data subject of the currently stored data is the same as the data subject already stored in the database, the current data is compared with the data stored in the database for data content similarity. When the similarity of the two data is greater than the preset threshold, it is considered that there may be duplicate data transmission, and a data duplication warning is issued. The operator confirms the data according to the prompt. When it is confirmed that the data is indeed sent repeatedly, the data is deleted. Otherwise, the data is processed according to the set plan (such as marking the data and saving it). Compared with traditional data comparison, the method of the present invention does not need to compare all data, which reduces the amount of computing tasks. At the same time, since the data of the present invention contains images, audio and other data, all data are compared, and the data comparison calculation efficiency is low.

[0027] The final data undergoes S105 data storage and S106 data backup. The data storage is divided into two parts, one of which stores the data content D in the security database, and the other stores the data header information K corresponding to the data content D and the database address V1 where D is located in the search database in the form of a K:V1 key-value pair; at night or when there are fewer computing tasks, the method of the present invention automatically backs up the data and maps the backed-up data access address value V2 with V1, and the mapping relationship table is saved in the search database. When a user needs to access data, the method of the present invention first verifies the user's identity. If the verification fails, the method of the present invention issues an illegal access warning. When the verification passes, the user can search for data by searching the information in the data header K in the database, thereby enabling the search of unstructured data such as pictures and audio. Among them, it is important to explain that, since K contains voice information with rich data content, the data security search of the present invention can realize fuzzy search of data. For example, if you need to search for pictures containing people, the method of the present invention can use NLP method to search for data with people in the subject information of the data header K and the data type is picture. For example, when searching for audio of interviews with teenagers larger than 10M, the corresponding data can be found according to the data volume and data subject, and the matched data can be obtained by addressing V1 corresponding to K in the security database.

[0028] S106 data backup is automatically performed during a time period when computer computing tasks are not intensive, generally in the evening after the operator gets off work. The backed up data will generate a data storage address, recorded as V2, and V2 will be mapped with V1. The mapping relationship table is saved in the search database. When normal access data cannot be correctly parsed according to the protocol, the present invention issues a data corruption warning. At this time, according to the V1 and V2 mapping tables, the corresponding backup data address V2 can be quickly found through V1, and the data can be restored. Embodiment 3

[0029] The present invention also provides an intelligent risk-aware data security storage system, which applies the above-mentioned data security storage method for data storage and data access, including: a data acquisition module, a data risk perception module, a data storage module, and a data access module, wherein the data acquisition module implements S101 original data acquisition; the data risk perception module implements S102-104 data information extraction, data verification and data deduplication, and when corresponding problems in S102-104 occur, the system issues corresponding warnings; the data storage module saves and backs up the data; the data access module implements user verification, and when the user verification passes, the matching of user access data and the addressing of data are implemented. At the same time, in order to improve data storage security and take into account data retrieval efficiency, the data security storage system divides the data storage module into several database sub-databases, such as Figure 2 As shown, the database sub-library includes: a security database sub-library, a search database sub-library, a model database sub-library and a backup database sub-library, wherein the security database sub-library has a higher security level and is used to store the original data content D after verification and deduplication to ensure data security; the search database sub-library has a higher data access speed and is used to store K, V1 key-value pair data, V1 / V2 mapping table data and different data protocols to provide fast retrieval of data access; the model database sub-library stores the deep learning neural network model for image summary generation and the natural language processing model for audio content recognition. Since the model and the corresponding parameters will be directly introduced into the cache when the system is running, the storage access speed requirement can be reduced; the backup database sub-library stores the backup data, and its security level is also high.

[0030] In summary, the system and method of the present invention can identify risks in the entire data storage process, including: risk of errors in the data content itself, risk of data transmission errors, data duplication, risk of illegal access, and risk of data corruption. Among them, the risk of errors in the data content itself comes from errors when the data is generated, and this type of error is generally difficult to detect.

[0031] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection claimed by the present invention.

Claims

1. An intelligent risk-aware data security storage method, characterized by: The data security storage method comprises: S101. Original data acquisition: obtaining source data from each data generating unit. The source data includes a data header and data content. The data header records brief information about the data. The brief information includes at least: data protocol code, data size, data checksum, data subject, and the data content format includes at least: numbers, text, images, and audio. S102. Data information extraction, extracting the source data header according to the universal data protocol and obtaining the local data protocol corresponding to the data content according to the data protocol code, parsing the data content, and issuing a data content error warning when the data content cannot be parsed according to the data protocol specified in the data header; S103. Data verification: Calculate the amount of data in the data content and compare it with the amount of data in the information header. The absolute value of the difference is recorded as Δd1. Use the data verification method of the protocol to calculate the data content check code and compare it with the check code in the information header. The absolute value of the difference is recorded as Δd2. When Δd1+Δd2>0, issue a data transmission error warning; S104. Data deduplication, verified data Compare the content similarity with the data with the same subject stored in the database. When the content similarity is greater than the threshold, a data duplication warning is issued. After the data is manually confirmed to be duplicate data, the data is deleted; S105. Data storage: record the data header after verification and deduplication as K, record the data content as D, save the data content D to the security database, generate a unique address value V1 for data access, and save the data header K and the data content access address V1 in the form of a key-value pair to the search database; S106. Data backup: when there are fewer computing tasks, the data is automatically backed up and the backed-up data access address value V2 is mapped to V1, and the mapping relationship table is saved in the search database.

2. The method for secure data storage based on intelligent risk perception according to claim 1 is characterized by: In the data information extraction of S102, the image data uses a deep learning neural network to obtain the summary information of the image and convert it into text. The audio data uses a natural language processing model to obtain the main information of the audio and convert it into text. When proceeding to step S103, the data verification also calculates the similarity value Tt between the image summary text, audio content text or original data text extracted from the information and the data subject text in the data header. When Tt is less than the set threshold, a data content error warning is issued; when the original data content is a number, no similarity calculation is performed.

3. The method for secure data storage based on intelligent risk perception according to claim 2 is characterized by: The data security storage method also includes iterative optimization and updating of the deep learning neural network model for image summary generation and the natural language processing model for audio content recognition, wherein the data source for iterative optimization is the data content stored in S105, the data is obtained by addressing address V1, and the corresponding data label is the data body text in the data header information K. When a set amount of data is added to the database, the data security storage method automatically performs fine-tuning training on the basis of the original model and updates the model parameters.

4. The method for secure data storage based on intelligent risk perception according to claim 3 is characterized by: The data security storage method also includes data security search. When a user needs to access data, the data security storage method verifies the identity of the user. When the identity verification fails, the data is not accessed and an illegal access warning is issued. When the identity verification passes, the data access task can be executed.

5. The method for secure data storage based on intelligent risk perception according to claim 4 is characterized by: Users who have passed identity verification can perform fuzzy searches based on the subject information in the data information header K, thereby achieving extensive searches of different types of data.

6. The method for secure data storage based on intelligent risk perception according to claim 5 is characterized by: When the accessed data cannot be correctly parsed according to the protocol, a data corruption warning is issued. At this time, the data security storage method automatically restores the data according to the mapping between V1 and V2.

7. An intelligent risk-aware data security storage system, wherein the data security storage system adopts the intelligent risk-aware data security storage method according to any one of claims 1 to 6 for data storage, comprising: The data acquisition module, data risk perception module, data storage module and data access module are characterized in that: the data acquisition module realizes the acquisition of original data of S101; the data risk perception module realizes the data information extraction, data verification and data deduplication of S102-104, and when the corresponding problems in S102-104 occur, the system issues corresponding warnings; the data storage module saves and backs up the data; the data access module realizes user verification, and when the user verification is passed, the matching of user access data and the addressing of data are realized.

8. The intelligent risk-aware data security storage system according to claim 7 is characterized by: The data storage module includes a security database sub-library, a search database sub-library, a model database sub-library and a backup database sub-library. The security database sub-library stores the original data content D after verification and deduplication, the search database sub-library stores K, V1 key-value pair data, V1 / V2 mapping table data and different data protocols, the model database sub-library stores the deep learning neural network model for image summary generation and the natural language processing model for audio content recognition, and the backup database sub-library stores the backup data.

Images