Sensitive data intelligent identification system and method based on multi-modal feature fusion

Abstract

The invention discloses a sensitive data intelligent identification system and method based on multi-modal feature fusion, and relates to the technical field of data security. According to the sensitive data intelligent identification system and method based on multi-modal feature fusion, visual features, semantic features and time sequence features are extracted from multi-modal data through a convolutional neural network, and the visual features, the semantic features and the time sequence features are fused into a multi-modal feature data set. Through abuse potential feature analysis, identifiability feature analysis and time dependence feature analysis methods, a data sensitivity index is calculated, and the sensitivity degree of a data set is quantified. And according to the data sensitivity index, identifying and screening out high-sensitivity data to form a sensitive data set. And secondary identification is performed on the sensitive data set through the convolutional neural network, and the sensitive data is marked, so that the security and privacy protection of the data are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data security, and specifically to an intelligent sensitive data recognition system and method based on multi-modal feature fusion. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, a large amount of multi-modal data has been generated in various industries, including images, texts, audios, videos, and time series, etc. These data are widely used in fields such as healthcare, finance, and security, especially in fields involving personal privacy and sensitive information. However, with the continuous improvement of the degree of informatization, the problems of leakage and abuse of sensitive data have become increasingly severe. How to intelligently identify and protect these sensitive data has become a technical problem to be solved urgently. The security management of sensitive data is not only related to the protection of personal privacy but also involves the security and trust of enterprises and society.

[0003] For example, the invention patent with the publication number CN110909224B discloses an automatic classification and recognition method and system for sensitive data based on artificial intelligence, which relates to the technical field of data security. An automatic classification and recognition method for sensitive data based on artificial intelligence includes the following steps: S1: obtaining a data training set; S2: classifying and establishing a sensitive data set; S3: identifying specific sensitive data; S4: obtaining test data and inputting it into the automatic classification model for sensitive data to classify the sensitive data, and then inputting it into the sensitive data recognition model to generate a feature recognition result. The automatic classification and recognition method and system for sensitive data based on artificial intelligence of the present invention apply artificial intelligence technology to the stage of sensitive data and associated relationship recognition, effectively solve the pain point that the performance and accuracy of the traditional regular method cannot be both obtained, and can also eliminate the maintenance of regular recognition rules by professionals, truly achieving the realization of no configuration and automatic functions, and bringing value improvement to users.

[0004] Existing methods cannot make full use of the correlation between various types of data when processing different modal data such as images, texts, and time series, resulting in poor recognition effects. Especially when dealing with data changes and complex scenarios, there are limitations in the recognition accuracy of existing methods and the scalability of the system. Therefore, how to improve the recognition accuracy of sensitive data and the robustness of the system through more advanced technologies, especially intelligent algorithms based on multi-modal feature fusion, has become a technical problem to be solved urgently. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention provides an intelligent sensitive data recognition system and method based on multi-modal feature fusion, which solves the problems in the above background art.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions: An intelligent recognition system for sensitive data based on multi-modal feature fusion, comprising the following steps: a feature fusion module, a feature analysis module, a data analysis module, and a data recognition module; the feature fusion module is used to extract visual features, semantic features, and time series features from multi-modal data through a convolutional neural network, and perform fusion through weighted summation to form a multi-modal feature data set; the feature analysis module is used to perform abuse potential feature analysis, identifiability feature analysis, and time dependence feature analysis on the multi-modal feature data set to obtain a data sensitivity index; the data analysis module is used to identify the multi-modal feature data set according to the data sensitivity index, screen out high-sensitivity data in the multi-modal data, and obtain a sensitive data set; the data recognition module is used to perform secondary recognition on the sensitive data set through a convolutional neural network and mark the sensitive data in the sensitive data set.

[0007] Further, the specific process of extracting visual features, semantic features, and time series features from multi-modal data through a convolutional neural network is as follows: The multi-modal data includes image data, text data, and time series data; preprocess the image data, perform convolution operations on the image through the convolutional layer of the convolutional neural network to extract the visual features of the image data, and perform downsampling on the convolutional output through the pooling layer; preprocess the text data, including word segmentation and stop word removal, convert each word in the text into an embedding vector through word embedding, perform convolution operations on the word embedding vectors through the convolutional layer to extract the semantic features of the text data, and perform pooling on the convolution result through the pooling layer to obtain the overall semantic representation of the text; standardize the time series data, perform convolution operations on the time series data through the temporal convolutional layer in the convolutional neural network to extract short-term and long-term dependencies, and perform downsampling on the convolution result through the pooling layer to obtain the time series data trend and change feature vector.

[0008] Further, the specific process of performing fusion through weighted summation to form a multi-modal feature data set is as follows: Map the visual features and semantic features extracted from the convolutional neural network into a common feature space, and perform weighted merging and integration on the visual features and semantic features through weighted summation; perform standardization processing on the fused visual and semantic feature vectors, and perform weighted summation on the time series features and the fused visual and semantic features; perform time window partitioning on the time series features to form a multi-modal feature data set.

[0009] Furthermore, the specific processes of performing abuse potential feature analysis, identifiability feature analysis, and time-dependence feature analysis on the multi-modal feature dataset are as follows: Conduct a risk assessment on each of the visual features, semantic features, and time series features, identify data abuse features according to the abuse potential assessment model, determine the sensitivity level of each feature, and generate an abuse potential index; Conduct an identifiability analysis on the visual features, semantic features, and time series features, evaluate the contribution of each feature to the data identification result through chi-square tests, and calculate the identifiability feature index based on the identification difficulty of the feature, the feasibility of the attack, and the exposure level of the target to quantify the likelihood of the features in the dataset being identified and abused; Conduct a time-dependence analysis on the time series features, identify the temporal patterns of the features and the sensitivity changes existing in the time window, capture the sensitivity patterns at different time points and time periods through a sliding window, analyze the changing trend of the data over time, evaluate the dependence relationship of the sensitive information in the time series according to the dynamic changes of the time features, and calculate the time-dependence feature index to quantify the sensitivity of the data over time.

[0010] Furthermore, the specific process of obtaining the data sensitivity index is as follows: Conduct a comprehensive evaluation of the abuse potential index, the identifiability feature index, and the time-dependence feature index; Perform fusion through weighted summation to obtain the data sensitivity index to quantify the sensitivity level of the dataset.

[0011] Furthermore, the specific process of identifying the multi-modal feature dataset according to the data sensitivity index is as follows: Sort the multi-modal feature dataset according to the data sensitivity index, and prioritize the features in the dataset from high to low sensitivity; According to the sorting result, divide the data in the dataset into two categories: high-sensitivity data and low-sensitivity data.

[0012] Furthermore, the specific process of screening the high-sensitivity data in the multi-modal data to obtain the sensitive dataset is as follows: Set a sensitivity threshold according to the data sensitivity index to identify which data belong to high-sensitivity data; Screen the sorted multi-modal feature dataset and select those feature data items with a sensitivity index higher than the threshold; Aggregate the selected high-sensitivity data items into a sensitive dataset.

[0013] Furthermore, the specific process of using a convolutional neural network to perform secondary identification on the sensitive data set is as follows: preprocess the selected sensitive data set and input the preprocessed data into the convolutional layer of the convolutional neural network; extract high-level features, including objects in images, semantic relationships in text, and trend changes in time series, through the stacking of multiple convolutional layers; perform pooling on the features output by the convolutional layer to reduce the feature dimension through max pooling and retain key information; input the pooled features into the fully connected layer to further comprehensively process the features through the fully connected layer to form a classification result; use the convolutional neural network to perform secondary identification on the sensitive data set and output the sensitivity label or classification result of each data item; adjust the parameters of the convolutional neural network, including the convolutional kernel size and the number of layers, according to the results of the secondary identification.

[0014] An intelligent identification method for sensitive data based on multi-modal feature fusion includes the following steps: S1. Extract visual features, semantic features, and time series features from multi-modal data through a convolutional neural network and fuse them through weighted summation to form a multi-modal feature data set; S2. Perform abuse potential feature analysis, identifiability feature analysis, and time dependence feature analysis on the multi-modal feature data set to obtain a data sensitivity index; S3. Identify the multi-modal feature data set according to the data sensitivity index, screen out the highly sensitive data in the multi-modal data, and obtain a sensitive data set; S4. Perform secondary identification on the sensitive data set through a convolutional neural network and mark the sensitive data in the sensitive data set.

[0015] The present invention has the following beneficial effects:

[0016] (1). The intelligent identification system for sensitive data based on multi-modal feature fusion can comprehensively and multi-dimensionally analyze and identify various forms of sensitive data by fusing visual features, semantic features, and time series features. This system can efficiently process data from different sources, improving the accuracy and robustness of identification. By using a convolutional neural network to achieve feature extraction and identification, it can automatically identify and mark sensitive information from a large amount of data, reducing the need for manual intervention, thereby improving the automation level of data protection and the overall performance of the system.

[0017] (2) The intelligent sensitive data recognition method based on multi-modal feature fusion provides a more refined means of sensitive data recognition by analyzing the features of data in terms of abuse potential, identifiability, and time dependence. By fusing features of different modalities through weighted summation, the processing ability for complex data is enhanced, and priority sorting can be performed according to the sensitivity of different features to ensure the timely recognition and protection of highly sensitive data. In addition, a convolutional neural network is used to perform secondary recognition on the data, further optimizing the accuracy and reliability of the data, effectively improving the sensitive data marking ability of the system, helping enterprises and users better protect privacy information, and preventing data leakage and abuse.

[0018] Of course, it is not necessary for any product implementing the present invention to achieve all the above-mentioned advantages simultaneously. Brief Description of the Drawings

[0019] Figure 1 It is a flowchart of the intelligent sensitive data recognition system based on multi-modal feature fusion of the present invention.

[0020] Figure 2 It is a flowchart of the intelligent sensitive data recognition method based on multi-modal feature fusion of the present invention. Detailed Embodiments

[0021] In the embodiments of the present application, through the intelligent sensitive data recognition system and method based on multi-modal feature fusion, the problems that the existing sensitive data recognition methods cannot fully fuse multi-modal data, lack dynamic processing ability, and have insufficient recognition accuracy are solved. By weighted fusion of visual features, semantic features, and time series features, and combined with abuse potential, identifiability, and time dependence analysis, this system can more comprehensively identify sensitive information in data, improve the recognition accuracy and robustness of sensitive data, and effectively cope with data changes and complexities in different scenarios. This method not only improves the recognition ability of the system, but also has strong adaptability and scalability, can be widely applied in multiple industry fields, and effectively guarantees data security and privacy protection.

[0022] The general idea of the solution in the embodiments of the present application is as follows:

[0023] Extract visual features, semantic features, and time series features from multi-modal data through a convolutional neural network, and fuse them through weighted summation to form a multi-modal feature data set.

[0024] Perform abuse potential feature analysis, identifiability feature analysis, and time dependence feature analysis on the multi-modal feature data set to obtain a data sensitivity index.

[0025] Identify the multi-modal feature data set according to the data sensitivity index, screen out the highly sensitive data in the multi-modal data, and obtain a sensitive data set.

[0026] The sensitive data set is secondarily identified through a convolutional neural network, and the sensitive data in the sensitive data set is marked.

[0027] Please refer to Figure 1 , an embodiment of the present invention provides a technical solution: a sensitive data intelligent recognition system based on multi-modal feature fusion, including the following steps: a feature fusion module, a feature analysis module, a data analysis module, and a data recognition module; the feature fusion module is used to extract visual features, semantic features, and time series features from multi-modal data through a convolutional neural network, and fuse them through weighted summation to form a multi-modal feature data set; the feature analysis module is used to perform abuse potential feature analysis, recognizability feature analysis, and time dependence feature analysis on the multi-modal feature data set to obtain a data sensitivity index; the data analysis module is used to identify the multi-modal feature data set according to the data sensitivity index, screen out high-sensitivity data in the multi-modal data, and obtain a sensitive data set; the data recognition module is used to secondarily identify the sensitive data set through a convolutional neural network, and mark the sensitive data in the sensitive data set.

[0028] In this implementation plan, the feature fusion module: Function: The main function of this module is to extract different types of feature information from multimodal data and fuse them. Specific implementation: Through a Convolutional Neural Network (CNN), three main types of features are extracted from multimodal data: Visual features: Visual information extracted from data such as images and videos, which may involve edges, textures, shapes, etc. Semantic features: Semantic information extracted from non-visual data such as text and audio, involving words, sentence structures, and context relationships. Time series features: Information extracted from time series data, such as dynamic change trends and patterns (e.g., financial data, sensor data, etc.). Then, these features are fused through weighted summation to obtain a comprehensive multimodal feature dataset for subsequent analysis and recognition. Feature analysis module: Function: This module deeply analyzes the fused multimodal feature dataset and evaluates the riskiness and sensitivity of different features. Specific implementation: Abuse potential feature analysis: Analyze whether each feature is easily abused and identify potential sensitive information. Identifiability feature analysis: Evaluate the ease of identifying each feature and whether it can be maliciously identified and attacked. Time-dependent feature analysis: Consider the change trend of features over time and evaluate whether some features will expose more sensitive information over time. Finally, the analysis results are converted into a data sensitivity index, which quantifies the sensitivity of the features and provides a basis for subsequent identification and screening. Data analysis module: Function: This module classifies and screens the multimodal feature dataset based on the data sensitivity index. Specific implementation: Data identification: According to the data sensitivity index, the system first sorts the dataset and prioritizes the features from high to low sensitivity. Screen high-sensitivity data: Screen out data items with higher sensitivity to form a sensitive dataset. This is to extract the parts of the data that are most likely to involve privacy leakage or abuse. Data identification module: Function: This module further intelligently identifies and marks the screened sensitive dataset. Specific implementation: Through a Convolutional Neural Network (CNN), the sensitive dataset is identified again. At this time, the system further analyzes the features of the sensitive data through a deep learning model to generate more accurate identification results. Mark the sensitive data to determine which data belongs to sensitive information for subsequent processing or protection.

[0029] Specifically, the specific process of extracting visual features, semantic features, and time series features from multi-modal data through a convolutional neural network is as follows: The multi-modal data includes image data, text data, and time series data; preprocess the image data, perform a convolution operation on the image through the convolutional layer of the convolutional neural network to extract the visual features of the image data, and perform downsampling on the convolutional output through the pooling layer; preprocess the text data, including word segmentation and stop word removal, convert each word in the text into an embedding vector through word embedding, perform a convolution operation on the word embedding vector through the convolutional layer to extract the semantic features of the text data, and perform pooling on the convolution result through the pooling layer to obtain the overall semantic representation of the text; perform normalization processing on the time series data, perform a convolution operation on the time series data through the temporal convolutional layer in the convolutional neural network to extract short-term and long-term dependencies, and perform downsampling on the convolution result through the pooling layer to obtain the time series data trend and change feature vector.

[0030] In this implementation, the processing of image data: Preprocessing: First, standardize and enhance the image data to improve the data quality and reduce the impact of noise. The image data may include various types of pictures, such as medical images, security surveillance images, etc. Convolution operation: Perform convolution operations on the image through the convolution layer of the convolutional neural network. The convolution operation extracts features from local regions of the image by sliding the convolution kernel, and can capture low-level features such as edges, textures, and shapes in the image. Pooling operation: After passing through the convolution layer, the output feature map will be downsampled through the pooling layer. The pooling layer reduces the size of the feature map through the max pooling operation, thereby retaining the most important feature information, while reducing the computational amount and improving the robustness of the model. The processing of text data: Preprocessing: Before the text data is input into the convolutional neural network, preprocessing is required. This includes word segmentation (segmenting the text into words or sub-words), removing stop words (such as meaningless words like "de", "shi", etc.), and word vectorization. Word embedding: Convert each word into a low-dimensional dense vector through word embedding. These embedding vectors can capture the semantic relationships between words. Convolution operation: Input the word embedding vectors in the text into the convolution layer, and extract semantic features through the convolution operation. The convolution layer can capture semantic patterns in phrases or sentences from local word vectors, such as collocation relationships or sentiment information between words. Pooling operation: The feature map output by the convolution layer will be downsampled through the pooling layer to obtain the overall semantic representation of the text. This process can extract the key information in the text while reducing the computational complexity. The processing of time series data: Standardization processing: Time series data usually needs to be standardized to eliminate the differences between different dimensions and ensure that the mean and variance of the data are within a reasonable range. Temporal convolution operation: The time series data passes through the temporal convolution layer in the convolutional neural network for convolution operation. Different from traditional convolution, the temporal convolution layer mainly processes the temporal dependence of the data and can capture short-term and long-term dependencies in the time series. For example, information such as trend changes and periodic fluctuations in the time series will be extracted by the convolution operation. Pooling operation: Similar to image and text data, after the time series data is convolved, it is downsampled through the pooling layer to extract the trend and change features of the data. This helps to capture the key patterns and features in the time series and reduce redundant information.

[0031] Specifically, fusion is performed through weighted summation, and the specific process of constructing the multi-modal feature dataset is as follows: Map the visual features and semantic features extracted from the convolutional neural network into a common feature space, and perform weighted combination and integration of the visual features and semantic features through weighted summation; Standardize the fused visual and semantic feature vectors, and perform weighted summation of the time series features and the fused visual and semantic features; Divide the time series features into time windows to form a multi-modal feature dataset.

[0032] In this implementation, the mapping of visual features and semantic features:

[0033] The visual features and semantic features extracted in the convolutional neural network respectively correspond to the key patterns or information in the image data and text data. These features need to be mapped into a unified feature space so that they can be effectively fused.

[0034] This mapping is usually achieved by converting the representations of each type of feature (such as the convolutional output of the image and the semantic embedding of the text) into a common vector space, ensuring that different types of features have the same scale and expressive power.

[0035] Fusing visual features and semantic features by weighted summation:

[0036] After mapping the visual features and semantic features into the common feature space, they need to be fused by weighted summation. Weighted summation means assigning a weight value to each feature (which can be set based on the importance of the feature or other criteria), then multiplying all features by their corresponding weights, and finally adding the results.

[0037] The goal of this operation is to combine the visual features and semantic features to obtain a unified feature vector containing multiple types of information. Weighted summation can effectively integrate features of different modalities, emphasize important information, and at the same time reduce the influence of unimportant features.

[0038] Normalization processing:

[0039] The fused visual and semantic feature vectors will be subjected to normalization processing. The purpose of normalization is to ensure that the scales of all features are consistent and to prevent certain features from dominating the entire model training process due to large numerical values. Usually, normalization is performed by subtracting the mean and dividing by the standard deviation, so that the mean of the feature data is 0 and the variance is 1.

[0040] Fusing time series features:

[0041] The features of time series data (such as trends, periodic changes, etc.) also need to be fused with visual features and semantic features. By means of weighted summation, the time series features are combined with the normalized visual and semantic features. This process also assigns weights to each feature and integrates the contribution values of different features into a unified representation vector.

[0042] This weighted summation operation combines the dynamic change information of the time series with the static visual and semantic features to generate an all-round multi-modal feature representation, which can provide richer data features for subsequent analysis.

[0043] Time window partitioning:

[0044] Time series data has temporal dependence. Therefore, before fusing it, it is usually necessary to perform time window partitioning. Time window partitioning refers to dividing the time series data into multiple subsequences according to a certain time length, and each subsequence represents the data trend and changes within a time period.

[0045] This operation can help the model better capture the temporal changes in the time series and avoid missing important time information. The size of the time window can be adjusted according to the characteristics of the data to ensure that key time patterns can be maximally captured.

[0046] Construct a multi-modal feature dataset:

[0047] After the above steps, the features of vision, semantics, and time series are effectively fused and divided into time windows. The finally formed multi-modal feature dataset contains information from different data modalities, such as the visual features of images, the semantic features of text, and the temporal dependence features of time series.

[0048] This dataset can comprehensively represent the key information in the original data, provide richer and more comprehensive input features for subsequent sensitive data identification and classification, and thus improve the accuracy and effectiveness of the system in sensitive data identification.

[0049] Specifically, the specific processes of analyzing the abuse potential features, identifiability features, and temporal dependence features of the multi-modal feature dataset are as follows: Conduct a risk assessment on each of the visual features, semantic features, and time series features, identify data abuse features according to the abuse potential assessment model, determine the sensitivity level of each feature, and generate an abuse potential index; Conduct an identifiability analysis on the visual features, semantic features, and time series features, evaluate the contribution of each feature to the data identification result through a chi-square test, and calculate the identifiability feature index according to the identification difficulty of the feature, the feasibility of the attack, and the exposure degree of the target to quantify the possibility that the features in the dataset are identified and abused; Conduct a temporal dependence analysis on the time series features, identify the temporal patterns of the features and the sensitivity changes existing in the time window, capture the sensitivity patterns at different time points and time periods through a sliding window, analyze the change trend of the data over time, evaluate the dependence relationship of sensitive information in the time series according to the dynamic changes of the time features, and calculate the temporal dependence feature index to quantify the sensitivity of the data changing over time.

[0050] In this implementation plan, the purpose of the abuse potential feature analysis is to evaluate the abuse risk of each feature and generate an abuse potential index The following is the formula and calculation process of the abuse potential assessment: Abuse potential assessment model: By constructing model M abuse (x i ), for each feature x iFor risk assessment, the model combines multiple factors, such as the likelihood of data abuse The feasibility of an attack And the degree of exposure Where: Is the risk assessment of the j-th type of abuse potential (such as illegal access risk, etc.); Is the risk assessment of the j-th type of attack; Is the risk assessment of the j-th type of exposure; ω 1,j , ω 2,j , ω 3,j Are the weight coefficients of the corresponding factors. Abuse potential index: Combining the above model, generate the abuse potential index for each feature: Where, δ 1 , δ 2 , δ 3 Are adjustment coefficients to ensure that the abuse potential index reflects multi-faceted evaluations. The analysis of identifiable features evaluates the contribution of each feature to the data identification result through the chi-square test, and further quantifies the identifiable feature index R index (x i ). AD is as follows: Chi-square test formula: For each feature, use the chi-square test to calculate its contribution χ 2 (x i ): Where: O k,i Is the observed frequency of feature x i In the k-th category; E k,i Is the expected frequency of feature x i In the k-th category. Identifiable feature index: Calculate the identifiable feature index through the chi-square value: R index (x i ) = α 1 ·χ 2 (x i ) + α 2 ·D difficulty (x i ) + α 3 ·A attack (x i ) + α 4 ·E exposure (x i ); Where, D difficulty (x i ) Is the identification difficulty of feature x i , A attack (x i ) Is the feasibility of feature x i Being attacked; E exposure (xi ) is the exposure degree of feature x i ; α 1 , α 2 , α 3 , α 4 is the adjustment coefficient to ensure that the index reflects the contributions of all dimensions. Time-dependent feature analysis: Time-dependent feature analysis evaluates the temporal patterns and their sensitivity changes in time series data and generates a time-dependent feature index The specific process is as follows: Time window partitioning: The time series data is divided into multiple time windows, and each time window contains k time points, denoted as , Sensitivity change capture: For each time window, calculate the sensitivity change metric within the window It can be calculated by the following formula: where is the mean of the data within the window. Time-dependent feature index: Through the sensitivity change metric and trend analysis, combined with the time dependence within the window, calculate the time-dependent feature index: where: is the trend metric of time window , reflecting the growth or decline trend of the time series; is the sensitivity change rate within the time window, measuring the sensitivity change speed of the time series in different time periods; γ 1 , γ 2 , γ 3 is the adjustment coefficient used to balance the weights of various indicators.

[0051] Specifically, the specific process of obtaining the data sensitivity index is as follows: Comprehensively evaluate the abuse potential index, identifiability feature index, and time-dependent feature index; Perform fusion through weighted summation to obtain the data sensitivity index, quantifying the sensitivity degree of the data set.

[0052] In this implementation scheme, the data sensitivity index formula: The process of obtaining the data sensitivity index is based on the weighted summation of the abuse potential index, identifiability feature index, and time-dependent feature index. The specific formula is as follows: where: is the data sensitivity index, used to quantify the overall sensitivity degree of the data set; is the abuse potential index of the a-th feature; R index,a is the identifiability feature index of the a-th feature; is the time-dependent feature index of the a-th feature. β 1 , β 2 , β 3: The weight coefficients for the abuse potential, identifiability, and time-dependence indices respectively, which are used to control the influence degree of these three features on the data sensitivity.

[0053] Specifically, the specific process of identifying the multi-modal feature data set according to the data sensitivity index is as follows: Sort the multi-modal feature data set according to the data sensitivity index, and prioritize the features in the data set from high to low sensitivity; According to the sorting result, divide the data in the data set into two categories: high-sensitivity data and low-sensitivity data.

[0054] In this implementation plan, data sorting: Sort all the features in the multi-modal feature data set according to the data sensitivity index of each feature. The data sensitivity index represents the comprehensive sensitivity of each feature in terms of abuse potential, identifiability, and time-dependence. After sorting, the features are prioritized from high to low sensitivity. Feature priority sorting: The sorted result divides the features into high and low sensitivity levels. Features with high sensitivity are listed as high-priority features, and features with low sensitivity are listed as low-priority features. This sorting process ensures that more sensitive features are given priority attention and protection. Classification processing: Based on the sorting result, the features in the data set are divided into two categories: High-sensitivity data: Refers to those features with a higher sensitivity index, representing the key privacy information in the data set or data that may pose high risks. Such data requires special protection measures, such as encryption, access control, etc. Low-sensitivity data: Refers to those features with a lower sensitivity index, indicating that such data is less involved in the risk of privacy leakage and is relatively less sensitive. Such data can relax the protection measures and be used and shared appropriately. According to the classification result, more stringent protection measures can be taken for high-sensitivity data, such as encryption, permission restrictions, etc., to ensure that it will not be leaked or misused; while low-sensitivity data can be analyzed or shared more under the premise of ensuring basic security.

[0055] Specifically, the specific process of screening high-sensitivity data in the multi-modal data and obtaining the sensitive data set is as follows: Set a sensitivity threshold according to the data sensitivity index to identify which data belong to high-sensitivity data; Screen the sorted multi-modal feature data set and select those feature data items with a sensitivity index higher than the threshold; Aggregate the selected high-sensitivity data items into a sensitive data set.

[0056] In this implementation, a sensitivity threshold is set: Based on the distribution of the data sensitivity index, a sensitivity threshold is set. This threshold is used to distinguish between high-sensitivity data and low-sensitivity data in the dataset. The setting of the threshold can be based on factors such as historical data, industry standards, and risk assessment to ensure that data that needs to be protected with emphasis can be effectively identified. Screening high-sensitivity data items: Screen the sorted multi-modal feature dataset. For each feature data item, check whether its corresponding data sensitivity index is higher than the set sensitivity threshold. If the sensitivity index of a certain feature data item exceeds the threshold, then this data is considered high-sensitivity data and should be included in the sensitive dataset. Aggregating into a sensitive dataset: During the screening process, all feature data items that meet the conditions (i.e., the sensitivity index is higher than the threshold) are aggregated to form a special sensitive dataset. This dataset contains those features with higher sensitivity in the multi-modal data, and these features usually require strict protection measures such as encryption and restricted access permissions.

[0057] Specifically, the specific process of secondary recognition of the sensitive dataset through a convolutional neural network is as follows: Preprocess the screened sensitive dataset and input the preprocessed data into the convolutional layer of the convolutional neural network; Through the stacking of multiple convolutional layers, extract high-level features, including objects in images, semantic relationships in text, and trend changes in time series; Perform pooling processing on the features output by the convolutional layer. Reduce the feature dimension through max pooling and retain key information; Input the pooled features into the fully connected layer, and further comprehensively process the features through the fully connected layer to form a classification result; The convolutional neural network performs secondary recognition on the sensitive dataset and outputs the sensitivity label or classification result of each data item; According to the results of the secondary recognition, adjust the parameters of the convolutional neural network, including the size of the convolutional kernel and the number of layers.

[0058] In this implementation, data preprocessing: Preprocess the selected sensitive data set. The preprocessing steps may include operations such as normalization, denoising, filling in missing values, etc., to ensure that the data format and quality are suitable for input into the convolutional neural network. For image data, it may also include operations such as resizing and color channel processing; for text data, it may be necessary to perform word segmentation, stop word removal, word vectorization, etc.; for time series data, it may be necessary to perform standardization or time alignment, etc. Input convolutional layer: The preprocessed data is input into the convolutional layer of the convolutional neural network. The convolutional layer performs a convolution operation on the input data by applying multiple convolutional kernels (filters). These convolutional kernels automatically learn and extract the basic features in the input data, such as edges and textures in images, keywords and syntactic patterns in text, and local trends and periodic changes in time series. Feature extraction: In the convolutional layer, by stacking multiple convolutional layers, the network can extract increasingly complex high-level features. In image data, the convolutional layer can identify features such as the shape and color of objects; in text data, the convolutional layer can identify semantic relationships or syntactic structures between words; in time series data, the convolutional layer can capture long-term and short-term trend changes. These features can be regarded as meaningful information in the original data, helping to distinguish sensitive and non-sensitive content. Pooling processing: Perform a pooling operation on the features output by the convolutional layer, usually using max pooling. The pooling operation has two main purposes: First, by reducing the dimension of the features, the computational complexity is reduced; second, by retaining the most significant feature information (such as the maximum value or average value), the transmission of key information is ensured. The role of the pooling layer is to enhance the translational invariance of the network and reduce the impact of local feature changes on the results. Fully connected layer: The pooled feature map is input into the fully connected layer. In the fully connected layer, the network further extracts the overall features of the data by comprehensively processing the features output by the pooling layer. This process is similar to pattern recognition, and the network comprehensively processes the features according to the weights learned during training, and finally forms a complete classification result or label. Secondary recognition and classification result: The convolutional neural network performs secondary recognition on the entire sensitive data set and outputs the classification result or sensitivity label of each data item. The classification result is labeled according to the prediction output in the training model, and may include different categories such as high sensitivity and low sensitivity. Each data item is labeled as sensitive or non-sensitive according to its performance in the network. Parameter adjustment and optimization: After secondary recognition, the convolutional neural network can be parameter-adjusted by evaluating the accuracy of the classification result. These parameters include the size of the convolutional kernel and the number of network layers. By adjusting these parameters, the performance of the model is optimized, and the accuracy and robustness of the recognition are improved. The adjustment process can be trained through the backpropagation algorithm to reduce the error of the prediction result.

[0059] An intelligent recognition method for sensitive data based on multi-modal feature fusion, comprising the following steps: S1. Extract visual features, semantic features, and time series features from multi-modal data through a convolutional neural network, and fuse them through weighted summation to form a multi-modal feature data set; S2. Conduct abuse potential feature analysis, identifiability feature analysis, and time-dependency feature analysis on the multi-modal feature data set to obtain a data sensitivity index; S3. Identify the multi-modal feature data set according to the data sensitivity index, screen out the highly sensitive data in the multi-modal data, and obtain a sensitive data set; S4. Conduct secondary identification on the sensitive data set through a convolutional neural network, and mark the sensitive data in the sensitive data set.

[0060] In this implementation plan, S1. Feature extraction and fusion: Extract visual features, semantic features, and time series features from multi-modal data (such as images, texts, time series, etc.) through a convolutional neural network (CNN). These features are fused through weighted summation to generate a multi-modal feature data set containing comprehensive information from different data sources. S2. Feature analysis and sensitivity index calculation: Conduct three aspects of analysis on the fused feature data: abuse potential feature analysis (evaluating the risk of data abuse), identifiability feature analysis (evaluating the difficulty of data identification), and time-dependency feature analysis (evaluating the impact of time changes on data sensitivity). The results of these analyses generate a data sensitivity index indicating the sensitivity of the data. S3. Data screening and identification: According to the data sensitivity index, sort the feature data and screen out the data items with higher sensitivity to form a sensitive data set. S4. Secondary identification and marking: Conduct further secondary identification on the screened sensitive data set through a convolutional neural network, output the sensitivity label or classification result of each data item, and mark the sensitive data.

[0061] In summary, this application has at least the following effects:

[0062] The intelligent recognition system and method for sensitive data based on multi-modal feature fusion can comprehensively and accurately identify the sensitivity of data by multi-modal feature fusion, combining visual, semantic, and time series features, avoiding the limitations that may exist in a single modality. Using the data sensitivity index to comprehensively evaluate the sensitivity of various data features realizes automated and efficient screening of sensitive data, ensuring the timely protection and management of sensitive information. The convolutional neural network can adaptively adjust network parameters in the secondary identification of data features, thereby continuously optimizing the recognition accuracy and improving the recognition ability for different types of sensitive data. Through the automated feature extraction, analysis, and recognition process, the need for manual intervention is greatly reduced, the possibility of human errors is lowered, and the efficiency of data processing is improved.

[0063] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0064] The present invention is described with reference to the flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0065] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0066] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0067] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.

[0068] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims

1. A sensitive data intelligent identification system based on multimodal feature fusion, characterized by: The following steps are involved: Feature fusion module, feature analysis module, data analysis module, data recognition module; The feature fusion module is used to extract visual features, semantic features and time series features from multimodal data through a convolutional neural network, and fuse them through weighted summation to form a multimodal feature data set; The feature analysis module is used to perform abuse potential feature analysis, identifiability feature analysis and time-dependency feature analysis on the multimodal feature data set to obtain a data sensitivity index; The data analysis module is used to identify the multimodal feature data set according to the data sensitivity index, screen the highly sensitive data in the multimodal data, and obtain the sensitive data set; The data identification module is used to perform secondary identification on the sensitive data set through a convolutional neural network and mark the sensitive data in the sensitive data set.

2. The sensitive data intelligent identification system based on multimodal feature fusion according to claim 1 is characterized in that: The specific process of extracting visual features, semantic features, and time series features from multimodal data through convolutional neural networks is as follows: The multimodal data includes image data, text data and time series data; Preprocess the image data, perform convolution operations on the image through the convolutional layer of the convolutional neural network, extract the visual features of the image data, and downsample the convolution output through the pooling layer; Preprocess the text data, including word segmentation and stop word removal. Convert each word in the text into an embedding vector through word embedding. Perform convolution operations on the word embedding vector through the convolution layer to extract the semantic features of the text data. Pool the convolution results through the pooling layer to obtain the overall semantic representation of the text. The time series data is standardized and convolved through the time convolution layer in the convolutional neural network to extract short-term and long-term dependencies. The convolution results are downsampled through the pooling layer to obtain the trend and change feature vectors of the time series data.

3. The sensitive data intelligent identification system based on multimodal feature fusion according to claim 2 is characterized in that: The specific process of fusion through weighted summation to form a multimodal feature data set is as follows: The visual features and semantic features extracted from the convolutional neural network are mapped to a common feature space respectively, and the visual features and semantic features are weighted and integrated by weighted summation; The fused visual and semantic feature vectors are normalized, and the time series features are weighted summed with the fused visual and semantic features; The time series features are divided into time windows to form a multimodal feature dataset.

4. The sensitive data intelligent identification system based on multimodal feature fusion according to claim 3 is characterized in that: The specific process of performing abuse potential feature analysis, identifiability feature analysis, and time-dependency feature analysis on the multimodal feature dataset is as follows: Perform risk assessment on each of the visual features, semantic features, and time series features, identify data abuse features based on the abuse potential assessment model, determine the sensitivity of each feature, and generate an abuse potential index; Perform identifiability analysis on visual features, semantic features, and time series features. Use chi-square test to evaluate the contribution of each feature to the data recognition result. Calculate the identifiability feature index based on the recognition difficulty of the feature, the feasibility of the attack, and the exposure of the target to quantify the possibility of the feature in the data set being recognized and abused. Perform time dependency analysis on time series features, identify the temporal patterns of features and sensitivity changes in time windows, capture sensitivity patterns at different time points and time periods through sliding windows, analyze the changing trends of data over time, evaluate the dependency of sensitive information in time series based on the dynamic changes of time features, and calculate the time dependency feature index to quantify the sensitivity of data changes over time.

5. The sensitive data intelligent identification system based on multimodal feature fusion according to claim 4 is characterized in that: The specific process of obtaining the data sensitivity index is as follows: Comprehensively evaluate the abuse potential index, identifiability characteristic index and time-dependent characteristic index; The data sensitivity index is obtained by weighted summation to quantify the sensitivity of the data set.

6. The sensitive data intelligent identification system based on multimodal feature fusion according to claim 5 is characterized in that: The specific process of identifying the multimodal feature data set according to the data sensitivity index is as follows: Sort the multimodal feature datasets according to the data sensitivity index, and prioritize the features in the dataset from high to low sensitivity; According to the sorting results, the data in the dataset are divided into two categories: high-sensitivity data and low-sensitivity data.

7. The sensitive data intelligent identification system based on multimodal feature fusion according to claim 6 is characterized in that: The specific process of screening highly sensitive data in multimodal data and obtaining sensitive data sets is as follows: According to the data sensitivity index, set a sensitivity threshold to identify which data is highly sensitive data; Screen the sorted multimodal feature data set and select those feature data items whose sensitivity index is higher than the threshold; The filtered highly sensitive data items are aggregated into sensitive data sets.

8. The sensitive data intelligent identification system based on multimodal feature fusion according to claim 7 is characterized in that: The specific process of secondary identification of sensitive data sets through convolutional neural networks is as follows: Preprocess the selected sensitive data sets and input the preprocessed data into the convolutional layer of the convolutional neural network; By stacking multiple convolutional layers, high-level features are extracted, including objects in images, semantic relationships in text, and trend changes in time series; Pool the features output by the convolutional layer, reduce the feature dimension through maximum pooling, and retain key information; The pooled features are input into the fully connected layer, and the features are further processed through the fully connected layer to form the classification results; The convolutional neural network performs secondary recognition on sensitive data sets and outputs the sensitivity label or classification result of each data item; According to the results of the secondary recognition, the parameters of the convolutional neural network are adjusted, including the convolution kernel size and the number of layers.

9. A method for intelligently identifying sensitive data based on multimodal feature fusion, applied to a system for intelligently identifying sensitive data based on multimodal feature fusion as claimed in any one of claims 1 to 8, characterized in that: The following steps are involved: S1. Extract visual features, semantic features and time series features from multimodal data through convolutional neural network, and fuse them through weighted summation to form a multimodal feature dataset; S2. Perform abuse potential feature analysis, identifiability feature analysis, and time-dependency feature analysis on the multimodal feature dataset to obtain a data sensitivity index; S3. Identify the multimodal feature data set according to the data sensitivity index, screen the highly sensitive data in the multimodal data, and obtain the sensitive data set; S4. Perform secondary recognition on sensitive data sets through convolutional neural networks and mark sensitive data in sensitive data sets.

Images