Deep learning-based automatic classification and secret setting method for image-text mixed document in manufacturing industry

Abstract

The invention discloses a deep learning-based automatic classification and secret setting method for a manufacturing industry image-text mixed document. The method comprises the following steps of S1, acquiring the image-text mixed document; s2, preprocessing a text part in the image-text mixed document; s3, generating a text feature vector; s4, preprocessing the image part in the document; s5, generating an image feature vector; s6, generating a comprehensive feature vector; s7, carrying out classification analysis on the comprehensive feature vector by utilizing a self-adaptive hierarchical classification model, and classifying the image-text mixed document into a predefined category; s8, carrying out sensitive information identification on the classified documents; s9, automatically judging the confidentiality level of the document through a confidentiality level judgment model based on dynamic risk assessment by using the identified sensitive information; and S10, outputting a classification and secret setting result, and storing the result in a document management system. According to the method, the efficiency and the accuracy of automatic classification and confidentiality grade judgment of the image-text mixed document are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology in the manufacturing industry, and in particular to a method for automatic classification and confidentiality determination of mixed text and image documents in the manufacturing industry based on deep learning. Background Technology

[0002] In the manufacturing sector, document management is a crucial aspect, especially when dealing with mixed text and image documents. These documents typically include important information such as design drawings, technical specifications, and operation manuals, playing a key role in a company's production and management. Traditional document management methods often rely on manual classification and security assignment, which is not only time-consuming but also prone to errors, especially when dealing with massive amounts of documents, making it difficult to guarantee efficiency and accuracy.

[0003] Existing document classification technologies primarily focus on processing pure text or pure images, with typical techniques including rule-based text classification algorithms and feature extraction-based image classification algorithms. However, these techniques have significant shortcomings when dealing with documents containing both text and images. Specifically, pure text classification methods typically rely on keyword matching or simple statistical methods, lacking an understanding of the deep semantics of the text and making it difficult to accurately identify the true category of the document content. On the other hand, pure image classification methods rely on the extraction and analysis of visual features of images. While they perform well when processing pure image documents, when images and text are mixed in the same document, the contextual information of the image portion is often ignored, leading to inaccurate classification results.

[0004] Furthermore, in the manufacturing industry, document confidentiality management is crucial, as many documents involve core technologies and confidential information. However, existing confidentiality management methods primarily rely on manual assessment, which has several major problems: First, manual assessment requires significant time and human resources, and different personnel may have different understandings of the confidentiality level of the same document, leading to subjectivity and inconsistency in the assessment results. Second, the accuracy of manual assessment is difficult to guarantee, especially when dealing with complex documents containing both text and images. It is challenging for humans to accurately identify all potentially sensitive information within a short timeframe, thus posing a significant risk of information leakage.

[0005] In the field of automatic classification and confidentiality management of mixed text and image documents, existing technologies still have the following key shortcomings:

[0006] Insufficient ability to comprehensively process text and image information: Existing technologies can usually only process a single type of information in text or images, lacking the ability to comprehensively process text and image information. This makes it impossible to effectively combine and classify text and image information when processing mixed text and image documents.

[0007] Lack of deep semantic analysis: Traditional text processing methods often rely on simple keyword matching or shallow syntactic analysis, making it difficult to understand the deep semantics of text, especially when dealing with complex technical documents. Image processing techniques are also usually limited to the extraction of visual features, ignoring the semantic information that images may contain.

[0008] Security level determination relies on manual judgment: Existing security management systems typically rely on manual judgment, which is not only inefficient but also prone to subjective bias, failing to ensure the accuracy and consistency of document security level determination.

[0009] Therefore, existing technologies have many shortcomings in the automatic classification and confidentiality management of mixed text and image documents. There is an urgent need for a method that can comprehensively process text and image information and achieve automated and accurate classification and confidentiality determination based on deep learning and dynamic risk assessment technologies, so as to improve the efficiency and security of document management in the manufacturing industry. Summary of the Invention

[0010] One objective of this invention is to propose an automatic classification and confidentiality determination method for mixed text and image documents in the manufacturing industry based on deep learning. This invention significantly improves the efficiency and accuracy of automatic classification and confidentiality level determination of mixed text and image documents.

[0011] An automatic classification and confidentiality determination method for mixed text and image documents in the manufacturing industry based on deep learning, according to an embodiment of the present invention, includes the following steps:

[0012] S1. Obtain the document containing mixed text and images, and initiate the automatic document confidentiality level identification process according to the method;

[0013] S2. Preprocess the text portion of the mixed text and image document, including noise reduction, word segmentation, and word vectorization.

[0014] S3. Use an adaptive multi-level bidirectional transform model to extract features from the preprocessed text and generate text feature vectors.

[0015] S4. Preprocess the image portion of the document, the preprocessing including grayscale conversion, noise reduction and normalization;

[0016] S5. Use a dynamic convolutional neural network to extract features from the preprocessed image and generate an image feature vector.

[0017] S6. Input the text feature vector and image feature vector into a multimodal fusion attention network to fuse features and generate a comprehensive feature vector;

[0018] S7. Use an adaptive hierarchical classification model to perform classification analysis on the comprehensive feature vector and classify the mixed text and image documents into predefined categories;

[0019] S8. Perform sensitive information identification on the classified documents, wherein the sensitive information identification includes keyword detection and semantic analysis;

[0020] S9. Using the identified sensitive information, the confidentiality level of the document is automatically determined by a confidentiality level determination model based on dynamic risk assessment.

[0021] S10. Output the classification and classification results, and save the results as needed.

[0022] Optionally, S1 includes the following steps:

[0023] S11. Obtain a mixed text and image document from a manufacturing-related document circulation or management system, wherein the mixed text and image document contains both text and image information types;

[0024] S12. The obtained mixed text and image document is parsed using the file parsing module to extract the text portion T from the mixed text and image document. d and image part I d ;

[0025] S13, Regarding the text portion T d and image part I d Perform formatting separately, and denote the formatted text portion as T. f The formatted image portion is denoted as I. f ;

[0026] S14. Format the text portion T f and image part I f The data is input into the preprocessing module of the deep learning model via a data transmission interface. The input process includes encoding the data and packaging it according to a predetermined data structure to generate the mixed text and image document dataset D8 to be processed.

[0027] D8={T f I f}

[0028] Optionally, S2 includes the following steps:

[0029] S21, Regarding the text portion T f Denoising is performed to remove noisy characters, meaningless symbols, and repetitive information from the text to generate clean text data T. olean ;

[0030] S22. For the denoised text data T clean Word segmentation is performed, dividing the text data T based on predefined language rules and a dictionary. cleanThe text is divided into several word units to generate a segmented text sequence T. req =(w1, w2, w n ), where w i This represents the i-th word unit after segmentation;

[0031] S23, the segmented text sequence T seg Word vectorization is performed, using a pre-trained word vector model to vectorize each word unit w. i Convert to the corresponding word vector v i Generate word vector order:

[0032] V T ={v1, v2, ..., v n};

[0033] Among them, v i w represents the word unit i The vector representation of .

[0034] Optionally, S3 includes the following steps:

[0035] S31. The text sequence V after word segmentation and word vectorization is... T The input to the input layer of the adaptive multilayer bidirectional transform model, which consists of multiple stacked bidirectional transform layers, is defined as follows:

[0036]

[0037] in, This represents element-wise addition of vectors. V represents the positional information vector at time step t, used to preserve the order information of the text sequence. in It is the sequence vector input to the bidirectional transform model;

[0038] S32. In each bidirectional transform layer, forward propagation and backward propagation are performed on the text sequence respectively. Forward propagation calculates the feature information from front to back:

[0039]

[0040] in, W represents the forward hidden state at time step t. f and U f These are the weight matrices in the forward propagation, b f Here, σ is the bias term, and σ is the activation function; backpropagation calculates the feature information from back to front:

[0041]

[0042] in, W represents the reverse hidden state at time step t. b and U b These are the weight matrices used in backpropagation, b b For bias terms;

[0043] S33, Transform the forward hidden state sequence With reverse hidden state sequence The hidden state sequence H is generated by splicing the hidden states together. t :

[0044]

[0045] Where || denotes the vector concatenation operation, H t This represents the comprehensive feature representation at time step t.

[0046] S34. Adaptively weight and summarize the output of each bidirectional transform layer to generate a global feature representation:

[0047]

[0048] Among them, H global This represents the comprehensive feature representation, where L is the number of bidirectional transform layers, and o1 is the adaptive weight coefficient for each layer. Let α represent the hidden state sequence of layer 1. l The text feature vector, obtained through model learning and used to highlight the importance of information at different levels, is ultimately generated as follows:

[0049]

[0050] Optionally, S4 includes the following steps:

[0051] S41, Regarding image portion I f Perform grayscale conversion to transform a color image into a grayscale image:

[0052] I gray (x, y) = 0.299·I r (x, y) + 0.587·I g (x, y) + 0.114·I b (x, y);

[0053] Among them, I gray (x, y) represents the grayscale value of a pixel at (x, y) in a grayscale image. r (xy), I g (x, y) and I b (x, y) represent the values ​​of the red, green, and blue channels of the corresponding pixel in the color image, respectively;

[0054] S42. For the grayscale image I gray Denoising is performed to remove noise interference from the image, resulting in the denoised image I. denoise ;

[0055] S43. For the denoised image I denoise Normalization is performed by scaling the image grayscale values ​​to a predetermined range [0, 1] to obtain the normalized image.

[0056] Optionally, S5 includes the following steps:

[0057] S51. Normalize the preprocessed image. The image feature data input to the input layer of the dynamic convolutional neural network is represented as a matrix I. input , where I input =I norm ;

[0058] S52. In the convolutional layer, multiple dynamic convolutional kernels are used to process the input image data I. input Perform convolution operations to generate feature maps F conv :

[0059]

[0060] Among them, F conv W represents the feature value generated by the k-th convolutional kernel at pixel (x, y). k (i,j) is the weight matrix of the k-th convolutional kernel, b k For the corresponding bias term, m and n are the sizes of the convolution kernel;

[0061] S53, Feature mapping F of the convolutional layer output conv Dynamic pooling is used to reduce feature dimensionality and optimize the model's robustness to interference.

[0062]

[0063] Among them, F pool (x′, y′, k) represents the maximum value of the k-th feature map within the pooling window P after pooling, and s is the pooling step size;

[0064] S54. Transfer the pooled feature map F pool The input is fed into subsequent dynamic convolutional and pooling layers for further feature extraction. The process of S52 and S53 is repeated until the predetermined feature dimension or number of extraction layers is reached.

[0065] S55. Flatten the high-level feature map generated after multi-layer convolution and pooling into a feature vector V1. The flattening process includes the following steps:

[0066]

[0067] Among them, F high (p, q, k) represents the feature value of the k-th feature map at position (p, q) in the high-level feature map, μ p，k and μ q，k Let σ represent the mean values ​​of the k-th feature map in directions p and q, respectively. p，k and σ q，k Let p and q represent the standard deviations of the k-th feature map in directions p and q, respectively. exp represents the natural exponential function, and Flatten represents the operation of flattening the multidimensional feature map into a one-dimensional feature vector.

[0068] Optionally, S6 includes the following steps:

[0069] S61. Transfer the text feature vector The image feature vector V1 is input into the input layer of the multimodal fusion attention network, and is represented as V... T and V1;

[0070] S62. Perform self-attention mechanism calculations on the text feature vector and image feature vector respectively to capture the correlation within the features:

[0071]

[0072] Where Q = W Q V T K=W K V T V=W V V T W represents the query, key, and value of the text feature vector, respectively. Q W K and W v Let d be a learnable weight matrix. k The dimension of the feature vector;

[0073] S63. Output A from the self-attention feature of the text. T With image feature vector V I Perform cross-attention calculation to generate fused feature representations:

[0074]

[0075] Among them, Q T =W QT A T K I =WKI V I V I =W VI V I W represents the query for text features and the key and value for image features, respectively. QT W KI and W VI The weight matrix is ​​a learnable matrix;

[0076] S64. Concatenate the output of the cross-attention calculation with the original text feature vector and image feature vector to generate a preliminary fused feature vector V. fused :

[0077] V fused =[CrossArtention(Q T K I V I )||A T ||V l ];

[0078] Where || denotes the vector concatenation operation;

[0079] S65, Initially fuse the feature vector V fused The input is fused into a fully connected layer to generate the final comprehensive feature vector V. composite :

[0080] V composite =W fu ·V fused +h fu ;

[0081] Among them, W fu h is the weight matrix of the fully connected layer. fu This is a bias term.

[0082] Optionally, S7 includes the following steps:

[0083] S71, Combine the feature vector V composite The input is fed into the input layer of the adaptive hierarchical classification model for preliminary feature classification. The adaptive hierarchical classification model includes multiple levels of classifiers, with each level of classifier handling different levels of feature complexity.

[0084] S72. In the primary classifier, a fully connected neural network is used to classify V. composite Perform preliminary classification and output the first-level classification result C1:

[0085]

[0086] in, This is the weight matrix of the primary classifier. C1 is the bias term, and C1 is the probability distribution of the first-level classification.

[0087] S73. The first-level classification result C1 is passed as input to the secondary classifier. The secondary classifier further refines the classification based on the primary classification result and outputs the second-level classification result C2:

[0088]

[0089] in, This is the weight matrix of the secondary classifier. C1 is the bias term, and C2 is the probability distribution of the second-level classification.

[0090] S74. Repeat steps S73, performing classification analysis through multiple levels of classifiers. Each level of classifier performs more refined classification based on the output of the previous level, finally outputting the top-level classification result C. l :

[0091]

[0092] in, This is the weight matrix of the top-level classifier. As a bias term, C L This is the final classification result.

[0093] Optionally, S8 includes the following steps:

[0094] S81. Sensitive information is identified in the categorized documents. Keyword detection is performed on the text portions. A predefined sensitive word library is used to match the text feature vectors, generating preliminary sensitive information tags M. key :

[0095]

[0096] Among them, w i M represents the i-th word unit in the text sequence. key For a predefined sensitive word library, δ is an indicator function, when w i If the word belongs to the sensitive word list, δ = 1; otherwise, δ = Q; μ key and u key γ represents the mean and standard deviation of words in the sensitive word database, respectively. i This is a weighting parameter used to adjust the impact of different words on sensitivity;

[0097] S82. Based on keyword detection, perform semantic analysis on the text content, use a pre-trained BERT model to perform semantic understanding on the text feature vectors, and generate a semantic representation S. semantic Calculate the sensitivity score S of the text. sens :

[0098]

[0099] Among them, S semantic For the semantic representation of text, w j b is the learnable weight vector in the semantic analysis model. j For the bias term, λ j The weight coefficient for each semantic feature, μ sem and σ sem These are the mean and standard deviation of the semantic features, respectively.

[0100] S83, Combining keyword detection results M key and semantic analysis score S send It integrates the results of keyword detection and semantic analysis, and takes into account the interaction between the two, to generate a comprehensive sensitive information tag M. send :

[0101]

[0102] Where α1 and β1 are the weight parameters for keyword detection and semantic analysis, and γ k and λ l η represents the weight coefficients of the corresponding keywords and semantic features. kl M is the interaction coefficient between keywords and semantic features. key，k and S sens，1 These represent the sensitivity scores of the k-th keyword and the first semantic feature, respectively.

[0103] S84, Mark the comprehensive sensitive information M sens The data is input into a security level determination model based on dynamic risk assessment. The model calculates the security level L of the document according to preset risk assessment standards. conf :

[0104]

[0105] Where ζ is the influence factor for sensitive information labeling, θ and v corf These are the threshold and standard deviation for determining the level of confidentiality, respectively; κ is the weight of contextual information in risk assessment; and ρ is the standard deviation of the security level. r Let R be the weight coefficient of the r-th contextual information feature. aontext，r This is a variable containing contextual information about the document.

[0106] The beneficial effects of this invention are:

[0107] (1) In the process of processing mixed text and image documents, this invention introduces a multimodal fusion attention network to effectively fuse text feature vectors and image feature vectors. Compared with the traditional single-modal processing method, the multimodal fusion mechanism can make full use of the complementary information of text and images, thereby achieving a more comprehensive and accurate analysis and classification of mixed text and image documents. Through the application of the adaptive hierarchical classification model, it can not only quickly locate the approximate category of the document in the primary classification, but also further accurately classify the document category through the refinement processing of the hierarchical classifier.

[0108] (2) The combined application of the adaptive multi-level bidirectional transform model and the dynamic convolutional neural network in this invention enables deep semantic analysis and feature extraction of text and image information. The adaptive multi-level bidirectional transform model can capture the contextual dependencies in text features and achieve a deep understanding of complex semantics through multi-level feature fusion; while the dynamic convolutional neural network can dynamically adjust the convolution kernel and pooling operation according to the complexity of image features, ensuring that the key features of image information are fully extracted. Through the combination of deep semantic analysis and adaptive feature extraction, this invention significantly improves the ability to process complex information in mixed text and image documents, making the classification results more accurate.

[0109] This invention combines a dynamic risk assessment model with sensitive information identification technology. By detecting keywords and performing semantic analysis on the text and analyzing features on the image, it automatically identifies potentially sensitive information in a document. Based on the identification results, the dynamic risk assessment model automatically determines the confidentiality level of the document. This not only improves the automation of confidentiality level determination but also greatly enhances the accuracy and consistency of the determination results, effectively preventing information leakage. Attached Figure Description

[0110] Figure 1 This is a flowchart of a deep learning-based method for automatic classification and confidentiality determination of mixed text and graphics documents in the manufacturing industry, as proposed in this invention. Detailed Implementation

[0111] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0112] refer to Figure 1 A deep learning-based method for automatic classification and confidentiality determination of mixed text and image documents in the manufacturing industry includes the following steps:

[0113] S1. Obtain a document containing both text and images;

[0114] S2. Preprocess the text portion of the mixed text and image document. Preprocessing includes noise reduction, word segmentation, and word vectorization.

[0115] S3. Use an adaptive multi-level bidirectional transform model to extract features from the preprocessed text and generate text feature vectors.

[0116] S4. Preprocess the image portion of the document, including grayscale conversion, noise reduction, and normalization.

[0117] S5. Use a dynamic convolutional neural network to extract features from the preprocessed image and generate an image feature vector.

[0118] S6. Input the text feature vector and image feature vector into a multimodal fusion attention network to fuse features and generate a comprehensive feature vector;

[0119] S7. Use an adaptive hierarchical classification model to perform classification analysis on the comprehensive feature vector and classify the mixed text and image documents into predefined categories;

[0120] S8. Perform sensitive information identification on the classified documents. Sensitive information identification includes keyword detection and semantic analysis.

[0121] S9. Using the identified sensitive information, the confidentiality level of the document is automatically determined by a confidentiality level determination model based on dynamic risk assessment.

[0122] S10. Output the classification and density determination results, and save the results.

[0123] In this embodiment, S1 includes the following steps:

[0124] S11. Obtain mixed text and image documents from manufacturing-related document circulation or management systems. Mixed text and image documents contain both text and image information types.

[0125] S12. The obtained mixed text and image document is parsed using the file parsing module to extract the text portion T from the mixed text and image document. d and image part I d ;

[0126] S13, Regarding the text portion T d and image part I d Perform formatting separately. The formatted text portion is denoted as Tf, and the formatted image portion is denoted as I. f ;

[0127] S14. Format the text portion T f and image part I f The data is input into the preprocessing module of the deep learning model via a data transmission interface. The input process includes encoding the data and packaging it according to a predetermined data structure to generate the mixed text and image document dataset D8 to be processed.

[0128] D8={T f ,f}.

[0129] In this embodiment, S2 includes the following steps:

[0130] S21, Regarding the text portion T f Denoising is performed to remove noisy characters, meaningless symbols, and repetitive information from the text to generate clean text data T. clean ;

[0131] S22. For the denoised text data T clean Word segmentation is performed, dividing the text data T based on predefined language rules and a dictionary. clean The text is divided into several word units to generate a segmented text sequence T. seg ={w1, w2, ..., w n}, where w i This represents the i-th word unit after segmentation;

[0132] S23, the segmented text sequence T seg Word vectorization is performed, using a pre-trained word vector model to vectorize each word unit w. i Convert to the corresponding word vector vi, and generate the word vector order:

[0133] V T ={v1, v2, ..., v n};

[0134] Among them, v i w represents the word unit i The vector representation of .

[0135] In this embodiment, S3 includes the following steps:

[0136] S31. Input the text sequence VT after word segmentation and word vectorization into the input layer of the adaptive multi-layer bidirectional transform model. The adaptive multi-layer bidirectional transform model consists of multiple stacked bidirectional transform layers. The input process of the text sequence is defined as follows:

[0137]

[0138] in, This represents element-wise addition of vectors. V represents the positional information vector at time step t, used to preserve the order information of the text sequence. in It is the sequence vector input to the bidirectional transform model;

[0139] S32. In each bidirectional transform layer, forward propagation and backward propagation are performed on the text sequence respectively. Forward propagation calculates the feature information from front to back:

[0140]

[0141] in, W represents the forward hidden state at time step t. i and U f These are the weight matrices in the forward propagation, b f Here, σ is the bias term, and σ is the activation function; backpropagation calculates the feature information from back to front:

[0142]

[0143] in, W represents the reverse hidden state at time step t. b and U b These are the weight matrices used in backpropagation, b b For bias terms;

[0144] S33, Transform the forward hidden state sequence With reverse hidden state sequence The hidden state sequence H is generated by splicing the hidden states together. t :

[0145]

[0146] Where || denotes the vector concatenation operation, H t This represents the comprehensive feature representation at time step t.

[0147] S34. Adaptively weight and summarize the output of each bidirectional transform layer to generate a global feature representation:

[0148]

[0149] Among them, H global This represents the comprehensive feature representation, where L is the number of bidirectional transform layers, and α l H represents the adaptive weight coefficients for each layer. t，l Let α represent the hidden state sequence of layer l. l The text feature vector, obtained through model learning and used to highlight the importance of information at different levels, is ultimately generated as follows:

[0150]

[0151] In this embodiment, S4 includes the following steps:

[0152] S41, Regarding image portion I f Perform grayscale conversion to transform a color image into a grayscale image:

[0153] Igray (x, y) = 0.299·I r (x, y) + 0.587·I g (x,,y)+0.114·I b (x, y);

[0154] Among them, I gray (x, y) represents the grayscale value of a pixel at (x, y) in a grayscale image. r (x, y), I g (x, y) and I b (x, y) represent the values ​​of the red, green, and blue channels of the corresponding pixel in the color image, respectively;

[0155] S42. For the grayscale image I gray Denoising is performed to remove noise interference from the image, resulting in the denoised image I. denoise ;

[0156] S43. For the denoised image I denoiie Normalization is performed to scale the image grayscale values ​​to a predetermined range [0, 1], resulting in the normalized image I. norm .

[0157] In this embodiment, S5 includes the following steps:

[0158] S51, The preprocessed normalized image I norm The image feature data input to the input layer of the dynamic convolutional neural network is represented as a matrix I. input , where I input =I norm ;

[0159] S52. In the convolutional layer, multiple dynamic convolutional kernels are used to process the input image data I. inpu t performs a convolution operation to generate a feature map P. conv :

[0160]

[0161] Among them, F conv W represents the feature value generated by the k-th convolutional kernel at pixel (x, y). k (1, j) is the weight matrix of the k-th convolutional kernel, b k For the corresponding bias term, m and n are the sizes of the convolution kernel;

[0162] S53, Feature mapping F of the convolutional layer output conv Dynamic pooling is used to reduce feature dimensionality and optimize the model's robustness to interference.

[0163]

[0164] Among them, F pool (x′, y′, k) represents the maximum value of the k-th feature map within the pooling window P after pooling, and s is the pooling step size;

[0165] S54. Transfer the pooled feature map F pool The input is fed into subsequent dynamic convolutional and pooling layers for further feature extraction. The process of S52 and S53 is repeated until the predetermined feature dimension or number of extraction layers is reached.

[0166] S55. Flatten the high-level feature map generated after multiple convolutions and pooling into a feature vector V. I The flattening process includes the following steps:

[0167]

[0168] Among them, F high (p, q, k) represents the feature value of the k-th feature map at position (p, q) in the high-level feature map, μ p，k and μ o，k Let σ represent the mean values ​​of the k-th feature map in directions p and q, respectively. p，k and σ q，k Let p and q represent the standard deviations of the k-th feature map in directions p and q, respectively. exp represents the natural exponential function, and Flatten represents the operation of flattening the multidimensional feature map into a one-dimensional feature vector.

[0169] In this embodiment, S6 includes the following steps:

[0170] S61. Transfer the text feature vector and image feature vector V l The input layers fed into the multimodal fusion attention network are denoted as V. T and V l ;

[0171] S62. Perform self-attention mechanism calculations on the text feature vector and image feature vector respectively to capture the correlation within the features:

[0172]

[0173] Where Q = W Q V T K=W K V T V=W V V T W represents the query, key, and value of the text feature vector, respectively. Q W K and Wv Let d be a learnable weight matrix. k The dimension of the feature vector;

[0174] S63. Combine the self-attention output AT of the text features with the image feature vector V. I Perform cross-attention calculation to generate fused feature representations:

[0175]

[0176] Among them, Q T =W QT A T K I =W κI V I V I =W VI V1 represents the query for text features and the key and value for image features, respectively; W QT W kI and W VI The weight matrix is ​​a learnable matrix;

[0177] S64. Concatenate the output of the cross-attention calculation with the original text feature vector and image feature vector to generate a preliminary fused feature vector V. fused :

[0178] V fused =[CrossAttention(Q T (K1, V1)‖A T ||V1];

[0179] Where || denotes the vector concatenation operation;

[0180] S65, Initially fuse the feature vector V fused The input is fused into a fully connected layer to generate the final comprehensive feature vector V. composite :

[0181] V composite =W fo ·V fused +b fo ;

[0182] Among them, W fo Let b be the weight matrix of the fully connected layer. fo This is a bias term.

[0183] In this embodiment, S7 includes the following steps:

[0184] S71, Combine the feature vector V compositeThe input is fed into the input layer of the adaptive hierarchical classification model for preliminary feature classification. The adaptive hierarchical classification model includes multiple levels of classifiers, with each level of classifier handling different levels of feature complexity.

[0185] S72. In the primary classifier, a fully connected neural network is used to classify C. composite Perform preliminary classification and output the first-level classification result C1:

[0186]

[0187] in, This is the weight matrix of the primary classifier. C1 is the bias term, and C1 is the probability distribution of the first-level classification.

[0188] S73. The first-level classification result C1 is passed as input to the secondary classifier. The secondary classifier further refines the classification based on the primary classification result and outputs the second-level classification result C2:

[0189]

[0190] in, This is the weight matrix of the secondary classifier. C1 is the bias term, and C2 is the probability distribution of the second-level classification.

[0191] S74. Repeat steps S73, performing classification analysis through multiple levels of classifiers. Each level of classifier performs more refined classification based on the output of the previous level, finally outputting the top-level classification result C. L :

[0192]

[0193] in, This is the weight matrix of the top-level classifier. As a bias term, C L This is the final classification result.

[0194] In this embodiment, S8 includes the following steps:

[0195] S81. Sensitive information is identified in the categorized documents. Keyword detection is performed on the text portions. A predefined sensitive word library is used to match the text feature vectors, generating preliminary sensitive information tags M. key :

[0196]

[0197] Among them, w i M represents the i-th word unit in the text sequence. lcoyFor a predefined sensitive word library, δ is an indicator function, when w i If the word belongs to the sensitive word list, δ = 1; otherwise, δ = 0; μ key and σ key γ represents the mean and standard deviation of words in the sensitive word database, respectively. i This is a weighting parameter used to adjust the impact of different words on sensitivity;

[0198] S82. Based on keyword detection, perform semantic analysis on the text content, use a pre-trained BERT model to perform semantic understanding on the text feature vectors, and generate a semantic representation S. semand Calculate the sensitivity score S of the text. sens :

[0199]

[0200] Among them, S semand For the semantic representation of text, w j b is the learnable weight vector in the semantic analysis model. j For the bias term, λ j The weight coefficient for each semantic feature, μ sem and σ sem These are the mean and standard deviation of the semantic features, respectively.

[0201] S83, Combining keyword detection results M key and semantic analysis score S sens It integrates the results of keyword detection and semantic analysis, and takes into account the interaction between the two, to generate a comprehensive sensitive information tag M. sens :

[0202]

[0203] Where α1 and β1 are the weight parameters for keyword detection and semantic analysis, and γ k λ1 and η are the weight coefficients of the corresponding keywords and semantic features. kl M is the interaction coefficient between keywords and semantic features. key，k and S sens，l These represent the sensitivity scores of the k-th keyword and the first semantic feature, respectively.

[0204] S84, Mark the comprehensive sensitive information M sens The data is input into a security level determination model based on dynamic risk assessment. The model calculates the security level L of the document according to preset risk assessment standards. conf :

[0205]

[0206] Where ζ is the influence factor for sensitive information labeling, θ and σ conf These are the threshold and standard deviation for determining the level of confidentiality, respectively; κ is the weight of contextual information in risk assessment; and ρ is the standard deviation of the security level. r Let R be the weight coefficient of the r-th contextual information feature. contest，r This is a variable containing contextual information about the document.

[0207] Example 1:

[0208] This embodiment verifies the present invention. In March 2024, Company A faced an urgent task: to classify and classify 5,000 internal technical documents to determine their confidentiality levels. These documents contained numerous complex design drawings and detailed technical specifications, involving a range of the company's core technologies and classified information. Previously, similar tasks typically required several weeks of manual work, and due to the complexity of the document content, the accuracy of classification and classification was difficult to guarantee. However, Company A decided to use the deep learning-based automatic classification and classification method for mixed text and graphics documents of this invention to improve work efficiency and accuracy.

[0209] On March 15, 2024, Company A input its first batch of 1,000 mixed text and image technology documents into the system. Each document contained an average of 15 pages of content. The system first separated the text and images, and then performed noise reduction, word segmentation, and word vectorization on the text. In one document, the system detected several key sensitive terms; these terms were successfully segmented and converted into corresponding word vectors.

[0210] Subsequently, the design drawings in the document were processed using this method, including grayscale conversion, noise reduction, and normalization. Using the design drawing with document number "2024A-001" as an example, the system automatically identified the main structural features of a specific device or Company A's network architecture diagram, and converted this information into image feature vectors. During the analysis, the system discovered that certain feature points in the design drawing matched entries in a predefined high-risk feature library, thus providing important reference for subsequent sensitive information identification.

[0211] On March 16, 2024, the system completed feature extraction for all documents and input the extracted text and image features into a multimodal fusion attention network for feature fusion. In document number "2024A-007", the system detected frequent occurrences of process design-related terms in the text and found multiple precision machining marks in the corresponding drawings. Through feature fusion, the system successfully categorized this document as a process technology document and marked it as potentially involving the company's core technologies.

[0212] On March 17, 2024, the system began assessing the confidentiality level of documents. Taking document number "2024A-015" as an example, the system first identified highly sensitive terms such as "XX model," "XX project," "non-public," and specific technical specifications through keyword detection. The system initially assigned a sensitivity score of 85. Simultaneously, the system used the BERT model to conduct a deep semantic analysis of the document, further confirming that it contained sensitive and confidential company information, with a semantic analysis score of 90. Combining the results of keyword detection and semantic analysis, the system determined the document's overall sensitivity score to be 88.

[0213] The system then inputs the comprehensive sensitivity markers into the dynamic risk assessment model to automatically determine the confidentiality level. Based on the model calculations and combined with the document's contextual risk information (document creation time, editor permissions, access records), the system ultimately classifies the document numbered "2024A-015" as "Confidential" and immediately restricts its access permissions, allowing only authorized personnel within the company's knowledge sphere to view it.

[0214] To verify the effectiveness of the method of the present invention, Company A conducted a comparative analysis between the method of the present invention and traditional manual methods. Using actual operational data from March 16th to March 20th, 2024, the system processed 5000 technical documents.

[0215] Classification efficiency comparison: Traditional manual methods take an average of about 4 minutes to classify each document, and it takes a total of 333 hours to process 5,000 documents; while the method of this invention takes an average of 15 seconds to classify each document, and the total time is only 20.8 hours, which is nearly 16 times more efficient.

[0216] Classification accuracy comparison: In 500 randomly selected sample documents, the classification accuracy of traditional manual methods is about 82%, with a lot of classification errors and duplicate classification problems; the classification accuracy of the method of the present invention reaches 96%, which significantly reduces classification errors, especially when dealing with complex mixed text and image content.

[0217] Comparison of classification determination: In the determination of confidentiality level, the accuracy rate of traditional manual methods is 78%, and common problems include unclear classification of confidentiality level and large subjective bias. The method of this invention improves the accuracy rate to 92% through dynamic risk assessment, and can automatically adapt to the risk changes of different types of documents, ensuring the objectivity and consistency of confidentiality level determination.

[0218] Through the application of this embodiment, Company A has significantly improved the efficiency and security of its document management. The high efficiency and accuracy demonstrated by the system in actual operation prove the practical application value of the method of this invention in the automatic classification and confidentiality determination of mixed text and graphics documents in the manufacturing industry. It effectively solves the shortcomings of traditional methods when processing complex documents and provides strong technical support for enterprise information management.

[0219] In the processing of mixed text and image documents, this invention introduces a multimodal fusion attention network to effectively fuse text feature vectors and image feature vectors. Compared with traditional single-modal processing methods, the multimodal fusion mechanism can make full use of the complementary information of text and images, thereby achieving a more comprehensive and accurate analysis and classification of mixed text and image documents. Through the application of an adaptive hierarchical classification model, it can not only quickly locate the approximate category of the document in the primary classification, but also further accurately classify the document category through the refinement processing of the hierarchical classifier.

[0220] The combined application of the adaptive multi-level bidirectional transform model and the dynamic convolutional neural network in this invention enables deep semantic analysis and feature extraction of text and image information. The adaptive multi-level bidirectional transform model can capture the contextual dependencies in text features and achieve a deep understanding of complex semantics through multi-level feature fusion; while the dynamic convolutional neural network can dynamically adjust the convolution kernel and pooling operations according to the complexity of image features, ensuring that key features of image information are fully extracted. By combining deep semantic analysis with adaptive feature extraction, this invention significantly improves the ability to process complex information in mixed text and image documents, resulting in more accurate classification results.

[0221] This invention combines a dynamic risk assessment model with sensitive information identification technology. By detecting keywords and performing semantic analysis on the text and analyzing features on the image, it automatically identifies potentially sensitive information in a document. Based on the identification results, the dynamic risk assessment model automatically determines the confidentiality level of the document. This not only improves the automation of confidentiality level determination but also greatly enhances the accuracy and consistency of the determination results, effectively preventing information leakage.

[0222] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for automatic classification and confidentiality determination of mixed graphic and text documents in manufacturing industry based on deep learning, characterized in that: The following steps are involved: S1. Obtaining a mixed text and image document; S2, preprocessing the text part in the mixed text and image document, wherein the preprocessing includes denoising, word segmentation and word vectorization; S3, using an adaptive multi-layer bidirectional transformation model to extract features from the preprocessed text part and generate a text feature vector; S4, preprocessing the image part in the mixed image and text document, wherein the preprocessing includes grayscale conversion, denoising and normalization; S5, using a dynamic convolutional neural network to extract features from the preprocessed image portion to generate an image feature vector; S6, inputting the text feature vector and the image feature vector into the multimodal fusion attention network for feature fusion to generate a comprehensive feature vector; S7, using an adaptive hierarchical classification model to classify and analyze the comprehensive feature vector, and classify the mixed text and image documents into predefined categories; S8. Perform sensitive information identification on the classified documents, wherein the sensitive information identification includes keyword detection and semantic analysis; S9, using the identified sensitive information, automatically determine the confidentiality level of the document through a confidentiality level determination model based on dynamic risk assessment; S10. Output classification and confidentiality results, and store the results in a document management system.

2. According to the method of automatic classification and confidentiality determination of mixed graphic and text documents in manufacturing industry based on deep learning in claim 1, it is characterized by: The S1 comprises the following steps: S11, obtaining a mixed text-and-graphic document from a manufacturing document management system, wherein the mixed text-and-graphic document contains two types of information: text and image; S12, parsing the acquired mixed text and image document through the file parsing module, extracting the text part T in the mixed text and image document d and image part I d ; S13, for the text part T d and image part I d Format them separately, and the formatted text part is recorded as T f , the formatted image part is recorded as I f ; S14. The formatted text part T f and image part I f The data is input into the preprocessing module of the deep learning model through the data transmission interface. The input process includes encoding the data and packaging it according to the predetermined data structure to generate the image-text mixed document dataset D to be processed. s : D s ={T f ,I f }。 3. According to the method of automatic classification and confidentiality determination of mixed graphic and text documents in manufacturing industry based on deep learning in claim 1, it is characterized by: The S2 comprises the following steps: S21. For the text part T f Perform denoising to remove noise characters, meaningless symbols, and repeated information in the text part to generate pure text data T clean ; S22, the denoised text data T clean Perform word segmentation based on predetermined language rules and dictionary word segmentation to convert text data into clean Split into several word units to generate a text sequence T after word segmentation seg ={w1, w2, ..., w i , ..., w n }, where w i Represents the i-th word unit after segmentation; S23, text sequence T after word segmentation seg Perform word vectorization and use the pre-trained word vector model to transform each word unit w i Convert to the corresponding word vector v i , generate word vector sequence: V T ={v1,v2,...,v i ,...,v n }; Among them, v i Represents word unit w i The vector representation of .

4. According to the method of automatic classification and confidentiality determination of mixed graphic and text documents in manufacturing industry based on deep learning in claim 1, it is characterized by: The S3 comprises the following steps: S31, the text sequence V after word segmentation and word vectorization T Input to the input layer of the adaptive multi-layer bidirectional transformation model, the adaptive multi-layer bidirectional transformation model includes multiple stacked bidirectional transformation layers, and the input process of the text sequence is defined as: in, represents the element-wise addition operation of a vector, Represents the position information vector of time step t, which is used to preserve the order information of the text sequence, V in is the sequence vector input to the bidirectional transformation model; S32. In each bidirectional transformation layer, the text sequence is forward propagated and backward propagated respectively. The forward propagation calculates the feature information from front to back: in, represents the forward hidden state at the tth time step, W f and U f are the weight matrices in forward propagation, b f is the bias term, σ is the activation function; back propagation calculates the feature information from back to front, V in，t Represents the input vector of the text sequence at time t: in, represents the reverse hidden state at the tth time step, W b and U b are the weight matrices in back propagation, b b is the bias term; S33, forward hidden state sequence With the reverse hidden state sequence Splice to generate the spliced ​​hidden state sequence H t : Among them, || represents the concatenation operation of vectors, H t represents the comprehensive feature representation of the t-th time step; S34, adaptively weight the output of each bidirectional transformation layer to generate a global feature representation: Among them, H glabal represents the comprehensive feature representation, L is the number of bidirectional transformation layers, α l is the adaptive weight coefficient of each layer, H i，l represents the hidden state sequence of the lth layer, α l It is obtained through model learning and is used to highlight the importance of information at different levels. The final generated text feature vector is:

5. According to the method of automatic classification and confidentiality determination of mixed graphic and text documents in manufacturing industry based on deep learning in claim 1, it is characterized by: The S4 comprises the following steps: S41, image part I f Perform grayscale processing to convert the color image into a grayscale image: I gray (x,y)=0.299·I r (x,y)+0.587·I g (x,y)+0.114·I b (x,y); Among them, I gray (x, y) represents the grayscale value of the grayscale image at the pixel (x, y), I r (x, y), I g (x, y) and I b (x, y) represent the values ​​of the red, green, and blue channels of the corresponding pixel of the color image respectively; S42, grayscale image I gray Perform denoising to remove the noise interference in the image and obtain the denoised image I denoise ; S43, the denoised image I denoise Normalize the image and scale the grayscale value to the predetermined range [0,1] to obtain the normalized image I norm .

6. According to the method of automatic classification and confidentiality determination of mixed graphic and text documents in manufacturing industry based on deep learning in claim 1, it is characterized by: The S5 comprises the following steps: S51, the preprocessed normalized image I norm Input to the input layer of the dynamic convolutional neural network, the input image feature data is represented as a matrix I input , where I input =I norm ; S52, in the convolution layer, using multiple dynamic convolution kernels to compare the input image data I input Perform convolution operation to generate feature map F conv : Among them, F conv represents the eigenvalue generated by the kth convolution kernel at the pixel point (x, y), W k (i, j) is the weight matrix of the kth convolution kernel, b k is the corresponding bias term, m and n are the sizes of the convolution kernel; S53, feature map F of the convolutional layer output conv Perform dynamic pooling to reduce feature dimensions and optimize the model's anti-interference ability: Among them, F pool (x′, y′, k) represents the maximum value of the kth feature map in the pooling window P after pooling, s is the pooling step size, and (x′, y′) represents the position index after the pooling operation; S54, the pooled feature map F pool The image is input to the subsequent dynamic convolution layer and pooling layer for further feature extraction, and the process of S52 and S53 is repeated until the predetermined feature dimension or number of extraction layers is reached to obtain a high-level feature mapping vector F. high ; S55, flatten the high-level feature map generated after multi-layer convolution and pooling into a feature vector V I , the flattening process includes the following steps: Among them, F high (p, q, k) represents the feature value of the kth feature map at position (p, q) in the high-level feature map, μ p，k and μ q，k Respectively represent the mean of the kth feature map in direction p and direction q, σ r，k and σ q，k They represent the standard deviation of the kth feature map in direction p and direction q respectively, exp represents the natural exponential function, and Flatten represents the operation of flattening a multi-dimensional feature map into a one-dimensional feature vector.

7. According to the method of automatic classification and confidentiality determination of mixed graphic and text documents in manufacturing industry based on deep learning in claim 1, it is characterized by: The S6 comprises the following steps: S61. Text feature vector and image feature vector V I Input to the input layer of the multimodal fusion attention network, denoted as V T and V I ; S62. Perform self-attention calculations on the text feature vector and the image feature vector to capture the internal correlation of the features: Where Q = W Q V T , K=W K V T 、V=W V V T Represent the query, key and value of the text feature vector respectively, W Q , W K and W V is the learnable weight matrix, d k is the dimension of the feature vector; S63, output the self-attention of the text feature A T With the image feature vector V I Perform cross-attention calculations to generate fused feature representations: Among them, Q T =W QT A T , K I =W KI V I 、V I =W VI V I Represents the query of text features and the key and value of image features respectively, W QT , W KI and W VI is a learnable weight matrix; S64, concatenate the output of the cross attention calculation with the original text feature vector and image feature vector to generate a preliminary fusion feature vector V fused : V fused -[CrossAllention(Q T ,K I ,V I ]||A T ||V I ]; Among them, || represents the vector concatenation operation; S65, initially fusion feature vector V fused Input to the fully connected layer for fusion to generate the final comprehensive feature vector V composite : V composite =W fo ·V fused +b fo ; Among them, W fo is the weight matrix of the fully connected layer, b fo is the bias term.

8. According to the method of automatic classification and confidentiality determination of mixed graphic and text documents in manufacturing industry based on deep learning in claim 1, it is characterized by: The S7 comprises the following steps: S71, the comprehensive feature vector V composite Inputting to the input layer of an adaptive hierarchical classification model for preliminary feature classification, the adaptive hierarchical classification model includes multiple levels of classifiers, each level of classifiers processing different levels of feature complexity; S72. In the primary classifier, a fully connected neural network is used to classify V composite Perform preliminary classification and output the first-level classification result C1: in, is the weight matrix of the primary classifier, is the bias term, C1 is the first-level classification probability distribution; S73, passing the first-level classification result C1 as input to the secondary classifier, which further refines the classification according to the primary classification result and outputs the second-level classification result C2: in, is the weight matrix of the secondary classifier, is the bias term, C2 is the second-level classification probability distribution; S74, repeat the step of S73, and perform classification analysis through multiple levels of classifiers in turn. Each level of classifier performs more detailed classification based on the output of the previous level, and finally outputs the top classification result C L : in, is the weight matrix of the top classifier, is the bias term, C L is the final classification result.

9. According to the method of automatic classification and confidentiality determination of mixed graphic and text documents in manufacturing industry based on deep learning in claim 1, it is characterized by: The S8 comprises the following steps: S81. Identify sensitive information on the classified documents, perform keyword detection on the text part, match the text feature vector using the predefined sensitive word library, and generate a preliminary sensitive information tag M. key : Among them, w i represents the i-th word unit in the text sequence, M key is a predefined sensitive word library, δ is an indicator function, when w i When it belongs to the sensitive word library, δ=1, otherwise δ=0; μ key and σ key They represent the mean and standard deviation of the words in the sensitive word library respectively, and γ1 is a weight parameter used to adjust the impact of different words on sensitivity; S82. Based on keyword detection, perform semantic analysis on the text content, use the pre-trained BERT model to understand the text feature vector semantically, and generate semantic representations semantio , calculate the sensitivity score S of the text sens : Among them, S semantio is the semantic representation of the text, w j is the learnable weight vector in the semantic analysis model, b j is the bias term, λ j is the weight coefficient of each semantic feature, μ sem and σ sem are the mean and standard deviation of the semantic features, respectively; S83, combined with keyword detection result M key and semantic analysis score S sens , combines the results of keyword detection and semantic analysis, and considers the interaction between the two to generate a comprehensive sensitive information marker M sens : Among them, α1 and β1 are the weight parameters of keyword detection and semantic analysis, γ k and λ l is the weight coefficient of the corresponding keyword and semantic feature, η kl is the interaction coefficient between keywords and semantic features, M key，k and S sens，l They represent the sensitivity score of the kth keyword and the sensitivity score of the lth semantic feature respectively; S84, mark the comprehensive sensitive information with M sens The confidentiality level determination model based on dynamic risk assessment is input, and the confidentiality level determination model calculates the confidentiality level L of the document according to the preset risk assessment criteria. conf : Among them, ζ is the impact factor of sensitive information labeling, θ and σ conf are the threshold and standard deviation of confidentiality level determination, κ is the weight of context information in risk assessment, and ρ r is the weight coefficient of the rth context information feature, R context，r The document's context information variable.

Images