Visual text feature fused image sensitive information automatic covering method

Abstract

The invention provides an image sensitive information automatic covering method based on visual text feature fusion, and belongs to the technical field of computer vision and artificial intelligence. The method comprises the following steps: acquiring image data, and preprocessing to obtain a preprocessed character image; performing visual feature extraction and text feature extraction on the character image, fusing visual features and text features to obtain cross-modal fusion features, and further obtaining unified feature representation after fusion; the multi-task identification module carries out AI identification and marking on the unified feature representation after fusion; the manual correction module is used for correcting marked handwritten characters, sensitive contents and sensitive content prefix word subsequent areas; and integrating AI identification and manual correction results, performing black covering processing and image synthesis processing, and outputting a final synthesized image. According to the invention, automatic identification and covering of the image sensitive information are realized, and the processing efficiency and accuracy are improved. The method provides a human-assisted correction function, ensures the processing quality, and is suitable for various business scenes.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and artificial intelligence, and particularly relates to an automatic image sensitive information covering method for visual text feature fusion. Background Art

[0002] In today's environment of rapid development of digital businesses, picture data containing sensitive information in business systems is increasingly widely used. These sensitive information cover key contents such as personal identity information, financial data, and business secrets. During the business process and when docking with third-party systems, covering sensitive or private information in pictures is an important link to ensure information security.

[0003] Currently, the covering process of image sensitive information in the industry mainly adopts manual operation methods. This method requires operators to manually identify sensitive areas in the picture and use image processing tools for covering. However, this manual processing method has many drawbacks. On the one hand, manual operation is inefficient. With the growth of the business volume, covering a large amount of picture data requires a large amount of human and time costs, seriously affecting the business processing efficiency. On the other hand, the accuracy of manual identification and covering is difficult to guarantee, and it is easy to miss sensitive information or miscover it, resulting in an increased risk of information leakage and unable to meet the increasingly strict information security compliance requirements. At the same time, manual operation lacks a standardized process, and there are differences in the processing methods and standards of different operators, further reducing the quality and reliability of the covering process.

[0004] In summary, the existing image sensitive information covering solutions based on manual operation can no longer meet the current business development's requirements for information security and processing efficiency. There is an urgent need for an automated image sensitive information covering solution to improve the efficiency, accuracy, and consistency of the covering process and ensure the security of sensitive information in business systems. Summary of the Invention

[0005] The purpose of the present invention is to provide an automatic image sensitive information covering method for visual text feature fusion, which can automatically identify and cover sensitive information in images, improving the efficiency and accuracy of image sensitive information processing. To solve the technical problems in the prior art that image covering solutions mainly rely on manual operation, resulting in low efficiency and insufficient accuracy.

[0006] To solve the above technical problems, the specific technical solution of the present invention is as follows:

[0007] An automatic image sensitive information covering method for visual text feature fusion, the method includes the following steps:

[0008] Step S1: Obtain image data, and preprocess the image data to obtain a character image after preprocessing;

[0009] Step S2: Extract visual features and text features from the character image, fuse the visual features and text features to obtain cross-modal fusion features, perform feature transformation and enhancement on the cross-modal fusion features to obtain a unified feature representation after fusion;

[0010] Step S3: The multi-task recognition module performs AI recognition and marking on the unified feature representation after fusion. The AI recognition includes handwritten character recognition, sensitive content recognition, and prefix recognition;

[0011] Step S4: The manual correction module corrects the subsequent areas of the marked handwritten characters, sensitive content, and sensitive content prefix words;

[0012] Step S5: Integrate the results of AI recognition and manual correction, perform black covering processing, perform image synthesis processing after completing the black covering processing operation, generate and save the final synthesized image, and output the final synthesized image.

[0013] Further, step S1 includes the following steps:

[0014] Step S11: Read the image data through the URL address, and the obtained image data is the original RGB image;

[0015] Step S12: Perform grayscale processing on the image data to convert the color image into a grayscale image;

[0016] Step S13: Perform edge detection on the grayscale image;

[0017] Step S14: Perform secondary detection on the image data to identify the positions of individual characters and obtain a set of character position coordinates;

[0018] Step S15: Perform region segmentation on the image data according to the set of character position coordinates to obtain the segmented character image.

[0019] Further, step S2 includes the following steps:

[0020] Step S21: The visual feature extraction module extracts visual features from the character image to obtain visual features;

[0021] Step S22: The text feature extraction module extracts text features from the character image to obtain text features;

[0022] Step S23: The multi-modal feature fusion module fuses the extracted visual features and text features to obtain cross-modal fusion features, performs feature transformation and enhancement on the cross-modal fusion features to obtain a unified feature representation after fusion.

[0023] Further, step S23 includes the following steps:

[0024] Step S231: Align the visual features and text features, map the visual features and text features to the same feature space, obtain the aligned visual features and the aligned text features, and preserve the feature semantic information of the visual modality and the text modality;

[0025] Step S232: The multi-modal feature fusion module calculates the attention weights between the aligned visual features and the aligned text features through a cross-modal attention mechanism, realizes the dynamic fusion of the two modal features, and obtains cross-modal fusion features;

[0026] Step S233: Use a feed-forward network to perform feature transformation and enhancement on the cross-modal fusion features to obtain a unified feature representation after fusion.

[0027] Furthermore, step S3 includes the following steps:

[0028] Step S31: The multi-task recognition module determines whether a character is a handwritten character through the handwritten character recognition task head and marks the handwritten characters;

[0029] Step S32: The multi-task recognition module judges the sensitive content through the sensitive content recognition task head and marks the sensitive content;

[0030] Step S33: The multi-task recognition module recognizes the sensitive content prefix word through the prefix recognition task head and marks the subsequent area of the sensitive content prefix word;

[0031] Step S34: Construct the loss function of the multi-task recognition module;

[0032] Step S35: Train the multi-task recognition module.

[0033] Furthermore, step S5 includes the following steps:

[0034] Step S51: Integrate the AI recognition and manual correction results and perform black covering processing;

[0035] Step S52: After completing the black covering processing operation, perform image synthesis processing to generate the final output image and output the processed image.

[0036] Compared with the prior art, the present invention has the following beneficial technical effects:

[0037] 1) The method proposed by the present invention integrates key technologies such as computer vision, multi-modal semantic understanding, and intelligent image processing, and automatically completes steps such as image preprocessing, character region recognition, sensitive information detection, and region masking. Through the cross-modal feature extraction mechanism, the image and language information are effectively combined, enhancing the discrimination ability for complex sensitive regions; by embedding an artificial auxiliary correction module, the processing quality based on automated processing is ensured, guaranteeing the integrity and accuracy of the masking results. Compared with the traditional manual processing method, the present invention can not only significantly improve the processing efficiency of image sensitive information, but also enhance the consistency and controllability of the results, and is suitable for a variety of complex business scenarios, including financial services, medical imaging, government information management, public opinion data release, etc. In addition, the technology of the present invention can flexibly configure sensitive discrimination strategies and region masking schemes, and has good scalability and engineering application value.

[0038] 2) The present invention realizes the automatic recognition and masking of image sensitive information, improving the processing efficiency and accuracy. The present invention provides an artificial auxiliary correction function to ensure the processing quality and is applicable to a variety of business scenarios.

[0039] 3) The present invention has strong innovation and practicality in the field of cross-modal intelligent recognition and information security, and has clear engineering value and economic benefits. Its technical route is scientific and the processing process is clear. It is expected that after the technology is implemented in related fields, it will play a positive role in improving the information security processing ability, reducing the risk of data leakage, and promoting the development of intelligent content review technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 It is a schematic flow chart of the automatic masking method for image sensitive information with visual text feature fusion of the present invention.

[0042] Figure 2 It is a schematic architecture diagram of the automatic masking method for image sensitive information with visual text feature fusion of the present invention.

[0043] Figure 3 It is a schematic architecture diagram of the AI recognition layer of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] The present invention proposes an automatic covering method for image sensitive information by fusing visual text features, as Figures 1 - 3 shown, the method includes the following steps:

[0046] Step S1: Obtain image data and preprocess the image data to obtain a character image after preprocessing.

[0047] Specifically, step S1 includes the following steps:

[0048] Step S11: Read the image data through the URL address, and the obtained image data is the original RGB image.

[0049] Step S12: Perform grayscale processing on the image data to convert the color image into a grayscale image.

[0050] The grayscale processing algorithm is expressed as follows:

[0051]

[0052] Among them, represents the grayscale value, which represents the brightness degree of the corresponding pixel point in the image under the grayscale image, and the value range is usually between 0 and 255, where 0 represents black and 255 represents white; , , respectively represent the red, green, and blue component values of the same pixel point in the color image, and the value range is also 0 to 255.

[0053] Step S13: Perform edge detection on the grayscale image.

[0054] Edge detection is used to locate the characters in the picture because the edges of the characters often show sudden changes in brightness; edge detection is also used to locate the bounding boxes of individual characters. For example, by clustering the pixels with high G values, the closed contours of each character are determined.

[0055] Edge detection is performed through the following edge detection algorithm:

[0056]

[0057]

[0058] Among them, represents the gradient magnitude of a certain pixel point in the image, reflecting the degree of sharp change in the gray value of this pixel point. The larger the value, the more obvious the gray change near this pixel point, and the more likely it is to be an edge point of the image. represents the gradient value of the pixel point in the horizontal direction ( axis direction), used to measure the gray change rate of the image in the horizontal direction. represents the gradient value of the pixel point in the vertical direction ( axis direction), used to measure the gray change rate of the image in the vertical direction. represents the gradient direction of the pixel point, that is, the direction in which the gray value changes fastest.

[0059] In the image, the gray value at the edge will undergo an obvious mutation, and the gradient can well describe this change. By calculating the gradient values in the horizontal and vertical directions and , the gradient magnitude is obtained using the Pythagorean theorem, thereby quantifying the intensity of the gray change; then the gradient direction is calculated through the arctangent function to clarify the trend of the gray change. In this way, based on the gradient magnitude and direction, the pixel points in the image that may have edges can be identified.

[0060] Step S14: Perform secondary detection on the image data to identify the positions of individual characters and obtain the set of character position coordinates.

[0061] On the basis of the edge detection to initially locate the text area, the secondary detection further accurately locates the bounding box of an individual character. Specifically, connected component labeling is performed on the high-gradient magnitude area, and the pixel points with continuous spatial positions are merged to form candidate character areas; noise is filtered according to the geometric features (aspect ratio, area, etc.) of the characters; the bounding box of the tilted character is corrected using the gradient direction θ (such as rotating the bounding box to the θ mean direction).

[0062] Specifically, on the basis of completing the edge detection, a method combining improved connected component analysis and morphological processing is used to achieve character positioning. The method combining improved connected component analysis and morphological processing includes the following steps:

[0063] Step S141: Perform adaptive threshold segmentation on the gradient magnitude to generate a binary image :

[0064]

[0065] where represents the threshold coefficient, generally takes a value of 1.2, represents the global gradient mean of the image. Indicates the image coordinates The gradient magnitude at this point can effectively eliminate the interference of uneven illumination through adaptive threshold segmentation.

[0066] Step S142: Use the 8-neighborhood scanning algorithm to perform connected component labeling on the binary image to generate a set of candidate character regions . And introduce the area - aspect ratio double - constraint condition to filter out non - character regions and obtain the retained regions.

[0067] The area - aspect ratio double - constraint condition is as follows:

[0068] 15 < < 3000 and

[0069] where, represents the candidate character region number, is the area of the candidate character region, is the aspect ratio of the bounding rectangle of the candidate character region, which can effectively exclude noise and large - area non - text regions.

[0070] Step S143: Perform iterative closing operations on the retained regions to solve the problem of character breakage and obtain the optimized connected components. The iterative closing operation is calculated as follows:

[0071]

[0072] where, represents the optimized connected region, represents the structuring element, and the structuring element adopts a 3×3 elliptical kernel, represents the dilation operation, represents the erosion operation. This process smooths the edges while maintaining the character shape.

[0073] Step S144: Extract the minimum bounding rotated rectangle from the optimized connected components to generate a set of character position coordinates.

[0074] The set of character position coordinates is represented as follows:

[0075]

[0076] where, represents the set of character position coordinates, is the rectangle rotation angle (-15° ≤ ≤ 15°), and respectively represent the minimum horizontal and vertical coordinates (the top - left vertex) of the bounding rectangle of the character region, and respectively represent the maximum horizontal and vertical coordinates (the bottom - right vertex), and these parameters together define the exact position and spatial orientation of the character in the image.

[0077] Step S15: Perform region segmentation on the image data according to the set of character position coordinates to obtain the segmented character images.

[0078] Crop the original RGB image according to the set of character position coordinates to obtain several character images. Subsequently, standardize each character image, and uniformly scale each character region to 64×64 pixels. The standardization uses the bicubic interpolation algorithm to preserve the character structure features and eliminate the influence of scale differences on subsequent recognition. The standardization is achieved as follows:

[0079]

[0080] where is the original RGB image. is the th standardized character image, indicating that the extracted sub - image is scaled to a size of 64×64 pixels.

[0081] Step S2: Extract visual features and text features from the character images, fuse the visual features and text features to obtain cross - modal fusion features, and perform feature transformation and enhancement on the cross - modal fusion features to obtain the unified feature representation after fusion.

[0082] Specifically, Step S2 includes the following steps:

[0083] Step S21: The visual feature extraction module extracts visual features from the character images to obtain visual features.

[0084] The visual feature extraction module adopts a hierarchical structure based on Vision Transformer, and through innovative architecture design and parameter configuration, realizes the efficient extraction and representation of image features.

[0085] In the input layer, the visual feature extraction module divides the input character image into image patches with a fixed size of 4×4 pixels. This division method reduces the complexity of model processing while retaining the detailed information of the image. Compared with the traditional division of larger - sized image patches, the 4×4 - pixel image patches can capture more subtle textures and local structure features in the image, laying a foundation for subsequent feature extraction.

[0086] The feature encoding layer performs dual operations on the divided image patches. On the one hand, a position encoding mechanism is introduced to assign position information to each image patch. Since the Transformer architecture itself does not have the ability to capture the sequence order, the position encoding enables the model to perceive the relative position relationship of the image patches in the original image by adding specific vectors, avoiding the loss of spatial information due to shuffling the order of the image patches. On the other hand, a linear mapping is performed on the image patches to map them to a 192-dimensional encoding space. The 192-dimensional encoding dimension has been verified through a large number of experiments. This dimension can not only ensure the richness of features and effectively represent the feature information of the image patches, but also achieve a good balance between computational resources and model performance.

[0087] The attention layer is the core part of the visual feature extraction module. Through the structural design of 24 attention heads and 24 layers, it deeply captures the local and global features of the image. Each attention head models the relationship between image patches from different perspectives. The 24 attention heads work in parallel, capable of capturing the feature interaction information of the image from multiple dimensions and enhancing the model's ability to understand complex image content. The 24-layer deep structure enables the model to continuously iterate and optimize the feature representation. As the number of layers increases, the model gradually abstracts more semantic global features from local features. The self-attention mechanism dynamically focuses on important regions in the image by calculating the attention weights between image patches, ensuring that both the minute details and the overall structure in the image can be fully attended to and extracted.

[0088] The output layer then integrates the features after multiple layers of processing and outputs visual features, which are high-dimensional visual feature representations. These high-dimensional visual features fuse the local details and global semantic information of the image. Their rich feature expression ability provides strong information support for downstream tasks, helping to improve the accuracy and reliability of the tasks.

[0089] Step S22: The text feature extraction module extracts text features from the character image to obtain text features.

[0090] The text feature extraction module adopts an enhanced pre-trained language model architecture. Through delicate structural design and parameter configuration, it realizes the efficient extraction and deep representation of text semantic features.

[0091] In the input layer, first, based on a pre-trained OCR (Optical Character Recognition) model, optical character recognition is performed on the character image to obtain the original text. Then, the original text is tokenized, splitting the continuous text sequence into individual lexical units. Subsequently, the tokenized words are encoded and mapped into numerical vectors that the model can process. The vocabulary size is set to 30,000, covering a large number of common words and professional terms in specific fields, ensuring that words in various texts can find corresponding encodings in the vocabulary, providing a basis for subsequent processing.

[0092] The attention layer adopts the self-attention mechanism with relative position encoding, which is the core innovation point of the text feature extraction module. Compared with the traditional absolute position encoding, relative position encoding can better capture the relative position relationship between words in the text, enabling the model to still accurately understand the dependency relationship between words when dealing with long texts. 16 attention heads work in parallel to model the lexical relationships in the text sequence from different perspectives. Each attention head focuses on different semantic information, and through the collaboration of multiple heads, the semantic associations and logical structures in the text are comprehensively captured. The self-attention mechanism dynamically focuses on the key information in the text by calculating the attention weights between words, and both long-distance dependent semantic connections and local syntactic structures can be effectively identified and extracted.

[0093] The feed-forward network layer receives the feature vectors output by the attention layer and performs a non-linear transformation on them. The hidden layer dimension is set to 1024. This dimension not only ensures that the model has sufficient capacity to learn complex semantic features but also avoids the problems of excessive computational burden and overfitting caused by too high a dimension. Through the action of the multi-layer perceptron and activation function, the feed-forward network layer further abstracts and refines the features, converts low-level semantic information into high-level features with stronger semantic representation ability, and enhances the semantic expression ability of text features.

[0094] The output layer integrates the features processed by the feed-forward network layer to generate text features, which are semantic feature representations. These semantic features integrate the syntactic structure, semantic information, and context of the text and can accurately express the connotation of the text. The maximum sequence length is set to 512 to ensure that the module can handle the text lengths in most actual application scenarios. Whether it is the fast feature extraction of short texts or the in-depth semantic analysis of long texts, high-quality semantic features can be stably output.

[0095] Step S23: The multi-modal feature fusion module fuses the extracted visual features and text features to obtain cross-modal fusion features, and performs feature transformation and enhancement on the cross-modal fusion features to obtain the unified feature representation after fusion.

[0096] The multi-modal feature fusion module uses the cross-modal attention mechanism to fuse the feature information of two different modalities, visual and text.

[0097] Step S231: Align the visual features and text features, map the visual features and text features to the same feature space to obtain the aligned visual features and the aligned text features, and maintain the feature semantic information of the visual modality and the text modality.

[0098] Feature alignment is a fundamental step in multimodal feature fusion. Since visual features and text features come from different data modalities, there are significant differences in their feature spaces. To achieve effective fusion of the two modal features, it is necessary to map visual and text features to the same feature space. This process not only ensures the consistency of feature dimensions but, more importantly, preserves the semantic information of each modal feature. Through feature alignment, it can be ensured that in the same feature space, the semantic information represented by visual and text features can correspond to each other, laying a foundation for subsequent cross-modal attention calculation.

[0099] The visual feature is represented as , and the text feature is represented as . After linear projection, they are mapped into the same-dimensional space to obtain the aligned visual feature and the aligned text feature . The specific expressions are as follows:

[0100]

[0101]

[0102] Among them, represents the weight matrix of the visual feature linear transformation, represents the weight matrix of the text feature linear transformation, represents the bias term of the visual feature, represents the bias term of the text feature.

[0103] Step S232: The multimodal feature fusion module calculates the attention weights between the aligned visual feature and the aligned text feature through a cross-modal attention mechanism to achieve dynamic fusion of the two modal features and obtain cross-modal fusion features.

[0104] The multimodal feature fusion module adaptively assigns weights to visual features and text features according to different input data. Cross-modal attention fusion is achieved through the following formula:

[0105]

[0106]

[0107]

[0108]

[0109] Among them, represents the cross-modal fusion feature, represents the context feature vector, represents the context adaptation factor, Represents the importance weight of the visual modality, Represents the importance weight of the text modality, Represents the aligned visual features, Represents the aligned text features, Represents the cross-modal interaction features, Represents the coefficient of the contribution degree of the cross-modal interaction features in the fusion result, which can be dynamically adjusted according to the context information; Represents the Sigmoid function, Represents for generating The matrix of Represents for generating The bias of Represents concatenating the context feature vector with the aligned visual features, Represents for generating The matrix of Represents for generating The bias of Represents concatenating the context feature vector with the aligned text features, Represents the tanh function, Represents for generating The matrix of Represents for generating The bias of.

[0110] Context feature vector Can contain global information from visual and text modalities as well as information related to the current task. The tanh function restricts the output value within [-1, 1] Between, enabling the context adaptation factor To dynamically adjust the overall fusion intensity according to the context information.

[0111] And Can be regarded as the weight coefficients of visual and text features in the fusion process. By concatenating the context feature vector With the corresponding modal features, and through linear transformation and Sigmoid activation, the importance of different modalities can be dynamically adjusted according to the context information.

[0112] Cross-modal interaction features Capture the interaction information between visual and text features. In the fusion process, considering the cross-modal interaction features can enable the model to better understand the correlation between the two modalities, such as the correspondence between objects in an image and text descriptions. The coefficient of the contribution degree of the cross-modal interaction features in the fusion result Can be dynamically adjusted according to the context information.

[0113] Step S233: Use a feed - forward network to perform feature transformation and enhancement on the cross - modal fusion features, obtaining a unified feature representation after fusion.

[0114] After completing cross - modal attention fusion, use a feed - forward network to further transform and enhance the cross - modal fusion features. The feed - forward network performs non - linear transformations on the input features through multiple fully - connected layers and activation functions, capable of uncovering more complex relationships between features. After being processed by the feed - forward network, a unified feature representation after fusion is finally obtained. This unified feature representation synthesizes information from both visual and text modalities, and takes into account context information and cross - modal interaction features, having stronger expressive power and robustness.

[0115] Step S3: The multi - task recognition module performs AI recognition and marking on the unified feature representation after fusion. The AI recognition includes handwritten character recognition, sensitive content recognition, and prefix recognition.

[0116] Specifically, design specialized task heads for different recognition tasks. Step S3 includes the following steps:

[0117] Step S31: The multi - task recognition module uses the handwritten character recognition task head to determine whether a character is handwritten and mark the handwritten characters.

[0118] The input of the handwritten character recognition task head is the unified feature representation after fusion. The unified feature representation after fusion integrates visual features and text features, containing both visual information such as the stroke morphology and structural layout of handwritten images, and also fusing prior knowledge at the text semantic level.

[0119] The middle layer of the handwritten character recognition task head adopts a multi - layer perceptron (MLP) architecture to build the core processing channel for feature transformation and pattern recognition. The multi - layer perceptron is stacked by multiple fully - connected layers. Each layer transforms the input features through a non - linear activation function, gradually mapping the original fused features to a higher - level semantic space. In the handwritten character recognition scenario, the multi - layer perceptron can effectively extract key features of handwritten characters, such as the turning of strokes, the morphology of connected strokes, and the contour of characters, through progressive feature abstraction. As the number of network layers increases, the model can learn more complex and discriminative feature representations, thus enhancing the ability to distinguish handwritten characters.

[0120] The output of the handwritten character recognition task head is based on the features processed by the multi - layer perceptron, and outputs the recognition probability of the handwritten characters.

[0121] To optimize the performance of the handwritten character recognition task head, its loss function uses cross - entropy. The cross - entropy loss function can effectively measure the difference between the model's predicted probability distribution and the true label distribution, and its formula is:

[0122]

[0123] Among them, represents the loss of handwritten character recognition, is the total number of characters in the current sample, is the true handwritten label of the p i (hw) ∈[0,1] th character (0 = non-handwritten, 1 = handwritten), is the probability that the model predicts the

[0124] th character as a handwritten character.

[0125] The input of the sensitive content recognition task head is the fused unified feature representation, which integrates multi-source information such as text semantics, image vision, and audio spectrum, and realizes deep interaction of cross-modal information through feature fusion technology.

[0126] The middle layer of the sensitive content recognition task head adopts an attention pooling layer, which can adaptively assign weights to different elements in the fused features based on the attention mechanism, focus on the key features related to sensitive content, and suppress the interference of redundant information.

[0127] The output of the sensitive content recognition task head generates a multi-label classification result, which is represented in the form of a probability vector. Each dimension corresponds to a predefined sensitive content category (such as violence, porn, politically sensitive, etc.), and the value range is [0, 1], indicating the probability that the input content belongs to this category.

[0128] To ensure classification accuracy, the sensitive content recognition task head adopts the binary cross-entropy loss function for multi-label classification, which is expressed as follows:

[0129]

[0130] Among them, represents the loss of sensitive content recognition, is the number of sensitive content categories, is the true label of sensitive classification, p ij (sc) ∈[0,1] is the probability that the th character predicted by the model belongs to the th class of sensitive content.

[0131] Step S33: The multi-task recognition module identifies specific sensitive content prefix words through the prefix recognition task head and marks the subsequent area of the sensitive content prefix words.

[0132] The input of the prefix recognition task head is the fused unified feature representation.

[0133] The middle layer of the prefix recognition task head adopts a sequence labeling layer. Based on sequence modeling techniques in deep learning (such as BiLSTM, Transformer encoder, etc.), this layer makes per-position predictions on the input fused feature sequence, and assigns corresponding labels (such as "prefix start", "prefix inside", "non-prefix", etc.) to each position. During the training process, the model learns to capture prefix-related syntactic patterns and semantic clues, such as the morphological features of specific lexical prefixes, context dependencies, etc., and then accurately identifies the specific positions of prefixes in the sequence, realizing fine-grained annotation of prefix information in complex texts.

[0134] The output of the prefix recognition task head generates prefix position information, presented in the form of a label sequence. Each label corresponds to a position in the input sequence. By parsing this sequence, the start and end positions of the target prefix in the original data can be directly obtained. This output design can flexibly adapt to prefixes of different lengths and forms. Whether it is a short-word prefix or a long-sentence prefix, accurate positioning can be achieved to meet diverse application requirements.

[0135] To improve the accuracy and robustness of the model in prefix recognition, the prefix recognition task head adopts a loss function that combines FocalLoss and position weighting. As a loss function that dynamically adjusts weights, Focal Loss can effectively alleviate the problem of class imbalance. When dealing with the prefix recognition task, it assigns higher weights to the prefix position labels of minority classes, enhancing the model's learning ability for prefix features. At the same time, a position weighting mechanism is introduced. Differentiated weights are set according to the different positions of the prefix in the sequence (such as the beginning, middle, and end of the sentence), focusing on the prediction accuracy of the prefix boundary positions, and further optimizing the model's ability to capture prefix position information. By combining the two, the loss function can guide the model to more accurately locate prefixes during training, improving the overall recognition performance.

[0136] Calculate the position weights, which are used to emphasize the importance of characters at different positions in the prefix recognition task.

[0137] The position weights are calculated as follows:

[0138]

[0139] where represents the position weight of the i-th character; represents the scaling factor; represents the distance from the current position to the start position of the prefix. The position weight represents the importance of a string at different positions. The earlier the position, the more important it is, and the greater the possibility of being a prefix.

[0140] The prefix recognition loss function is expressed as follows:

[0141]

[0142] Wherein, represents the prefix recognition loss function, represents the position weight factor, is the label indicating whether the -th character is a prefix, p i (pr) ∈[0,1] is the probability that the model predicts the -th character as a prefix, represents the focusing parameter (default value is 2).

[0143] Step S34: Construction of the loss function of the multi-task recognition module.

[0144] The loss function of the multi-task recognition module combines the handwritten character recognition loss function, the sensitive content recognition loss function, and the prefix recognition loss function. By combining the three loss functions, it balances the difficulty of the three tasks by itself, prevents a single task from dominating the training, and speeds up the convergence rate.

[0145] The multi-task recognition loss function is expressed as follows:

[0146]

[0147] Wherein, is the time step of the current training, represents the multi-task recognition loss at the time step, , , respectively represent the weight coefficients of the handwritten character recognition task, the sensitive content recognition task, and the prefix recognition task at the -th time step, and the initial setting is . , , respectively represent the losses of the handwritten character recognition task, the sensitive content recognition task, and the prefix recognition task at the -th time step.

[0148] The dynamic weight mechanism is used to update the weight system of each task. The dynamic weight mechanism is expressed as follows:

[0149]

[0150]

[0151]

[0152] Among them, 、 、 respectively represent the basic weights of the handwritten recognition task, sensitive content recognition task, and prefix recognition task; 、 、 respectively represent the balance factors of the handwritten recognition task, sensitive content recognition task, and prefix recognition task; 、 、 respectively represent the losses of the handwritten recognition task, sensitive content recognition task, and prefix recognition task at the 0th time step; represents the multi-task recognition loss at the 0th time step.

[0153] The calculation formula for the balance factor is as follows:

[0154]

[0155]

[0156]

[0157] Among them, 、 、 respectively represent the basic balance coefficients (default value 0.5) of the handwritten recognition task, sensitive content recognition task, and prefix recognition task; 、 、 respectively represent the balance coefficient scaling factors of the handwritten recognition task, sensitive content recognition task, and prefix recognition task; 、 、 respectively represent the loss change rates of the handwritten recognition task, sensitive content recognition task, and prefix recognition task; 、 、 respectively represent the sensitivity factors of the handwritten recognition task, sensitive content recognition task, and prefix recognition task; represents the relevance factor of task to task where represents the index of different tasks (task 1 = handwritten recognition, task 2 = sensitive content recognition, task 3 = prefix recognition).

[0158] The loss change rates of the handwritten recognition task, sensitive content recognition task, and prefix recognition task are calculated as follows:

[0159]

[0160]

[0161]

[0162] 、 、 respectively represent the losses of the handwritten character recognition task, sensitive content recognition task, and prefix recognition task at the th time step.

[0163] The sensitivity factors of the handwritten character recognition task, sensitive content recognition task, and prefix recognition task are calculated as follows:

[0164]

[0165]

[0166]

[0167] where represents the sigmoid activation function, 、 、 、 、 、 are learnable parameters; 、 、 respectively represent the variances of the losses of the handwritten character recognition task, sensitive content recognition task, and prefix recognition task.

[0168] The task relevance factor is calculated as follows:

[0169]

[0170]

[0171] where 、 are the gradients of different tasks, which are the update directions obtained by backpropagation and are a tensor, is the correlation weight (default is 0.2), represents the task gradient and the cosine similarity between them; represents the L2 norm of the gradient of task and represents the L2 norm of the gradient of task 。

[0172] Step S35: Train the multi-task recognition module.

[0173] Step S351: Set the optimization objective.

[0174]

[0175] Among them, represents the optimal parameter solution; represents the parameter value when the objective function obtains the minimum value, represents taking as the multi-task recognition loss calculated by the model with parameters.

[0176] Step S352: Calculate the gradient and update the model parameters.

[0177]

[0178] Among them, represents the parameter at the (t + 1)-th time step, represents the parameter at the t-th time step, represents the learning rate, represents the number of iterations; represents the gradient of the total loss function with respect to the model parameter θ at the current iteration step t.

[0179] Step S353: Dynamic weight adjustment. By real-time quantifying the loss convergence speed and gradient correlation of each task, adaptively balance the weight distribution of multi-task learning, thereby alleviating task conflicts and improving the overall performance of the model.

[0180] Step S354: Gradient equalization. By normalizing and weighted adjusting the gradients of different tasks, ensure that the optimization directions of each task in multi-task learning are coordinated and consistent, and avoid a single task dominating the parameter update.

[0181]

[0182] Among them, represents the gradients of each task, represents the median operation, represents task the square of the L2 norm of the gradient, represents task the square of the L2 norm of the gradient.

[0183] Step S4: The manual correction module corrects the marked handwritten characters, sensitive content, and the subsequent areas of specific sensitive content prefix words.

[0184] Through the graphical interface design, the marked handwritten characters, sensitive content, and subsequent areas of specific sensitive content prefixes are displayed on the graphical interface for easy manual correction.

[0185] The graphical interface realizes the accurate positioning and synchronous covering of sensitive areas in images under different display sizes, different resolutions, or cross-platform application scenarios through a coordinate mapping algorithm. Based on the standard image coordinate system, combined with normalization processing and mapping matrix operations, this algorithm can accurately map the sensitive information areas identified in the original image to the target display area with high precision.

[0186] The graphical interface also provides a region editing function, which allows users to manually add, delete, adjust the position and size of the covered area after sensitive information detection, ensuring the controllability and flexibility of the system generation results. The region editing function includes but is not limited to the following operations:

[0187] (1) Adding a covered area: Users can customize and add a new rectangular covered area on the image by clicking and dragging. The system automatically incorporates the coordinates of the newly added area into the coordinate mapping algorithm for unified management.

[0188] (2) Adjusting the covered area: Users can select an existing covered area and adjust its size by dragging the edge nodes, or drag the entire area to change its position. The system real-time updates the region coordinate data and automatically synchronizes the covering rendering layer.

[0189] (3) Deleting the covered area: Users can click on an existing covered area and perform a delete operation. The system removes the area from the covered area list and cancels the visual covering process for it.

[0190] In addition, to avoid misoperations, the present invention also provides an "editing confirmation mechanism", that is, all editing operations take effect only after the user clicks the "Save" button. The changes in the area before and after editing will be visually marked with differences for easy comparison and confirmation by the user. The region editing function can be implemented through various interaction methods such as a mouse and a touch screen, and is applicable to different device platforms such as PC terminals, mobile terminals, and embedded terminals, effectively improving the operability and user-friendliness of the system in practical applications.

[0191] Step S5: Integrate the AI recognition and manual correction results, perform black covering processing, and perform image synthesis processing after the black covering processing operation is completed to generate and save the final synthesized image, and output the final synthesized image.

[0192] Step S51: Integrate the AI recognition and manual correction results, and perform black covering processing.

[0193] The black covering process aims to perform configurable image masking on the identified sensitive information areas to achieve controllable hiding of sensitive information. The black covering process can adapt to different types of sensitive area features and support multiple covering methods, including but not limited to blurring, pixelation (mosaic), black block filling, text replacement covering, etc. The specific steps are as follows:

[0194] Step S511: Covering type selection. Select the covering type according to the configuration parameters or user settings. If it is set to blurring, a Gaussian blur kernel is used to perform a convolution operation on the sensitive area; if it is set to pixelation, the area image is downsampled and then upsampled to form a blocky pixel effect.

[0195] Step S512: Area covering process. Perform covering processing on the image content within each sensitive area coordinate. The form of the processing function is as follows:

[0196] Blurring: Apply a two-dimensional Gaussian convolution function to the image area R .

[0197] Pixelation: Perform average block resampling on the area R with a step size of s.

[0198] Black block filling: Set all pixel values in the area R to (0, 0, 0).

[0199] Text covering: Use a unified identification text (such as "covered") to be drawn at the center of the original area, and the font color has a distinct contrast with the background.

[0200] Step S52: After completing the black covering process operation, perform image composition processing to generate the final output image and output the processed image.

[0201] After completing the black covering process operation, perform image composition processing to generate the final output image, ensuring that the covered area is naturally integrated with the original image and does not affect the overall content display.

[0202] Step S521: Layer separation. The original image and the covering image (or covering layer) are respectively processed into independent layers. The covering layer can be represented by a transparent background or an alpha mask, and only has opaque covering content at the sensitive areas.

[0203] Step S522: Covering layer superposition. Based on the coordinate mapping result, accurately align and superpose the covering layer on the corresponding position of the original image. If the alpha blending function is enabled, perform the following pixel-level composition operation:

[0204]

[0205] Among them, represents the pixel value of the final output image at the position and Represents the abscissa of the pixel point, Represents the ordinate of the pixel point, Represents the original image at the position The pixel value at the position, Represents the covering layer at the position The pixel value at the position, Is the transparency coefficient at the corresponding position.

[0206] Step S523: Edge processing and anti-aliasing. Smooth the covering boundary in the composite image, such as using a 3×3 or 5×5 median filter kernel, to make the edge transition of the covering area natural and avoid an obvious sense of fragmentation.

[0207] Step S524: Output formatting. The final composite image can be exported as a mainstream image format such as PNG, JPG, or WebP as required, support embedding a transparent channel or compression processing, and retain the covering metadata of the sensitive area for subsequent auditing or revocation.

[0208] It can be understood that the present invention is described through some embodiments. Those skilled in the art know that without departing from the spirit and scope of the present invention, various changes or equivalent replacements can be made to these features and embodiments. Additionally, under the teaching of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the scope protected by the present invention.

Claims

1. An automatic covering method for sensitive information in images by fusing visual text features, characterized in that, The method includes the following steps: Step S1: Obtain image data, perform preprocessing on the image data, and obtain the character image after preprocessing; Step S2: Extract visual features and text features from the character image, fuse the visual features and text features to obtain cross-modal fusion features, perform feature transformation and enhancement on the cross-modal fusion features, and obtain a unified feature representation after fusion; Step S3: The multi-task recognition module performs AI recognition and marking on the unified feature representation after fusion. The AI recognition includes handwritten character recognition, sensitive content recognition, and prefix recognition; Step S4: The manual correction module corrects the subsequent areas of the marked handwritten characters, sensitive content, and sensitive content prefix words; Step S5: Integrate the AI recognition and manual correction results, perform black covering processing, perform image synthesis processing after completing the black covering processing operation, generate and save the final synthesized image, and output the final synthesized image.

2. The automatic covering method for image sensitive information with visual text feature fusion according to claim 1, characterized in that Step S1 includes the following steps: Step S11: Read the image data through the URL address, and the obtained image data is the original RGB image; Step S12: Perform grayscale processing on the image data to convert the color image into a grayscale image; Step S13: Perform edge detection on the grayscale image; Step S14: Perform secondary detection on the image data to identify the positions of individual characters and obtain a set of character position coordinates; Step S15: Perform region segmentation on the image data according to the set of character position coordinates to obtain the segmented character image.

3. The automatic masking method for image sensitive information with visual text feature fusion according to claim 2, characterized in that, Step S2 includes the following steps: Step S21: The visual feature extraction module extracts visual features from the character image to obtain visual features; Step S22: The text feature extraction module extracts text features from the character image to obtain text features; Step S23: The multi-modal feature fusion module fuses the extracted visual features and text features to obtain cross-modal fusion features, performs feature transformation and enhancement on the cross-modal fusion features, and obtains a unified feature representation after fusion.

4. The automatic covering method for image sensitive information with visual text feature fusion according to claim 3, wherein Step S23 includes the following steps: Step S231: Align the visual features and text features, map the visual features and text features to the same feature space, obtain the aligned visual features and the aligned text features, and preserve the feature semantic information of the visual modality and the text modality; Step S232: The multi-modal feature fusion module calculates the attention weights between the aligned visual features and the aligned text features through the cross-modal attention mechanism, realizes the dynamic fusion of the two modal features, and obtains cross-modal fusion features; Step S233: Use a feed-forward network to perform feature transformation and enhancement on the cross-modal fusion features to obtain a unified feature representation after fusion.

5. The automatic covering method for image sensitive information with visual text feature fusion according to claim 4, characterized in that, Step S3 includes the following steps: Step S31: The multi-task recognition module judges whether the character is a handwritten character through the handwritten character recognition task head and marks the handwritten character; Step S32: The multi-task recognition module judges the sensitive content through the sensitive content recognition task head and marks the sensitive content; Step S33: The multi-task recognition module identifies the sensitive content prefix word through the prefix recognition task head and marks the subsequent area of the sensitive content prefix word; Step S34: Construction of the loss function of the multi-task recognition module; Step S35: Training the multi-task recognition module.

6. The automatic covering method for image sensitive information with visual text feature fusion according to claim 5, characterized in that, Step S5 includes the following steps: Step S51: Integrate the AI recognition and manual correction results and perform black covering processing; Step S52: After completing the black covering processing operation, perform image synthesis processing to generate the final output image and output the processed image.

Images