An automatic masking method for image sensitive information based on visual-text feature fusion

Abstract

The present invention proposes a method for automatically masking image-sensitive information by fusing visual and text features, which belongs to the field of computer vision and artificial intelligence technology. The method includes: acquiring image data, preprocessing to obtain a character image after preprocessing; performing visual feature extraction and text feature extraction on the character image, fusing the visual features and text features to obtain cross-modal fusion features, and then obtaining a unified feature representation after fusion; a multi-task recognition module performs AI recognition and marking on the unified feature representation after fusion; a manual correction module corrects the marked handwritten characters, sensitive content, and subsequent areas of the prefix words of the sensitive content; integrating the AI ​​recognition and manual correction results, performing black masking processing and image synthesis processing, and outputting the final synthetic image. The present invention realizes the automatic recognition and masking of image-sensitive information, and improves processing efficiency and accuracy. The present invention provides a manual assisted correction function to ensure processing quality and is applicable to a variety of business scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and in particular relates to a method for automatically masking image-sensitive information by fusing visual and textual features. Background Art

[0002] In today's rapidly developing digital business environment, image data containing sensitive information is becoming increasingly common in business systems. This sensitive information includes key content such as personal identity information, financial data, and trade secrets. During business transactions and when interfacing with third-party systems, masking sensitive or private information in images is crucial for ensuring information security.

[0003] Currently, the industry primarily relies on manual processing to mask sensitive image information. This method requires operators to manually identify sensitive areas in images and apply image processing tools to mask them. However, this manual process has numerous drawbacks. Firstly, manual processing is inefficient. As business volume grows, masking large amounts of image data requires significant manpower and time costs, severely impacting business processing efficiency. Secondly, manual identification and masking accuracy is difficult to guarantee, making it easy for sensitive information to be missed or mistakenly masked, increasing the risk of information leakage and failing to meet increasingly stringent information security compliance requirements. Furthermore, manual processing lacks standardized procedures, and different operators employ varying methods and standards, further reducing the quality and reliability of masking.

[0004] To sum up, the existing image sensitive information masking solutions based on manual operations can no longer meet the current business development needs for information security and processing efficiency. An automated image sensitive information masking solution is urgently needed to improve the efficiency, accuracy and consistency of masking processing and ensure the security of sensitive information in business systems. Summary of the Invention

[0005] The present invention aims to provide a method for automatically masking sensitive information in images by integrating visual and textual features. This method can automatically identify and mask sensitive information in images, improving the efficiency and accuracy of sensitive information processing. This method aims to address the technical issues of existing image masking solutions, which rely primarily on manual operations, resulting in low efficiency and insufficient accuracy.

[0006] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0007] A method for automatically masking sensitive image information by integrating visual and textual features, the method comprising the following steps:

[0008] Step S1: acquiring image data and preprocessing the image data to obtain a character image after preprocessing;

[0009] Step S2: extracting visual features and text features from the character image, fusing the visual features and text features to obtain cross-modal fusion features, performing feature conversion and enhancement on the cross-modal fusion features to obtain a unified feature representation after fusion;

[0010] Step S3: The multi-task recognition module performs AI recognition and labeling on the fused unified feature representation. AI recognition includes handwriting recognition, sensitive content recognition, and prefix recognition.

[0011] Step S4: The manual correction module corrects the marked handwritten characters, sensitive content, and the area following the prefix of the sensitive content;

[0012] Step S5: Integrate the AI ​​recognition and manual correction results, perform black masking, perform image synthesis after completing the black masking operation, generate and save the final synthesized image, and output the final synthesized image.

[0013] Furthermore, step S1 includes the following steps:

[0014] Step S11: Read image data through the URL address, and the acquired image data is the original RGB image;

[0015] Step S12: grayscale processing is performed on the image data to convert the color image into a grayscale image;

[0016] Step S13: performing edge detection on the grayscale image;

[0017] Step S14: performing secondary detection on the image data to identify the position of a single character and obtain a set of character position coordinates;

[0018] Step S15: performing region segmentation on the image data according to the character position coordinate set to obtain a segmented character image.

[0019] Furthermore, step S2 includes the following steps:

[0020] Step S21: The visual feature extraction module extracts visual features from the character image to obtain visual features;

[0021] Step S22: The text feature extraction module extracts text features from the character image to obtain text features;

[0022] Step S23: The multimodal feature fusion module fuses the extracted visual features and text features to obtain cross-modal fusion features, performs feature conversion and enhancement on the cross-modal fusion features, and obtains a unified feature representation after fusion.

[0023] Furthermore, step S23 includes the following steps:

[0024] Step S231: aligning visual features and text features, mapping the visual features and text features to the same feature space, obtaining aligned visual features and aligned text features, and maintaining feature semantic information of the visual modality and the text modality;

[0025] Step S232: The multimodal feature fusion module calculates the attention weights between the aligned visual features and the aligned text features through a cross-modal attention mechanism, thereby dynamically fusing the two modal features to obtain cross-modal fusion features.

[0026] Step S233: Use a feedforward network to perform feature conversion and enhancement on the cross-modal fusion features to obtain a unified feature representation after fusion.

[0027] Furthermore, step S3 includes the following steps:

[0028] Step S31: The multi-task recognition module determines whether the character is handwritten through the handwriting recognition task head and marks the handwritten character;

[0029] Step S32: The multi-task identification module identifies the task header through sensitive content, determines the sensitive content, and marks the sensitive content;

[0030] Step S33: The multi-task recognition module identifies the task header through the prefix recognition, identifies the sensitive content prefix word, and marks the area following the sensitive content prefix word;

[0031] Step S34: constructing the loss function of the multi-task recognition module;

[0032] Step S35: training the multi-task recognition module.

[0033] Furthermore, step S5 includes the following steps:

[0034] Step S51: Integrate AI recognition and manual correction results to perform black masking;

[0035] Step S52: After the black masking operation is completed, an image synthesis process is performed to generate a final output image, and the processed image is output.

[0036] Compared with the prior art, the present invention has the following beneficial technical effects:

[0037] 1) The method proposed in the present invention integrates key technologies such as computer vision, multimodal semantic understanding, and intelligent image processing, and automatically completes steps such as image preprocessing, character area recognition, sensitive information detection, and area masking. Through the cross-modal feature extraction mechanism, it effectively combines image and language information, and improves the ability to distinguish complex sensitive areas; by embedding a manual auxiliary correction module, it ensures the processing quality based on automated processing and guarantees the integrity and accuracy of the masking results. Compared with traditional manual processing methods, the present invention can not only significantly improve the processing efficiency of image sensitive information, but also enhance the consistency and controllability of the results, and adapt to a variety of complex business scenarios, including financial services, medical imaging, government information management, public opinion data release, etc. In addition, the technology of the present invention can flexibly configure sensitive discrimination strategies and area masking solutions, and has good scalability and engineering application value.

[0038] 2) This invention enables automated recognition and masking of sensitive image information, improving processing efficiency and accuracy. It also provides a manual correction function to ensure processing quality and is applicable to a variety of business scenarios.

[0039] 3) This invention demonstrates strong innovation and practicality in the field of cross-modal intelligent recognition and information security, with clear engineering value and economic benefits. Its technical approach is scientific and its processing flow is clear. It is expected that its implementation in relevant fields will positively impact information security processing capabilities, reduce data leakage risks, and advance the development of intelligent content review technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 This is a flow chart of the method for automatically masking image-sensitive information by integrating visual and textual features of the present invention.

[0042] Figure 2 Schematic diagram of the architecture of the method for automatically masking image-sensitive information by integrating visual and textual features of the present invention.

[0043] Figure 3 Schematic diagram of the AI ​​recognition layer architecture of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] The present invention proposes a method for automatically masking image sensitive information by integrating visual and text features. Figure 1-3 As shown, the method includes the following steps:

[0046] Step S1: Acquire image data, and pre-process the image data to obtain a character image after pre-processing.

[0047] Specifically, step S1 includes the following steps:

[0048] Step S11: Read image data through the URL address, and the acquired image data is the original RGB image.

[0049] Step S12: grayscale processing is performed on the image data to convert the color image into a grayscale image.

[0050] The grayscale processing algorithm is expressed as follows:

[0051]

[0052] in, Grayscale value represents the brightness of the corresponding pixel in the image under the grayscale image. The value range is usually between 0 and 255, where 0 represents black and 255 represents white. 、 、 They represent the red, green, and blue component values ​​of the same pixel in a color image, and their value range is also 0 to 255.

[0053] Step S13: performing edge detection on the grayscale image.

[0054] Edge detection is used to locate characters in an image because the edges of characters often show sudden changes in brightness. Edge detection is also used to locate the bounding box of a single character. For example, by clustering pixels with high G values, the closed contour of each character is determined.

[0055] Edge detection is performed using the following edge detection algorithm:

[0056]

[0057]

[0058] in, Indicates the gradient amplitude of a pixel in the image, reflecting the intensity of the grayscale value change of the pixel. The larger the value, the more obvious the grayscale change near the pixel is, and the more likely it is an edge point of the image. Indicates that the pixel is in the horizontal direction ( The gradient value in the horizontal direction is used to measure the grayscale change rate of the image in the horizontal direction; Indicates that the pixel is in the vertical direction ( The gradient value in the vertical direction is used to measure the grayscale change rate of the image in the vertical direction; Indicates the gradient direction of the pixel, that is, the direction in which the grayscale changes fastest.

[0059] In an image, the grayscale value at the edge will have a significant mutation, and the gradient can well describe this change. By calculating the gradient values ​​in the horizontal and vertical directions and , using the Pythagorean theorem to get the gradient amplitude , thereby quantifying the intensity of grayscale changes; then the gradient direction is calculated by the inverse tangent function , clarify the direction of grayscale change. In this way, based on the gradient amplitude and direction, the pixels that may have edges in the image can be identified.

[0060] Step S14: Perform secondary detection on the image data to identify the position of a single character and obtain a set of character position coordinates.

[0061] Secondary detection, based on the initial localization of the text area by edge detection, further precisely locates the bounding boxes of individual characters. Specifically, connected domains are marked for regions with high gradient amplitudes, and spatially contiguous pixels are merged to form candidate character regions. Noise is filtered based on the geometric characteristics of the characters (aspect ratio, area, etc.). The bounding boxes of tilted characters are corrected using the gradient direction θ (e.g., rotating the bounding boxes to the mean direction of θ).

[0062] Specifically, on the basis of completing edge detection, the character positioning is realized by combining the improved connected domain analysis with morphological processing. The improved connected domain analysis combined with morphological processing method includes the following steps:

[0063] Step S141: Perform adaptive threshold segmentation on the gradient amplitude to generate a binary image :

[0064]

[0065] in, represents the threshold coefficient, The general value is 1.2. represents the global gradient mean of the image, Represents image coordinates The gradient amplitude at , can effectively eliminate the interference of uneven illumination through adaptive threshold segmentation.

[0066] Step S142: Use the 8-neighborhood scanning algorithm to mark the connected domains of the binary image and generate a set of candidate character regions. . And the area-aspect ratio dual constraint is introduced to filter out the non-character area and obtain the retained area.

[0067] The area-aspect ratio dual constraint is:

[0068] 15< <3000 and

[0069] in, Indicates the candidate character area number, is the area of ​​the candidate character region, The aspect ratio of the rectangle enclosing the candidate character area is used to effectively exclude noise and large non-text areas.

[0070] Step S143: Perform an iterative closing operation on the retained area to solve the character breakage problem and obtain the optimized connected domain. The iterative closing operation is calculated as follows:

[0071]

[0072] in, represents the optimized connected area, Represents structural element, structural element Using a 3×3 elliptical kernel, represents the expansion operation, Represents the erosion operation. This process smoothes the edges while preserving the shape of the character.

[0073] Step S144: extracting the minimum circumscribed rotated rectangle from the optimized connected domain to generate a character position coordinate set.

[0074] The character position coordinate set is represented as follows:

[0075]

[0076] in, Represents a set of character position coordinates, is the rectangle rotation angle (-15°≤ ≤15°), and Respectively represent the minimum horizontal and vertical coordinates of the circumscribed rectangle of the character area (upper left corner vertex), and Represent the maximum horizontal and vertical coordinates (the lower right vertex), respectively. These parameters together define the precise position and spatial orientation of the character in the image.

[0077] Step S15: performing region segmentation on the image data according to the character position coordinate set to obtain a segmented character image.

[0078] The original RGB image is cropped based on the character position coordinates to produce several character images. Each character image is then normalized, with each character region uniformly scaled to 64×64 pixels. Normalization uses a bicubic interpolation algorithm to preserve the character structure and eliminate the impact of scale differences on subsequent recognition. Normalization is achieved through the following methods:

[0079]

[0080] in, is the original RGB image. For the A standardized character image, Indicates that the extracted sub-image is scaled to a size of 64×64 pixels.

[0081] Step S2: extract visual features and text features from the character image, fuse the visual features and text features to obtain cross-modal fusion features, perform feature conversion and enhancement on the cross-modal fusion features, and obtain a unified feature representation after fusion.

[0082] Specifically, step S2 includes the following steps:

[0083] Step S21: The visual feature extraction module extracts visual features from the character image to obtain visual features.

[0084] The visual feature extraction module adopts a hierarchical structure based on Vision Transformer, and achieves efficient extraction and representation of image features through innovative architecture design and parameter configuration.

[0085] At the input layer, the visual feature extraction module divides the input character image into fixed-size 4×4 pixel blocks. This partitioning method preserves image details while reducing model processing complexity. Compared to traditional larger block sizes, 4×4 pixel blocks can capture more subtle textures and local structural features in the image, laying the foundation for subsequent feature extraction.

[0086] The feature encoding layer performs a dual operation on the divided image blocks. On the one hand, a position encoding mechanism is introduced to assign position information to each image block. Since the Transformer architecture itself does not have the ability to capture sequence order, position encoding enables the model to perceive the relative position relationship of image blocks in the original image by adding specific vectors, avoiding the loss of spatial information due to disrupting the order of image blocks. On the other hand, the image blocks are linearly mapped to a 192-dimensional encoding space. The 192-dimensional encoding dimension has been verified by a large number of experiments. This dimension can not only ensure the richness of features and effectively represent the feature information of image blocks, but also achieve a good balance between computing resources and model performance.

[0087] The attention layer is the core of the visual feature extraction module. Through its 24 attention heads and 24-layer architecture, it deeply captures both local and global image features. Each attention head models the relationship between image blocks from a different perspective. Working in parallel, the 24 attention heads can capture interactive image features from multiple dimensions, enhancing the model's understanding of complex image content. The 24-layer deep architecture enables the model to continuously iteratively optimize feature representations. As the number of layers increases, the model gradually abstracts more semantically relevant global features from local features. The self-attention mechanism dynamically focuses on important areas of the image by calculating attention weights between image blocks, ensuring that both minute details and overall structure within the image are fully attended to and extracted.

[0088] The output layer integrates the features processed by multiple layers and outputs visual features, which are high-dimensional visual feature representations. These high-dimensional visual features combine local image details with global semantic information. Their rich feature representation provides strong information support for downstream tasks, helping to improve their accuracy and reliability.

[0089] Step S22: The text feature extraction module extracts text features from the character image to obtain text features.

[0090] The text feature extraction module adopts an enhanced pre-trained language model architecture, and through sophisticated structural design and parameter configuration, it achieves efficient extraction and in-depth representation of text semantic features.

[0091] At the input layer, optical character recognition (OCR) is first performed on the character images based on a pre-trained OCR model to generate the original text. This text is then segmented, breaking the continuous text sequence into individual vocabulary units. The segmented words are then encoded and mapped into numerical vectors that the model can process. The vocabulary is set to 30,000 words, covering a wide range of common words as well as specialized terminology in specific fields. This ensures that all words in the text can be encoded in the vocabulary, providing a foundation for subsequent processing.

[0092] The attention layer uses a self-attention mechanism based on relative position encoding, which is the core innovation of the text feature extraction module. Compared with traditional absolute position encoding, relative position encoding can better capture the relative position relationship between words in the text, allowing the model to accurately understand the dependencies between words when processing long texts. 16 attention heads work in parallel, modeling the lexical relationships in the text sequence from different perspectives. Each attention head focuses on different semantic information. Through multi-head collaboration, the semantic associations and logical structures in the text are fully captured. The self-attention mechanism dynamically focuses on key information in the text by calculating the attention weights between words. Whether it is long-distance dependent semantic connections or local grammatical structures, they can be effectively identified and extracted.

[0093] The feedforward network layer receives the feature vector output by the attention layer and performs a nonlinear transformation on it. The hidden layer dimension is set to 1024, which ensures the model has sufficient capacity to learn complex semantic features while avoiding the computational burden and overfitting problems caused by excessive dimensions. Through the interaction of multi-layer perceptrons and activation functions, the feedforward network layer further abstracts and refines the features, transforming low-level semantic information into higher-level features with greater semantic representation capabilities, thereby enhancing the semantic expressiveness of text features.

[0094] The output layer integrates the features processed by the feedforward network layer to generate text features, which represent semantic features. These semantic features integrate the text's grammatical structure, semantic information, and context, accurately expressing the text's meaning. The maximum sequence length is set to 512, ensuring that the module can handle text lengths in most practical application scenarios. Whether it is fast feature extraction of short texts or in-depth semantic analysis of long texts, it can stably output high-quality semantic features.

[0095] Step S23: The multimodal feature fusion module fuses the extracted visual features and text features to obtain cross-modal fusion features, performs feature conversion and enhancement on the cross-modal fusion features, and obtains a unified feature representation after fusion.

[0096] The multimodal feature fusion module uses a cross-modal attention mechanism to fuse feature information from two different modalities: vision and text.

[0097] Step S231: Align the visual features and the text features, map the visual features and the text features to the same feature space, obtain aligned visual features and aligned text features, and maintain the feature semantic information of the visual modality and the text modality.

[0098] Feature alignment is a fundamental step in multimodal feature fusion. Because visual and textual features originate from different data modalities, their feature spaces differ significantly. To effectively fuse features from both modalities, visual and textual features must be mapped into the same feature space. This process not only ensures consistency in feature dimensions but, more importantly, preserves the semantic information of features from each modality. Feature alignment ensures that the semantic information represented by visual and textual features corresponds to each other within the same feature space, laying the foundation for subsequent cross-modal attention computation.

[0099] The visual features are represented as , the text feature is represented as , mapped to the same dimensional space after linear projection, and the aligned visual features are obtained Text features after alignment , the specific expression is as follows:

[0100]

[0101]

[0102] in, represents the visual feature linear transformation weight matrix, represents the linear transformation weight matrix of text features, represents the visual feature bias term, Represents the text feature bias.

[0103] Step S232: The multimodal feature fusion module calculates the attention weights between the aligned visual features and the aligned text features through the cross-modal attention mechanism, realizes the dynamic fusion of the two modal features, and obtains the cross-modal fusion features.

[0104] The multimodal feature fusion module adaptively assigns weights to visual features and text features based on different input data. Cross-modal attention fusion is achieved through the following formula:

[0105]

[0106]

[0107]

[0108]

[0109] in, represents cross-modal fusion features, represents the context feature vector, represents the context adaptation factor, represents the importance weight of the visual modality, represents the importance weight of the text modality, represents the aligned visual features, represents the aligned text features, represents cross-modal interaction features, The coefficient that indicates the contribution of cross-modal interaction features to the fusion result can be dynamically adjusted according to context information; represents the Sigmoid function, Indicates the use to generate The matrix, Indicates the use to generate The bias, Indicates concatenating the context feature vector with the aligned visual features. Indicates the use to generate The matrix, Indicates the use to generate The bias, Indicates concatenating the context feature vector with the aligned text feature. represents the tanh function, Indicates the use to generate The matrix, Indicates the use to generate Bias.

[0110] Context feature vector It can include global information from visual and textual modalities as well as relevant information for the current task. The tanh function limits the output value to [-1, 1] , so that the context adaptation factor Ability to dynamically adjust the strength of overall fusion based on contextual information.

[0111] and It can be regarded as the weight coefficient of visual and text features in the fusion process. After being concatenated with the corresponding modal features and undergoing linear transformation and Sigmoid activation, the importance of different modalities can be dynamically adjusted according to contextual information.

[0112] Cross-modal interaction features It captures the interaction information between visual and text features. In the fusion process, considering cross-modal interaction features can enable the model to better understand the relationship between the two modalities, such as the correspondence between objects in the image and text descriptions. The coefficient of the contribution of cross-modal interaction features to the fusion result Can be dynamically adjusted based on context information.

[0113] Step S233: Use a feedforward network to perform feature conversion and enhancement on the cross-modal fusion features to obtain a unified feature representation after fusion.

[0114] After cross-modal attention fusion, a feedforward network is used to further transform and enhance the cross-modal fused features. The feedforward network uses multiple fully connected layers and activation functions to perform nonlinear transformations on the input features, enabling the discovery of more complex relationships between features. After processing by the feedforward network, a unified fused feature representation is ultimately obtained. This unified feature representation integrates information from both visual and textual modalities, taking into account contextual information and intermodal interactions, resulting in greater expressiveness and robustness.

[0115] Step S3: The multi-task recognition module performs AI recognition and marking on the fused unified feature representation. AI recognition includes handwriting recognition, sensitive content recognition, and prefix recognition.

[0116] Specifically, a dedicated task head is designed for different recognition tasks. Step S3 includes the following steps:

[0117] Step S31: The multi-task recognition module determines whether the character is handwritten through the handwriting recognition task head and marks the handwritten character.

[0118] The input of the handwriting recognition task head is a fused unified feature representation, which integrates visual features and text features. It not only contains visual information such as the stroke shape and structural layout of the handwriting image, but also integrates prior knowledge at the text semantic level.

[0119] The middle layer of the handwriting recognition task head utilizes a multi-layer perceptron (MLP) architecture, forming the core processing pipeline for feature conversion and pattern recognition. The MLP is composed of multiple stacked fully connected layers. Each layer transforms the input features using a nonlinear activation function, gradually mapping the original fused features into a higher-level semantic space. In the handwriting recognition scenario, the MLP effectively extracts key features of handwritten characters, such as stroke transitions, connected strokes, and character outlines, through progressive feature abstraction. As the number of network layers increases, the model learns more complex and discriminative feature representations, thereby enhancing its ability to distinguish handwritten characters.

[0120] The output of the handwriting recognition task head is based on the features processed by the multi-layer perceptron and outputs the recognition probability of the handwriting.

[0121] In order to optimize the performance of the handwriting recognition task head, its loss function uses cross entropy. The cross entropy loss function can effectively measure the difference between the model's predicted probability distribution and the true label distribution. Its formula is:

[0122]

[0123] in, represents the handwriting recognition loss, is the total number of characters in the current sample, For the The true handwritten labels of the characters (0 = not handwritten, 1 = handwritten), p i (hw) ∈[0,1] The model predicts The probability that a character is handwritten.

[0124] Step S32: The multi-task identification module identifies the sensitive content through the sensitive content identification task header, judges the sensitive content, and marks the sensitive content.

[0125] The input of the sensitive content recognition task head is the fused unified feature representation, which integrates multi-source information such as text semantics, image vision, and audio spectrum, and realizes deep interaction of cross-modal information through feature fusion technology.

[0126] The middle layer of the sensitive content recognition task head adopts the attention pooling layer. This layer is based on the attention mechanism and can adaptively assign weights to different elements in the fusion features, focusing on key features related to sensitive content and suppressing redundant information interference.

[0127] The output of the sensitive content identification task head generates a multi-label classification result, which is expressed in the form of a probability vector. Each dimension corresponds to a predefined sensitive content category (such as violence, pornography, political sensitivity, etc.), and the value range is [0, 1], indicating the probability that the input content belongs to that category.

[0128] To ensure classification accuracy, the sensitive content recognition task head adopts the binary cross entropy loss function for multi-label classification, which is expressed as follows:

[0129]

[0130] in Indicates the loss of sensitive content recognition, is the number of sensitive content categories, is the true label of sensitive classification, p ij (sc) ∈[0,1] The model predicts Characters belong to The probability of sensitive content.

[0131] Step S33: The multi-task recognition module identifies the task header through the prefix recognition, identifies the specific sensitive content prefix word, and marks the subsequent area of ​​the sensitive content prefix word.

[0132] The input of the prefix recognition task head is the fused unified feature representation.

[0133] The middle layer of the prefix recognition task head uses a sequence annotation layer. This layer, based on sequence modeling techniques from deep learning (such as BiLSTM and Transformer encoders), predicts each position of the input fused feature sequence and assigns a corresponding label (such as "prefix start," "prefix inside," or "non-prefix") to each position. During training, the model learns to capture grammatical patterns and semantic clues related to prefixes, such as the morphological characteristics and contextual dependencies of specific lexical prefixes. This allows it to accurately identify the specific position of prefixes in the sequence, enabling refined annotation of prefix information in complex text.

[0134] The output of the prefix recognition task head generates prefix location information, presented as a sequence of labels. Each label corresponds to a position in the input sequence. By parsing this sequence, the starting and ending positions of the target prefix in the original data can be directly obtained. This output design can flexibly adapt to prefixes of varying lengths and forms, accurately locating both short word prefixes and long sentence prefixes, meeting diverse application needs.

[0135] To improve the model's accuracy and robustness in prefix recognition, the prefix recognition task head uses a loss function that combines FocalLoss and position weighting. As a loss function that dynamically adjusts weights, Focal Loss effectively alleviates the problem of class imbalance. When processing prefix recognition tasks, it assigns higher weights to the prefix position labels of minority categories, enhancing the model's ability to learn prefix features. At the same time, a position weighting mechanism is introduced, setting differentiated weights based on the different positions of the prefix in the sequence (such as the beginning, middle, and end of a sentence), focusing on the prediction accuracy of prefix boundaries and further optimizing the model's ability to capture prefix position information. By combining these two, the loss function can guide the model to more accurately locate prefixes during training, improving overall recognition performance.

[0136] Calculate position weights, which are used to emphasize the importance of characters at different positions in the prefix recognition task.

[0137] Position weights are calculated as follows:

[0138]

[0139] in, represents the position weight of the i-th character; represents the scaling factor; Indicates the distance between the current position and the starting position of the prefix. Position weight indicates the importance of a string at different positions. The closer to the front, the more important it is, and the greater the possibility that it is a prefix.

[0140] The prefix recognition loss function is expressed as follows:

[0141]

[0142] in, represents the prefix recognition loss function, represents the position weight factor, For the Is the character a prefix tag? p i (pr) ∈[0,1] The model predicts The probability of a character being a prefix, Indicates the focus parameter (the default value is 2).

[0143] Step S34: constructing the loss function of the multi-task recognition module.

[0144] The loss function of the multi-task recognition module integrates the handwriting recognition loss function, the sensitive content recognition loss function and the prefix recognition loss function. By combining the three loss functions, it balances the difficulty of the three tasks, prevents a single task from dominating the training, and speeds up the convergence.

[0145] The multi-task recognition loss function is expressed as follows:

[0146]

[0147] in, is the current training time step, express Multi-task recognition loss of time steps, 、 、 Respectively represent The weight coefficients of the handwriting recognition task, sensitive content recognition task, and prefix recognition task for each time step are initialized. . 、 、 Respectively represent The handwriting recognition task loss, sensitive content recognition task loss, and prefix recognition task loss are calculated for each time step.

[0148] The dynamic weight mechanism is used to update the weight system of each task. The dynamic weight mechanism is expressed as follows:

[0149]

[0150]

[0151]

[0152] in, 、 、 Represent the basic weights of the handwriting recognition task, sensitive content recognition task, and prefix recognition task respectively; 、 、 They represent the balance factors of the handwriting recognition task, sensitive content recognition task, and prefix recognition task respectively; 、 、 They represent the handwriting recognition task loss, sensitive content recognition task loss, and prefix recognition task loss at the 0th time step respectively; represents the multi-task recognition loss at the 0th time step.

[0153] The calculation formula of the balance factor is as follows:

[0154]

[0155]

[0156]

[0157] in, 、 、 Respectively represent the basic balance coefficients of the handwriting recognition task, sensitive content recognition task, and prefix recognition task (the default value is 0.5); 、 、 Represent the balance coefficient scaling factors of the handwriting recognition task, sensitive content recognition task, and prefix recognition task respectively; 、 、 Respectively represent the loss change rates of the handwriting recognition task, sensitive content recognition task, and prefix recognition task; 、 、 They represent the sensitivity factors of the handwriting recognition task, sensitive content recognition task, and prefix recognition task respectively; Indicates a task Task The correlation factor of Indicates the index of different tasks (Task 1 = handwriting recognition, Task 2 = sensitive content recognition, Task 3 = prefix recognition).

[0158] The loss change rates for the handwriting recognition task, sensitive content recognition task, and prefix recognition task are calculated as follows:

[0159]

[0160]

[0161]

[0162] 、 、 Respectively represent The handwriting recognition task loss, sensitive content recognition task loss, and prefix recognition task loss are calculated for each time step.

[0163] The sensitivity factors for the handwriting recognition task, sensitive content recognition task, and prefix recognition task are calculated as follows:

[0164]

[0165]

[0166]

[0167] in, represents the sigmoid activation function, 、 、 、 、 、 are learnable parameters; 、 、 They represent the variance of the loss of handwriting recognition task, sensitive content recognition task, and prefix recognition task respectively.

[0168] The task relevance factor is calculated as follows:

[0169]

[0170]

[0171] in, 、 is the gradient of different tasks, is the update direction obtained by back propagation, is a tensor, is the relevance weight (default is 0.2), Table Task Gradient and The cosine similarity between Indicates a task L2 norm of the gradient, Indicates a task L2 norm of the gradient.

[0172] Step S35: training the multi-task recognition module.

[0173] Step S351: Set the optimization target.

[0174]

[0175] in, represents the optimal parameter solution; Indicates the parameter value that makes the objective function reach the minimum value. Indicates The multi-task recognition loss calculated for the model with parameters .

[0176] Step S352: Calculate gradients and update model parameters.

[0177]

[0178] in, represents the parameters of the t+1th time step, represents the parameters of the t-th time step, represents the learning rate, Indicates the number of iterations; Represents the gradient of the total loss function with respect to the model parameters θ at the current iteration step t.

[0179] Step S353: Dynamic weight adjustment, by quantifying the loss convergence speed and gradient correlation of each task in real time, adaptively balancing the weight distribution of multi-task learning, thereby alleviating task conflicts and improving the overall performance of the model.

[0180] Step S354: Gradient balancing, by normalizing and weighting the gradients of different tasks, ensures that the optimization directions of each task in multi-task learning are coordinated and consistent, and avoids a single task dominating the parameter update.

[0181]

[0182] in, represents the gradient of each task, Indicates the median operation. Indicates a task The L2 norm squared of the gradient, Indicates a task The squared L2 norm of the gradient.

[0183] Step S4: The manual correction module corrects the marked handwritten characters, sensitive content, and the area following the specific sensitive content prefix.

[0184] Through the graphical interface design, the marked handwritten characters, sensitive content, and the subsequent areas of specific sensitive content prefixes are displayed on the graphical interface to facilitate manual correction.

[0185] The graphical interface uses a coordinate mapping algorithm to accurately locate and mask sensitive areas in images across different display sizes, resolutions, and across multiple platforms. Based on a standard image coordinate system, combined with normalization and mapping matrix operations, this algorithm accurately maps sensitive information areas identified in the original image to the target display area.

[0186] The graphical interface also provides a region editing function, which allows users to manually add, delete, adjust the position and size of the masked area after sensitive information detection, ensuring the controllability and flexibility of the system-generated results. The region editing function includes but is not limited to the following operations:

[0187] (1) Adding a mask area: Users can add a new rectangular mask area on the image by clicking and dragging. The system automatically incorporates the coordinates of the newly added area into the coordinate mapping algorithm for unified management.

[0188] (2) Adjust the mask area: Users can select an existing mask area and adjust its size by dragging edge nodes, or drag the entire area to change its position. The system updates the area coordinate data in real time and automatically synchronizes the mask rendering layer.

[0189] (3) Delete the masked area: The user can click on an existing masked area and perform the delete operation. The system will remove the area from the masked area list and cancel the visual masking process for it.

[0190] To prevent misoperation, the present invention also provides an "edit confirmation mechanism," requiring the user to click the "Save" button before all edits take effect. Region changes before and after editing are visually marked with differences, facilitating user comparison and confirmation. Region editing can be implemented through various interactive methods, such as a mouse and touch screen, and is applicable to various device platforms, including PCs, mobile devices, and embedded terminals, effectively improving the system's operability and user-friendliness in practical applications.

[0191] Step S5: Integrate the AI ​​recognition and manual correction results, perform black masking, perform image synthesis after completing the black masking operation, generate and save the final synthesized image, and output the final synthesized image.

[0192] Step S51: Integrate AI recognition and manual correction results to perform black masking processing.

[0193] Black masking is designed to perform configurable image masking on identified sensitive information areas to achieve controllable hiding of sensitive information. Black masking can adapt to different types of sensitive area characteristics and supports multiple masking methods, including but not limited to blurring, pixelation (mosaic), black block filling, and text replacement masking. The specific steps are as follows:

[0194] Step S511: Select a masking type. Select a masking type based on configuration parameters or user settings. If set to blur, a Gaussian blur kernel is used to convolve the sensitive area. If set to pixelate, the area image is downsampled and then upsampled to create a blocky pixel effect.

[0195] Step S512: Area masking. Mask the image content within each sensitive area coordinate. The processing function is as follows:

[0196] Blur processing: Apply a two-dimensional Gaussian convolution function to the image region R .

[0197] Pixelation: Perform average block resampling on the region R with a step size of s.

[0198] Black block filling: Set all pixel values ​​in area R to (0,0,0).

[0199] Text masking: Use a unified identification text (such as "masked") to draw in the center of the original area, and the font color should contrast clearly with the background.

[0200] Step S52: After the black masking operation is completed, an image synthesis process is performed to generate a final output image, and the processed image is output.

[0201] After completing the black masking operation, image synthesis processing is performed to generate the final output image, ensuring that the masked area blends naturally with the original image and does not affect the overall content display.

[0202] Step S521: Layer separation. The original image and the mask image (or mask layer) are processed as independent layers. The mask layer can be represented by a transparent background or an alpha mask, with opaque mask content only in sensitive areas.

[0203] Step S522: Overlay the mask layer. Based on the coordinate mapping results, the mask layer is precisely aligned and overlaid on the corresponding position of the original image. If the alpha fusion function is enabled, the following pixel-level synthesis operations are performed:

[0204]

[0205] in, Indicates that the final output image is at position The pixel value at Indicates the horizontal coordinate of the pixel point, Indicates the vertical coordinate of the pixel point, Indicates that the original image is at position The pixel value at Indicates that the mask layer is at position The pixel value at is the transparency coefficient of the corresponding position.

[0206] Step S523: Edge processing and anti-aliasing: Smoothing is performed on the masked boundary in the synthesized image, such as using a 3×3 or 5×5 median filter kernel, to make the masked area edge transition smooth and avoid obvious fragmentation.

[0207] Step S524: Output formatting. The final composite image can be exported as needed to mainstream image formats such as PNG, JPG, or WebP. It supports embedding transparent channels or compression processing, and retains sensitive area masking metadata for subsequent audit or recall.

[0208] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A method for automatically masking sensitive image information by integrating visual and textual features, characterized in that: The method comprises the following steps: Step S1: acquiring image data and preprocessing the image data to obtain a character image after preprocessing; Step S2: extracting visual features and text features from the character image, fusing the visual features and text features to obtain cross-modal fusion features, performing feature conversion and enhancement on the cross-modal fusion features to obtain a unified feature representation after fusion; Step S3: The multi-task recognition module performs AI recognition and labeling on the fused unified feature representation. AI recognition includes handwriting recognition, sensitive content recognition, and prefix recognition. Step S4: The manual correction module corrects the marked handwritten characters, sensitive content, and the area following the prefix of the sensitive content; Step S5: Integrate the AI ​​recognition and manual correction results, perform black masking, perform image synthesis after the black masking operation is completed, generate and save the final synthesized image, and output the final synthesized image; Step S3 includes the following steps: Step S31: The multi-task recognition module determines whether the character is handwritten through the handwriting recognition task head and marks the handwritten character; Step S32: The multi-task identification module identifies the task header through sensitive content, determines the sensitive content, and marks the sensitive content; Step S33: The multi-task recognition module identifies the task header through the prefix recognition, identifies the sensitive content prefix word, and marks the area following the sensitive content prefix word; Step S34: constructing the loss function of the multi-task recognition module; Step S35: training the multi-task recognition module.

2. The method for automatically masking image sensitive information by integrating visual and textual features according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: Read image data through the URL address, and the acquired image data is the original RGB image; Step S12: grayscale processing is performed on the image data to convert the color image into a grayscale image; Step S13: performing edge detection on the grayscale image; Step S14: performing secondary detection on the image data to identify the position of a single character and obtain a set of character position coordinates; Step S15: performing region segmentation on the image data according to the character position coordinate set to obtain a segmented character image.

3. The method for automatically masking image sensitive information by integrating visual and textual features according to claim 2, characterized in that: Step S2 includes the following steps: Step S21: The visual feature extraction module extracts visual features from the character image to obtain visual features; Step S22: The text feature extraction module extracts text features from the character image to obtain text features; Step S23: The multimodal feature fusion module fuses the extracted visual features and text features to obtain cross-modal fusion features, performs feature conversion and enhancement on the cross-modal fusion features, and obtains a unified feature representation after fusion.

4. The method for automatically masking image sensitive information by integrating visual and textual features according to claim 3 is characterized in that: Step S23 includes the following steps: Step S231: aligning visual features and text features, mapping the visual features and text features to the same feature space, obtaining aligned visual features and aligned text features, and maintaining feature semantic information of the visual modality and the text modality; Step S232: The multimodal feature fusion module calculates the attention weights between the aligned visual features and the aligned text features through a cross-modal attention mechanism, thereby dynamically fusing the two modal features to obtain cross-modal fusion features. Step S233: Use a feedforward network to perform feature conversion and enhancement on the cross-modal fusion features to obtain a unified feature representation after fusion.

5. The method for automatically masking image sensitive information by integrating visual and textual features according to claim 4 is characterized in that: Step S5 includes the following steps: Step S51: Integrate AI recognition and manual correction results to perform black masking; Step S52: After the black masking operation is completed, an image synthesis process is performed to generate a final output image, and the processed image is output.

Images