A method and system for processing sensitive information of engineering data

Abstract

The application discloses a sensitive information processing method and system for engineering materials, and relates to the technical fields of multi-modal intelligent identification and industrial information security. The application adopts an offline preparation and online processing decoupling architecture, fine-tunes the text and handwriting detection model in the offline stage, and constructs a project-specific RAG knowledge base; in the online stage, the engineering materials are analyzed by OCR, and the text stream and image stream processing are executed in parallel: the text stream calls a large language model to complete semantic recognition in combination with the knowledge base, and the image stream realizes accurate detection of handwritten sensitive information through a differentiable binarization, multi-scale branch convolution and hybrid attention module; coordinate fusion is completed through semantic and spatial double verification, and finally the automatic covering of sensitive information is realized. The application solves the problems of low efficiency, poor handwriting recognition accuracy and difficult rule adaptation of existing schemes, can effectively prevent information leakage risks in the circulation of engineering materials, and is suitable for security desensitization processing in scenes such as engineering drawings and technical documents.

Description

Technical Field

[0001] This invention belongs to the field of image recognition and information security technology, specifically relating to a method and system for processing sensitive information in engineering data. Background Technology

[0002] In supply chain collaborations within the manufacturing and engineering construction industries, companies frequently need to share core documents such as engineering drawings and technical specifications with suppliers or partners for processing and reference. These engineering documents often contain a significant amount of sensitive information, including but not limited to core design parameters, confidential company data, handwritten signatures of project leaders, and contact information. If such sensitive information is leaked without proper processing, it can lead to the disclosure of intellectual property rights, weaken core competitiveness, and even trigger serious legal and security risks. Therefore, accurately identifying and automatically masking (de-identifying) sensitive information in engineering documents before data sharing is a critical issue that urgently needs to be addressed in the field of industrial data security.

[0003] Existing sensitive information processing technologies are mainly divided into manual processing, rule-based traditional automated processing, and deep learning-based intelligent processing methods.

[0004] Manual processing: This method relies on staff to manually select and pixel-mask or blur sensitive areas using image editing software. While it offers high flexibility, it is extremely inefficient when dealing with massive amounts of engineering drawings. Furthermore, staff fatigue or misunderstandings of confidentiality rules can easily lead to the omission or over-masking of sensitive information, affecting the readability and subsequent use of the drawings.

[0005] Traditional rule-based automation methods typically involve string matching by building a sensitive word library. While this method is simple to implement, it is costly to maintain and cannot handle semantic variations, abbreviations, or contextual dependencies of sensitive information. It also has a high false positive and false negative rate and cannot identify non-textual graphical sensitive data (such as signatures and seals).

[0006] With the development of computer vision and optical character recognition (OCR) technologies, automated solutions combining object detection and OCR are increasingly being applied to document anonymization. However, traditional object detection algorithms lack deep semantic understanding capabilities, making it difficult to distinguish between publicly available technical parameters and confidential company information in drawings. Furthermore, traditional OCR technology has limited accuracy when dealing with slanted text, blurred handwriting, or handwritten content commonly found in engineering drawings, and it struggles to logically correlate the recognized text content with specific confidentiality rules.

[0007] In recent years, data privacy protection technologies based on Large Language Models (LLM) have begun to emerge. Some existing technologies attempt to use pre-trained models for text classification and sensitive information identification, or combine generative large models for content review. In the patent field, there are also solutions that propose key information masking methods based on large models. For example, patent document CN119314179A discloses a method for masking sensitive information in ID card images, and patent document CN120257368A discloses a method and device for masking sensitive personal information of clinical trial participants.

[0008] While the aforementioned technologies have made some progress in general text processing, the following major technical problems still exist when directly applied to engineering drawing scenarios:

[0009] 1) Insufficient semantic understanding and rule adaptation capabilities: The general-purpose model lacks understanding of the confidentiality rules for specific engineering projects, and the definitions of sensitive information vary greatly between different projects. Existing static models are difficult to adapt to the unique rules of new projects in a low-cost manner, resulting in high false positive and false negative rates.

[0010] 2) Weak multimodal fusion processing capability and low handwriting localization accuracy: Engineering drawings contain multiple modal information such as text, lines, and handwritten signatures. Existing solutions often separate text detection from image detection. Especially for handwritten signatures and annotations, if a general multimodal large model is used for detection, although it has a certain generalization ability, the large number of parameters leads to high inference latency, and it is not accurate enough in locating the fine handwritten stroke edges in the drawings, easily misclassifying printed text or drawing frames as handwritten text. If a traditional lightweight object detection small model is used, it is easily interfered with by the complex background of the drawings (such as grid lines, background maps, and seals), resulting in a "scale submersion" effect during multi-scale feature fusion, leading to missed detections or false detections.

[0011] 3) Low coordinate mapping accuracy: Some methods based on large models can only output text labels for sensitive content, lacking visual positioning capabilities and making it difficult to accurately map back to the specific coordinate positions of the original image for precise masking. This discrepancy between semantic and visual position results in inaccurate masking box positions, severely impacting the engineering usability of the drawings.

[0012] In summary, existing technologies cannot simultaneously achieve compatibility with confidentiality rules, multimodal recognition accuracy, handwritten area positioning accuracy, and batch processing efficiency. Therefore, there is an urgent need to develop a processing method and system for engineering data that can efficiently identify sensitive text and handwritten information and perform automated desensitization. Summary of the Invention

[0013] In view of this, the purpose of this invention is to provide a method and system for processing sensitive information in engineering data, aiming to solve the problems of low processing efficiency, weak semantic recognition capability, poor detection accuracy of handwritten sensitive information, difficulty in adapting to project confidentiality rules, and inaccurate matching of masking coordinates in existing solutions, so as to realize intelligent, accurate and automated desensitization processing of sensitive information in engineering data and ensure information security during the flow of engineering data.

[0014] To achieve the above objectives, the present invention adopts the following technical solution:

[0015] A method and system for processing sensitive information in engineering data, comprising the following steps:

[0016] S1. Collect historical engineering data and complete the annotation of sensitive information, construct a training dataset, and use the training dataset to fine-tune the pre-trained large language model and lightweight visual model respectively to obtain the text sensitive information detection model and the handwritten data detection model.

[0017] S2. Based on the preset knowledge base template, input the sensitive information rules and whitelist rules corresponding to the engineering project, vectorize the rule text, and build a project-specific retrieval enhancement generation knowledge base;

[0018] S3. Obtain the engineering data to be processed, parse it using optical character recognition and image positioning algorithms, extract the text content and corresponding physical coordinates, and generate a coordinate mapping table;

[0019] S4. Sensitive information identification and location are performed using a parallel processing method of text stream and image stream;

[0020] The text stream invokes the text sensitive information detection model and combines it with the retrieval enhancement knowledge base to perform semantic analysis and identify text-based sensitive information;

[0021] The image stream calls the handwritten data detection model to identify handwritten sensitive information and output the corresponding coordinates; through semantic similarity and spatial location dual verification, the sensitive information and coordinate mapping table are matched and fused to obtain a list of coordinates of all sensitive information.

[0022] After completing the recognition of image stream and text stream, the text-type sensitive information and handwritten sensitive information are matched and fused with the coordinate mapping table through dual verification of semantic similarity and spatial bounding box overlap rate, and after deduplication, a list of coordinates of all sensitive information is obtained.

[0023] S5. Based on the list of sensitive information coordinates, perform masking processing on the corresponding area of ​​the engineering data and output the desensitized engineering data;

[0024] Steps S1-S2 are the offline preparation stage, and steps S3-S5 are the online business processing stage.

[0025] Furthermore, in step S1, the lightweight visual model is a DRSC model, with ResNet18 as the residual backbone network. When fine-tuning the lightweight visual model, the GIoU loss function is used to optimize the coordinate regression process.

[0026] The formula for calculating the GIoU loss function is as follows:

[0027]

[0028] In the formula: The intersection-union ratio (IoU) of the model's predicted bounding boxes and the ground truth labeled bounding boxes; The area of ​​the smallest closed bounding rectangle that can simultaneously enclose both the predicted bounding box and the ground truth bounding box; This represents the area of ​​the union of the predicted bounding box and the ground truth bounding box. This represents the area of ​​the background region within the smallest bounding rectangle that is not covered by the predicted bounding box or the ground truth bounding box.

[0029] Furthermore, in step S2, the knowledge base template includes the project name, confidentiality level, specific sensitive word rules, exception whitelist rules, and sensitive information definition description.

[0030] Furthermore, the image stream processing in step S4, namely, inputting the engineering data image into the handwritten data detection model, and performing the following processing sequentially:

[0031] I. By predicting the pixel probability map and adaptive threshold map through the differentiable binarization module, a binary mask is generated to filter out the complex background of the engineering data and obtain the handwritten foreground feature map.

[0032] The differentiable binarization module generates a binary mask according to the following formula:

[0033]

[0034] In the formula, For the generated binary mask at pixel position The value at this location is used to filter out the background of the drawing; A probability map of pixels predicted by the network as belonging to handwritten strokes; A dynamic threshold map for network adaptive learning, enabling the model to automatically adjust the binarization threshold based on the depth of the local background of the drawing; A magnification factor to control the degree of binarization approximation;

[0035] Will Multiplying each element of the original drawing yields the handwritten foreground feature map. ;

[0036] II. Input the handwritten foreground feature map into the residual backbone network for feature extraction to obtain the initial feature map;

[0037] III. Input the initial feature map into three different receptive field branches: global feature convolutional layer, local feature convolutional layer, and detail feature convolutional layer, to obtain feature maps of the corresponding scales. Input each scale feature map into the matched global hybrid attention module, local hybrid attention module, and detail hybrid attention module, respectively. Use selective gating mechanism to suppress background noise, complete single-scale feature enhancement, and obtain multi-scale enhanced feature maps.

[0038] Among them, the global feature convolutional layer, the local feature convolutional layer, and the detail feature convolutional layer all adopt dilated convolutional structure, and each branch is configured with a different dilation rate to distinguish the size of the receptive field.

[0039] The multi-scale feature extraction formula for the corresponding scale feature map is:

[0040]

[0041] In the formula, For the corresponding scale branch in spatial location The output feature map at the location, where ; These are the weight parameters of the convolution kernel; This represents the expansion rate of the corresponding branch;

[0042] The hybrid attention module sequentially performs mask generation and attention weighting operations, and achieves spatial dimension noise filtering and feature purification through element-wise multiplication.

[0043] Mask generation formula:

[0044]

[0045] Attention-weighted formula:

[0046]

[0047] In the formula, For selective gating masks, use 1×1 convolution and the Sigmoid activation function. generate; Each is derived from the input features The query matrix, key matrix, and value matrix obtained through linear mapping; Key matrix The dimension; This indicates element-wise multiplication; The output feature map after attention weighting and gating filtering;

[0048] IV. The multi-scale enhanced feature maps are concatenated along the channel dimension and then input into the channel position attention module to complete the cross-scale channel and spatial adaptive feature fusion, resulting in the final fused feature map;

[0049] The channel position attention module calculates the channel weight map and the spatial position weight map respectively, and combines the element-wise multiplication of broadcast to complete the secondary feature enhancement.

[0050] V. Input the final fused feature map into the coordinate regression module, and output the location coordinates of the handwritten sensitive information.

[0051] Furthermore, in step S5, the masking processing method includes any one or more of the following: solid color block filling, Gaussian blur, and mosaic processing.

[0052] The present invention also provides a sensitive information processing system for engineering data, including a preprocessing module, a model training module, a knowledge base management module, an identification and analysis module, and a masking execution module;

[0053] The knowledge base management module is used to store knowledge base templates, input project-specific sensitive information rules and complete vectorization processing, and build and maintain a retrieval-enhanced generated knowledge base.

[0054] The preprocessing module has built-in optical character recognition algorithm and image positioning algorithm, which are used to parse the engineering data to be processed, extract text content and position coordinates, generate a coordinate mapping table and send it to the recognition and analysis module.

[0055] The recognition and analysis module is communicatively connected to the preprocessing module, the model training module, and the knowledge base management module, respectively. It loads the model weight file and performs text stream processing and image stream processing in parallel. The recognition and analysis module integrates a differentiable binarization module, a residual backbone network, a multi-scale convolutional branch, a hybrid attention module, a channel position attention module, and a coordinate regression module to complete the detection of handwritten sensitive information. It also combines semantic and spatial dual verification to complete coordinate fusion and output a list of sensitive information coordinates.

[0056] The masking execution module is connected to the identification and analysis module and is used to mask the corresponding area of ​​the engineering data according to the list of sensitive information coordinates and output desensitized engineering data.

[0057] Furthermore, the recognition and analysis module includes a text stream processing unit, an image stream processing unit, and a coordinate fusion unit;

[0058] The text stream processing unit calls the fine-tuned large language model and combines it with the retrieval enhancement to generate a knowledge base to complete the semantic recognition of sensitive information in the text.

[0059] The image stream processing unit integrates a lightweight DRSC visual model, and outputs the coordinates of handwritten sensitive information through differentiable binarization, feature extraction, multi-scale branch enhancement, cross-scale fusion, and coordinate regression in sequence.

[0060] The coordinate fusion unit, through dual verification of semantic similarity and spatial bounding box overlap rate, fuses the coordinates of text-sensitive information and handwritten-sensitive information to generate a final list of sensitive information coordinates.

[0061] Furthermore, the system also includes a human-computer interaction review module, which is used to display de-identified engineering data, receive user review and adjustment instructions, and feed them back to the masking execution module.

[0062] Beneficial effects:

[0063] 1. This invention employs a lightweight visual model and a large language model to construct a dual-stream parallel processing architecture for image and text streams, simultaneously detecting handwritten and text-based sensitive information in engineering documents. The lightweight visual model, combined with differentiable binarization, multi-scale feature extraction, and attention fusion mechanisms, effectively filters out complex background interference such as drawing textures, grid lines, and seals, significantly improving the recognition and localization accuracy of handwritten signatures and annotations. The large language model, combined with retrieval-enhanced generation technology and project-specific confidentiality rules, possesses powerful semantic understanding capabilities, effectively identifying sensitive word variations, abbreviations, and context-related confidential content, significantly reducing the false negative and false positive rates of sensitive information detection.

[0064] 2. This invention builds a project-specific retrieval enhancement knowledge base, which supports the rapid input of confidentiality rules and whitelist content for different projects. It can adapt to differentiated confidentiality requirements without repeated fine-tuning of the model. The rule configuration is flexible and the scenario adaptability is strong, which effectively reduces the cost of model iteration and rule maintenance. It can meet the desensitization processing needs of multiple types and multiple project engineering data.

[0065] 3. This invention uses a dual verification mechanism of semantic similarity and spatial location to accurately map and fuse the recognition results with the original coordinates of the engineering data. At the same time, it combines an optimized coordinate regression algorithm to output the boundary coordinates of the handwritten area, ensuring that the covered area and the sensitive information area are highly consistent. This avoids problems such as offset or omission of the covered area, and also prevents excessive covering from damaging the content of the data. While achieving information desensitization, it also protects the original format and usability of the engineering data.

[0066] 4. This invention uses a customized lightweight visual model, abandoning the general multimodal large model with a large number of parameters. While ensuring recognition accuracy, it effectively shortens the model inference time, has excellent batch processing capabilities, and can adapt to the business scenario of enterprises routinely sending out large amounts of engineering data, taking into account both detection accuracy and operating efficiency.

[0067] 5. This invention divides the overall process into an offline model building stage and an online business processing stage, decoupling model training, knowledge base maintenance, and online anonymization processes. Each functional module has a clear division of labor and operates independently, facilitating subsequent model upgrades, rule updates, and functional expansions, resulting in better overall system stability, maintainability, and scalability.

[0068] 6. This invention achieves fully automated processing of sensitive information identification, location, and masking, replacing the traditional manual operation mode. It not only saves a lot of labor costs, but also avoids security risks caused by human negligence and subjective judgment bias. From a technical perspective, it prevents the risk of leakage of trade secrets and intellectual property rights during the external transfer of engineering data, and provides reliable protection for information security in the engineering field.

[0069] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0070] Figure 1 This is a flowchart of a method for processing sensitive information in engineering data according to the present invention;

[0071] Figure 2 The flowchart for the Lightweight Visual Modeling (DRSC) algorithm;

[0072] Figure 3 This is an architecture diagram of the sensitive information processing system of the present invention. Detailed Implementation

[0073] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0074] like Figure 1 As shown, this invention provides a method for processing sensitive information in engineering data. This method runs on the server side and aims to solve the technical problem that traditional rule-based or single OCR technologies cannot effectively identify handwritten signatures, non-standard annotations, and context-related sensitive semantics in engineering drawings. It includes an offline model building stage and an online business processing stage.

[0075] 1. Offline model building stage

[0076] Step S1: Construct a sensitive information processing model library.

[0077] This step aims to obtain a specialized model capable of identifying sensitive information in the engineering field, specifically including:

[0078] Dataset Construction: Collect historical engineering drawings (PDF or CAD exported drawings) from the company and manually annotate sensitive information areas. Annotations include: sensitive text content, sensitive types (such as "core technical parameters," "personal signature," and "contact information"), and the coordinates of the bounding boxes of sensitive areas. Construct a training dataset containing positive examples (information that needs to be masked) and negative examples (information that does not need to be masked).

[0079] Fine-tuning of the text sensitive information detection model: Select a pre-trained large language model as the base model (e.g., Qwen-7B / 14B / 72B series, Llama series, etc.). Use the constructed training dataset to fine-tune the base model with instructions, enabling it to determine whether the input text content and context constitute sensitive information and output sensitive type labels.

[0080] Construction and Fine-tuning of the Handwritten Data Detection Model (Lightweight Visual Model): Abandoning the large-scale, general-purpose multimodal model with numerous parameters, a dedicated lightweight visual model (referred to as the DRSC model in this embodiment) is constructed. This model uses ResNet18 as the residual backbone network and innovatively designs a network topology structure that integrates multi-scale branch independent enhancement and attention fusion. The model is fine-tuned using a dataset labeled with the coordinates of handwritten signatures and annotation regions. During training, a coordinate regression loss function (such as GIoULoss) is introduced, enabling the model to directly output the physical coordinates of the handwritten sensitive region in the image.

[0081] Model storage: Save the fine-tuned model weight file to the model repository to generate versioned model files for loading and use in the online phase.

[0082] Step S2: Build a project-specific knowledge base.

[0083] This step aims to achieve dynamic adaptation of confidentiality rules, resolving inconsistencies in confidentiality standards across different projects. Rule Template Design: Design a structured knowledge base file template (e.g., JSON format). Template fields include: project name, confidentiality level, specific sensitive word rules (supporting regular expressions), exception whitelist rules, and sensitive information definition descriptions. Knowledge Base Vectorization: For new engineering projects, security administrators input the project-specific sensitive information rules into the template. The system uses a text embedding model to convert the rule text into vectors, storing them in a vector database (such as Milvus or Faiss), forming a project-specific Retrieval Enhanced Generation (RAG) knowledge base.

[0084] 2. Online business processing stage

[0085] Step S3: Engineering drawing preprocessing and feature extraction.

[0086] File parsing: Obtain the engineering drawing file to be processed and convert it into a high-resolution image format (such as PNG or TIFF).

[0087] OCR Recognition and Coordinate Extraction: The drawing image is scanned and analyzed using an Optical Character Recognition (OCR) engine. In a preferred embodiment, the OCR engine can be selected from MinerU, PaddleOCR, etc.

[0088] Generate a coordinate mapping table: Extract all text information from the drawing and record the physical location information of each text block within the drawing. The physical location information includes the top-left corner coordinates (x, y), width w, and height h. The system binds the text content with the location information to generate a coordinate mapping table.

[0089] Example data structure: {"test":"Contact number","bbox":[100,200,50,10],"confidence":0.98}

[0090] Step S4: Intelligent Identification and Location of Sensitive Information (Core Step). This step employs a dual-stream parallel processing mechanism to process semantically sensitive information in printed text and visually sensitive information in handwritten text, respectively.

[0091] Text stream processing (semantic analysis): The text information extracted in step S3 is input into the text sensitive information detection model. Simultaneously, based on the current project's metadata (such as the project ID), the system retrieves the Top-K most relevant confidentiality rule vectors from the knowledge base constructed in step S2, and inputs them as part of the contextual prompts into the model. The model combines knowledge base rules with semantic understanding to output a list of sensitive information content and its sensitivity type.

[0092] Technical effect: Through RAG technology, the model can understand that "the A-type parameter in this project is confidential", rather than just matching keywords.

[0093] Image stream processing (visual inspection): in parallel, such as Figure 2 As shown, the drawing image is input into the lightweight visual model (DRSC). This model addresses the challenges of complex backgrounds and varying handwriting scales in engineering drawings by employing rigorous mathematical logic. Its forward inference and backward training processes are as follows:

[0094] A. Adaptive Foreground Separation (DB Module):

[0095] After the drawing is input, it first passes through the differentiable binarization (DB) module. To accommodate handwritten strokes of varying thickness and contrast in the drawing, the DB module uses a network to predict a probability map and an adaptive threshold map, and generates a binary mask using the following formula:

[0096]

[0097] In the formula: For the generated binary mask at pixel position The value at this location is used to filter out the background of the drawing; A probability map of pixels predicted by the network as belonging to handwritten strokes; A dynamic threshold map for network adaptive learning, enabling the model to automatically adjust the binarization threshold based on the depth of the local background of the drawing; The amplification factor is used to control the degree of binarization approximation (during the training phase). Set it to a small value to ensure gradient differentiability during the inference phase. Set a larger value to approximate the standard step function). Multiplying the original drawing element by element yields a clean handwritten foreground feature map. .

[0098] B. Feature Extraction and Multi-Scale Convolution:

[0099] Handwritten foreground feature map After initial features are extracted by the residual backbone network, they are input into the global, local, and detail branches, respectively. To obtain receptive fields of different sizes without increasing computational cost, this invention employs dilated convolution to construct a multi-scale feature extraction formula:

[0100]

[0101] In the formula: For the corresponding scale branch in spatial location The output feature map at the location, where ; These are the weight parameters of the convolution kernel; This represents the dilation rate for the corresponding branch. In this embodiment, global branch settings are used. To capture the context of the page with a large receptive field; local branch settings Capture character-level outlines; detailed branching settings To focus on the edges of tiny strokes.

[0102] C. Branch Independent Enhancement and Gated Filtering (SeSA Module):

[0103] To avoid the "scale overload" problem caused by direct fusion of multi-scale features, the features of the three branches are fed into their respective hybrid attention modules (SeSA). SeSA introduces a selective gating mechanism, and its computation process includes two steps: mask generation and attention weighting.

[0104] Mask generation formula:

[0105]

[0106] Attention-weighted formula:

[0107]

[0108] In the formula: For selective gating masks, use 1×1 convolution and the Sigmoid activation function. Generates a tool to dynamically suppress low-response background noise such as grid lines and textures in engineering drawings; Each is derived from the input features The query matrix, key matrix, and value matrix obtained through linear mapping; Key matrix The dimension is used to scale the dot product result to prevent gradient vanishing. This represents element-wise multiplication (Hadamard product), which enables the gating mask. It can directly filter irrelevant features in the spatial dimension, achieving "feature purification" at various scales.

[0109] D. Cross-scale adaptive fusion (CPAM module):

[0110] The purified features at three scales are fused together using a multi-scale splicing node (Concat). The input channel position attention module (CPAM) then performs dual weight allocation:

[0111] Channel attention calculation:

[0112]

[0113] Spatial attention calculation:

[0114]

[0115] Final feature fusion:

[0116]

[0117] In the formula: and These are the generated channel weight map and spatial location weight map, respectively; and These are global average pooling and global max pooling operations, respectively. A multilayer perceptron, incorporating both dimensionality reduction and dimensionality expansion, is used to capture nonlinear dependencies between channels; It is a standard 7×7 convolutional layer used to extract contextual features in the spatial dimension; This indicates element-wise multiplication based on a broadcast mechanism, ensuring that features are adaptively enhanced in both channel and spatial dimensions.

[0118] E. Coordinate Regression and Loss Function Optimization:

[0119] Final fused feature map The input is fed into a coordinate regression module containing fully connected layers, and the output is the coordinates of the handwritten bounding box. Addressing the issue of large aspect ratio differences and dense arrangement of handwritten signatures and annotations in engineering drawings, this invention abandons the traditional SmoothL1Loss during the model fine-tuning stage and introduces the Generalized Intersection over Union (GIoULoss) loss function for bounding box regression optimization.

[0120]

[0121] In the formula: The Intersection over Union (IoU) ratio between the model's predicted bounding box and the ground truth labeled bounding box is calculated. The area of ​​the smallest closed bounding rectangle that can simultaneously enclose both the predicted bounding box and the ground truth bounding box; This represents the area of ​​the union of the predicted bounding box and the ground truth bounding box. This represents the area of ​​the background region within the smallest bounding rectangle that is not covered by the predicted bounding box or the ground truth bounding box.

[0122] Technical effect: This loss function not only penalizes the non-overlapping parts of the predicted box and the ground truth box, but also additionally penalizes the "blank background" area between them. It effectively guides the model to generate more compact bounding boxes that fit the edges of handwritten strokes in dense drawing areas, and completely solves the gradient vanishing problem of traditional regression loss functions in extreme aspect ratio cases.

[0123] Coordinate fusion and matching: Since the text stream model outputs sensitive text labels, and masking requires physical coordinates, the system executes the following matching logic:

[0124] A. Semantic matching: Calculate the semantic similarity between the sensitive text output by the model and the text blocks in the coordinate mapping table (e.g., use cosine similarity to calculate the cosine value of the angle between vectors).

[0125] B. Spatial verification: For candidates with semantic similarity higher than the threshold (e.g., 0.85), further verification is performed to determine whether their location overlaps with the region of interest of the model.

[0126] C. Coordinate merging: The coordinates in the successfully matched coordinate mapping table (for printed text) are merged with the handwritten area coordinates (for handwritten text) output by the handwritten data detection model to remove duplicates, forming the final list of sensitive information coordinates.

[0127] Step S5: Automated masking processing and output.

[0128] Generate a masking layer: Based on the coordinates of the sensitive information determined in step S4, generate an independent transparent masking layer on top of the original engineering drawing.

[0129] Perform masking: Perform pixel processing on the coordinate region. Processing methods include, but are not limited to: solid color block filling (such as a black rectangle), Gaussian blurring, or mosaic processing. In this embodiment, solid color block filling is preferred to ensure that the information is irretrievable.

[0130] File Composition and Output: Combine the masking layer with the original drawing layer to generate a de-identified engineering drawing file (PDF format).

[0131] Manual review (optional): The system provides a visual review interface, displaying comparison images before and after the masking. Staff can manually adjust the position of the masking frame or remove incorrectly masked areas. After confirmation, the sensitive information processing is complete.

[0132] 3. Effect verification and performance testing: In order to verify the effectiveness of the method described in this embodiment, the inventors conducted comparative experiments and ablation experiments.

[0133] Experimental setup: Historical project drawings (842 pages in total) containing 1174 pieces of sensitive information, including handwritten signatures (636), technical parameters (526), ​​and contact information (12), were selected as the test set.

[0134] Control group:

[0135] Method A: Traditional regular expression keyword matching.

[0136] Method B: Single OCR recognition + keyword matching.

[0137] Method C (this application): Lightweight visual model + large language model + RAG knowledge base + two-stream processing.

[0138] Method D (Comparative Example): General multimodal large model (such as Qwen-VL) + large language model + RAG knowledge base + dual-stream processing.

[0139]

[0140] Experimental conclusions: This application (method C) achieves the best results in both sensitive information recall rate and handwritten signature recognition rate. Moreover, thanks to the customized network structure of the lightweight visual model (DRSC), its single-page inference time is much lower than that of the general multimodal large model (method D), which greatly meets the real-time requirements of batch desensitization of engineering drawings.

[0141] like Figure 3 As shown, this invention also provides a system for processing sensitive engineering data based on the collaboration of a lightweight vision and a large language model. This system is used to execute the method described in Embodiment 1. The system is deployed on a server cluster and includes:

[0142] Preprocessing module: Integrates OCR algorithm library and image positioning algorithm, used to receive original engineering drawings, extract text information and position coordinates from the drawings, and generate coordinate mapping table.

[0143] Support Module: This module is the logical collection within the dashed box, containing:

[0144] Model training module: Equipped with GPU computing resources, it stores historical drawing data and sensitive information rules, performs fine-tuning training on the large language model and the lightweight visual model, and generates model weight files. These model weight files are transmitted to the recognition and analysis module via a secure channel.

[0145] Knowledge Base Management Module: Used to store fixed templates for knowledge base files, providing a user interface to support the entry of unique sensitive information rules when creating a new project, and calling the embedded model to generate a unique RAG knowledge base vector index for the project.

[0146] The recognition and analysis module, as the core processing unit of the system, loads the model weight file. It connects to the knowledge base management module and the preprocessing module, respectively. It is used to call the fine-tuned large language model combined with the RAG knowledge base to perform semantic analysis (text stream) on text information, and to call the fine-tuned lightweight visual model to perform high-precision detection of handwritten regions (image stream). After coordinate fusion, it outputs a list of sensitive information coordinates.

[0147] Masking Execution Module: Connected to the recognition and analysis module, it is used to draw masking boxes on the original drawing based on the list of sensitive information coordinates, using an image processing library (such as OpenCV) and perform mosaic processing to output desensitized drawings.

[0148] User / External System Interface: Used to receive the original drawings uploaded by the user and return the final de-identified drawings to the user.

[0149] As a preferred embodiment, the identification and analysis module includes:

[0150] The text stream processing unit calls the fine-tuned large language model and combines it with retrieval enhancement to generate a knowledge base to complete the semantic recognition of sensitive information in the text.

[0151] The image stream processing unit integrates a lightweight DRSC visual model, which sequentially outputs the coordinates of handwritten sensitive information through differentiable binarization, feature extraction, multi-scale branch enhancement, cross-scale fusion, and coordinate regression.

[0152] The coordinate fusion unit, through dual verification of semantic similarity and spatial bounding box overlap rate, merges the coordinates of text-sensitive information and handwritten-sensitive information to generate the final list of sensitive information coordinates.

[0153] As a preferred embodiment, the system further includes a human-computer interaction review module, which is used to display desensitized engineering data, receive user review and adjustment instructions, and feed them back to the masking execution module.

[0154] It is hereby declared that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method and system for processing sensitive information in engineering data, characterized in that, Includes the following steps: S1. Collect historical engineering data and complete the annotation of sensitive information, construct a training dataset, and use the training dataset to fine-tune the pre-trained large language model and lightweight visual model respectively to obtain the text sensitive information detection model and the handwritten data detection model. S2. Based on the preset knowledge base template, input the sensitive information rules and whitelist rules corresponding to the engineering project, vectorize the rule text, and build a project-specific retrieval enhancement generation knowledge base; S3. Obtain the engineering data to be processed, parse it using optical character recognition and image positioning algorithms, extract the text content and corresponding physical coordinates, and generate a coordinate mapping table; S4. Sensitive information identification and location are performed using a parallel processing method of text stream and image stream; The text stream invokes the text sensitive information detection model and combines it with the retrieval enhancement knowledge base to perform semantic analysis and identify text-based sensitive information; The image stream calls the handwritten data detection model to identify handwritten sensitive information and output the corresponding coordinates; through semantic similarity and spatial location dual verification, the sensitive information and coordinate mapping table are matched and fused to obtain a list of coordinates of all sensitive information. After completing the recognition of image stream and text stream, the text-type sensitive information and handwritten sensitive information are matched and fused with the coordinate mapping table through dual verification of semantic similarity and spatial bounding box overlap rate, and after deduplication, a list of coordinates of all sensitive information is obtained. S5. Based on the list of sensitive information coordinates, perform masking processing on the corresponding area of ​​the engineering data and output the desensitized engineering data; Steps S1-S2 are the offline preparation stage, and steps S3-S5 are the online business processing stage.

2. The method for processing sensitive information in engineering data according to claim 1, characterized in that: In step S1, the lightweight visual model is a DRSC model, with ResNet18 as the residual backbone network. When fine-tuning the lightweight visual model, the GIoU loss function is used to optimize the coordinate regression process. The formula for calculating the GIoU loss function is as follows: In the formula: The intersection-union ratio (IoU) of the model's predicted bounding boxes and the ground truth labeled bounding boxes; The area of ​​the smallest closed bounding rectangle that can simultaneously enclose both the predicted bounding box and the ground truth bounding box; This represents the area of ​​the union of the predicted bounding box and the ground truth bounding box. This represents the area of ​​the background region within the smallest bounding rectangle that is not covered by the predicted bounding box or the ground truth bounding box.

3. The method for processing sensitive information in engineering data according to claim 2, characterized in that: In step S2, the knowledge base template includes the project name, confidentiality level, specific sensitive word rules, exception whitelist rules, and sensitive information definition description.

4. The method for processing sensitive information in engineering data according to claim 3, characterized in that, The image stream processing in step S4 involves inputting the engineering data image into the handwritten data detection model and performing the following processing sequentially: I. By predicting the pixel probability map and adaptive threshold map through the differentiable binarization module, a binary mask is generated to filter out the complex background of the engineering data and obtain the handwritten foreground feature map. The differentiable binarization module generates a binary mask according to the following formula: In the formula, For the generated binary mask at pixel position The value at this location is used to filter out the background of the drawing; A probability map of pixels predicted by the network as belonging to handwritten strokes; A dynamic threshold map for network adaptive learning, enabling the model to automatically adjust the binarization threshold based on the depth of the local background of the drawing; A magnification factor to control the degree of binarization approximation; Will Multiplying each element of the original drawing yields the handwritten foreground feature map. ; II. Input the handwritten foreground feature map into the residual backbone network for feature extraction to obtain the initial feature map; III. Input the initial feature map into three different receptive field branches: global feature convolutional layer, local feature convolutional layer, and detail feature convolutional layer, to obtain feature maps of the corresponding scales. Input each scale feature map into the matched global hybrid attention module, local hybrid attention module, and detail hybrid attention module, respectively. Use selective gating mechanism to suppress background noise, complete single-scale feature enhancement, and obtain multi-scale enhanced feature maps. Among them, the global feature convolutional layer, the local feature convolutional layer, and the detail feature convolutional layer all adopt dilated convolutional structure, and each branch is configured with a different dilation rate to distinguish the size of the receptive field. The multi-scale feature extraction formula for the corresponding scale feature map is: In the formula, For the corresponding scale branch in spatial location The output feature map at the location, where ; These are the weight parameters of the convolution kernel; This represents the expansion rate of the corresponding branch; The hybrid attention module sequentially performs mask generation and attention weighting operations, and achieves spatial dimension noise filtering and feature purification through element-wise multiplication. Mask generation formula: Attention-weighted formula: In the formula, For selective gating masks, use 1×1 convolution and the Sigmoid activation function. generate; Each is derived from the input features The query matrix, key matrix, and value matrix obtained through linear mapping; Key matrix The dimension; This indicates element-wise multiplication; The output feature map after attention weighting and gating filtering; IV. The multi-scale enhanced feature maps are concatenated along the channel dimension and then input into the channel position attention module to complete the cross-scale channel and spatial adaptive feature fusion, resulting in the final fused feature map; The channel position attention module calculates the channel weight map and the spatial position weight map respectively, and combines the element-wise multiplication of broadcast to complete the secondary feature enhancement. V. Input the final fused feature map into the coordinate regression module, and output the location coordinates of the handwritten sensitive information.

5. A method for processing sensitive information in engineering data according to claim 4, characterized in that: In step S5, the masking processing method includes any one or more of the following: solid color block filling, Gaussian blur, and mosaic processing.

6. A sensitive information processing system for engineering data, characterized in that: It includes a preprocessing module, a model training module, a knowledge base management module, a recognition and analysis module, and a masking execution module; The model training module is used to store historical engineering data and sensitive information annotation data, fine-tune the pre-trained large language model and lightweight visual model, and output the model weight file. The knowledge base management module is used to store knowledge base templates, input project-specific sensitive information rules and complete vectorization processing, and build and maintain a retrieval-enhanced generated knowledge base. The preprocessing module has built-in optical character recognition algorithm and image positioning algorithm, which are used to parse the engineering data to be processed, extract text content and position coordinates, generate a coordinate mapping table and send it to the recognition and analysis module. The recognition and analysis module is communicatively connected to the preprocessing module, the model training module, and the knowledge base management module, respectively. It loads the model weight file and performs text stream processing and image stream processing in parallel. The recognition and analysis module integrates a differentiable binarization module, a residual backbone network, a multi-scale convolutional branch, a hybrid attention module, a channel position attention module, and a coordinate regression module to complete the detection of handwritten sensitive information. It also combines semantic and spatial dual verification to complete coordinate fusion and output a list of sensitive information coordinates. The masking execution module is connected to the identification and analysis module and is used to mask the corresponding area of ​​the engineering data according to the list of sensitive information coordinates and output desensitized engineering data.

7. The sensitive information processing system for engineering data according to claim 6, characterized in that: The recognition and analysis module includes a text stream processing unit, an image stream processing unit, and a coordinate fusion unit. The text stream processing unit calls the fine-tuned large language model and combines it with the retrieval enhancement to generate a knowledge base to complete the semantic recognition of sensitive information in the text. The image stream processing unit integrates a lightweight DRSC visual model, and outputs the coordinates of handwritten sensitive information through differentiable binarization, feature extraction, multi-scale branch enhancement, cross-scale fusion, and coordinate regression in sequence. The coordinate fusion unit, through dual verification of semantic similarity and spatial bounding box overlap rate, fuses the coordinates of text-sensitive information and handwritten-sensitive information to generate a final list of sensitive information coordinates.

8. The sensitive information processing system for engineering data according to claim 7, characterized in that: The system also includes a human-computer interaction review module, which is used to display desensitized engineering data, receive user review and adjustment instructions, and feed them back to the masking execution module.

Images