Image desensitization method and electronic device based on model fine-tuning

By fine-tuning the large language model and the potential diffusion model, accurate identification and natural redrawing of sensitive information in images were achieved, solving the problems of inaccurate positioning of sensitive areas and destruction of image structure in existing technologies, and ensuring the security and usability of images in the power industry.

CN123116331APending Publication Date: 2026-11-17STATE GRID SHANGHAI MUNICIPAL ELECTRIC POWER CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611370628.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-04
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies often fail to accurately locate sensitive areas during image desensitization, and the structural integrity and visual continuity of the desensitized image are difficult to maintain. This is especially true in the power industry, where variations in equipment models and context-dependent sensitivities are difficult to identify.

Method used

A low-rank adaptation method is used to fine-tune the basic large language model to obtain a first target model with the ability to identify sensitive text information. This model is then combined with a latent diffusion model to perform concept erasure, thereby achieving accurate localization and natural redrawing of sensitive areas.

Benefits of technology

It significantly improves the positioning accuracy of sensitive information, maintains the structural integrity and visual continuity of the desensitized image, and solves the problems of inaccurate positioning of sensitive areas and image damage in existing technologies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116331A_ABST
    Figure CN123116331A_ABST
Patent Text Reader

Abstract

This application proposes an image desensitization method and electronic device based on model fine-tuning, relating to the field of information security technology. The method includes: acquiring an image to be processed; performing text recognition on the image to obtain text content and corresponding text location information; inputting the text content into a first target model for sensitive recognition to obtain sensitive text data; determining sensitive text regions in the image to be processed based on the sensitive text data and text location information; performing target object detection on the image to be processed to obtain sensitive object regions; and inputting the image to be processed, the sensitive text regions, and the sensitive object regions into a second target model, which then redraws the content of the sensitive text regions and sensitive object regions to obtain the desensitized image. This technical solution can accurately locate sensitive information in an image while maintaining the structural integrity and visual continuity of the desensitized image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of information security technology, and more specifically, to an image desensitization method and electronic device based on model fine-tuning. Background Technology

[0002] With the rapid development of artificial intelligence technology, image data, as an important information carrier, is widely used in business scenarios such as equipment inspection, on-site operation recording, and security monitoring. Image data desensitization refers to the process of identifying and removing sensitive information (such as faces, equipment nameplates, and geographic location tags) contained in images. Its purpose is to reduce the risk of privacy leakage in image data while ensuring the usability of the images.

[0003] Currently, image desensitization primarily employs a combination of rule-based text desensitization and image blurring. Specifically, existing solutions typically first perform optical character recognition (OCR) on the image, then match the identified text content with a pre-defined sensitive word library using keywords or regular expressions to locate sensitive text regions in the image. Simultaneously, face detection algorithms are used to locate face regions in the image. Finally, both the located sensitive text and face regions are uniformly masked using mosaic or Gaussian blur.

[0004] However, keyword matching and regular expression matching can only identify sensitive words that are completely identical to those in the preset thesaurus. They cannot identify variations such as device model variants or line abbreviations, nor can they determine the sensitivity of the same word in different business scenarios, resulting in low accuracy in locating sensitive areas. Furthermore, the located sensitive areas are uniformly covered using destructive processing methods such as mosaic or blurring, leading to obvious regional fragmentation and content discontinuity in the desensitized image, thus destroying the structural integrity and visual continuity of the image.

[0005] Therefore, how to accurately locate sensitive information in an image while maintaining the structural integrity and visual continuity of the desensitized image has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The purpose of this application is to provide an image desensitization method and electronic device based on model fine-tuning, which can accurately locate sensitive information in an image while maintaining the structural integrity and visual continuity of the desensitized image.

[0007] This application is implemented as follows: In a first aspect, this application provides an image desensitization method based on model fine-tuning, comprising the following steps: acquiring an image to be processed; performing text recognition on the image to be processed to obtain text content and corresponding text location information; inputting the text content into a first target model for sensitive recognition to obtain sensitive text data, wherein the first target model is a model with the ability to recognize sensitive text information obtained by pre-adjusting a basic large language model using a low-rank adaptation method; determining sensitive text regions in the image to be processed based on the sensitive text data and text location information; performing target object detection on the image to be processed to obtain sensitive object regions; inputting the image to be processed, sensitive text regions, and sensitive object regions into a second target model, wherein the second target model redraws the content of the sensitive text regions and sensitive object regions to obtain a desensitized image, wherein the second target model is a model that loses the ability to generate sensitive visual concepts obtained by pre-adjusting a basic latent diffusion model using a concept erasure method.

[0008] In some implementations, the first target model is obtained by: acquiring a training dataset, which includes instruction data, input text data, and expected output data, wherein the expected output data is used to indicate whether the input text data contains sensitive information and the content of the sensitive information; freezing the original weight matrix of the basic large language model and injecting a low-rank matrix into the attention layer and / or feedforward network layer of the basic large language model; updating the parameters of the low-rank matrix with the training dataset and fine-tuning the instruction with the expected output data as a supervision signal to obtain the first target model, wherein the instruction fine-tuning is to calculate the loss only for the expected output data, and the losses of the instruction data and the input text data are masked by a mask matrix so that the basic large language model learns to generate answers based on instructions rather than memorizing the input text.

[0009] In some implementations, low-rank matrices are injected into the attention layer and / or feedforward network layer of the underlying large language model, including: injecting low-rank matrices into at least one of the query projection matrix, key projection matrix, value projection matrix, and output projection matrix of the attention layer of the underlying large language model; and / or, injecting low-rank matrices into at least one of the gating matrix, up projection matrix, and down projection matrix of the feedforward network layer of the underlying large language model.

[0010] In some implementations, the second target model is obtained as follows: a basic latent diffusion model and a set of sensitive concepts to be erased are obtained; target parameters to be fine-tuned are determined in the cross-attention layer of the basic latent diffusion model; for each target concept in the set of sensitive concepts, target anchor concepts corresponding to the target concepts are determined in a pre-defined anchor concept search space; a loss function is constructed based on the target concepts and target anchor concepts, the loss function constraining the output of the fine-tuned basic latent diffusion model under the target concepts to approach the output of the basic latent diffusion model under the target anchor concepts; the target parameters are updated with the goal of minimizing the loss function, thus obtaining the second target model.

[0011] In some implementations, the pre-defined anchor concept search space is constructed as follows: a pre-defined image dataset is obtained, and the embedding vectors of each image sample in the pre-defined image dataset are extracted; the distance between the embedding vector of each image sample and the embedding vector of the target concept is calculated; the concepts corresponding to the K nearest image samples are selected as the anchor concept candidate set corresponding to the target concept, where K is a positive integer.

[0012] In some implementations, text recognition is performed on the image to be processed to obtain the text content and the corresponding text location information. This includes: calling an optical character recognition engine to perform text detection and recognition on the image to be processed, and obtaining at least one text detection box and the text content and pixel coordinate information corresponding to each text detection box.

[0013] In some implementations, target object detection is performed on the image to be processed to obtain sensitive object regions. This includes: performing face detection on the image to be processed to obtain face detection results, which include face location information and face attribute information, including face confidence, face rotation angle, and the proportion of face area to the entire image; determining sensitive faces from the face detection results based on preset privacy judgment rules and face attribute information; and generating sensitive object regions based on the face location information corresponding to the sensitive faces.

[0014] In some implementations, the steps of inputting the image to be processed, the sensitive text region, and the sensitive object region into a second target model, and having the second target model redraw the content of the sensitive text region and the sensitive object region, include: generating a text mask image based on the sensitive text region, and generating an object mask image based on the sensitive object region, wherein the text mask image and the object mask image have a first pixel value in their respective corresponding regions and a second pixel value in other regions; performing a logical OR operation on the text mask image and the object mask image to obtain a target mask image; and inputting the image to be processed and the target mask image together into the second target model, whereby the second target model only redraws the content of the regions in the target mask image that have the first pixel value.

[0015] In some implementations, this model-based fine-tuning image desensitization method further includes the following steps: performing text recognition and sensitivity recognition on the desensitized image again to obtain desensitized sensitive text data; performing target object detection on the desensitized image again to obtain desensitized sensitive object data; and determining the desensitization effect evaluation result based on the differences between the sensitive text data and the desensitized sensitive text data, as well as the differences between the sensitive object data and the desensitized sensitive object data.

[0016] In a second aspect, this application provides an electronic device including a memory for storing one or more programs; a processor; and, when the one or more programs are executed by the processor, implementing the method as described in any one of the first aspects above.

[0017] Compared with the prior art, this application has at least the following advantages or beneficial effects: This application decouples the identification and removal of sensitive information in an image into two independent stages. It leverages the semantic understanding capabilities of a large language model to improve the accuracy of sensitive information localization, and leverages the image generation capabilities of a latent diffusion model to improve the visual effect after sensitive area removal.

[0018] Specifically, this application replaces the traditional keyword matching method with the semantic understanding capability of the first target model, which can identify variant expressions and context-related sensitive information, thereby significantly improving the localization accuracy of sensitive areas. At the same time, the directional redrawing of the second target model replaces the traditional mosaic or blurring masking method, so that the filling content of the sensitive area is naturally connected with the surrounding image in terms of texture, color and structure, effectively avoiding the problems of image area fragmentation and content discontinuity. While ensuring that sensitive information is effectively removed, the overall structural integrity and visual continuity of the image are maintained. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of an embodiment of an image desensitization method based on model fine-tuning according to this application; Figure 2 This is a flowchart illustrating the specific method for obtaining the first target model in one embodiment of this application; Figure 3 This is a flowchart illustrating the specific method for obtaining the second target model in one embodiment of this application; Figure 4This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0021] Icons: 201, Processor; 202, Memory; 203, Communication Interface. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0023] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0024] To facilitate understanding of the technical solutions provided in this application, some concepts will be introduced below.

[0025] 1. Low-rank adaptation method Low-rank adaptation is an existing model fine-tuning technique that injects a low-rank decomposition matrix next to the original weight matrix of the basic large language model to simulate the weight update amount. During training, only the parameters of the injected low-rank matrix are updated, while the original weight matrix is ​​frozen and remains unchanged. Thus, it achieves a model adaptation effect similar to full fine-tuning of a large language model with a very low number of training parameters.

[0026] 2. Concept erasure method Concept erasure is an existing targeted fine-tuning technique for diffusion models. It designs a loss function to guide the diffusion model so that when it receives a textual prompt corresponding to the target concept, its output approximates the output generated when it receives an anchor concept, thus causing the diffusion model to lose its ability to generate visual content related to the target concept. For example, during training, "face" is used as the target concept to be erased, and "landscape" is used as the anchor concept. Adversarial optimization is used to make the model's output when the prompt "face" is input approximate the output when "landscape" is input. Simultaneously, adversarial search is performed on the selection of anchor concepts to ensure that the anchor concepts are not synonyms of the target concept. This maintains the quality of the diffusion model's generation of other concepts while erasing sensitive concepts.

[0027] 3. Potential diffusion model Latent diffusion models are image generation models that perform denoising in a low-dimensional latent space. They use variational autoencoders to compress the image into a low-dimensional latent feature representation, perform an iterative denoising process within this latent space, and then use a decoder to restore the denoised latent features to a pixel-space image, thereby generating high-quality images with low computational cost. For example, the Stable Diffusion model is a commonly used latent diffusion model.

[0028] 4. Basic Large Language Model The basic large language model refers to a large language model that has been pre-trained on a general large-scale corpus without domain-adaptive fine-tuning. The term "basic" here means that the model has not been specifically fine-tuned for the sensitive information identification task of this application, but is only used as the initial weight source for subsequent low-rank adaptation fine-tuning.

[0029] 5. Basic Potential Diffusion Model The basic latent diffusion model refers to a latent diffusion model that has been pre-trained on a general image dataset without concept erasure fine-tuning. The term "basic" here means that the model has not been specifically fine-tuned for the sensitive visual concept erasure task of this application, and still retains the complete ability to generate various visual concepts such as faces, but only serves as the initial weight source for subsequent concept erasure fine-tuning.

[0030] With the rapid development of artificial intelligence (AI) technology, image data, as a crucial information carrier, is widely used in business scenarios such as equipment inspection, on-site operation recording, and security monitoring. Taking the power industry as an example, as a vital foundational industry related to national energy security and the operation of the national economy, the power industry is actively promoting the deep integration of AI technology with business scenarios. In areas such as equipment operation and maintenance, production scheduling management, customer service, and knowledge management, large-scale modeling technology can effectively improve information processing efficiency and decision support. However, power industry data has higher sensitivity and security requirements. When large-scale models are used for inference, sensitive information such as equipment parameters, power grid structure, and operating status may be explicitly or implicitly included in the input. When using on-site images and business data to train models or build knowledge bases, the sensitive content may be reproduced in subsequent generation processes. Effectively processing sensitive information to achieve secure access, secure sharing, and secure utilization of data, supporting the compliant development of AI applications, has become a critical issue that urgently needs to be addressed in the digital transformation of the power industry.

[0031] Therefore, it is necessary to use image data desensitization technology to desensitize images. Specifically, this refers to the process of identifying and removing sensitive information (such as faces, device nameplates, geolocation markers, etc.) contained in images. The purpose is to reduce the risk of privacy leakage in image data while ensuring the usability of the images.

[0032] Currently, existing technologies for desensitizing images typically involve first using Optical Character Recognition (OCR) to extract the text content from the image, then locating sensitive text regions through keyword matching or regular expression matching, and finally combining this with face detection to locate sensitive object regions. These regions are then uniformly processed by mosaic or blurring.

[0033] However, on the one hand, keyword matching and regular expression matching are literal-level hard matching, lacking semantic understanding capabilities. They cannot identify variant expressions or judge the strength of sensitivity in conjunction with context, thus resulting in significantly insufficient accuracy in locating sensitive text regions. Taking the power industry as an example, existing keyword matching and regular expression matching solutions are relatively stable in recognizing information with fixed formats, but they are prone to missed recognition when faced with equipment model variations, line abbreviations, and noisy text generated by image OCR. Rule matching lacks semantic understanding capabilities and cannot utilize implicit information in the context. Sensitivity in power business is often not determined by a single field, but by a combination of "field + context + business scenario." For example, the same equipment number may be risk-free in publicly available training materials, but may constitute critical infrastructure information in fault analysis, operating mode, or geographic location context. Existing methods also struggle to keep up with the continuous evolution of new equipment, new projects, new naming habits, and new attack methods, leading to high rule maintenance costs and delayed updates.

[0034] On the other hand, mosaic or blurring are globally destructive masking methods that fail to consider the semantic continuity and texture consistency of image content, resulting in visual content fragmentation in the desensitized image. Taking the power industry as an example, photos of power equipment inspections often contain sensitive content such as equipment nameplates, geographical location markers, and facial information of workers. Simply adding mosaics makes it difficult to achieve a natural transition of image content, leading to obvious regional fragmentation and content fragmentation in the desensitized image, thus disrupting the structural integrity and visual continuity of the image.

[0035] Based on the above analysis, this application provides an image desensitization method based on model fine-tuning. This method primarily employs a low-rank adaptation method to fine-tune a basic large language model, obtaining a first target model with semantic understanding capabilities to replace traditional keyword matching for sensitive text recognition. This leverages the semantic understanding capabilities of the large language model to achieve accurate location of sensitive text. Furthermore, a concept erasure method is used to fine-tune a basic latent diffusion model, obtaining a second target model that loses the ability to generate sensitive visual concepts. This model then performs targeted redrawing of sensitive areas, utilizing the image generation capabilities of the latent diffusion model to achieve semantic-level restoration of sensitive areas. This removes sensitive information while maintaining the overall structural integrity and visual continuity of the image. Therefore, this application can accurately locate sensitive information in an image while maintaining the structural integrity and visual continuity of the desensitized image.

[0036] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0037] Please see Figure 1 The model-based image desensitization method includes the following steps: Step S101: Obtain the image to be processed.

[0038] It should be noted that the image to be processed can be obtained from local storage, network storage, or by reading the original image data to be de-identified from an image acquisition device. The image to be processed serves as the foundational data source for subsequent text recognition and target object detection, and is the input object for the entire de-identification process.

[0039] Step S102: Perform text recognition on the image to be processed to obtain the text content and the corresponding text location information.

[0040] When performing text recognition on images, an Optical Character Recognition (OCR) engine can be used to perform text detection and recognition operations. The OCR engine scans the image region by region, detecting areas containing text, recognizing the text content in each region, and recording the pixel coordinates of each text element within the image. This converts the text information in the image into structured text data that can be processed by a computer, while simultaneously preserving the spatial location information of each text element, providing data support for subsequent semantic recognition and spatial localization of sensitive information.

[0041] Step S103: Input the text content into the first target model for sensitive identification to obtain sensitive text data. The first target model is a model with the ability to identify sensitive text information, which is obtained by fine-tuning the basic large language model in advance using the low-rank adaptation method.

[0042] It should be noted that the first target model can utilize the semantic understanding capabilities of the large language model itself, combined with contextual information, to determine the sensitivity of each field in the text content, identify text fragments belonging to the sensitive information category, and output sensitive text data in the form of structured data. Specifically, during the inference phase, the first target model uses its self-attention mechanism to capture the global dependencies between each word and other words in the text content, and combines this with domain knowledge learned in the fine-tuning phase to dynamically calculate the probability distribution of each word belonging to the sensitive category, ultimately outputting the sensitive text data in a structured manner. That is, step S103 uses a finely tuned large language model to replace the traditional keyword matching method, elevating the identification of sensitive information from the literal matching level to the semantic understanding level. This effectively identifies variant expressions and context-related sensitivities, thereby significantly improving the accuracy of sensitive text identification.

[0043] Step S104: Determine the sensitive text region in the image to be processed based on the sensitive text data and text location information.

[0044] Step S104 associates the sensitive text data output in step S103 with the text location information output in step S102. Specifically, the system built using the image desensitization method of this application maintains a one-to-one correspondence between the text content and text location information established in step S102. When step S103 returns the sensitive text data, the system iterates through each sensitive term in the data, using it as an index to find the corresponding relationship and lock its precise bounding box in the image to be processed, thereby obtaining the sensitive text region. That is, step S104 completes the mapping from the sensitive text recognition result to the image pixel space, providing precise spatial constraints for subsequent targeted redrawing.

[0045] Step S105: Perform target object detection on the image to be processed to obtain the sensitive object region.

[0046] Step S105, independent of the text processing path (steps S102 to S104), detects and locates non-textual sensitive objects in the image that may carry privacy information. This allows the desensitization operation to cover sensitive information types that text recognition cannot reach, such as faces, thus compensating for the coverage blind spots of the pure text recognition path. Specifically, a convolutional neural network can be used to perform forward inference on the image to be processed, extracting high-level semantic feature maps. Through a region proposal network and a classification regression branch, the category labels and corresponding bounding box coordinates of all potential targets in the image are output. Sensitive targets are retained according to preset sensitive category filtering rules, and finally, the sensitive object regions are output.

[0047] It should be noted that after acquiring the image to be processed and before inputting it into the second target model, the image desensitization method of this application executes two processing paths in parallel or sequentially: one processing path is steps S102 to S104, used to identify sensitive text regions in the image to be processed; the other processing path is step S105, used to identify sensitive object regions in the image to be processed. It is understood that steps S105 and S102 do not depend on each other's outputs, therefore they can be executed synchronously or asynchronously in any order, and this application does not impose any limitations on this.

[0048] Step S106: Input the image to be processed, the sensitive text region, and the sensitive object region into the second target model. The second target model redraws the content of the sensitive text region and the sensitive object region to obtain the desensitized image. The second target model is a model that loses the ability to generate sensitive visual concepts after the basic latent diffusion model is fine-tuned by the concept erasure method.

[0049] Understandably, during inference, the second target model identifies the locations of sensitive text and object regions within the image to be processed. It uses the latent features of the image as the initial background and iteratively performs the denoising generation process only within the regions indicated by the sensitive text and object regions, regenerating the pixel content within those regions while preserving the image content outside these regions. Since the second target model has had its ability to generate sensitive visual concepts erased during fine-tuning, the redrawn content does not contain sensitive information. It should be noted that the architecture of general latent diffusion models allows text, images, edge maps, depth maps, etc., to be injected as conditions into the denoising process. Some models, trained only with text pairs, may appear to only be able to generate images from text, but in reality, general models like Stable Diffusion explicitly introduce image conditions during training or inference, thus supporting image-to-image generation.

[0050] In other words, step S106 uses the image generation capability of the latent diffusion model to replace traditional mosaic or blurring processing, so that the filling content of the sensitive area transitions naturally with the surrounding area in terms of texture, color and structure, effectively removing sensitive information while maintaining the overall structural integrity and visual continuity of the image.

[0051] In summary, this application decouples the identification and removal of sensitive information in an image into two independent stages. It leverages the semantic understanding capabilities of a large language model to improve the accuracy of sensitive information localization, and utilizes the image generation capabilities of a latent diffusion model to improve the visual effect after sensitive area removal. Specifically, this application replaces the traditional keyword matching method with the semantic understanding capabilities of the first target model, enabling the identification of variant expressions and context-dependent sensitive information, thereby significantly improving the localization accuracy of sensitive areas. Simultaneously, it replaces the traditional mosaic or blurring masking methods with the directional redrawing of the second target model, allowing the filled content of the sensitive area to naturally connect with the surrounding image in terms of texture, color, and structure. This effectively avoids the problems of image region fragmentation and content discontinuity, ensuring that sensitive information is effectively removed while maintaining the overall structural integrity and visual continuity of the image.

[0052] For example, in practical applications, the image desensitization method provided in this application can be executed according to the following exemplary process.

[0053] After fine-tuning the basic large language model and the basic latent diffusion model, the system framework is complete. For text data T in the image to be processed (denoted as original image data I), text data T is sent to the fine-tuned first target model, which returns a set of sensitive text fragments that reveal privacy, Sensitive_Words. Then, the sensitive text fragments in text data T are masked to obtain the desensitized result. .

[0054] For RGB format image data with height H and width W First, PaddleOCR (an open-source OCR library) is used to extract the text and the pixel coordinates of the text boxes in the image. ,in The text data T is sent to the fine-tuned first target model, which returns a set of sensitive words, Sensitive_Words. This determines the pixel coordinates of the sensitive text in the image. Each sensitive text box corresponds to a vector. :

[0055] The framework then calls the InsightFace framework (an open-source face recognition toolkit) to identify face tuples in the image. ,in For facial recognition, , , , These are the pixel coordinates corresponding to the face. For confidence level, The face is the angle. A set of Sensitive Faces, which reveals sensitive facial information, is extracted using judgment rules. Each sensitive face bounding box corresponds to a vector. :

[0056] Generate the text mask image words_mask and the object mask image face_mask:

[0057]

[0058] The framework sends the original image data I, the text mask image words_mask, and the object mask image face_mask to the fine-tuned second target model. The second target model only redraws the areas with a pixel value of 1 in the mask image, thus achieving image desensitization.

[0059] Based on the aforementioned solution, please refer to Figure 2 In some implementations of this application, the first target model is obtained in the following ways: Step S201: Obtain a training dataset, which includes instruction data, input text data, and expected output data. The expected output data is used to indicate whether the input text data contains sensitive information and the content of the sensitive information. Step S202: Freeze the original weight matrix of the basic large language model and inject a low-rank matrix into the attention layer and / or feedforward network layer of the basic large language model. Step S203: Update the parameters of the low-rank matrix with the training dataset and perform instruction fine-tuning with the expected output data as a supervision signal to obtain the first target model. Here, the instruction fine-tuning is to calculate the loss only for the expected output data. The losses of the instruction data and the input text data are masked by a mask matrix so that the basic large language model learns to generate answers based on instructions rather than memorizing the input text.

[0060] In this implementation, a training dataset is first acquired. Each sample in this dataset consists of three parts: instruction data, input text data, and expected output data. Instruction data informs the model of the task type to be performed. Input text data is the original text to be identified for sensitive information. Expected output data indicates whether the input text contains sensitive information and the specific content of that information. During training, the original weight matrix of the basic large language model is frozen and kept unchanged. Only a low-rank matrix is ​​injected into the attention layer or feedforward network layer of the basic large language model, and only the parameters of this low-rank matrix are updated during training. Simultaneously, during instruction fine-tuning, the loss is calculated only for the expected output data; the losses for the instruction data and input text data are masked using a mask matrix. The purpose of this is to teach the model how to generate correct answers based on instructions, rather than simply memorizing the input text itself. In other words, the model is trained to learn the ability to "distinguish sensitive information," rather than merely memorizing a specific piece of input text.

[0061] Through the above method, this implementation obtains a first target model with the ability to identify sensitive information in text. This first target model can identify sensitive information in any input text according to instructions. Since only the low-rank matrix is ​​updated during training, rather than all parameters of the first target model, the number of training parameters is significantly reduced, and the demand for computing resources is also significantly reduced, making it possible to complete domain adaptation of the first target model for basic large languages ​​under limited hardware conditions. At the same time, since the loss of instruction data and input text data is masked, the first target model does not over-memorize the specific input content in the training samples, but truly learns how to perform the sensitive information identification task according to instructions, thus ensuring that the first target model also has good generalization recognition ability for unseen text content.

[0062] It should be noted that this implementation involves fine-tuning the basic large language model. The fine-tuned first target model can make full use of the context of the input data to achieve semantic understanding, identify sensitive information in specific scenarios (such as power scenarios), and perform preliminary desensitization of text information.

[0063] Specifically, this implementation uses the Low-Rank Adaptation (LoRA) method to fine-tune the instructions of the basic large language model. During training, for training sample triples... The training objective is to maximize the conditional log-likelihood. .in, C represents instruction data. For input text data, Here, θ represents the desired output data, and θ represents the model parameters. Fine-tuning instructions apply only to the desired output portion. Calculate the loss, instruction data and input data Part of the loss is masked by a mask matrix, a mechanism that ensures the base language model learns how to generate responses based on instructions, rather than simply memorizing the input text itself. The low-rank adaptation method significantly reduces the number of trainable parameters by performing low-rank decomposition on the weight update matrix. For any linear layer weight in the base language model... , The weight matrix has its parameter update amount constrained to be... .in , ,rank , For the input dimension of the linear layer, For the output dimension of the linear layer, For rank, Representing the set of real numbers, the forward propagation process is as follows: ,in For scaling hyperparameters, take The finely tuned first target model can not only desensitize ordinary text fragments, but also identify sensitive information in text in image OCR, providing support for multimodal desensitization.

[0064] Based on the aforementioned scheme, in some implementations of this application, low-rank matrices are injected into the attention layer and / or feedforward network layer of the basic large language model, including: injecting low-rank matrices into at least one of the query projection matrix, key projection matrix, value projection matrix and output projection matrix of the attention layer of the basic large language model; and / or, injecting low-rank matrices into at least one of the gating matrix, up projection matrix and down projection matrix of the feedforward network layer of the basic large language model.

[0065] It should be noted that the attention layer is responsible for handling the dependencies between different positions in the input sequence, while the feedforward network layer is responsible for independently transforming the features at each position. These two layers play different functional roles in the basic large language model. By selecting injection positions from multiple projection matrices in the attention and feedforward network layers, the combination of injection positions in the low-rank matrix can be flexibly configured according to the needs of specific fine-tuning tasks. This allows the basic large language model to simultaneously adjust multiple aspects such as attention calculation and feature transformation when adapting to sensitive information recognition tasks, thereby improving the accuracy of the primary target model in recognizing various variant expressions and context-related sensitive information in power scenarios.

[0066] Based on the aforementioned solution, please refer to Figure 3In some implementations of this application, the second target model is obtained in the following ways: Step S301: Obtain the basic latent diffusion model and the set of sensitive concepts to be erased; Step S302: Determine the target parameters to be fine-tuned in the cross-attention layer of the basic latent diffusion model; Step S303: For each target concept in the set of sensitive concepts, determine the target anchor concept corresponding to the target concept in the preset anchor concept search space; Step S304: Construct a loss function based on the target concept and the target anchor concept. The loss function is used to constrain the output of the fine-tuned basic latent diffusion model under the target concept to approach the output of the basic latent diffusion model under the target anchor concept; Step S305: Update the target parameters with the goal of minimizing the loss function to obtain the second target model.

[0067] In this implementation, the basic latent diffusion model and a set of sensitive concepts to be erased are first obtained. These sensitive concepts may include faces, specific device nameplate text styles, etc. Then, the target parameters to be fine-tuned are determined in the cross-attention layer of the basic latent diffusion model. This layer is responsible for converting text prompts into image features and is a key step in controlling the correspondence between the generated content and the text prompts. Next, for each target concept in the sensitive concept set, a target anchor concept corresponding to the target concept is determined in a pre-defined anchor concept search space. This anchor concept search space can contain multiple semantically similar but non-sensitive concepts to the target concept; the most suitable one is selected as the target anchor concept. After determining the target anchor concept, a loss function is constructed based on the target concept and the target anchor concept. This loss function is used to constrain the output of the fine-tuned basic latent diffusion model under the target concept to approach the output of the basic latent diffusion model under the target anchor concept. Finally, with the goal of minimizing the loss function, the determined target parameters are updated to obtain the second target model.

[0068] This implementation maintains the basic latent diffusion model's ability to generate other irrelevant concepts by determining the target parameters to be fine-tuned only in the cross-attention layer, rather than updating all model parameters. Simultaneously, by introducing an anchored concept search space to determine the target anchored concept, the loss function construction has optional optimization directions, avoiding the degradation in generation quality caused by simply mapping the target concept to random concepts. This allows the second target model to maintain the generation quality for other non-sensitive visual concepts while erasing sensitive visual concepts.

[0069] It should be noted that the principle behind the above concept erasure and fine-tuning process is as follows: For image desensitization, a fine-tuning of the basic latent diffusion model (such as the Stable Diffusion model) is performed to erase the model's ability to generate sensitive concepts within the target scene (e.g., an electricity scene). Regenerating images containing sensitive information using the fine-tuned basic latent diffusion model can remove the sensitive information from the image, while achieving a more natural transition compared to directly adding mosaic effects. It is in the Noisy image representation at each diffusion time step Representing the basic potential diffusion model Under conditions The noise to be predicted, Representing the basic potential diffusion model Noise predicted without conditions, To guide the strength, the goal of fine-tuning is to improve the underlying potential diffusion model after fine-tuning. The noise prediction becomes , This represents the fine-tuned underlying latent diffusion model, where the model forgets the concepts. and the concept Mapping to Concepts Above. Among them, the concept... These are the sensitive concepts involved in the power scene images. This refers to anchor concepts. Using the Adaptive Guided Erasure algorithm to find the optimal anchor concepts can effectively reduce the impact of concept erasure on the quality of the images generated by the model. Solve the following problem in the search space C:

[0070] The parameters in the above formula are defined as follows: : The fine-tuned underlying potential diffusion model, i.e. the target model that is currently being trained and whose parameters will change. The basic potential diffusion model is the original, frozen, and parameter-unchanged basic model. Anchor concept: A variable in the search space C, representing a non-sensitive concept used to guide the output direction of the fine-tuned model when it receives a sensitive concept. C: The search space of anchor concepts, i.e., the set of all candidate anchor concepts. : Noise in the model prediction. E: Expectation. : The fine-tuned basic potential diffusion model Upon receiving the target concept and noisy images The noise predicted in time. Original basic potential diffusion model Upon receiving the anchoring concept and noisy images The noise predicted in time. : The fine-tuned basic potential diffusion model Upon receiving the anchoring concept e and the noisy image The noise predicted in time. : Weighting coefficient, used to balance the importance of the L1 and L2 losses. ||·||2²: Squared L2 norm, used to measure the Euclidean distance between two vectors.

[0071] Among them, the inner max pair The optimization aims to find, within the search space C, the anchor concept that achieves the optimal overall effect of L1 (the output difference between the target concept and the anchor concept) and L2 (the output consistency of the anchor concept itself). ; outer layer min pair The purpose of optimization is to update the fine-tuned model parameters after selecting the anchor concept, so that the model's output under the target concept is close to the output of the original model under the anchor concept, thereby erasing sensitive concepts.

[0072] That is, this concept of erase fine-tuning is... Minimize target This ensures that the fine-tuned underlying potential diffusion model is consistent with the target concept. The output on the above is close to the basic latent diffusion model in anchoring concepts. The output above, thereby achieving the target concept The erasure. At the same time, for Minimize target Ensure that the two models have the same input concepts The outputs remain similar across layers, thus preserving the generative capacity of the underlying latent diffusion model on other concepts. The outer minimization problem applies to the fine-tuned parameters of the underlying latent diffusion model. Optimization is performed to simultaneously erase the target concept and retain the functionality of other concepts. On the other hand, for... Maximize the goal Ensure that the optimal solution is not the target concept. Synonyms; and for Maximize the goal Then, we find the concept most sensitive to the change in model parameters θ→θ, which helps to identify the target concept. The most closely related concept.

[0073] It should be noted that the conditions This refers broadly to all textual guidance information relied upon in the diffusion model's denoising process, the content of which changes with the input prompts; while the concept... This specifically refers to sensitive concepts to be erased in this application (such as "face"). The use of the same letter for both is merely a simplified writing convention in formula derivation, and those skilled in the art can clearly distinguish them in context, without affecting the clear understanding of the technical solution.

[0074] Based on the aforementioned scheme, in some implementations of this application, the preset anchor concept search space is constructed in the following way: obtain a preset image dataset, extract the embedding vector of each image sample in the preset image dataset; calculate the distance between the embedding vector of each image sample and the embedding vector of the target concept; select the concepts corresponding to the K closest image samples as the anchor concept candidate set corresponding to the target concept, where K is a positive integer.

[0075] Understandably, this implementation selects multiple concepts from a pre-defined image dataset that are closest to the target concept in the embedding space, forming a candidate set of anchor concepts. This provides an optional search range for subsequently determining the target anchor concept. Specifically, first, a pre-defined image dataset is obtained, containing a large number of image samples, each corresponding to a concept. The embedding vectors of each image sample in the pre-defined image dataset are extracted, along with the embedding vector of the target concept. Then, the distance between the embedding vector of each image sample and the embedding vector of the target concept is calculated; this distance measures the semantic similarity between the two concepts. Finally, the concepts corresponding to the K closest image samples are selected as the candidate set of anchor concepts corresponding to the target concept, where K is a positive integer, and its value can be set according to actual needs.

[0076] In other words, this implementation calculates the distance in the embedding space and selects multiple concepts that are closest to the target concept as candidates, so that the anchoring concepts in the candidate set have a high semantic similarity to the target concept. This allows for selection from semantically similar concepts when determining the target anchoring concept in the future, which helps to reduce the negative impact of concept erasure on the generation quality of the basic latent diffusion model on other related concepts.

[0077] Based on the aforementioned scheme, in some implementations of this application, text recognition is performed on the image to be processed to obtain the text content and the corresponding text location information, including: calling an optical character recognition engine to perform text detection and recognition on the image to be processed, and obtaining at least one text detection box and the text content and pixel coordinate information corresponding to each text detection box.

[0078] Specifically, this implementation first calls an optical character recognition engine to process the image to be processed. The engine first performs text detection, locating regions containing text in the image and generating at least one text detection box, with each text detection box corresponding to a text region in the image. Then, the engine performs recognition on the text within each text detection box, converting the visual text into computer-readable text content, and simultaneously outputting the pixel coordinates of each text detection box in the image coordinate system as the text position information corresponding to the text content.

[0079] Based on the aforementioned scheme, in some implementations of this application, target object detection is performed on the image to be processed to obtain sensitive object regions, including: performing face detection on the image to be processed to obtain face detection results, the face detection results including face location information and face attribute information, the face attribute information including face confidence, face rotation angle, and the proportion of face area to the entire image; determining sensitive faces from the face detection results according to preset privacy judgment rules and face attribute information; and generating sensitive object regions according to the face location information corresponding to the sensitive faces.

[0080] Among them, face confidence score is used to characterize the credibility of the detected face as a real face, face deflection angle is used to characterize the degree of deflection of the face relative to the frontal direction, and face area ratio of the whole image is used to characterize the visual size of the face in the image. The preset privacy judgment rule can be set based on one or more of the face confidence score threshold, face deflection angle threshold, and face area ratio threshold.

[0081] Compared to the approach of indiscriminately desensitizing all detected faces, this implementation method introduces multi-dimensional facial attribute information such as face confidence, face rotation angle, and the proportion of face area in the entire image. Combined with preset privacy judgment rules, it can effectively distinguish the degree of facial privacy risk in different scenarios, avoid unnecessary desensitization of non-sensitive faces or low-quality detection results, and thus reduce excessive image processing while ensuring privacy protection.

[0082] Based on the aforementioned scheme, in some implementations of this application, the step of inputting the image to be processed, the sensitive text region, and the sensitive object region into a second target model, and having the second target model redraw the content of the sensitive text region and the sensitive object region, includes: generating a text mask image based on the sensitive text region, and generating an object mask image based on the sensitive object region, wherein the text mask image and the object mask image have a first pixel value in their respective corresponding regions and a second pixel value in other regions; performing a logical OR operation on the text mask image and the object mask image to obtain a target mask image; and inputting the image to be processed and the target mask image together into the second target model, wherein the second target model only redraws the content of the regions in the target mask image that have the first pixel value.

[0083] This implementation identifies sensitive regions in the image to be processed by generating text and object masks. The two masks are then merged into a target mask, enabling the second target model to accurately locate and redraw only the sensitive regions based on the target mask. Specifically, a text mask is first generated based on the sensitive text regions. This mask is a binary image with the same size as the image to be processed, containing a first pixel value at the pixel location corresponding to the sensitive text region and a second pixel value at other locations. Similarly, an object mask is generated based on the sensitive object regions, containing a first pixel value at the pixel location corresponding to the sensitive object region and a second pixel value at other locations. Then, a logical OR operation is performed on the text and object masks to merge them into a single target mask. In the target mask, any location marked as a sensitive region in either the text or object mask is marked as the region requiring redrawing. Finally, the image to be processed and the target mask image are input into the second target model. The second target model identifies the regions in the target mask image that have the first pixel value and only redraws the pixel content in these regions, while keeping the regions with the second pixel value unchanged.

[0084] In other words, this implementation merges the text mask image and the object mask image into a single target mask image. This allows the second target model to receive only one mask image as input and simultaneously obtain information about all regions that need to be redrawn, avoiding the additional computational overhead of calling the model multiple times. Furthermore, the second target model only redraws the regions indicated by the mask image, ensuring that the content of non-sensitive areas in the image to be processed is completely preserved. This approach maximizes the retention of usable information in the original image while removing sensitive information.

[0085] Based on the aforementioned scheme, in some implementations of this application, the model-based image desensitization method further includes the following steps: performing text recognition and sensitivity recognition on the desensitized image again to obtain desensitized sensitive text data; performing target object detection on the desensitized image again to obtain desensitized sensitive object data; and determining the desensitization effect evaluation result based on the differences between the sensitive text data and the desensitized sensitive text data, as well as the differences between the sensitive object data and the desensitized sensitive object data.

[0086] This implementation method performs sensitivity recognition and target object detection on the desensitized image again, and compares the desensitized sensitive text data and sensitive object data obtained from the second detection with the original sensitive text data and sensitive object data, thereby quantitatively evaluating the effect of the desensitization process.

[0087] Specifically, after the image to be processed is anonymized, text recognition and sensitivity detection are first performed again on the anonymized image to obtain anonymized sensitive text data. This anonymized sensitive text data reflects the remaining sensitive text information in the anonymized image. Then, target object detection is performed again on the anonymized image to obtain anonymized sensitive object data, which reflects the remaining sensitive object information in the anonymized image. Subsequently, the anonymized sensitive text data and the anonymized sensitive text data are compared, and the anonymized sensitive object data and the anonymized sensitive object data are also compared. Based on the results of these two comparisons, the anonymization effect is evaluated. For example, if the anonymized sensitive text data contains multiple sensitive terms while the anonymized sensitive text data does not contain any sensitive terms, it indicates that the text anonymization effect is good.

[0088] In this way, by performing the same recognition process on the desensitized image as before desensitization, the change in the amount of sensitive information before and after desensitization can be objectively obtained, thus providing a quantifiable evaluation basis for the desensitization effect. This evaluation result can be used as a reference for subsequent processes, helping to determine whether the current desensitization treatment has met the expected requirements, and also providing users with traceable feedback on the quality of desensitization.

[0089] For example, in some implementations of this application, the specific processing and principles for providing quantifiable evaluation criteria for desensitization effects may include: The anonymized text and images are then subjected to the aforementioned sensitive identification process (the sensitive identification process involved in step S103, and the process of "target object detection in the image to be processed" involved in step S105) for evaluation. By comparing the number of sensitive texts and faces before and after anonymization, the anonymization effect can be preliminarily measured. To more effectively refine the degree of privacy leakage of images, this application proposes a privacy leakage scoring function. ,in and The confidence score of the face and the proportion of the face pixel box to the entire image area are identified using InsightFace.

[0090]

[0091]

[0092] The function can be viewed as a translation and scaling operation along the horizontal axis. The function is used to characterize the strength of the model's ability to detect the presence of a face. The function takes the face area into account; if the area ratio exceeds a preset threshold... (For example When a significant risk of privacy breach is identified, the final score is enhanced accordingly.

[0093] Furthermore, the cosine similarity of the embedded images before and after anonymization can be used to determine whether the people in the two images can be considered the same person. This application embeds the unanonymized and anonymized images using a ResNet (residual network) face recognition model trained on ArcFace (a deep learning face recognition model based on additive angular margin loss). The higher the calculated cosine similarity, the more likely the model is to identify the two images as the same person; the lower the similarity, the more successfully the anonymization process has weakened the identifiability. This metric does not simply measure pixel changes, but rather evaluates the privacy protection effect from the model's discriminative perspective. If the cosine similarity between the anonymized and unanonymized images is significantly reduced, it indicates that key identity features have been effectively destroyed, achieving the privacy protection objective.

[0094] To enable those skilled in the art to more intuitively understand this application, a specific example will be provided below. This example is an exemplary demonstration combining the overall technical paradigm of this application with some optional implementation details. It should be noted that the following demonstration is intended to aid understanding and does not constitute an exhaustive list of all embodiments of this application, nor does it imply that this application must include all the details described below in its specific implementation.

[0095] This example uses the power industry as an example to illustrate the image desensitization process of this application, which includes steps one through four.

[0096] Step 1: Low-rank adaptation fine-tuning of large language models In the specific fine-tuning, the base model chosen was Qwen2.5-7B-Instruct. This model has a moderate number of parameters, sufficient native Chinese pre-training corpus, and a strong prior understanding of Chinese terminology in the power industry. The LoRA injection layer uses the projection matrix of the attention layer. , and the gating matrix of the FFN layer , covering the FFN layer can significantly improve the model's adaptability to domain knowledge, which is particularly important for the understanding of power terminology. LoRA training parameters .

[0097] Training data format: <|im_start|>system You are a data security auditor for the power system, responsible for identifying sensitive private information in texts. The sensitive information types include: geographical locations (place names, city names, county names), full names and abbreviations of companies or entities, transmission line names, license plate numbers, employee numbers or personnel identification numbers, uniquely identified equipment models, names of institutions such as dispatch centers and substations.

[0098] <|im_end|> <|im_start|>user Please determine whether the following text contains sensitive private information, and output the result strictly in JSON format: Text: "……Operator EMP-2024-0031 is driving……Destination Wuhan Dispatch Center……License plate E A·12345" <|im_end|> <|im_start|>assistant {"has_privacy": true, "invalid_terms": ["EMP-2024-0031", "Wuhan Dispatch Center", "E A·12345"]} <|im_end|> Learning rate= , AdamW is selected as the optimizer, and Focal Loss is introduced to reduce the weight of easily classified samples, , where, is the Focal Loss value, which is used to measure the difference between the model prediction and the ground truth label; t is the position index in the output sequence, which traverses from the 1st output token to the |Y|-th output token; is the class weight coefficient, which is used to balance the contribution ratio of different class samples in the loss calculation, and can be set according to the sample distribution; is the prediction probability of the model for the correct token; is the modulation factor, when the model can already predict a certain token with high confidence, , the gradient contributed by this sample is close to zero, the training focuses on hard-to-classify samples, which helps to improve the recognition accuracy of rare power terminologies.

[0099] Step 2: Concept erasure fine-tuning of latent diffusion model Stable Diffusion was used as the base latent diffusion model. The fine-tuning parameters were selected from the to_k and to_v parameters in the cross-attention layer of U-Net within Stable Diffusion. These cross-attention parameters are responsible for converting text prompts into image features; they serve as the interface between text and image. The erased concepts were specified as ["words", "face"]. The number of fine-tuning rounds was set to 1000, the learning rate to 1e-5, the optimizer to Adam, the denoising steps to 50, the guiding strength to 3.0, and the negative guiding strength to 1.0. The space for searching anchor concepts used the ImageNet dataset, and the Top-K nearest neighbors of each concept to be erased were retrieved using L2 similarity.

[0100] Fine-tuning first involves instantiating the original frozen model θ and the trainable model on two separate GPU devices. For each concept to be erased, the system constructs a differentiable discrete distribution using Gumbel-Softmax (a differentiable discrete sampling approximation technique). It dynamically learns the optimal set of concepts to retain from a pre-computed ImageNet embedding matrix, achieving adversarial optimization by erasing the target concept while maximizing the preservation of semantically similar concepts. In each optimization iteration, the conditional embedding of the current word to be erased is sampled. Preservative embeddings weighted by Gumbel Using fixed starting noise At the same time step Intermediate latent variables are generated separately, and teacher denoising predictions are extracted using a frozen model. Extract student predictions through a trainable model This is used to construct the loss function. Here, GumbelSoftmax(π) represents the Gumbel-Softmax sampling operator, and π is the learnable class probability distribution. The pre-computed candidate anchoring concept embedding matrix consists of embedding vectors of multiple concepts semantically similar to the target concept retrieved from the ImageNet dataset, with each row corresponding to the embedding representation of a candidate concept. The optimizer alternately updates the U-Net cross-attention parameters and the Gumbel distribution weights. .Every The evaluation sampling is carried out step by step, and the latent variables are efficiently transferred between devices through a zero-copy mechanism to ensure training efficiency.

[0101] Step 3: Desensitizing Multimodal Data In image optical character recognition (OCR), the parameters of the PaddleOCR model are set as follows: Enable angle classification to improve the accuracy of tilted text recognition. Specify Chinese recognition language. Turn off document correction to adapt to various shooting angles and lighting conditions that may occur in construction site photos.

[0102] In face detection, FaceAnalysis is used for face detection and pose estimation. FaceAnalysis refers to two types of face attribute analysis tools based on the InsightFace / DLib open-source model: one is the ComfyUI plugin for developers (used for face similarity evaluation and feature extraction in AI drawing workflows), and the other is the Face Analysis WebUI for ordinary users (an out-of-the-box browser-based system that provides visual analysis of age, gender, pose, etc.). [Selection / Use] The face analysis model scales the input image to... Resolution is chosen to balance detection accuracy and computational efficiency. This is based on the confidence level returned by the model. facial angle The proportion of the face area in the entire image A multi-dimensional threshold strategy is used to determine whether facial privacy has been leaked:

[0103] In image redrawing, the input image and binary mask Preprocessing is performed based on the maximum side length. Scale proportionally and align to integer multiples of 8 to fit the VAE latent space. in, , , The DDIM sampler is used, and the number of inference steps is set to... The classifier guidance strength is set to This is combined with negative cues to suppress residual sensitive features. The optimization strategy uses xformers (an efficient Transformer component library based on PyTorch) to reduce the standard attention complexity from... Optimize to near linear and enable mixed-precision inference using torch.autocast (a context management tool provided in the PyTorch deep learning framework that automatically performs mixed-precision training and inference), which will significantly improve inference speed while ensuring quality.

[0104] Step 4: Assess the degree of privacy breach before and after anonymization When calculating the cosine similarity of faces before and after anonymization, the InsightFace face analysis model is first instantiated. Obtain images before and after desensitization. Corresponding face embedding vector and If no valid face is detected in the processed image, it is determined that the identity information has been completely removed, and the privacy leakage score is set to zero. For successfully detected face samples, the following steps are first performed: Normalization:

[0105] Then, the cosine similarity between the two vectors is calculated in the ArcFace feature space.

[0106] in To facilitate subsequent statistical analysis, the system further maps similarity to intervals.

[0107]

[0108] in The larger the value, the more identity information is retained in the processed image, and the higher the risk of privacy leakage; conversely, the smaller the value, the more significant the de-identification effect.

[0109] Please see Figure 4 This application provides an electronic device including at least one processor 201 and at least one memory 202. The processor 201 and memory 202 are directly connected to each other, or communicate with each other through a communication interface 203, or are electrically connected through one or more communication buses or signal lines to achieve data transmission or interaction. The memory 202 stores program instructions executable by the processor 201, which can call the program instructions to execute them, thereby implementing any of the model-based image desensitization methods provided by the various implementations described above.

[0110] The memory 202 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0111] The processor 201 can be an integrated circuit chip with signal processing capabilities. The processor 201 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0112] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0113] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. An image desensitization method based on model fine-tuning, characterized in that, include: Obtain the image to be processed; The image to be processed is subjected to text recognition to obtain the text content and the corresponding text location information; The text content is input into the first target model for sensitive identification to obtain sensitive text data. The first target model is a model with the ability to identify sensitive text information, which is obtained by fine-tuning the basic large language model in advance using the low-rank adaptation method. Based on the sensitive text data and the text location information, the sensitive text region in the image to be processed is determined; Target object detection is performed on the image to be processed to obtain the sensitive object region; The image to be processed, the sensitive text region, and the sensitive object region are input into the second target model. The second target model redraws the content of the sensitive text region and the sensitive object region to obtain the desensitized image. The second target model is a model that loses the ability to generate sensitive visual concepts after the basic latent diffusion model has been fine-tuned by the concept erasure method.

2. The image desensitization method based on model fine-tuning according to claim 1, characterized in that, The first target model was obtained in the following way: Obtain a training dataset, which includes instruction data, input text data, and expected output data. The expected output data is used to indicate whether the input text data contains sensitive information and the content of the sensitive information. The original weight matrix of the basic large language model is frozen, and a low-rank matrix is ​​injected into the attention layer and / or feedforward network layer of the basic large language model. The parameters of the low-rank matrix are updated using the training dataset, and fine-tuning is performed using the expected output data as a supervision signal to obtain the first target model. The fine-tuning of the instructions involves calculating the loss only for the expected output data. The losses of the instruction data and the input text data are masked by a mask matrix so that the basic large language model learns to generate answers based on instructions rather than memorizing the input text.

3. The image desensitization method based on model fine-tuning according to claim 2, characterized in that, The injection of a low-rank matrix into the attention layer and / or feedforward network layer of the basic large language model includes: A low-rank matrix is ​​injected into at least one of the query projection matrix, key projection matrix, value projection matrix, and output projection matrix of the attention layer of the basic large language model; and / or, A low-rank matrix is ​​injected into at least one of the gating matrix, the up-projection matrix, and the down-projection matrix of the feedforward network layer of the basic large language model.

4. The image desensitization method based on model fine-tuning according to claim 1, characterized in that, The second target model is obtained in the following way: Obtain the basic potential diffusion model and the set of sensitive concepts to be erased; The target parameters to be fine-tuned are determined in the cross-attention layer of the basic potential diffusion model; For each target concept in the set of sensitive concepts, a target anchor concept corresponding to the target concept is determined in a preset anchor concept search space; A loss function is constructed based on the target concept and the target anchoring concept. The loss function is used to constrain the output of the fine-tuned basic potential diffusion model under the target concept to approach the output of the basic potential diffusion model under the target anchoring concept. With the goal of minimizing the loss function, the target parameters are updated to obtain the second target model.

5. The image desensitization method based on model fine-tuning according to claim 4, characterized in that, The preset anchor concept search space is constructed in the following way: Obtain a preset image dataset and extract the embedding vector of each image sample in the preset image dataset; Calculate the distance between the embedding vector of each image sample and the embedding vector of the target concept; The concepts corresponding to the K nearest image samples are selected as the anchor concept candidate set corresponding to the target concept, where K is a positive integer.

6. The image desensitization method based on model fine-tuning according to claim 1, characterized in that, The step of performing text recognition on the image to be processed to obtain text content and corresponding text location information includes: calling an optical character recognition engine to perform text detection and recognition on the image to be processed, and obtaining at least one text detection box and the text content and pixel coordinate information corresponding to each text detection box.

7. The image desensitization method based on model fine-tuning according to claim 1, characterized in that, The step of detecting target objects in the image to be processed to obtain sensitive object regions includes: Face detection is performed on the image to be processed to obtain face detection results. The face detection results include face location information and face attribute information. The face attribute information includes face confidence, face rotation angle, and the proportion of face area to the entire image. Sensitive faces are identified from the face detection results based on preset privacy judgment rules and the face attribute information. A sensitive object region is generated based on the facial location information corresponding to the sensitive face.

8. The image desensitization method based on model fine-tuning according to claim 1, characterized in that, The step of inputting the image to be processed, the sensitive text region, and the sensitive object region into the second target model, and having the second target model redraw the content of the sensitive text region and the sensitive object region, includes: A text mask is generated based on the sensitive text region, and an object mask is generated based on the sensitive object region. The text mask and the object mask have a first pixel value in their respective corresponding regions and a second pixel value in other regions. Perform a logical OR operation on the text mask image and the object mask image to obtain the target mask image; The image to be processed and the target mask image are input together into the second target model, and the second target model only redraws the content of the region with the first pixel value in the target mask image.

9. The image desensitization method based on model fine-tuning according to claim 1, characterized in that, The method further includes: The desensitized image is then subjected to text recognition and sensitivity recognition again to obtain desensitized sensitive text data. The desensitized image is subjected to target object detection again to obtain desensitized sensitive object data; The desensitization effect evaluation result is determined based on the differences between the sensitive text data and the desensitized sensitive text data, and the differences between the sensitive object data and the desensitized sensitive object data.

10. An electronic device, characterized in that, include: Memory, used to store one or more programs; processor; When the one or more programs are executed by the processor, the method as described in any one of claims 1-9 is implemented.