Method for realizing multi-modal large-model fine-grained privacy grading protection in edge-cloud collaborative inference system

Abstract

The invention relates to a method for realizing multi-modal large-model fine-grained privacy grading protection in an edge-cloud collaborative inference system, belongs to the technical field of large-model data security and privacy protection, and aims to solve the problems that privacy grading is rough, inference intermediate features are easy to leak and precision is damaged by protection measures in the prior art. According to the method, multi-modal data such as images and texts are received by edge computing nodes, deep semantic analysis and fine-grained segmentation are performed by using a local pre-trained multi-modal large model, and privacy scores are calculated to divide high and low privacy levels; uploading low-privacy data to a cloud computing center, and locally processing high-privacy data; and selectively shielding a middle layer feature map F generated by the edge based on the correlation degree and the contribution degree, then completing local reasoning, and uploading a result to be fused with the cloud. According to the method, precise fine-grained protection is achieved, the sensitive information leakage risk is remarkably reduced, and meanwhile the model reasoning precision and the system real-time performance are kept to the maximum degree.

Description

Technical Field

[0001] This invention belongs to the field of large model data security and privacy protection technology, and relates to a method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system. Background Technology

[0002] With the deep integration of artificial intelligence (AI) technology with the Internet of Things (IoT) and 5G / 6G communication technologies, multimodal data-driven intelligent systems have become a core support for key areas such as the Industrial Internet, smart healthcare, autonomous driving, and intelligent security. Multimodal large-scale models can simultaneously process multiple data types, including text, images, voice, and sensor signals, thereby achieving accurate understanding and decision-making in complex real-world scenarios. To meet the demands of low-latency, high-reliability real-time applications, large-scale model deployment is rapidly extending to the network edge. Edge computing, with its advantages of proximity processing, low latency, and high bandwidth efficiency, has become an important carrier for the deployment of multimodal large-scale models.

[0003] In the field of privacy protection in edge deployment scenarios of multimodal large models, existing technologies mainly include two types of solutions. One type is data-level privacy protection solutions, typically represented by the combination of federated learning (FL) and differential privacy (DP) technology, which prevents information leakage by adding noise to local data or gradients. The other type is deployment-level privacy isolation solutions, which use model compression or distillation techniques to lightweight the complete large model and deploy it fully on edge devices, so that the data does not leave the local machine.

[0004] However, existing technologies still have significant shortcomings when dealing with large edge models driven by multimodal data. First, the granularity of privacy classification is too coarse, only achieving binary high-low privacy classification for the entire data or a single modality, failing to finely distinguish different semantic units within the same data. This leads to either over-protection wasting computing power or insufficient protection leaving leakage risks. Second, there is a lack of effective protection for the intermediate processes of edge inference. In common edge-cloud collaborative inference models, intermediate features uploaded by edge nodes still contain reversibly reconstructable sensitive information, making them vulnerable to network interception or malicious inference attacks. Finally, there is a serious contradiction between existing privacy protection methods and model inference accuracy. Methods such as global noise addition, sensitive modality masking, or encrypted transmission all destroy the semantic correlation between multimodal features, resulting in a significant decrease in inference performance and failing to meet the dual requirements of high security and accuracy in practical applications.

[0005] Therefore, there is an urgent need for a new privacy protection method that can achieve fine-grained privacy classification, effectively protect intermediate features of inference, and maintain the inference accuracy of multimodal large models to the greatest extent in an edge-cloud collaborative environment. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for implementing fine-grained privacy-graded protection for multimodal large models in an edge-cloud collaborative inference system, the method being collaboratively executed by edge computing nodes and cloud computing centers, includes the following steps: Edge computing nodes receive multimodal data to be processed, which contains at least image modalities and text modalities; Edge computing nodes input the multimodal data into a locally deployed pre-trained multimodal large model, and use the collaborative capabilities of its visual encoder and language model to perform deep semantic parsing, identify and segment information regions with independent semantics; Edge computing nodes calculate the inherent sensitivity factor, contextual relevance factor, and accessibility factor of each information region based on the semantic parsing results, obtain a privacy score, and then label the information region as high privacy level or low privacy level after comparing it with the configurable sensitivity threshold. Edge computing nodes upload low-privacy data elements to the cloud computing center via the network, while keeping high-privacy data elements locally. Edge computing nodes input high-privacy data into the underlying encoder deployed only locally for feature extraction, resulting in an intermediate layer feature map F; The edge computing node calculates the correlation degree with the high privacy region and the contribution to the final task for each feature unit of the intermediate layer feature map F, obtains the fusion score, and generates a purification mask according to a settable threshold to selectively mask the intermediate layer feature map F, and continues to complete the remaining inference locally to obtain the edge inference result. The edge computing node uploads the edge inference results to the cloud computing center, which then merges these results with the inference results from low-privacy data to output the global inference results.

[0008] Furthermore, the correlation between each feature unit of the intermediate layer feature map F and the high privacy region is calculated by using Gaussian blur to process the high privacy region mask. The larger the value, the higher the correlation between the feature unit and the privacy information.

[0009] Furthermore, the contribution of each feature channel to the final task is represented by the absolute value after global average pooling and normalized to the range of 0~1.

[0010] Furthermore, the fusion score comprehensively considers privacy relevance and task contribution, and the generation of the sanitization mask satisfies the condition that if the fusion score is greater than or equal to a set threshold, the mask is 0, otherwise it is 1.

[0011] Furthermore, the configurable sensitivity threshold is set according to the application scenario requirements. If more emphasis is placed on confidentiality, it is set to a lower value; if more emphasis is placed on inference efficiency, it is set to a higher value.

[0012] Furthermore, the multimodal data also includes voice modalities and / or sensor signal modalities.

[0013] Furthermore, the underlying encoder is the first few layers of encoders responsible for basic feature extraction in the multimodal large model.

[0014] The beneficial effects of this invention are as follows: (1) This invention utilizes the deep semantic understanding capabilities of the multimodal large model to achieve automated fine-grained privacy classification of different semantic units within the same data, thereby improving the protection precision from the file level or modality level to the pixel level and semantic element level, effectively avoiding the problems of over-protection or under-protection, and making privacy protection more accurate and efficient.

[0015] (2) Based on the results of fine-grained classification, this invention constructs a dynamic collaborative mechanism for fully local processing of high privacy data at the edge and secure uploading of low privacy data to the cloud for processing. This significantly reduces the number of times sensitive information is transmitted across the network and the risk of leakage from the source. At the same time, it fully leverages the powerful computing power of the cloud, avoids the shortage of computing resources at the edge nodes, and improves the overall security and operating efficiency of the system.

[0016] (3) In the intermediate feature transmission stage of edge inference, the present invention selectively masks based on the correlation of high privacy regions and the contribution of tasks, and only cleans up high privacy low contribution features, abandoning crude methods such as global noise addition or modal masking. While completely blocking the leakage of sensitive information, it retains the features that are crucial to inference performance to the greatest extent, thereby achieving a good balance between strict privacy protection and high-precision inference.

[0017] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall process of a multimodal data-driven edge large model fine-grained privacy classification protection method of the present invention; Figure 2 This is a schematic diagram of the selective shielding process of the intermediate layer feature map F of the edge node in this invention. Detailed Implementation

[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0021] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0022] Please see Figure 1 One embodiment of the present invention provides a method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system, the method being as follows: Step 1: Input the multimodal data to be processed into a pre-trained multimodal large model. This model utilizes the collaborative capabilities of its visual encoder and language model to perform deep semantic parsing of the data, automatically identifying and segmenting information regions with independent semantics contained in the data.

[0023] Step 2: Privacy scoring via images and text With configurable sensitivity threshold The privacy level assessment includes the following steps: Step 2.1: Based on the semantic understanding results, set the inherent sensitivity factors for images and text according to the scene requirements. Contextual factors The weighting relationship between them, combined with the accessibility factor Get privacy rating :

[0024] Step 2.2: Set up according to requirements If greater emphasis is placed on data confidentiality, then... The regulations are in If more emphasis is placed on the efficiency of large models, then... The regulations are in . Step 2.3: Comparison and ,like This indicates low privacy, and vice versa. This ensures high privacy. Furthermore, it utilizes the built-in segmentation module of a multimodal large model to segment each identified independent information region. Generate a binary mask And obtain a unified mask that covers all high-privacy areas. :

[0025] Step 3: Transfer low-privacy data to the cloud for direct reading and processing; send high-privacy data to edge nodes for processing.

[0026] Step 4: Obtain the correlation degree of each location based on the intermediate feature map of the high-privacy data. and contribution Selective masking is then performed, specifically including the following steps: Step 4.1: High-privacy data is input and deployed to the underlying encoder of the edge node, utilizing the multimodal feature alignment capability of the large model itself to obtain the intermediate feature map F. For each spatial location of the feature map F... The correlation of high-privacy regions is obtained by using the image smoothing filtering technique Gaussian Blur. :

[0027] Step 4.2: Use the absolute value after global average pooling As a contributing agent:

[0028] and absolute value Normalization process is performed to obtain :

[0029] Step 4.3: For each feature unit in the image Calculate a fusion score :

[0030] And according to With threshold Generate a binary mask based on the size relationship. :

[0031] Step 4.4: As Figure 2 As shown, the intermediate feature map F is combined with the obtained mask matrix. Perform element-wise multiplication:

[0032] Step 5: Obtain the purified characteristics It is then fed into the next layer of the model to continue the entire inference of the data at the edge nodes.

[0033] Step 6: Upload the results obtained from the edge nodes to the cloud and merge them with the results of the low-privacy data processed in the cloud to obtain the global result.

[0034] Example 1: Multimodal Data Privacy Classification Protection Process Based on Holistic Approach Edge computing nodes first receive multimodal data containing at least image and text modalities. For example, in a smart security scenario, the input could be images captured by a surveillance camera and accompanying log text data. The data is then fed into a locally deployed, pre-trained multimodal large model, which leverages the collaborative capabilities of a visual encoder and a language model for deep semantic parsing. The parsing process automatically identifies and segments independent semantic information regions, such as segmenting a face from an image or extracting key phrases from text. Each region is assigned a binary mask identifier.

[0035] Figure 1 This document demonstrates the overall workflow of this embodiment, from data input to result fusion. Edge nodes calculate a privacy score for each information region based on the semantic parsing results. The scoring formula integrates inherent sensitivity factors, contextual relevance factors, and accessibility factors, with weight parameters dynamically adjustable; for example, higher weights are assigned to sensitivity factors in medical scenarios. After comparing the score with a configurable sensitivity threshold, the region is labeled as either high-privacy or low-privacy. Low-privacy data is uploaded to the cloud computing center for processing, while high-privacy data is retained locally. Edge nodes perform feature extraction and selective masking on the high-privacy data, ultimately completing inference locally and uploading the results to the cloud for fusion. This embodiment achieves a balance between privacy and efficiency through hierarchical collaboration.

[0036] Example 2: Feature Processing Flow for Correlation and Contribution Calculation When processing high-privacy data at edge nodes, each feature unit in the intermediate layer feature map F needs to calculate its correlation with the high-privacy region. The correlation calculation employs a Gaussian blurring method to smooth the high-privacy region mask. The kernel size can be set to 5×5 with a standard deviation of 1.0, generating a continuous correlation map. A larger value indicates a higher correlation between the feature unit and the privacy information, requiring stronger protection. Contribution calculation uses global average pooling, pooling the absolute value of each feature channel and normalizing it to the zero-to-one range to reflect the feature's importance to the task.

[0037] Figure 2 The selective masking process is illustrated. The fusion score is composed of correlation and contribution, and a binary mask is generated based on a settable threshold. If the fusion score is greater than or equal to the threshold, the feature is masked; otherwise, it is retained. The cleaned feature map continues to be used for local inference, ensuring high privacy data security. This embodiment maximizes inference accuracy while protecting privacy through refined feature analysis.

[0038] Example 3: Dynamic Threshold and Multimodal Extended Application Process During the privacy grading phase, configurable sensitivity thresholds can be dynamically set according to the application scenario. For example, in smart home scenarios, where inference efficiency is prioritized, a lower threshold such as 0.3 is set; while in financial surveillance, where stricter protection is desired, a threshold of 0.6 is set. Multimodal data is extended to voice and sensor modalities, with edge nodes uniformly parsing all modalities. High-privacy data is processed locally, while low-privacy data is processed in the cloud, with results fused and output. This embodiment adapts to diverse scenarios through dynamic parameters, enhancing system flexibility.

[0039] Example 4: Low-level encoder optimization and real-time processing flow The bottom-level encoder is defined as the first few layers of a multimodal large model, responsible for basic feature extraction. Only these lightweight encoders are deployed at edge nodes to reduce computational load. High-privacy data is input into the encoder to generate a feature map F, which undergoes selective masking. The optimized encoder supports real-time applications, such as video streaming. This embodiment achieves efficient privacy protection in edge environments through resource optimization.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system, characterized in that: This method is executed collaboratively by edge computing nodes and cloud computing centers, and includes the following steps: Edge computing nodes receive multimodal data to be processed, which contains at least image modalities and text modalities; Edge computing nodes input the multimodal data into a locally deployed pre-trained multimodal large model, and use the collaborative capabilities of its visual encoder and language model to perform deep semantic parsing, identify and segment information regions with independent semantics; Edge computing nodes calculate the inherent sensitivity factor, contextual relevance factor, and accessibility factor of each information region based on the semantic parsing results, obtain a privacy score, and then label the information region as high privacy level or low privacy level after comparing it with the configurable sensitivity threshold. Edge computing nodes upload low-privacy data elements to the cloud computing center via the network, while keeping high-privacy data elements locally. Edge computing nodes input high-privacy data into the underlying encoder deployed only locally for feature extraction, resulting in an intermediate layer feature map F; The edge computing node calculates the correlation degree with the high privacy region and the contribution to the final task for each feature unit of the intermediate layer feature map F, obtains the fusion score, and generates a purification mask according to a settable threshold to selectively mask the intermediate layer feature map F, and continues to complete the remaining inference locally to obtain the edge inference result. The edge computing node uploads the edge inference results to the cloud computing center, which then merges these results with the inference results from low-privacy data to output the global inference results.

2. The method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system according to claim 1, characterized in that: The correlation between each feature unit of the intermediate layer feature map F and the high privacy region is calculated by using Gaussian blur to process the high privacy region mask. The larger the value, the higher the correlation between the feature unit and the privacy information.

3. The method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system according to claim 1 or 2, characterized in that: The contribution of each feature channel to the final task is represented by the absolute value after global average pooling and normalized to the range of 0~1.

4. The method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system according to claim 1 or 2, characterized in that: The fusion score takes into account both privacy relevance and task contribution. The cleanup mask is generated such that if the fusion score is greater than or equal to a set threshold, the mask is 0; otherwise, it is 1.

5. The method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system according to claim 1 or 2, characterized in that: The configurable sensitivity threshold is set according to the application scenario requirements. If more emphasis is placed on confidentiality, it is set to a lower value; if more emphasis is placed on inference efficiency, it is set to a higher value.

6. The method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system according to claim 1 or 2, characterized in that: The multimodal data also includes voice modalities and / or sensor signal modalities.

7. The method for implementing fine-grained privacy classification protection for multimodal large models in an edge-cloud collaborative inference system according to claim 1 or 2, characterized in that: The underlying encoder is the first few layers of encoders responsible for basic feature extraction in the multimodal large model.

Images