A smart glasses control method and system based on multimodal privacy protection

Abstract

The present application relates to a smart glasses control method and system based on multimodal privacy protection, including collecting multimodal sensory data of users, preprocessing the multimodal sensory data, generating sensory shared data; using a multimodal privacy feature extraction algorithm to analyze the sensory shared data, generating a privacy feature vector; classifying the privacy feature vector based on multi-layer privacy judgment rules, identifying and obtaining judgment results; processing data based on the processing logic in the multi-layer privacy judgment rules, generating shared data after privacy protection processing; storing the shared data after privacy protection processing in the cloud, retrieving the corresponding shared data after privacy protection processing from the cloud according to the sharing demand information of the target device, and generating encrypted shared data; decrypting and reconstructing the encrypted shared data in the target device to generate shared display data. The present application has the effect of improving the accuracy of smart glasses privacy data recognition.

Description

Technical Field

[0001] The present application relates to the technical field of smart wearable devices, and in particular to a smart glasses control method and system based on multimodal privacy protection. Background Art

[0002] In recent years, with the rapid development of smart wearable devices, smart glasses, as an important branch, have gradually become a hot spot for technology research and market application. Smart glasses not only have the visual correction function of traditional glasses, but also realize a variety of intelligent functions such as image acquisition, voice interaction, and motion monitoring by integrating multiple sensors such as cameras, microphones, and accelerometers. These functions significantly improve the user's interactive experience, making smart glasses widely used in augmented reality, remote collaboration, health monitoring and other fields. However, in the process of smart glasses collecting and processing user sensory data, how to protect user privacy data while providing personalized services has become an important issue that needs to be solved. Especially in the scenario of sensory sharing, the user's private information may be inadvertently collected and shared, which will cause major challenges to data security and privacy protection.

[0003] The above-mentioned existing technical solutions have the following defects: the existing privacy recognition technology of smart glasses has misjudgment and missed judgment in complex and changeable scenarios, and it is difficult to reliably identify sensitive information, so there is room for improvement. Summary of the invention

[0004] In order to improve the accuracy of smart glasses' privacy data identification, the present application provides a smart glasses control method and system based on multimodal privacy protection.

[0005] The above-mentioned invention objective of the present application is achieved through the following technical solutions:

[0006] A smart glasses control method based on multimodal privacy protection, the smart glasses control method based on multimodal privacy protection comprising:

[0007] Collecting multimodal sensory data of a user, and preprocessing the multimodal sensory data to generate sensory sharing data;

[0008] Analyze the sensory shared data using a multimodal privacy feature extraction algorithm to generate a privacy feature vector;

[0009] Classifying the privacy feature vectors based on multi-layer privacy determination rules, adjusting the determination thresholds of the multi-layer privacy determination rules in combination with dynamic environment perception and interactive feedback from the user, and identifying and obtaining determination results;

[0010] Marking the privacy data in the determination result, and performing data processing based on the processing logic in the multi-layer privacy determination rule to generate shared data after privacy protection processing;

[0011] The shared data after the privacy protection processing is stored in the cloud, and according to the sharing requirement information of the target device, the corresponding shared data after the privacy protection processing is retrieved from the cloud for encryption to generate encrypted shared data;

[0012] The encrypted shared data is transmitted to the target device, the encrypted shared data is decrypted and reconstructed in the target device to generate shared display data, and the shared display data is displayed in a multi-channel interactive manner.

[0013] By adopting the above technical solutions, the differences in data of different modalities are effectively eliminated by collecting the user's visual, auditory and environmental data and performing standardization and formatting preprocessing; the multimodal privacy feature extraction algorithm is used to deeply analyze and integrate visual, auditory and contextual information, which helps to generate a unified privacy feature vector, improve the accuracy of privacy judgment and the comprehensiveness of data analysis; through the combination of multi-layer privacy judgment rules with dynamic environmental perception and user interaction feedback mechanism, the judgment threshold is dynamically adjusted to effectively adapt to the privacy needs in different scenarios, and the flexibility and accuracy of privacy judgment are improved; by marking the privacy data and blurring, shielding or limiting the scope according to the sharing priority, privacy leakage is effectively prevented, while ensuring that non-privacy data can be shared normally; by classifying and storing the processed shared data in the cloud and encrypting and protecting the highly sensitive data, the security of data storage and the efficiency of retrieval are effectively enhanced; by decrypting in the target device and dynamically adjusting the data format and resolution according to the hardware capabilities, high-quality data reconstruction is achieved; by displaying data in a multi-channel interactive manner, the richness and immersion of the user experience are significantly improved.

[0014] In a preferred example, the present application may be further configured as follows: the use of a multimodal privacy feature extraction algorithm to analyze the sensory shared data to generate a privacy feature vector includes:

[0015] Recognizing visual data in the sensory shared data by a convolutional neural network to generate a sensitive area of ​​the visual data;

[0016] Using a speech recognition algorithm to detect the auditory data in the sensory sharing data, and obtaining speech features and sensitive keywords of the auditory data;

[0017] Detecting the sensory shared data based on a context-aware algorithm to obtain situational information;

[0018] The sensitive areas of the visual data, the speech features and sensitive keywords of the auditory data, and the context information are uniformly represented by using a feature fusion network to generate the privacy feature vector.

[0019] By adopting the above technical solutions, the visual data is feature extracted through the convolutional neural network, and the areas containing sensitive information are identified, which helps to accurately mark the privacy-related visual features; the auditory data is converted into text through the speech recognition algorithm, and the speech features and sensitive keywords are extracted at the same time, which helps to quickly locate the privacy-sensitive speech content and execute the corresponding privacy protection strategy according to the needs; the environmental data and user behavior data are comprehensively analyzed through the context-aware algorithm to generate dynamic situational information, which helps to understand the user's current environmental state and provide a basis for the environmental risk level for privacy judgment, so as to dynamically adjust the privacy protection strategy; the data of different modes are uniformly processed and represented in high dimensions through the feature fusion network to generate privacy feature vectors, which effectively solves the problem of difference in feature dimensions of multimodal data and realizes the comprehensive, unified and efficient expression of privacy information.

[0020] In a preferred example, the present application may be further configured as follows: the step of adjusting the determination threshold of the multi-layer privacy determination rule in combination with dynamic environment perception and the user's interactive feedback includes:

[0021] Based on the built-in sensors of smart glasses, the privacy risk factors in the current environment are detected and the privacy risk level is generated;

[0022] Dynamically setting the privacy identification sensitivity in the multi-layer privacy determination rule according to the privacy risk level;

[0023] Acquiring the user's intention to adjust the current privacy setting through the touch interface, voice command or gesture recognition of the smart glasses;

[0024] The sensitivity threshold of the multi-layer privacy determination rule is updated in real time according to the user's adjustment intention to the current privacy setting.

[0025] By adopting the above technical solutions, the built-in sensors of smart glasses can collect environmental data in real time, detect privacy risk factors in the current environment, and generate privacy risk levels, which is helpful to accurately evaluate the privacy security level of the user's environment and provide basic data for dynamically adjusting the privacy protection strategy; by utilizing the generated privacy risk level, the privacy recognition sensitivity in the multi-layer privacy determination rules can be dynamically set, which is helpful to dynamically optimize the privacy determination logic according to the environment, enhance privacy protection in high-risk environments, and improve the flexibility of data sharing in low-risk environments; through the user touch interface, voice commands or gesture recognition input adjustment intentions, the user's personalized needs for privacy settings can be captured in real time, which helps smart glasses to quickly respond to user privacy preferences and improve the adaptability of privacy protection strategies and user satisfaction; by updating the sensitivity thresholds of the multi-layer privacy determination rules in real time, effectively combining user preferences with dynamic environmental factors, it helps to improve the flexibility and accuracy of the privacy protection scheme and realize intelligent privacy protection.

[0026] In a preferred example, the present application can be further configured as follows: the privacy feature vector is classified based on the multi-layer privacy determination rule, and the determination threshold of the multi-layer privacy determination rule is adjusted in combination with dynamic environment perception and the interactive feedback of the user, and the determination result obtained by identification includes:

[0027] Preliminarily screening the sensitive features in the privacy feature vector using the first layer privacy determination rule in the multi-layer privacy determination rule, identifying sensitive areas in the visual features and sensitive keywords in the auditory features, and obtaining sensitive feature data;

[0028] The second layer of privacy determination rules in the multi-layer privacy determination rules are used to further classify the sensitive feature data, determine the privacy sensitivity label of the sensitive feature data, and identify to obtain the determination result.

[0029] By adopting the above technical solution, the visual features and auditory features in the privacy feature vector are quickly screened through the first layer of rules in the multi-layer privacy determination rules, and the areas containing sensitive information are identified and extracted, which helps to efficiently filter non-sensitive data, narrow the processing scope of subsequent classification, and improve the speed and accuracy of privacy determination; the sensitive feature data preliminarily screened out are refined and classified through the second layer of rules in the multi-layer privacy determination rules, and privacy sensitivity labels are assigned to the data according to specific privacy sensitivity levels, which helps to accurately define the privacy attributes of each sensitive data.

[0030] In a preferred example, the present application may be further configured as follows: marking the privacy data in the determination result, and performing data processing based on the processing logic in the multi-layer privacy determination rule to generate shared data after privacy protection processing includes:

[0031] marking the private data in the determination result, and assigning a security level according to the sharing priority of the private data in the determination result;

[0032] According to the security level, a corresponding processing method is selected from the processing logic in the multi-layer privacy determination rules to process the data, so as to generate the shared data after the privacy protection processing, wherein the processing logic in the multi-layer privacy determination rules includes but is not limited to obfuscation, shielding and limiting the transmission scope.

[0033] By adopting the above technical solution, by marking the privacy data in the judgment results and assigning sharing priorities, and at the same time assigning security levels according to the sharing priorities, it is helpful to formulate differentiated processing strategies for different privacy data and improve the flexibility and efficiency of privacy protection; by selecting matching processing methods from multi-layer privacy judgment rules according to the security level, privacy data can be protected in a targeted manner, effectively reducing the risk of privacy leakage, while ensuring that non-sensitive data can be shared within a reasonable range, balancing data protection and sharing needs.

[0034] In a preferred example, the present application may be further configured as follows: storing the shared data after the privacy protection processing in the cloud includes:

[0035] Classify and store the shared data after the privacy protection processing according to the access rights, storage duration and data type of the shared data after the privacy protection processing to obtain stored data;

[0036] For the highly sensitive data in the stored data, a zero-knowledge encryption method is used to encrypt the highly sensitive data to prevent unauthorized access;

[0037] During the cloud storage process, the storage status and access frequency of the data are dynamically monitored, and storage priority is adjusted or an automatic cleanup mechanism is triggered according to the user's usage needs and access behavior.

[0038] By adopting the above technical solution, by classifying and storing shared data according to the access rights, storage duration and data type after privacy protection processing, the organization and retrieval efficiency of cloud storage are effectively improved, ensuring that data of different types and permissions can be quickly located and securely managed; by adopting zero-knowledge encryption method to encrypt highly sensitive data, unauthorized access is effectively prevented, ensuring that even if the data is stolen during transmission or storage, it cannot be decrypted, significantly improving the security and privacy protection of the data; by dynamically monitoring the storage status and access frequency of the data, adjusting the storage priority or triggering the automatic cleanup mechanism according to user needs and access behavior, it helps to optimize the utilization of cloud storage resources, prevent unnecessary data from occupying space, and improve user experience and system efficiency.

[0039] In a preferred example, the present application may be further configured as follows: decrypting and reconstructing the encrypted shared data in the target device to generate shared display data includes:

[0040] Generate a key using a distributed storage method based on blockchain, decrypt the encrypted shared data according to the key and a preset end-to-end key decryption algorithm, and obtain the decrypted shared data;

[0041] According to the display requirements of the target device, the decrypted shared data is formatted to obtain converted data;

[0042] The resolution, sound effect level or tactile feedback strength of the converted data is dynamically adjusted according to the hardware data of the target device to obtain the shared display data.

[0043] By adopting the above technical solution, keys are generated through a distributed storage method based on blockchain, ensuring the security and traceability of the keys during the generation, transmission and storage process; the encrypted shared data is decrypted in combination with a preset end-to-end key decryption algorithm, effectively preventing data leakage during transmission and storage, and improving the security and reliability of data protection; by converting the format of the decrypted shared data according to the display requirements of the target device, the data display requirements of different devices are effectively adapted, and the compatibility and user experience of shared data among multiple devices are improved; by dynamically adjusting the resolution, sound level or tactile feedback intensity of the shared data according to the hardware data of the target device, the display effect and interactive experience of the data on the target device are effectively optimized, making full use of the device performance and improving the quality and immersion of the sensory experience.

[0044] The second object of the invention is achieved by the following technical solutions:

[0045] A smart glasses control system based on multimodal privacy protection, the smart glasses control system based on multimodal privacy protection comprising:

[0046] A multimodal data collection module, used to collect multimodal sensory data of the user, and pre-process the multimodal sensory data to generate sensory sharing data;

[0047] A privacy feature analysis module, used to analyze the sensory shared data using a multimodal privacy feature extraction algorithm to generate a privacy feature vector;

[0048] A privacy determination rule module, used to classify the privacy feature vector based on the multi-layer privacy determination rule, and adjust the determination threshold of the multi-layer privacy determination rule in combination with dynamic environment perception and the interactive feedback of the user, and identify and obtain the determination result;

[0049] A privacy data processing module, used to mark the privacy data in the determination result, and perform data processing based on the processing logic in the multi-layer privacy determination rule to generate shared data after privacy protection processing;

[0050] An encrypted data transmission module is used to store the shared data after the privacy protection processing to the cloud, and retrieve the corresponding shared data after the privacy protection processing from the cloud according to the sharing requirement information of the target device, encrypt it, and generate encrypted shared data;

[0051] The decryption and display module is used to transmit the encrypted shared data to the target device, decrypt and reconstruct the encrypted shared data in the target device, generate shared display data, and display the shared display data in a multi-channel interactive manner.

[0052] By adopting the above technical solutions, the differences of different modal data are effectively eliminated by collecting the user's visual, auditory and environmental data and performing standardization and formatting preprocessing, providing a unified data basis for subsequent privacy feature extraction and judgment; the multimodal privacy feature extraction algorithm deeply analyzes and integrates visual, auditory and contextual information, which helps to generate a unified privacy feature vector, improve the accuracy of privacy judgment and the comprehensiveness of data analysis; through the combination of multi-layer privacy judgment rules with dynamic environmental perception and user interaction feedback mechanism, the judgment threshold is dynamically adjusted to effectively adapt to the privacy needs in different scenarios, and the flexibility and accuracy of privacy judgment are improved; by marking the privacy data and blurring, shielding or limiting the scope according to the sharing priority, privacy leakage is effectively prevented, while ensuring that non-privacy data can be shared normally; by classifying and storing the processed shared data in the cloud and encrypting and protecting the highly sensitive data, the security of data storage and the efficiency of retrieval are effectively enhanced; by decrypting in the target device and dynamically adjusting the data format and resolution according to the hardware capabilities, high-quality data reconstruction is achieved; by displaying data in a multi-channel interactive manner, the richness and immersion of the user experience are significantly improved.

[0053] In summary, the present application includes at least one of the following beneficial technical effects:

[0054] 1. By collecting the user's visual, auditory and environmental data and performing standardization and formatting preprocessing, the differences in data of different modalities are effectively eliminated; the multimodal privacy feature extraction algorithm is used to deeply analyze and integrate visual, auditory and contextual information, which helps to generate a unified privacy feature vector, improve the accuracy of privacy judgment and the comprehensiveness of data analysis; through the combination of multi-layer privacy judgment rules with dynamic environmental perception and user interaction feedback mechanism, the judgment threshold is dynamically adjusted to effectively adapt to the privacy needs in different scenarios, and the flexibility and accuracy of privacy judgment are improved;

[0055] 2. By marking private data and blurring, shielding or limiting the scope according to sharing priority, privacy leakage is effectively prevented while ensuring that non-privacy data can be shared normally; by classifying and storing the processed shared data in the cloud and encrypting and protecting highly sensitive data, the security of data storage and the efficiency of retrieval are effectively enhanced; by decrypting in the target device and dynamically adjusting the data format and resolution according to hardware capabilities, high-quality data reconstruction is achieved; by displaying data in a multi-channel interactive manner, the richness and immersion of the user experience are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a smart glasses control method based on multimodal privacy protection in one embodiment of the present application;

[0057] Figure 2 is a flowchart for implementing step S20 in a smart glasses control method based on multimodal privacy protection in an embodiment of the present application;

[0058] Figure 3 is another implementation flow chart of step S30 in a smart glasses control method based on multimodal privacy protection in an embodiment of the present application;

[0059] Figure 4 is a flowchart for implementing step S30 in a smart glasses control method based on multimodal privacy protection in an embodiment of the present application;

[0060] Figure 5 is a flowchart for implementing step S40 in a smart glasses control method based on multimodal privacy protection in an embodiment of the present application;

[0061] Figure 6 is a flowchart for implementing step S50 in a smart glasses control method based on multimodal privacy protection in an embodiment of the present application;

[0062] Figure 7 is a flowchart for implementing step S60 in a smart glasses control method based on multimodal privacy protection in an embodiment of the present application;

[0063] Figure 8 It is a principle block diagram of a smart glasses control system based on multimodal privacy protection in one embodiment of the present application. DETAILED DESCRIPTION

[0064] The present application is further described in detail below in conjunction with the accompanying drawings.

[0065] In one embodiment, if Figure 1As shown, the present application discloses a smart glasses control method based on multimodal privacy protection, which specifically includes the following steps:

[0066] S10: Collect multimodal sensory data of the user, and pre-process the multimodal sensory data to generate sensory sharing data.

[0067] Specifically, various sensor devices, such as cameras, microphones, accelerometers, temperature sensors, etc., are used to synchronously collect users' multimodal data, which include different types of information such as vision, hearing, touch, and movement. The multimodal sensory data is then preprocessed, such as data synchronization and alignment, data cleaning, data standardization, and feature extraction. The preprocessed multimodal data is then integrated to generate sensory shared data.

[0068] S20: Use the multimodal privacy feature extraction algorithm to analyze the sensory shared data and generate a privacy feature vector.

[0069] Specifically, by combining data from different modalities, comprehensive privacy-related features are extracted through multimodal privacy feature extraction algorithms, such as multimodal neural networks. Features with high privacy relevance are selected, such as features involving sensitive information such as personal identity information and behavioral patterns. The extracted privacy-related features are converted into high-dimensional vector representations to form privacy feature vectors, which will be used for subsequent privacy classification and processing.

[0070] S30: Classify the privacy feature vectors based on the multi-layer privacy determination rules, and adjust the determination thresholds of the multi-layer privacy determination rules in combination with dynamic environment perception and user interaction feedback to obtain the determination results.

[0071] Specifically, the system operating environment is monitored in real time, such as network status, device performance, etc., to adjust the sensitivity of the judgment rules. According to user feedback on privacy protection, such as adjusting privacy settings, manually marking data, etc., the judgment thresholds of the multi-layer privacy judgment rules are dynamically adjusted to ensure that the judgment rules are consistent with user needs. According to the different levels of privacy risks (such as low, medium, and high), corresponding judgment rules are formulated. The rules at each level include specific thresholds and conditions. For example, if certain features exceed specific values, they are judged to have high privacy risks. Finally, according to the multi-layer judgment rules and dynamic adjustments, the privacy level and processing requirements of each privacy feature vector are determined, and finally the recognition results are integrated to obtain the judgment results.

[0072] S40: Mark the privacy data in the determination result, and process the data based on the processing logic in the multi-layer privacy determination rule to generate shared data after privacy protection processing.

[0073] Specifically, based on the judgment results, the part that belongs to the private data is marked, such as adding metadata tags to indicate its privacy level and processing requirements, and the data with high privacy levels is masked, such as blurring and data desensitization, to prevent the leakage of sensitive information; data that requires high protection is encrypted to ensure security during storage and transmission; data access rights are set according to the privacy level to restrict unauthorized access and use. After the above processing, shared data with privacy protection is generated.

[0074] S50: storing the shared data processed with privacy protection in the cloud, retrieving the corresponding shared data processed with privacy protection from the cloud according to the sharing requirement information of the target device, encrypting the data, and generating encrypted shared data.

[0075] Specifically, upload the shared data after privacy protection processing to the cloud storage service to ensure the accessibility and persistence of the data, collect the sharing demand information of the target device, including the required data type, access rights, usage scenarios, etc., retrieve the corresponding shared data after privacy protection processing from the cloud according to the sharing needs, perform secondary encryption on the retrieved data, and use appropriate encryption algorithms, such as AES, RSA, etc., to generate encrypted shared data to ensure the security of data during transmission.

[0076] S60: Transmit the encrypted shared data to the target device, decrypt and reconstruct the encrypted shared data in the target device, generate shared display data, and display the shared display data in a multi-channel interactive manner.

[0077] Specifically, the encrypted shared data is sent to the target device through a secure transmission protocol, such as HTTPS, SSL / TLS. On the target device, the received encrypted data is decrypted using the corresponding decryption key to restore the original shared data after privacy protection processing. The decrypted data is formatted and reorganized to ensure that it meets the display and processing requirements of the target device. For example, image data is converted into a format suitable for display, structured data is loaded into an application, etc. The reconstructed data is the shared display data that can be displayed. The shared display data is displayed on the target device through a variety of interactive methods, such as touch screen, voice commands, gesture recognition, etc., to improve user experience and data accessibility.

[0078] By adopting the above technical solutions, the differences in data of different modalities are effectively eliminated by collecting the user's visual, auditory and environmental data and performing standardization and formatting preprocessing; the multimodal privacy feature extraction algorithm is used to deeply analyze and integrate visual, auditory and contextual information, which helps to generate a unified privacy feature vector, improve the accuracy of privacy judgment and the comprehensiveness of data analysis; through the combination of multi-layer privacy judgment rules with dynamic environmental perception and user interaction feedback mechanism, the judgment threshold is dynamically adjusted to effectively adapt to the privacy needs in different scenarios, and the flexibility and accuracy of privacy judgment are improved; by marking the privacy data and blurring, shielding or limiting the scope according to the sharing priority, privacy leakage is effectively prevented, while ensuring that non-privacy data can be shared normally; by classifying and storing the processed shared data in the cloud and encrypting and protecting the highly sensitive data, the security of data storage and the efficiency of retrieval are effectively enhanced; by decrypting in the target device and dynamically adjusting the data format and resolution according to the hardware capabilities, high-quality data reconstruction is achieved; by displaying data in a multi-channel interactive manner, the richness and immersion of the user experience are significantly improved.

[0079] In one embodiment, if Figure 2 As shown, in step S20, the sensory shared data is analyzed using a multimodal privacy feature extraction algorithm to generate a privacy feature vector, which specifically includes:

[0080] S21: Recognize visual data in sensory shared data through convolutional neural network and generate sensitive areas of visual data.

[0081] Specifically, the visual data in the sensory shared data is recognized through a convolutional neural network, and the input visual data is preprocessed, such as adjusting the image size and normalization. The trained CNN model is used to perform layer-by-layer convolution and pooling operations on the visual data to extract high-level features. According to the model classification results, sensitive areas in the visual data, such as faces and license plates, are located, and regional markers or masks, i.e., sensitive areas of the visual data, are generated.

[0082] S22: Using a speech recognition algorithm to detect auditory data in the sensory sharing data, and obtaining speech features and sensitive keywords of the auditory data.

[0083] Specifically, the audio data is preprocessed, such as denoising, framing and Fourier transform to extract time-frequency features. A speech recognition model, such as RNN and Transformer, is used to transcribe the preprocessed data. A keyword detection algorithm is used to extract keywords related to sensitive content from the transcribed text, and the spectral features of the corresponding speech are extracted as speech feature outputs, ultimately obtaining the speech features and sensitive keywords of the auditory data.

[0084] S23: Detect the sensory shared data based on the context perception algorithm to obtain situational information.

[0085] Specifically, the association between visual and auditory data is comprehensively analyzed, such as detecting the match between people's expressions in vision and tone in hearing, and applying context-aware models in combination with time series data to extract situational clues, such as scene categories, relationships between participants, etc., and finally output situational information, which includes current time, location characteristics, nature of events, etc.

[0086] S24: Use the feature fusion network to uniformly represent the sensitive areas of the visual data, the speech features and sensitive keywords of the auditory data, and the contextual information to generate a privacy feature vector.

[0087] Specifically, visual, auditory and contextual information are embedded and represented separately and unified into a high-dimensional feature space. For example, cross-modal feature alignment is performed through a shared attention mechanism, and a feature fusion network is used to perform multi-level interactive learning on the input features to generate a fused privacy feature representation, namely a privacy feature vector.

[0088] In one embodiment, if Figure 3 As shown, in step S30, the determination threshold of the multi-layer privacy determination rule is adjusted in combination with dynamic environment perception and user's interactive feedback, specifically including:

[0089] S31: Detect privacy risk factors in the current environment based on the built-in sensors of the smart glasses and generate a privacy risk level.

[0090] Specifically, sensors are used to capture environmental information, such as images, sounds, locations, and crowd density around the device. Based on pre-defined risk factors, such as whether there is a facial recognition camera and whether there are sensitive conversations in the ambient audio, the environmental data is analyzed through a machine learning model. A risk assessment algorithm is used to perform weighted scoring on environmental risk factors, and a privacy risk level is generated based on the time and space context.

[0091] S32: Dynamically set the privacy identification sensitivity in the multi-layer privacy determination rules according to the privacy risk level.

[0092] Specifically, the sensitivity threshold of the privacy identification module is set through the risk level mapping table. For example, when the privacy risk level is "high", stricter privacy detection rules are enabled, such as blurring the face in real time, while when the risk level is "low", more relaxed detection is allowed.

[0093] S33: Obtain the user's intention to adjust the current privacy setting through the touch interface, voice command or gesture recognition of the smart glasses.

[0094] Specifically, real-time monitoring records the commands issued by users through sliding, clicking or long pressing the touch interface, such as adjusting the privacy level or turning off specific rules; or the built-in voice recognition module captures user voice commands, such as "increase privacy sensitivity" or "turn off privacy protection"; or the user gestures are captured through the smart glasses camera and gesture recognition algorithm, such as gesture commands triggering the switch of privacy rules, to obtain the user's intention to adjust the current privacy settings.

[0095] S34: updating the sensitivity threshold of the multi-layer privacy determination rule in real time according to the user's adjustment intention to the current privacy setting.

[0096] Specifically, based on the instruction content input by the user, the privacy rule level and target threshold that need to be adjusted are parsed, and according to the parsing results, the parameters of the corresponding privacy rules are modified in real time. For example, the sensitivity threshold is lowered to reduce privacy interference, or the sensitivity is increased to strengthen privacy protection, that is, the sensitivity threshold of the multi-layer privacy judgment rule is updated.

[0097] In one embodiment, if Figure 4 As shown, in step S30, the privacy feature vectors are classified based on the multi-layer privacy determination rules, and the determination thresholds of the multi-layer privacy determination rules are adjusted in combination with dynamic environment perception and user interaction feedback, and the determination results are identified, which specifically include:

[0098] S35: Preliminarily screen the sensitive features in the privacy feature vector through the first layer privacy determination rule in the multi-layer privacy determination rule, identify the sensitive areas in the visual features and the sensitive keywords in the auditory features, and obtain sensitive feature data.

[0099] Specifically, a convolutional neural network is used in combination with a rule-based screening algorithm to generate mask data containing sensitive areas. Preliminary screening of sensitive areas in visual data is performed using predefined visual privacy judgment rules, such as detecting areas containing faces, license plates, document text, etc. A speech recognition model is used in combination with a sensitive keyword dictionary to perform sensitivity detection on keywords in speech data, extract potential privacy-sensitive keywords, and finally output sensitive feature data.

[0100] S36: Use the second layer privacy determination rule in the multi-layer privacy determination rule to further classify the sensitive feature data, determine the privacy sensitivity label of the sensitive feature data, and identify the determination result.

[0101] Specifically, based on the privacy classification algorithm, the identified visually sensitive areas are further analyzed to distinguish specific privacy levels. For example, the region refinement algorithm is used to determine whether a face is clearly identifiable or whether a document contains sensitive terms. Combined with contextual semantic analysis, sensitive keywords are analyzed in context, such as distinguishing between "home address" and general location descriptions. The privacy semantic classification algorithm is used to annotate keywords with different privacy sensitivity labels, and the sensitivity labels are output to represent the data classification results of each sensitive feature. All sensitivity labels are integrated to generate the final judgment result.

[0102] In one embodiment, if Figure 5 As shown, in step S40, the privacy data in the determination result is marked, and data processing is performed based on the processing logic in the multi-layer privacy determination rule to generate shared data after privacy protection processing, specifically including:

[0103] S41: Mark the private data in the determination result, and assign a security level according to the sharing priority of the private data in the determination result.

[0104] Specifically, the identified privacy data in the judgment results is marked, such as marking sensitive areas, marking sensitive keywords and their context timestamps, etc., a unique identifier is assigned to each data item, and privacy attributes such as type and sensitivity level are attached. The sharing priority of the privacy data is determined based on its purpose, type and risk assessment, and a security level is assigned based on the sharing priority of the privacy data in the judgment results, and the sharing priority is mapped to the security level.

[0105] S42: According to the security level, a corresponding processing method is selected from the processing logic in the multi-layer privacy determination rules to process the data, and the shared data after privacy protection processing is generated, wherein the processing logic in the multi-layer privacy determination rules includes but is not limited to obfuscation, shielding and limiting the transmission scope.

[0106] Specifically, the applicable processing logic is matched according to the security level, and the corresponding processing method is selected from the processing logic in the multi-layer privacy judgment rules to process the data, so as to generate shared data after privacy protection processing. For example, the sharing scope of sensitive data is restricted according to the authority and location of the data sharer; the processing method is dynamically adjusted in combination with the environmental context information in the judgment result, and the processed visual and auditory data are reassembled to generate multimodal shared data after privacy protection.

[0107] In one embodiment, if Figure 6 As shown, in step S50, the shared data after the privacy protection processing is stored in the cloud, which specifically includes:

[0108] S51: Classify and store the shared data after the privacy protection processing according to the access rights, storage duration, and data type of the shared data after the privacy protection processing to obtain stored data.

[0109] Specifically, the shared data after privacy protection processing is classified and stored according to the access rights, storage duration and data type of the shared data after privacy protection processing. The data is stored in different logical or physical storage spaces according to the classification results. For example, highly sensitive data is stored in encrypted partitions or secure databases, and low-sensitivity data is stored in ordinary storage pools to improve access efficiency. Finally, the stored data is obtained, and each piece of stored data is accompanied by metadata description, including classification labels and processing records.

[0110] S52: For highly sensitive data in the stored data, a zero-knowledge encryption method is used to encrypt the highly sensitive data to prevent unauthorized access.

[0111] Specifically, the data is encrypted through an encryption algorithm to generate encrypted ciphertext and access credentials. In particular, for highly sensitive data in stored data, the zero-knowledge encryption method is used to encrypt the highly sensitive data. The highly sensitive data is encrypted through an encryption algorithm to generate ciphertext that cannot be directly deciphered, so as to prevent unauthorized access.

[0112] S53: Dynamically monitor the storage status and access frequency of data during cloud storage, and adjust the storage priority or trigger the automatic cleanup mechanism according to the user's usage requirements and access behavior.

[0113] Specifically, during the cloud storage process, the occupancy of the data storage space, encryption status, access records, etc. are tracked in real time, the access frequency of each piece of data is recorded, and an access frequency curve is generated. The storage priority is adjusted according to the user's usage needs and access behavior. The storage priority of data with high access frequency is increased, such as storing it in a cache or a server node closer to the user; the priority of data that has not been accessed for a long time is lowered, such as moving it to cold storage or archive storage; cleanup conditions are set according to the storage duration, data sensitivity and user behavior, and the target data is cleaned when the trigger conditions are met.

[0114] In one embodiment, if Figure 7 As shown, in step S60, the encrypted shared data is decrypted and reconstructed in the target device to generate shared display data, which specifically includes:

[0115] S61: Generate a key using a distributed storage method based on blockchain, decrypt the encrypted shared data according to the key and a preset end-to-end key decryption algorithm, and obtain the decrypted shared data.

[0116] Specifically, an encryption key is generated through a consensus mechanism in the blockchain network, and the generated key is stored in different nodes of the blockchain in a distributed manner. The encrypted shared data is decrypted according to the key in the blockchain distributed storage and a preset end-to-end key decryption algorithm to restore the original shared data, i.e., the decrypted shared data.

[0117] S62: According to the display requirements of the target device, the decrypted shared data is format converted to obtain converted data.

[0118] Specifically, the system first identifies the specific display requirements of the target device, understands the data formats and display capabilities it supports, and converts the decrypted shared data into a format suitable for the target device based on the requirements, such as compressing or adjusting high-resolution images to a resolution and format suitable for the device screen; adjusting the video encoding format, resolution and frame rate to adapt to the device's playback capabilities, etc. After completing the format conversion, it generates conversion data that is adapted to the target device to ensure that it can be displayed correctly and efficiently on the device.

[0119] S63: Dynamically adjust the resolution, sound effect level, or tactile feedback strength of the converted data according to the hardware data of the target device to obtain shared display data.

[0120] Specifically, the system obtains hardware information of the target device through an interface or sensor, including current resource usage and supported functions, such as battery power, current running load, etc., and dynamically adjusts the image or video resolution of the converted data according to the display resolution and processing power of the device; adjusts the sound effect level of the audio data according to the audio output capability of the device and the user's current volume setting; for devices that support tactile feedback, adjusts the intensity and frequency of tactile feedback according to hardware capabilities and user preferences. Through the above adjustments, shared display data is generated that is adapted to the current device hardware status and user needs, ensuring that the data is presented in the best state on the device.

[0121] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0122] In one embodiment, a smart glasses control system based on multimodal privacy protection is provided, and the smart glasses control system based on multimodal privacy protection corresponds one-to-one to the smart glasses control method based on multimodal privacy protection in the above embodiment. Figure 8 As shown, the smart glasses control system based on multimodal privacy protection includes a multimodal data acquisition module, a privacy feature analysis module, a privacy determination rule module, a privacy data processing module, an encrypted data transmission module and a decryption and display module. The functional modules are described in detail as follows:

[0123] A multimodal data collection module is used to collect multimodal sensory data of users, and pre-process the multimodal sensory data to generate sensory sharing data;

[0124] A privacy feature analysis module is used to analyze sensory shared data using a multimodal privacy feature extraction algorithm to generate a privacy feature vector;

[0125] The privacy determination rule module is used to classify the privacy feature vectors based on the multi-layer privacy determination rules, and adjust the determination thresholds of the multi-layer privacy determination rules in combination with dynamic environment perception and user interaction feedback to obtain the determination results;

[0126] The privacy data processing module is used to mark the privacy data in the determination results, and process the data based on the processing logic in the multi-layer privacy determination rules to generate shared data after privacy protection processing;

[0127] The encrypted data transmission module is used to store the shared data after the privacy protection processing to the cloud, retrieve the corresponding shared data after the privacy protection processing from the cloud according to the sharing demand information of the target device, encrypt it, and generate encrypted shared data;

[0128] The decryption and display module is used to transmit the encrypted shared data to the target device, decrypt and reconstruct the encrypted shared data in the target device, generate shared display data, and display the shared display data through a multi-channel interactive method.

[0129] Optionally, the privacy feature analysis module includes:

[0130] A visual recognition submodule is used to recognize visual data in the sensory shared data through a convolutional neural network and generate sensitive areas of visual data;

[0131] A speech recognition submodule is used to detect the auditory data in the sensory sharing data using a speech recognition algorithm to obtain the speech features and sensitive keywords of the auditory data;

[0132] The context detection submodule is used to detect the sensory shared data based on the context perception algorithm to obtain the situational information;

[0133] The fusion submodule is used to use the feature fusion network to uniformly represent the sensitive areas of the visual data, the speech features and sensitive keywords of the auditory data, and the contextual information to generate a privacy feature vector.

[0134] Optionally, the privacy determination rule module includes:

[0135] The risk level identification submodule is used to detect the privacy risk factors in the current environment based on the built-in sensors of the smart glasses and generate the privacy risk level;

[0136] The sensitivity adjustment submodule is used to dynamically set the privacy identification sensitivity in the multi-layer privacy determination rules according to the privacy risk level;

[0137] The adjustment intention recognition submodule is used to obtain the user's intention to adjust the current privacy settings through the touch interface of the smart glasses, voice commands or gesture recognition;

[0138] The threshold adjustment submodule is used to update the sensitivity threshold of the multi-layer privacy determination rule in real time according to the user's adjustment intention to the current privacy setting;

[0139] The first-layer determination submodule is used to preliminarily screen the sensitive features in the privacy feature vector through the first-layer privacy determination rule in the multi-layer privacy determination rule, identify the sensitive areas in the visual features and the sensitive keywords in the auditory features, and obtain sensitive feature data;

[0140] The second-layer determination submodule is used to further classify the sensitive feature data using the second-layer privacy determination rules in the multi-layer privacy determination rules, determine the privacy sensitivity label of the sensitive feature data, and identify the determination result.

[0141] Optionally, the privacy data processing module includes:

[0142] A marking submodule, used to mark the private data in the determination result and assign a security level according to the sharing priority of the private data in the determination result;

[0143] The logic processing submodule is used to select the corresponding processing method from the processing logic in the multi-layer privacy determination rules according to the security level to process the data and generate shared data after privacy protection processing. The processing logic in the multi-layer privacy determination rules includes but is not limited to obfuscation, shielding and limiting the transmission scope.

[0144] Optionally, the encrypted data transmission module includes:

[0145] A storage submodule, used to classify and store the shared data after the privacy protection processing according to the access rights, storage duration and data type of the shared data after the privacy protection processing, so as to obtain stored data;

[0146] The encryption submodule is used to encrypt highly sensitive data in the stored data by using a zero-knowledge encryption method to prevent unauthorized access;

[0147] The storage priority adjustment submodule is used to dynamically monitor the storage status and access frequency of data during cloud storage, and adjust the storage priority or trigger the automatic cleanup mechanism according to the user's usage needs and access behavior.

[0148] Optionally, the decryption and display module includes:

[0149] A key generation submodule is used to generate a key using a distributed storage method based on blockchain, decrypt the encrypted shared data according to the key and a preset end-to-end key decryption algorithm, and obtain the decrypted shared data;

[0150] The format conversion submodule is used to convert the format of the decrypted shared data according to the display requirements of the target device to obtain converted data;

[0151] The display submodule is used to dynamically adjust the resolution, sound effect level or tactile feedback strength of the converted data according to the hardware data of the target device to obtain shared display data.

[0152] For the specific definition of a smart glasses control system based on multimodal privacy protection, please refer to the definition of a smart glasses control method based on multimodal privacy protection in the above text, which will not be repeated here. Each module in the above-mentioned smart glasses control system based on multimodal privacy protection can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0153] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0154] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A smart glasses control method based on multimodal privacy protection, characterized in that: The smart glasses control method based on multimodal privacy protection includes: Collecting multimodal sensory data of a user, and preprocessing the multimodal sensory data to generate sensory sharing data; The sensory shared data is analyzed using a multimodal privacy feature extraction algorithm to generate a privacy feature vector; the analysis of the sensory shared data using a multimodal privacy feature extraction algorithm to generate a privacy feature vector includes: identifying visual data in the sensory shared data by a convolutional neural network to generate a sensitive area of ​​the visual data; detecting auditory data in the sensory shared data by a speech recognition algorithm to obtain speech features and sensitive keywords of the auditory data; detecting the sensory shared data based on a context-aware algorithm to obtain context information; and using a feature fusion network to uniformly represent the sensitive areas of the visual data, the speech features and sensitive keywords of the auditory data, and the context information to generate the privacy feature vector; The privacy feature vectors are classified based on the multi-layer privacy determination rules, and the determination thresholds of the multi-layer privacy determination rules are adjusted in combination with dynamic environment perception and the interactive feedback of the user to identify and obtain the determination results; the determination thresholds of the multi-layer privacy determination rules are adjusted in combination with dynamic environment perception and the interactive feedback of the user, including: detecting the privacy risk factors in the current environment based on the built-in sensors of the smart glasses to generate a privacy risk level; dynamically setting the privacy recognition sensitivity in the multi-layer privacy determination rules according to the privacy risk level; obtaining the user's adjustment intention for the current privacy setting through the touch interface of the smart glasses, voice commands or gesture recognition; and updating the sensitivity thresholds of the multi-layer privacy determination rules in real time according to the user's adjustment intention for the current privacy setting; The privacy data in the determination result is marked, and data processing is performed based on the processing logic in the multi-layer privacy determination rule to generate shared data after privacy protection processing; the marking of the privacy data in the determination result, and data processing is performed based on the processing logic in the multi-layer privacy determination rule to generate shared data after privacy protection processing includes: marking the privacy data in the determination result, and assigning a security level according to the sharing priority of the privacy data in the determination result; according to the security level, selecting a corresponding processing method from the processing logic in the multi-layer privacy determination rule to perform data processing to generate shared data after privacy protection processing, wherein the processing logic in the multi-layer privacy determination rule includes fuzzification, shielding and limiting the transmission range; The shared data after the privacy protection processing is stored in the cloud, and according to the sharing requirement information of the target device, the corresponding shared data after the privacy protection processing is retrieved from the cloud for encryption to generate encrypted shared data; The encrypted shared data is transmitted to the target device, the encrypted shared data is decrypted and reconstructed in the target device to generate shared display data, and the shared display data is displayed in a multi-channel interactive manner.

2. The method for controlling smart glasses based on multimodal privacy protection according to claim 1, characterized in that: The privacy feature vector is classified based on the multi-layer privacy determination rule, and the determination threshold of the multi-layer privacy determination rule is adjusted in combination with dynamic environment perception and the interactive feedback of the user, and the determination result obtained by identification includes: Preliminarily screening the sensitive features in the privacy feature vector using the first layer privacy determination rule in the multi-layer privacy determination rule, identifying sensitive areas in the visual features and sensitive keywords in the auditory features, and obtaining sensitive feature data; The second layer of privacy determination rules in the multi-layer privacy determination rules are used to further classify the sensitive feature data, determine the privacy sensitivity label of the sensitive feature data, and identify to obtain the determination result.

3. The method for controlling smart glasses based on multimodal privacy protection according to claim 1, characterized in that: The storing the shared data after the privacy protection processing in the cloud includes: Classify and store the shared data after the privacy protection processing according to the access rights, storage duration and data type of the shared data after the privacy protection processing to obtain stored data; For the highly sensitive data in the stored data, a zero-knowledge encryption method is used to encrypt the highly sensitive data to prevent unauthorized access; During the cloud storage process, the storage status and access frequency of the data are dynamically monitored, and storage priority is adjusted or an automatic cleanup mechanism is triggered according to the user's usage needs and access behavior.

4. The method for controlling smart glasses based on multimodal privacy protection according to claim 1, characterized in that: The decrypting and reconstructing the encrypted shared data in the target device to generate shared display data comprises: Generate a key using a distributed storage method based on blockchain, decrypt the encrypted shared data according to the key and a preset end-to-end key decryption algorithm, and obtain the decrypted shared data; According to the display requirements of the target device, the decrypted shared data is formatted to obtain converted data; The resolution, sound effect level or tactile feedback strength of the converted data is dynamically adjusted according to the hardware data of the target device to obtain the shared display data.

5. A smart glasses control system based on multimodal privacy protection, characterized in that: The smart glasses control system based on multimodal privacy protection includes: A multimodal data collection module, used to collect multimodal sensory data of the user, and pre-process the multimodal sensory data to generate sensory sharing data; A privacy feature analysis module, used to analyze the sensory shared data using a multimodal privacy feature extraction algorithm to generate a privacy feature vector; A privacy determination rule module, used to classify the privacy feature vector based on the multi-layer privacy determination rule, and adjust the determination threshold of the multi-layer privacy determination rule in combination with dynamic environment perception and the interactive feedback of the user, and identify and obtain the determination result; A privacy data processing module, used to mark the privacy data in the determination result, and perform data processing based on the processing logic in the multi-layer privacy determination rule to generate shared data after privacy protection processing; An encrypted data transmission module is used to store the shared data after the privacy protection processing to the cloud, and retrieve the corresponding shared data after the privacy protection processing from the cloud according to the sharing requirement information of the target device, encrypt it, and generate encrypted shared data; a decryption and display module, configured to transmit the encrypted shared data to the target device, decrypt and reconstruct the encrypted shared data in the target device, generate shared display data, and display the shared display data in a multi-channel interactive manner; The privacy feature analysis module includes: a visual recognition submodule, which is used to recognize the visual data in the sensory shared data through a convolutional neural network to generate a sensitive area of ​​the visual data; a speech recognition submodule, which is used to detect the auditory data in the sensory shared data using a speech recognition algorithm to obtain the speech features and sensitive keywords of the auditory data; a context detection submodule, which is used to detect the sensory shared data based on a context perception algorithm to obtain context information; a fusion submodule, which is used to use a feature fusion network to uniformly represent the sensitive areas of the visual data, the speech features and sensitive keywords of the auditory data, and the context information to generate the privacy feature vector; The privacy determination rule module includes: a risk level identification submodule, which is used to detect privacy risk factors in the current environment based on the built-in sensors of the smart glasses and generate a privacy risk level; a sensitivity adjustment submodule, which is used to dynamically set the privacy identification sensitivity in the multi-layer privacy determination rule according to the privacy risk level; an adjustment intention identification submodule, which is used to obtain the user's adjustment intention for the current privacy setting through the touch interface of the smart glasses, voice commands or gesture recognition; and a threshold adjustment submodule, which is used to update the sensitivity threshold of the multi-layer privacy determination rule in real time according to the user's adjustment intention for the current privacy setting; The privacy data processing module includes: a marking submodule, which is used to mark the privacy data in the determination result and assign a security level according to the sharing priority of the privacy data in the determination result; a logic processing submodule, which is used to select a corresponding processing method from the processing logic in the multi-layer privacy determination rule according to the security level to process the data and generate the shared data after the privacy protection processing, wherein the processing logic in the multi-layer privacy determination rule includes fuzzification, shielding and limiting the transmission range.

Images