Data processing method and device for privacy protection of intelligent equipment with body

Abstract

The invention discloses a data processing method and device for privacy protection of an intelligent device, and the method comprises the steps: inputting multi-modal data into a preset dynamic privacy sandbox under the condition of receiving the multi-modal data collected by a sensor preset in the intelligent device, the preset dynamic privacy sandbox is an isolated execution environment constructed by a trusted execution environment chip; in a preset dynamic privacy sandbox, training a locally deployed intention recognition model according to the multi-modal data to obtain small model update parameters, constructing desensitization features according to the multi-modal data, and outputting the small model update parameters and the desensitization features; encrypting the small model update parameters, and sending the parameters to the cloud to update a global model of the cloud; and inputting the desensitization features into the intention recognition model, and outputting a decision instruction corresponding to the multi-modal data. Therefore, by adopting the embodiment of the invention, the security of the data of the intelligent equipment is improved, and meanwhile, the real-time interaction requirement of the intelligent equipment can be met.

Description

Technical Field

[0001] The present application relates to the technical field of privacy protection of embodied smart devices, and in particular to a data processing method and apparatus for privacy protection of embodied smart devices. Background Art

[0002] In smart home environments, embodied intelligent devices (such as smart robots and smart speakers) are widely used in a variety of fields, including home automation, health monitoring, and security. These devices use technologies such as voice recognition, facial expression analysis, and body motion capture to achieve natural interaction with users, providing convenient and personalized services.

[0003] On the one hand, collected multimodal data is typically uploaded to the cloud for global model training, leveraging the cloud's powerful computing power to continuously improve the intelligence of embodied smart devices. However, when user data is uploaded to the cloud for model training, if the cloud storage is attacked, the user's private information is at risk of being leaked, thus reducing the security of the embodied smart device data.

[0004] Furthermore, to ensure data security, encryption algorithms are used during data processing. However, this static encryption method cannot adapt to the real-time interaction requirements of embodied smart devices. For example, voice commands require instant responses, and static encryption introduces processing delays, impacting the user experience. Summary of the Invention

[0005] The embodiments of this application provide a data processing method and apparatus for privacy protection in embodied smart devices. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is provided below. This summary is not intended to be a comprehensive review, identify key or important elements, or delineate the scope of protection for these embodiments. Its sole purpose is to present some concepts in a simplified form, serving as a prelude to the detailed description that follows.

[0006] In a first aspect, embodiments of the present application provide a data processing method for privacy protection of an embodied smart device, which is applied to the embodied smart device and includes: Upon receiving multimodal data collected by sensors pre-installed in the embodied smart device, the multimodal data is input into a preset dynamic privacy sandbox, which is an isolated execution environment built by a trusted execution environment chip; In the preset dynamic privacy sandbox, the locally deployed intent recognition model is trained based on multimodal data to obtain small model update parameters. Desensitization features are constructed based on the multimodal data, and the small model update parameters and desensitization features are output. Encrypt the updated parameters of the small model and send them to the cloud to update the global model in the cloud; input the desensitized features into the intent recognition model and output the decision instructions corresponding to the multimodal data.

[0007] In a second aspect, an embodiment of the present application provides a data processing method for privacy protection of an embodied smart device, which is applied to the cloud and includes: Receive encrypted small model update parameters from each embodied smart device. The small model update parameters are obtained by training the intent recognition model deployed in the embodied smart device based on the multimodal data collected by the embodied smart device within a preset dynamic privacy sandbox, which is an isolated execution environment built by the Trusted Execution Environment chip. Using homomorphic encryption technology, the encrypted small model update parameters of multiple embodied smart devices are aggregated to obtain aggregated information; Based on the aggregated information, the global model in the cloud is trained to obtain the global parameters of the large model; Send the global parameters of the large model to each embodied intelligent device.

[0008] In a third aspect, an embodiment of the present application provides a data processing device for privacy protection of an embodied smart device, the device comprising: A multimodal data input module, configured to, upon receiving multimodal data collected by sensors pre-installed in the embodied smart device, input the multimodal data into a preset dynamic privacy sandbox, which is an isolated execution environment built by a trusted execution environment chip; The information output module is used to train the locally deployed intent recognition model based on multimodal data within a preset dynamic privacy sandbox, obtain small model update parameters, construct desensitization features based on the multimodal data, and output the small model update parameters and desensitization features; The model update module is used to encrypt the small model update parameters and send them to the cloud to update the global model in the cloud; it inputs the desensitized features into the intent recognition model and outputs the decision instructions corresponding to the multimodal data.

[0009] The technical solutions provided by the embodiments of the present application may have the following beneficial effects: In the embodiments of the present application, on the one hand, by constructing a dynamic privacy sandbox locally on the embodied smart device and performing local model training in the sandbox, only the encrypted model update parameters are uploaded to the cloud for global model aggregation, which effectively avoids the direct exposure of the user's original data and significantly improves the security of the data. At the same time, the dynamic privacy sandbox is constructed by a trusted execution environment chip, which can provide hardware-level isolation protection, further improving the security of the embodied smart device data. Therefore, while ensuring data security, the present application also fully utilizes the computing power of the cloud and achieves a continuous improvement in the level of intelligence. On the other hand, by performing local feature extraction and model training in the dynamic privacy sandbox and generating desensitized features for subsequent interactive decision-making, the direct use of original data and frequent encryption and decryption operations are avoided, thereby greatly reducing processing delays and improving the real-time and smoothness of interactions.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] Figure 1 This is a flowchart of a data processing method for privacy protection of an embodied smart device provided by an embodiment of the present application; Figure 2 This is a schematic diagram of sensors deployed on an embodied smart device provided in an embodiment of the present application; Figure 3 This is a schematic diagram of an application scenario provided by an embodiment of the present application; Figure 4 This is a model architecture diagram of an intent recognition model provided in an embodiment of the present application; Figure 5 This is a schematic diagram of a scenario in which different intelligent agents interact with a cloud service platform, as provided in an embodiment of the present application; Figure 6 This is a flowchart of another data processing method for privacy protection of an embodied smart device provided by an embodiment of the present application; Figure 7 This is a flowchart of a method for training an intent recognition model provided in an embodiment of the present application; Figure 8 This is a structural diagram of a data processing device for privacy protection of embodied smart devices provided in an embodiment of the present application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following description and the drawings sufficiently illustrate specific embodiments of the application to enable those skilled in the art to practice them.

[0014] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0015] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0016] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0017] Currently, collected multimodal data is typically uploaded to the cloud for global model training, leveraging the cloud's powerful computing power to continuously improve the intelligence of embodied smart devices. To ensure data security, encryption algorithms are used during data processing.

[0018] The applicants of this application recognize that when uploading user data to the cloud for model training, if the cloud storage is attacked, the user's private information will be at risk of being leaked, thereby reducing the security of the data in the embodied smart device. Furthermore, static encryption methods cannot adapt to the real-time interaction requirements of embodied smart devices. For example, voice commands require instant responses, and static encryption introduces processing delays, affecting the user experience.

[0019] In order to solve the above problems, the present application provides a data processing method and device for privacy protection of embodied smart devices to solve the problems existing in the above-mentioned related technical problems. In the embodiment of the present application, on the one hand, by constructing a dynamic privacy sandbox locally on the embodied smart device and performing local model training in the sandbox, only the encrypted model update parameters are uploaded to the cloud for global model aggregation, which effectively avoids the direct exposure of the user's original data and significantly improves the security of the data. At the same time, the dynamic privacy sandbox is constructed by a trusted execution environment chip, which can provide hardware-level isolation protection, further improving the security of the embodied smart device data. Therefore, while ensuring data security, the present application also makes full use of the computing power of the cloud and achieves a continuous improvement in the level of intelligence. On the other hand, by performing local feature extraction and model training in the dynamic privacy sandbox and generating desensitized features for subsequent interactive decision-making, the direct use of original data and frequent encryption and decryption operations are avoided, thereby greatly reducing processing delays and improving the real-time and smoothness of interaction. The following is a detailed description using an exemplary embodiment.

[0020] The following will be combined with the Figure 1 -Attached Figure 7 This article details the data processing method for privacy protection of embodied smart devices provided in the embodiments of this application. This method can be implemented using a computer program and run on a data processing device for privacy protection of embodied smart devices based on the von Neumann architecture. This computer program can be integrated into an application or run as a standalone tool application.

[0021] See Figure 1 , provides a flow chart of a data processing method for privacy protection of embodied smart devices according to an embodiment of the present application, which is applied to embodied smart devices. Figure 1 As shown, the method of the embodiment of the present application includes the following steps: S101, upon receiving multimodal data collected by sensors pre-installed in an embodied smart device, inputting the multimodal data into a preset dynamic privacy sandbox, where the preset dynamic privacy sandbox is an isolated execution environment constructed by a trusted execution environment chip; Among them, embodied intelligent devices refer to intelligent devices with physical forms and perception capabilities, which can interact with the environment through sensors and actuators. Embodied intelligent devices include robots, smart speakers, smart cameras, etc. In the fields of smart homes, medical care, education, etc., embodied intelligent devices interact naturally with users through multimodal data (such as voice, images, actions, etc.) to provide intelligent services. Multimodal data refers to multiple types of data from different sensors, including voice, images, videos, text, etc. Multimodal data can provide more comprehensive user information, enabling embodied intelligent devices to understand user intentions more accurately and provide more personalized services. For example Figure 2 As shown, the robot is equipped with different sensors, such as cameras, microphones, and tactile sensors, and different sensors can provide different modal data.

[0022] The dynamic privacy sandbox is an isolated execution environment for securely processing multimodal data locally on embodied smart devices. It dynamically adjusts resource access permissions based on task requirements, ensuring that data is protected from unauthorized access or leakage during processing. The TEE chip (Trusted Execution Environment) ensures the security of the dynamic privacy sandbox through hardware-level isolation, preventing data from being accessed by malware or attackers.

[0023] In some embodiments of the present application, sensors on the embodied smart device (such as microphones, cameras, accelerometers, etc.) collect multimodal data from the user, including voice, facial expressions, body movements, etc. The collected multimodal data is transmitted to a dynamic privacy sandbox local to the embodied smart device. The dynamic privacy sandbox is constructed by a TEE chip to ensure the secure processing of data within the sandbox. For example, the TEE chip initializes and constructs the dynamic privacy sandbox. The code and data running in the sandbox are completely isolated from the rest of the device, ensuring data security. The dynamic privacy sandbox dynamically adjusts resource access permissions based on task requirements. For example, only the voice recognition module is allowed to access microphone data, and only the image processing module is allowed to access camera data.

[0024] For example, when a user says "play music" to a smart speaker, the microphone collects voice data; when a user makes a gesture in front of the camera, the camera collects image data. The central processing unit (CPU) of the embodied smart device transmits the collected voice and image data through a secure channel to the dynamic privacy sandbox built by the TEE chip. The TEE chip uses hardware-level isolation technology (such as ARM TrustZone or Intel SGX) to create a secure execution environment within which all data processing occurs.

[0025] When voice recognition is required, the dynamic privacy sandbox only grants the voice recognition module permission to access microphone data; when image recognition is required, the dynamic privacy sandbox only grants the image processing module permission to access camera data.

[0026] S102: In a preset dynamic privacy sandbox, a locally deployed intent recognition model is trained based on the multimodal data to obtain updated parameters for the small model, and desensitization features are constructed based on the multimodal data. The updated parameters and desensitization features for the small model are output. The locally deployed intent recognition model refers to a machine learning model that runs locally on the embodied smart device and is used to identify user intent. This model analyzes multimodal data (such as voice and images) to understand user needs. Based on user input (such as voice commands and gestures), the intent recognition model generates corresponding decision instructions, enabling the device to respond accordingly.

[0027] Small model update parameters refer to model parameter updates generated during local training. By uploading these small model update parameters to the cloud, global model aggregation and optimization can be performed, thereby improving the performance and intelligence of the entire system.

[0028] Desensitized features are abstracted features generated by processing raw multimodal data. These features replace the original multimodal data for subsequent interaction decisions, protecting user privacy while ensuring the device's intelligent interaction capabilities.

[0029] Among them, the dynamic privacy sandbox is preset to generate dynamic security parameters when it is started; for example Figure 4 As shown, the intent recognition model includes a shared encoder, an intent recognition main task decoder, and a privacy task decoder.

[0030] In some embodiments of the present application, the specific process of training a locally deployed intent recognition model based on multimodal data to obtain small model update parameters includes: randomly reorganizing the modal data in the multimodal data to generate random data features; generating random abstract features based on the body movement trajectory and voice intonation changes in user behavior; splicing the random data features with the random abstract features to obtain small model training samples; inputting the small model training samples into a shared encoder to output a unified hidden layer representation; inputting the unified hidden layer representation into the intent recognition main task decoder and the privacy task decoder respectively, and outputting the main task loss value for intent recognition and the privacy task loss value for performing privacy-related tasks; calculating the total loss value of the model based on the dynamic security parameters, the main task loss value and the privacy task loss value, combined with the preset game loss function; the preset game loss function is used to minimize the main task loss value of the intent recognition main task decoder and maximize the privacy task loss value of the privacy task decoder; and generating small model update parameters based on the total model loss value.

[0031] Shuffling refers to randomly adjusting the order of each modal data in multimodal data to generate new data features. Shuffling can disrupt the original data structure, increase its randomness, and thus reduce its identifiability, thereby protecting user privacy. Random abstract features refer to randomized features generated based on user behavior, such as body movement trajectories and voice intonation variations. Small model training samples refer to training samples generated by concatenating shuffled data features with random abstract features. The shared encoder refers to the encoder portion shared within the model, used to encode input data into a unified latent representation. The intent recognition main task decoder refers to the decoder portion used for the intent recognition task. The privacy task decoder refers to the decoder portion used for the privacy protection task. Dynamic security parameters refer to security parameters that are dynamically adjusted based on the real-time monitored device operating status. The preset game loss function refers to the loss function used to balance the intent recognition main task and the privacy protection task. The total model loss refers to the total model loss calculated by combining the main task loss, the privacy task loss, and the dynamic security parameters. The small model update parameters refer to model parameter updates generated based on the total model loss and are used to update the local intent recognition model to improve its performance and privacy protection capabilities.

[0032] In an embodiment of the present application, by randomly reorganizing the modal data in the multimodal data to generate disordered data features, and generating random abstract features based on the body movement trajectory and voice intonation changes in the user behavior, the randomness and unrecognizable nature of the data are further enhanced. The small model training samples generated by splicing these features are input into the shared encoder to output a unified latent layer representation, which not only retains the key information of the data but also further reduces the risk of privacy leakage. By inputting the unified latent layer representation into the intent recognition main task decoder and the privacy task decoder respectively, optimizing the intent recognition and privacy protection tasks at the same time, and combining the preset game loss function to calculate the total loss value of the model, a balance is achieved between the privacy protection and intent recognition tasks. This method not only improves the privacy protection capability of the model, but also enhances the robustness of the model under different security threats by introducing dynamic security parameters.

[0033] Among them, the dynamic security parameters include privacy task weight and game training coefficient. The privacy task weight is used to control the training intensity of the privacy task. The initial value of the privacy task weight is set by the preset dynamic privacy sandbox according to the preset security level. The game training coefficient is used to adjust the intensity of the loss to suppress the leakage of private information.

[0034] Specifically, the specific process of calculating the total loss value of the model based on the dynamic security parameters, the main task loss value and the privacy task loss value, combined with the preset game loss function, includes: multiplying the privacy task weight, the game training coefficient and the privacy task loss value to obtain the product result; subtracting the main task loss value from the product result to obtain the total loss value of the model.

[0035] Among them, the expression of the preset game loss function is:

[0036] in, is the total loss of game training, is the loss value of the main task, is the game training coefficient. The larger the game training coefficient, the stronger the privacy protection. is the privacy task weight, is the privacy task loss value; The loss function of the decoder for the main task of intent recognition is:

[0037] in, is the total number of intent categories, is the one-hot encoding of the true intent label, The model predicts Class intent probability; The loss function of the privacy task decoder is:

[0038] in, is the number of privacy categories, is the one-hot encoding of the privacy label, The model predicts Class privacy attribute probability.

[0039] Among them, the dynamic security parameters include the gradient clipping threshold.

[0040] Specifically, the specific process of generating the small model update parameters based on the total loss value of the model includes: determining the current gradient of the total loss value of the model to the parameters of the intent recognition model; using the gradient clipping threshold to clip the current gradient to obtain the target gradient; using the target gradient to update the network parameters of the intent recognition main task decoder and the privacy task decoder, and obtain the first main task decoder and the first privacy task decoder with updated parameters; inputting the pre-labeled verification sample into the first main task decoder and the first privacy task decoder, and outputting the main task prediction result and the privacy task prediction result; determining the main task accuracy and the privacy task accuracy based on the actual label of the verification sample, the main task prediction result and the privacy task prediction result; when the main task accuracy and the privacy task accuracy are greater than the preset threshold, the updated network parameters are used as the small model update parameters.

[0041] In one possible implementation, assuming a high security level, the dynamic privacy sandbox generates a privacy task weight of 0.8, a game training coefficient of 1.2, and a gradient clipping threshold of 1.0. A user says "play music" to the smart speaker, and the microphone collects voice data. The user gestures in front of the camera, and the camera collects image data. The voice data is shuffled and reassembled to generate shuffled speech features. Body movement trajectories and voice intonation variations are processed to generate random abstract features. The shuffled speech features and random abstract features are concatenated to generate small model training samples. The shared encoder extracts common features from the small model training samples and generates a unified hidden layer representation. The intent recognition main task decoder calculates the main task loss, and the privacy task decoder calculates the privacy task loss.

[0042] For example, the main task loss value =0.5, privacy task loss value is 0.3, the game training coefficient The privacy task weight is 1.2. is 0.8, based on the expression: Can be calculated =0.5 - 1.2 × 0.8 × 0.3 = 0.5 - 0.288 = 0.212.

[0043] Specifically, the specific process of determining the current gradient of the total loss value of the model with respect to the parameters of the intent recognition model includes: the total loss value of the model is back-propagated to calculate the gradient of the total loss value with respect to the model parameters. Specifically, the automatic differentiation mechanism of PyTorch can be used to call the backward() method of the neural network to calculate the gradient.

[0044] Among them, the preset dynamic privacy sandbox generates dynamic security parameters when it is started, and the dynamic security parameters include the current privacy level of the device.

[0045] In some embodiments of the present application, the specific process of constructing desensitizing features based on multimodal data includes: performing modal separation on multimodal data to obtain multiple independent modal streams; performing basic feature extraction on each independent modal stream to obtain a modal behavior feature vector corresponding to each independent modal stream; inputting the modal behavior feature vector corresponding to each independent modal stream into a preset privacy information detector, and outputting a sensitive privacy vector corresponding to each independent modal stream; adding privacy perturbations to the sensitive privacy vectors corresponding to each independent modal stream to obtain privacy protection features corresponding to each independent modal stream; splicing the modal behavior feature vector corresponding to each independent modal stream with the privacy protection features corresponding to each independent modal stream to obtain enhanced input information; and constructing desensitizing features based on the enhanced input information.

[0046] Modal separation refers to separating the different modalities (such as speech, image, and text) in multimodal data into multiple independent modal streams. Modal behavior feature vectors are feature vectors extracted from each independent modal stream that reflect user behavior. These feature vectors can capture user behavior patterns across different modalities, providing a basis for privacy protection and intent recognition. A preset privacy information detector is a predefined model used to detect sensitive privacy information in modal behavior feature vectors. This preset privacy information detector can be set to different types of privacy-related information. Privacy perturbation refers to adding noise or perturbations to sensitive privacy vectors to protect user privacy.

[0047] In an embodiment of the present application, multimodal data is modally separated to obtain multiple independent modal streams. Basic features are then extracted from each independent modal stream to obtain modal behavior feature vectors. These modal behavior feature vectors are further input into a preset privacy information detector to detect sensitive privacy vectors. Privacy perturbations are added to these sensitive privacy vectors to generate privacy-preserving features. By concatenating the modal behavior feature vectors with the privacy-preserving features, enhanced input information is obtained, and finally, desensitized features are constructed. This method not only effectively protects the user's privacy information through the privacy information detector and privacy perturbations, but also retains key information about user behavior by enhancing the input information, ensuring interactive performance.

[0048] In some embodiments of the present application, the specific process of constructing a desensitized feature based on the enhanced input information includes: inputting the enhanced input information into the adversarial generative network to maximize the retention of intent recognition information and minimize sensitive information to obtain an adversarial feature; setting a feature compression ratio according to the current privacy level of the device to control the privacy strength; and compressing the adversarial feature based on the feature compression ratio to obtain a desensitized feature.

[0049] A generative adversarial network (GAN) is a type of generative adversarial network consisting of a generator and a discriminator. The generator generates data, while the discriminator determines the data's authenticity. The two compete against each other, ultimately allowing the generator to generate data that deceives the discriminator. In this application, GANs are used to generate adversarial features, generating more robust features by maximizing the retention of intent recognition information and minimizing sensitive information. The feature compression ratio refers to the ratio of the compressed feature dimensions to the original feature dimensions during feature compression.

[0050] For example Figure 3 As shown, Figure 3This is a scenario diagram of an application scenario provided by this application. The embodied intelligent device collects multimodal data (such as visual sequences, voice sequences, and tactile sequences), and then inputs the multimodal data into a preset dynamic privacy sandbox including an isolated execution environment built by a trusted execution environment chip. Finally, it can output the updated parameters and desensitized features of the small model.

[0051] S103: Encrypt the updated parameters of the small model and send them to the cloud to update the global model in the cloud; input the desensitized features into the intent recognition model and output the decision instructions corresponding to the multimodal data.

[0052] The cloud refers to a remote server or server cluster that provides computing resources and services over the internet. The cloud is used to store and process small model update parameters from multiple embodied intelligent devices. By aggregating these parameters, the global model is updated to improve the overall performance of the system. The global model refers to a model maintained in the cloud that is aggregated from the local model update parameters of multiple embodied intelligent devices. By integrating the updated parameters of multiple devices, the global model can better adapt to the needs of different users and improve the intelligence and generalization capabilities of the system. The decision instruction refers to the final output generated by the intent recognition model based on the desensitized features, instructing the device on the specific operation to be performed.

[0053] In some embodiments, a homomorphic encryption environment is initialized and the small model update parameters are encrypted using a homomorphic encryption algorithm to ensure data security during transmission. The encrypted small model update parameters are sent to the cloud via a secure communication channel. Homomorphic encryption is a special encryption technique that allows computation to be performed directly on ciphertext without first decrypting it, thus avoiding the risk of data leakage in the cloud.

[0054] In some embodiments, the global parameters of the large model sent from the cloud are received; based on the global parameters of the large model, the intent recognition model is updated. For example, the received global parameters are parsed from a string or dictionary format into a format that can be directly used by the model. An HTTP request is used to obtain global parameters from the cloud. Assuming that the cloud provides global parameters through an API interface, these parameters are usually returned in JSON format. The received global parameters in JSON format are parsed into a Python dictionary or list for subsequent processing. All parameters of the model are traversed using the named_parameters method of PyTorch. For each parameter, the corresponding value is obtained from the global parameter according to its name (such as weight or bias), and the copy_ method is used to copy the value of the global parameter to the parameter of the local model.

[0055] In some embodiments, desensitized features are input into an intent recognition model, which generates decision instructions based on these features. The model's output is parsed and converted into specific decision instructions. The model outputs probabilities or log-odds for each intent category, which need to be converted into final decision instructions using a softmax function or other methods. Based on the generated decision instructions, the corresponding actions are executed. These actions can include playing music, adjusting lighting, and so on.

[0056] For example Figure 5 As shown, Figure 5 This application provides a schematic diagram of a scenario in which different intelligent agents interact with a cloud service platform. Different intelligent agents output small model update parameters and desensitized features, encrypt the small model update parameters, and send them to the cloud to update the cloud's global model. The cloud receives the encrypted small model update parameters sent by each intelligent agent and aggregates the encrypted small model update parameters from multiple embodied intelligent devices using homomorphic encryption technology to obtain aggregated information. Based on the aggregated information, the cloud's global model is trained to obtain the global parameters of the large model, which are then sent to each intelligent agent. The intelligent agents receive the global parameters of the large model sent by the cloud and, based on the global parameters of the large model, update the intent recognition model.

[0057] In some embodiments of the present application, the operating status of the embodied smart device can be monitored in real time; when the operating status indicates that the embodied smart device has abnormal behavior, the isolation level of the preset dynamic privacy sandbox is upgraded to adjust the security policy.

[0058] In the embodiments of the present application, on the one hand, by constructing a dynamic privacy sandbox locally on the embodied smart device and performing local model training in the sandbox, only the encrypted model update parameters are uploaded to the cloud for global model aggregation, which effectively avoids the direct exposure of the user's original data and significantly improves the security of the data. At the same time, the dynamic privacy sandbox is constructed by a trusted execution environment chip, which can provide hardware-level isolation protection, further improving the security of the embodied smart device data. Therefore, while ensuring data security, the present application also fully utilizes the computing power of the cloud and achieves a continuous improvement in the level of intelligence. On the other hand, by performing local feature extraction and model training in the dynamic privacy sandbox and generating desensitized features for subsequent interactive decision-making, the direct use of original data and frequent encryption and decryption operations are avoided, thereby greatly reducing processing delays and improving the real-time and smoothness of interactions.

[0059] See Figure 6 , provides a flow chart of a data processing method for embodied smart device privacy protection, which is applied to the cloud. Figure 6 As shown, the method of the embodiment of the present application includes the following steps: S201, receiving encrypted small model update parameters sent by each embodied smart device, where the small model update parameters are obtained by training an intent recognition model deployed in the embodied smart device based on multimodal data collected by the embodied smart device within a preset dynamic privacy sandbox, where the preset dynamic privacy sandbox is an isolated execution environment constructed by a trusted execution environment chip; S202, aggregating the encrypted small model update parameters of multiple embodied smart devices using homomorphic encryption technology to obtain aggregated information; S203, training a global model in the cloud based on the aggregated information to obtain global parameters of the large model; S204: Send the global parameters of the large model to each embodied intelligent device.

[0060] In the embodiments of the present application, on the one hand, by constructing a dynamic privacy sandbox locally on the embodied smart device and performing local model training in the sandbox, only the encrypted model update parameters are uploaded to the cloud for global model aggregation, which effectively avoids the direct exposure of the user's original data and significantly improves the security of the data. At the same time, the dynamic privacy sandbox is constructed by a trusted execution environment chip, which can provide hardware-level isolation protection, further improving the security of the embodied smart device data. Therefore, while ensuring data security, the present application also fully utilizes the computing power of the cloud and achieves a continuous improvement in the level of intelligence. On the other hand, by performing local feature extraction and model training in the dynamic privacy sandbox and generating desensitized features for subsequent interactive decision-making, the direct use of original data and frequent encryption and decryption operations are avoided, thereby greatly reducing processing delays and improving the real-time and smoothness of interactions.

[0061] See Figure 7 , which is a flow chart of a method for training an intent recognition model according to an embodiment of the present application. Figure 7 As shown, the method of the embodiment of the present application may include the following steps: S301, reorganizing the modal data in the multimodal data in a disordered order to generate disordered data features; S302, generating random abstract features based on the body movement trajectory and voice intonation changes in the user's behavior; S303, concatenating the out-of-order data features with the random abstract features to obtain a small model training sample; S304, input the small model training sample into the shared encoder and output a unified hidden layer representation; S305: Input the unified hidden layer representation into the intent recognition main task decoder and the privacy task decoder respectively, and output the main task loss value for intent recognition and the privacy task loss value for performing the privacy-related task; S306, calculating the total loss of the model based on the dynamic security parameters, the main task loss value, and the privacy task loss value, combined with a preset game loss function; the preset game loss function is used to minimize the main task loss value of the intent recognition main task decoder and maximize the privacy task loss value of the privacy task decoder; S307: Generate small model update parameters based on the total loss value of the model.

[0062] In an embodiment of the present application, by randomly reorganizing the modal data in the multimodal data to generate disordered data features, and generating random abstract features based on the body movement trajectory and voice intonation changes in the user behavior, the randomness and unrecognizable nature of the data are further enhanced. The small model training samples generated by splicing these features are input into the shared encoder to output a unified latent layer representation, which not only retains the key information of the data but also further reduces the risk of privacy leakage. By inputting the unified latent layer representation into the intent recognition main task decoder and the privacy task decoder respectively, optimizing the intent recognition and privacy protection tasks at the same time, and combining the preset game loss function to calculate the total loss value of the model, a balance is achieved between the privacy protection and intent recognition tasks. This method not only improves the privacy protection capability of the model, but also enhances the robustness of the model under different security threats by introducing dynamic security parameters.

[0063] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0064] See Figure 8 , which shows a schematic diagram of the structure of a data processing device for privacy protection of embodied smart devices, provided by an exemplary embodiment of the present application. The data processing device for privacy protection of embodied smart devices can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device 1 includes a multimodal data input module 10, an information output module 20, and a model update module 30.

[0065] A multimodal data input module 10 is configured to, upon receiving multimodal data collected by sensors pre-installed in the embodied smart device, input the multimodal data into a preset dynamic privacy sandbox, which is an isolated execution environment constructed by a trusted execution environment chip; Information output module 20, used to train a locally deployed intent recognition model based on multimodal data within a preset dynamic privacy sandbox, obtain small model update parameters, construct desensitization features based on the multimodal data, and output the small model update parameters and desensitization features; The model update module 30 is used to encrypt the small model update parameters and send them to the cloud to update the global model in the cloud; input the desensitized features into the intent recognition model and output the decision instructions corresponding to the multimodal data.

[0066] It should be noted that the data processing apparatus for embodied smart device privacy protection provided in the above embodiments, when executing the data processing method for embodied smart device privacy protection, only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing apparatus for embodied smart device privacy protection provided in the above embodiments and the data processing method for embodied smart device privacy protection embodiments are based on the same concept. The implementation process is detailed in the method embodiments and will not be repeated here.

[0067] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0068] In the embodiments of the present application, on the one hand, by constructing a dynamic privacy sandbox locally on the embodied smart device and performing local model training in the sandbox, only the encrypted model update parameters are uploaded to the cloud for global model aggregation, which effectively avoids the direct exposure of the user's original data and significantly improves the security of the data. At the same time, the dynamic privacy sandbox is constructed by a trusted execution environment chip, which can provide hardware-level isolation protection, further improving the security of the embodied smart device data. Therefore, while ensuring data security, the present application also fully utilizes the computing power of the cloud and achieves a continuous improvement in the level of intelligence. On the other hand, by performing local feature extraction and model training in the dynamic privacy sandbox and generating desensitized features for subsequent interactive decision-making, the direct use of original data and frequent encryption and decryption operations are avoided, thereby greatly reducing processing delays and improving the real-time and smoothness of interactions.

[0069] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implements the data processing method for privacy protection of embodied smart devices provided by the above-mentioned various method embodiments.

[0070] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the data processing method for embodied smart device privacy protection of each of the above method embodiments.

[0071] See Figure 9 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 9 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0072] The communication bus 1002 is used to implement the connection and communication between these components.

[0073] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0074] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0075] The processor 1001 may include one or more processing cores. The processor 1001 utilizes various interfaces and circuits to connect various components within the electronic device 1000. It executes instructions, programs, code sets, or instruction sets stored in the memory 1005, and accesses data stored in the memory 1005 to perform various functions and process data within the electronic device 1000. Optionally, the processor 1001 may be implemented in hardware using at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 1001 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the display; and the modem handles wireless communications. It is understood that the modem may also be implemented independently of the processor 1001 and implemented on a separate chip.

[0076] Among them, the memory 1005 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 9 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data processing application for privacy protection of the embodied smart device.

[0077] exist Figure 9 In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain user input data; and the processor 1001 can be used to call the data processing application for embodied smart device privacy protection stored in the memory 1005 and specifically perform the following operations: Upon receiving multimodal data collected by sensors pre-installed in the embodied smart device, the multimodal data is input into a preset dynamic privacy sandbox, which is an isolated execution environment built by a trusted execution environment chip; In the preset dynamic privacy sandbox, the locally deployed intent recognition model is trained based on multimodal data to obtain small model update parameters. Desensitization features are constructed based on the multimodal data, and the small model update parameters and desensitization features are output. Encrypt the updated parameters of the small model and send them to the cloud to update the global model in the cloud; input the desensitized features into the intent recognition model and output the decision instructions corresponding to the multimodal data.

[0078] In one embodiment, the processor 1001 further performs the following operations: Real-time monitoring of the operating status of embodied smart devices; When the operating status indicates that the embodied smart device has abnormal behavior, the isolation level of the preset dynamic privacy sandbox is upgraded to adjust the security policy.

[0079] In one embodiment, when the processor 1001 trains the locally deployed intent recognition model based on the multimodal data and obtains the small model update parameters, it specifically performs the following operations: The modal data in the multimodal data are reorganized in random order to generate random data features; Generate random abstract features based on the body movement trajectory and voice intonation changes in user behavior; The out-of-order data features are concatenated with random abstract features to obtain small model training samples; Input the small model training samples into the shared encoder and output a unified hidden layer representation; The unified hidden layer representation is input into the intent recognition main task decoder and the privacy task decoder respectively, and the main task loss value for intent recognition and the privacy task loss value for performing privacy-related tasks are output; The total loss of the model is calculated based on the dynamic security parameters, the main task loss value, and the privacy task loss value, combined with the preset game loss function. The preset game loss function is used to minimize the main task loss value of the intent recognition main task decoder and maximize the privacy task loss value of the privacy task decoder. Generate small model update parameters based on the total loss value of the model.

[0080] In one embodiment, when the processor 1001 calculates the total loss value of the model based on the dynamic security parameters, the main task loss value, and the privacy task loss value, combined with a preset game loss function, it specifically performs the following operations: Product the privacy task weight, the game training coefficient, and the privacy task loss value to obtain the product result; Subtract the main task loss value from the product result to get the total model loss value.

[0081] In one embodiment, when the processor 1001 generates the small model update parameters according to the total loss value of the model, it specifically performs the following operations: Determine the current gradient of the model's total loss with respect to the parameters of the intent recognition model; Use the gradient clipping threshold to clip the current gradient and obtain the target gradient; Using the target gradient, update the network parameters of the intent recognition main task decoder and the privacy task decoder to obtain the first main task decoder and the first privacy task decoder with updated parameters; Input the pre-labeled verification samples into the first main task decoder and the first privacy task decoder, and output the main task prediction results and the privacy task prediction results; Determine the accuracy of the main task and the privacy task based on the actual labels of the verification samples, the prediction results of the main task, and the prediction results of the privacy task. When the accuracy of the main task and the privacy task is greater than the preset threshold, the updated network parameters are used as the small model update parameters.

[0082] In one embodiment, when constructing a desensitization feature based on multimodal data, the processor 1001 specifically performs the following operations: Perform modal separation on multimodal data to obtain multiple independent modal streams; Perform basic feature extraction on each independent modal flow to obtain the modal behavior feature vector corresponding to each independent modal flow; Input the modal behavior feature vector corresponding to each independent modal stream into a preset privacy information detector, and output the sensitive privacy vector corresponding to each independent modal stream; Add privacy perturbations to the sensitive privacy vectors corresponding to each independent modal stream to obtain the privacy protection features corresponding to each independent modal stream; The modal behavior feature vector corresponding to each independent modal stream is concatenated with the privacy protection feature corresponding to each independent modal stream to obtain enhanced input information; Construct desensitizing features based on the enhanced input information.

[0083] In one embodiment, when constructing a desensitization feature based on enhanced input information, the processor 1001 specifically performs the following operations: The enhanced input information is fed into the adversarial generative network to maximize the retention of intent recognition information and minimize sensitive information, thereby obtaining adversarial features. Set the feature compression ratio based on the device's current privacy level to control privacy strength; Based on the feature compression ratio, the adversarial features are compressed to obtain desensitized features.

[0084] In one embodiment, the processor 1001 further performs the following operations: Receive the global parameters of the large model sent by the cloud; Update the intent recognition model based on the global parameters of the large model.

[0085] In the embodiments of the present application, on the one hand, by constructing a dynamic privacy sandbox locally on the embodied smart device and performing local model training in the sandbox, only the encrypted model update parameters are uploaded to the cloud for global model aggregation, which effectively avoids the direct exposure of the user's original data and significantly improves the security of the data. At the same time, the dynamic privacy sandbox is constructed by a trusted execution environment chip, which can provide hardware-level isolation protection, further improving the security of the embodied smart device data. Therefore, while ensuring data security, the present application also fully utilizes the computing power of the cloud and achieves a continuous improvement in the level of intelligence. On the other hand, by performing local feature extraction and model training in the dynamic privacy sandbox and generating desensitized features for subsequent interactive decision-making, the direct use of original data and frequent encryption and decryption operations are avoided, thereby greatly reducing processing delays and improving the real-time and smoothness of interactions.

[0086] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The data processing program for embodied smart device privacy protection can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium for the data processing program for embodied smart device privacy protection can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0087] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method for privacy protection of embodied smart devices, characterized in that: Applied to an embodied intelligent device, the method includes: Upon receiving multimodal data collected by sensors pre-installed in the embodied smart device, inputting the multimodal data into a preset dynamic privacy sandbox, which is an isolated execution environment constructed by a trusted execution environment chip; In a preset dynamic privacy sandbox, a locally deployed intent recognition model is trained based on the multimodal data to obtain small model update parameters, and desensitization features are constructed based on the multimodal data, and the small model update parameters and the desensitization features are output; Encrypt the small model update parameters and send them to the cloud to update the global model in the cloud; input the desensitized features into the intent recognition model, and output the decision instructions corresponding to the multimodal data.

2. The method according to claim 1, characterized in that The method further comprises: monitoring the operating status of the embodied smart device in real time; When the operating status indicates that the embodied smart device has abnormal behavior, the isolation level of the preset dynamic privacy sandbox is upgraded to adjust the security policy.

3. The method according to claim 1, characterized in that The preset dynamic privacy sandbox generates dynamic security parameters when it is started; the intent recognition model includes a shared encoder, an intent recognition main task decoder, and a privacy task decoder; The step of training a locally deployed intent recognition model based on the multimodal data to obtain updated parameters of a small model includes: Rearranging the modal data in the multimodal data in a disordered order to generate disordered data features; Generate random abstract features based on the body movement trajectory and voice intonation changes in user behavior; The disordered data features are concatenated with the random abstract features to obtain a small model training sample; Inputting the small model training sample into the shared encoder and outputting a unified hidden layer representation; Inputting the unified hidden layer representation into the intention recognition main task decoder and the privacy task decoder respectively, and outputting the main task loss value for intention recognition and the privacy task loss value for performing the privacy-related task; Calculate the total loss value of the model based on the dynamic security parameter, the main task loss value, and the privacy task loss value in combination with a preset game loss function; the preset game loss function is used to minimize the main task loss value of the intent recognition main task decoder and maximize the privacy task loss value of the privacy task decoder; Generate small model update parameters based on the total loss value of the model.

4. The method according to claim 3, characterized in that The dynamic security parameters include a privacy task weight and a game training coefficient. The privacy task weight is used to control the training intensity of the privacy task. The initial value of the privacy task weight is set by the preset dynamic privacy sandbox according to the preset security level. The game training coefficient is used to adjust the intensity of the loss to suppress the leakage of private information. The calculating of the total loss value of the model according to the dynamic security parameter, the main task loss value, and the privacy task loss value in combination with a preset game loss function includes: Producting the privacy task weight, the game training coefficient, and the privacy task loss value to obtain a product result; Subtract the main task loss value from the product result to obtain the total model loss value; wherein, Among them, the expression of the preset game loss function is: in, is the total loss of game training, is the loss value of the main task, is the game training coefficient. The larger the game training coefficient, the stronger the privacy protection. is the privacy task weight, is the privacy task loss value; The loss function of the intention recognition main task decoder is: in, is the total number of intent categories, is the one-hot encoding of the true intent label, The model predicts Class intent probability; The loss function of the privacy task decoder is: in, is the number of privacy categories, is the one-hot encoding of the privacy label, The model predicts Class privacy attribute probability.

5. The method according to claim 3, characterized in that The dynamic security parameters include a gradient clipping threshold; Generating small model update parameters according to the total loss value of the model includes: Determining a current gradient of the total model loss value with respect to parameters of the intent recognition model; Using the gradient clipping threshold, clipping the current gradient to obtain a target gradient; Using the target gradient, updating the network parameters of the intent recognition main task decoder and the privacy task decoder to obtain a first main task decoder and a first privacy task decoder with updated parameters; Input the pre-labeled verification samples into the first main task decoder and the first privacy task decoder, and output the main task prediction results and the privacy task prediction results; Determine the accuracy of the main task and the accuracy of the privacy task based on the actual label of the verification sample, the main task prediction result, and the privacy task prediction result; When the accuracy of the main task and the accuracy of the privacy task are greater than a preset threshold, the updated network parameters are used as small model update parameters.

6. The method according to claim 1, characterized in that The preset dynamic privacy sandbox generates dynamic security parameters when it is started, and the dynamic security parameters include the current privacy level of the device; The constructing of desensitization features according to the multimodal data includes: Performing modal separation on the multimodal data to obtain multiple independent modal streams; Perform basic feature extraction on each independent modal flow to obtain the modal behavior feature vector corresponding to each independent modal flow; Inputting the modal behavior feature vector corresponding to each independent modal stream into a preset privacy information detector, and outputting a sensitive privacy vector corresponding to each independent modal stream; Adding privacy perturbations to the sensitive privacy vectors corresponding to each independent modal stream to obtain privacy protection features corresponding to each independent modal stream; concatenating the modal behavior feature vector corresponding to each independent modal stream with the privacy protection feature corresponding to each independent modal stream to obtain enhanced input information; A desensitized feature is constructed based on the enhanced input information.

7. The method according to claim 6, characterized in that The step of constructing a desensitization feature based on the enhanced input information includes: Inputting the enhanced input information into a generative adversarial network to maximize the retention of intent recognition information and minimize sensitive information, thereby obtaining adversarial features; Setting a feature compression ratio according to the current privacy level of the device to control the privacy strength; Based on the feature compression ratio, feature compression is performed on the adversarial feature to obtain a desensitized feature.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Receive the global parameters of the large model sent by the cloud; Based on the large model global parameters, the intent recognition model is updated.

9. A data processing method for privacy protection of embodied smart devices, characterized in that: Applied to the cloud, the method includes: Receive encrypted small model update parameters sent by each embodied smart device, where the small model update parameters are obtained by training an intent recognition model deployed in the embodied smart device based on multimodal data collected by the embodied smart device within a preset dynamic privacy sandbox, where the preset dynamic privacy sandbox is an isolated execution environment constructed by a trusted execution environment chip; Using homomorphic encryption technology, the encrypted small model update parameters of multiple embodied smart devices are aggregated to obtain aggregated information; Based on the aggregated information, a global model in the cloud is trained to obtain global parameters of the large model; The global parameters of the large model are sent to each of the embodied intelligent devices.

10. A data processing device for privacy protection of embodied smart devices, characterized in that: The device comprises: A multimodal data input module, configured to, upon receiving multimodal data collected by sensors pre-installed in the embodied smart device, input the multimodal data into a preset dynamic privacy sandbox, which is an isolated execution environment constructed by a trusted execution environment chip; An information output module is configured to train a locally deployed intent recognition model based on the multimodal data within a preset dynamic privacy sandbox to obtain small model update parameters, construct desensitization features based on the multimodal data, and output the small model update parameters and the desensitization features; The model update module is used to encrypt the small model update parameters and send them to the cloud to update the global model in the cloud; input the desensitized features into the intent recognition model, and output the decision instructions corresponding to the multimodal data.

Images