Method and system for processing body-interaction data

Abstract

The application discloses a processing method and system of embodied interaction data. The processing method is applied to an embodied intelligent device, the embodied intelligent device is provided with a multi-source sensor array and an edge computing unit, and the processing method comprises the following steps: acquiring embodied interaction data between the embodied intelligent device and a target user through the multi-source sensor array, and sending the embodied interaction data to the edge computing unit for local storage; pre-processing the embodied interaction data through the edge computing unit to generate pre-processed data, and locally storing the pre-processed data in the edge computing unit; determining a user personality vector based on the pre-processed data through the edge computing unit; and uploading the user personality vector to a cloud server through the edge computing unit. The embodied interaction data and the pre-processed data are locally stored, and only the user personality vector which cannot reverse the user interaction information is encrypted and uploaded to the cloud, so that the privacy leakage risk of data in the transmission and cloud storage process can be effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a method and system for processing embodied interactive data. Background Technology

[0002] With the rapid development of technology, embodied intelligent devices (such as humanoid robots, robotic dogs, and intelligent dolls) are being widely used in scenarios such as family companionship, social entertainment, and elderly care. These embodied intelligent devices continuously collect multimodal embodied interaction data from users, including facial expressions, voice, and touch, through various sensors such as vision, hearing, and touch, in order to perform operations such as user personality modeling.

[0003] In data processing solutions for embodied smart devices, a common approach is to upload all data to the cloud. This means the embodied smart device directly uploads all raw sensor data it collects to a cloud server, which then handles data storage and personality modeling. However, this approach has significant drawbacks: all private embodied interaction data between the user and the embodied smart device in environments such as the home is exposed to network transmission and cloud storage, posing a serious risk of privacy breaches.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for processing embodied interactive data, so as to solve the problem of privacy leakage risk caused by uploading all embodied interactive data to a cloud server.

[0006] In a first aspect, the present invention provides a method for processing embody interactive data. The processing method is applied to an embody smart device, the embody smart device being equipped with a multi-source sensor array and an edge computing unit, and the processing method includes: The multi-source sensor array acquires embodied interaction data with the target user, and sends the embodied interaction data to the edge computing unit for local storage. The edge computing unit preprocesses the embodied interaction data to generate preprocessed data, and stores the preprocessed data locally in the edge computing unit. The edge computing unit determines the user personality vector of the target user based on the preprocessed data. The user personality vector is encrypted and uploaded to the cloud server through the edge computing unit.

[0007] In one implementation of the first aspect, determining the user personality vector of the target user based on the preprocessed data by the edge computing unit includes: Extract the specific interaction features of the target user from the preprocessed data; Local feature data is generated based on the embodied interaction features, and the local feature data is stored locally in the edge computing unit. The user personality vector of the target user is generated based on the local feature data.

[0008] In one implementation of the first aspect, the user personality vector is encrypted and uploaded to a cloud server via the edge computing unit, including: The user personality vector is uploaded to the cloud server through an encrypted channel; wherein the encrypted channel is an encrypted transmission channel based on a symmetric encryption algorithm, an encrypted transmission channel based on an asymmetric encryption algorithm, or an encrypted transmission channel based on the Secure Sockets Layer protocol.

[0009] In one implementation of the first aspect, the user personality vector is encrypted and uploaded to a cloud server via the edge computing unit, including: When the network connection between the edge computing unit and the cloud server is interrupted, the edge computing unit records the current upload progress and caches user personality vector data that has not yet been uploaded; Once the edge computing unit detects that the network connection has been restored, it continues to upload the cached but not yet uploaded user personality vector data to the cloud server according to the current upload progress.

[0010] In one implementation of the first aspect, the method further includes: When the local storage of the edge computing unit reaches a preset capacity threshold, the local data stored in the local storage is processed according to a preset data lifecycle management strategy. The local data includes at least one of the embodied interaction data, the preprocessed data, and the local feature data.

[0011] In one implementation of the first aspect, the data lifecycle management strategy includes at least one of a data timeliness-based cleanup strategy, a data importance-based hierarchical storage strategy, and an incremental update-based differential storage strategy.

[0012] In one implementation of the first aspect, the preprocessing includes at least one of data cleaning, data alignment, and noise suppression.

[0013] In one implementation of the first aspect, the edge computing unit preprocesses the embodied interaction data to generate preprocessed data, including: The data from each modal sensor in the embodied interaction data are subjected to time-series alignment processing based on hardware timestamps. The sensor data of each mode after time alignment processing are filtered and denoised. Normalization is performed on the sensor data of each mode after filtering and denoising.

[0014] In one implementation of the first aspect, the edge computing unit is any one of an embedded edge computing module, a system-on-a-chip, a field-programmable gate array, and an application-specific integrated circuit.

[0015] Secondly, the present invention provides a system for processing embodied interactive data. The system includes an embodied intelligent device, which is equipped with a multi-source sensor array and an edge computing unit. The edge computing unit includes a preprocessing module, a personality vector generation module, and an encrypted upload module. The multi-source sensor array is used to acquire embodied interaction data with the target user and send the embodied interaction data to the edge computing unit for local storage. The preprocessing module is used to preprocess the embodied interaction data to generate preprocessed data, and to store the preprocessed data locally in the edge computing unit; The personality vector generation module is used to determine the user personality vector of the target user based on the preprocessed data; The encrypted upload module is used to encrypt and upload the user personality vector to the cloud server.

[0016] The beneficial effects of this invention are as follows: By offloading all processing operations for generating user personality vectors based on embodied interaction data to the edge computing unit of the embodied smart device, and storing privacy data such as embodied interaction data and preprocessed data locally on the embodied smart device, only the user personality vectors, which cannot be reversed to reconstruct the original user interaction information, are encrypted and uploaded to the cloud server. Compared with the processing mode of uploading all data to the cloud, this invention effectively mitigates the risk of privacy leakage during data transmission and cloud storage, allowing users to confidently allow embodied smart devices to collect embodied interaction data in private environments such as homes, reducing the risk of data leakage. Furthermore, compared to uploading massive amounts of original embodied interaction data, this invention reduces the data transmission volume by several orders of magnitude by uploading only low-dimensional user personality vectors, thereby significantly reducing network bandwidth consumption and effectively alleviating the storage pressure on the cloud server. Attached Figure Description

[0017] Figure 1 This is a system architecture diagram of a personalized interactive data processing system disclosed in an embodiment of the present invention.

[0018] Figure 2This is a flowchart of the steps of a method for processing embodied interactive data according to an embodiment of the present invention.

[0019] Figure 3 Schematic illustration Figure 2 Flowchart of the sub-steps in step 102.

[0020] Figure 4 Schematic illustration Figure 2 The flowchart of the sub-step in step 104.

[0021] Figure 5 Schematic illustration Figure 2 The flowchart of the sub-step in step 106.

[0022] Figure 6 This is an application example diagram of a method for processing embodied interactive data disclosed in an embodiment of the present invention. Detailed Implementation

[0023] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0024] First, a definition of the terminology used in this application is provided: Embodied intelligence technology refers to a comprehensive technological system that enables intelligent agents to dynamically interact with the real environment through physical entities, achieving intelligent emergence and continuous evolution through closed-loop feedback of perception, cognition, decision-making, and execution. The core of embodied intelligence technology lies in the deep integration of artificial intelligence algorithms with the physical body, motion control, and multimodal sensing, enabling machines to autonomously learn, adapt to the environment, and complete tasks through real-time interaction with the physical world and humans.

[0025] Embossed intelligent devices refer to physical machines equipped with artificial intelligence systems, possessing multi-source perception capabilities and physical execution capabilities. They can autonomously collect information, understand scenes, and perform physical actions in real environments, achieving dynamic interaction with the physical world and humans. Typical forms include humanoid robots, multi-legged robots, intelligent service robots, and embodied intelligent dolls.

[0026] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of the present invention by those skilled in the art, the relevant technologies are described below: With the rapid development of embodied intelligence technology, embodied intelligent devices (such as bipedal humanoid robots, multi-legged robots, and intelligent dolls) are increasingly widely used as interactive objects for users in the physical world, in scenarios such as family companionship, social entertainment, and elderly care. Embodied intelligent devices continuously collect multimodal embodied interaction data generated by users during their interactions with the devices through multi-source sensor arrays deployed on the device (such as visual sensors, auditory sensors, tactile sensors, motion sensors, and environmental sensors). This embodied interaction data includes multi-dimensional information such as the user's facial expressions, body posture, voice content, touch behavior, and spatial location, which contains rich personality trait information and is the core data foundation for realizing user personality modeling and social matching.

[0027] The inventors learned that most data processing solutions for embodied intelligent devices employ a full data upload to the cloud. Specifically, the embodied intelligent device uploads all collected raw embodied interaction data to a cloud server, where data storage, feature extraction, and personality modeling are performed. However, this processing model has several drawbacks: (1) The private interaction data between users and embodied smart devices in the home environment (such as facial expressions, voice content, touch behavior, etc.) are all exposed in network transmission and cloud storage, which poses a serious risk of privacy leakage.

[0028] (2) The processing of embodied interaction data is highly dependent on the computing resources of cloud servers. When the network connection is unstable or the bandwidth is limited, the data upload delay will lead to a decrease in the real-time performance of personality modeling; when the cloud server is overloaded, the data processing delay will be further aggravated. This cloud-dependent model is difficult to meet the real-time requirements of embodied intelligent social scenarios.

[0029] (3) Lack of effective data hierarchical processing and local storage mechanisms. Data at different levels, such as raw embodied interaction data, intermediate processing data, and final user personality vectors, are all uploaded to cloud servers, which not only consumes a large amount of network bandwidth and cloud storage resources, but also increases the attack surface exposed to the data.

[0030] (4) When performing social matching for users, cloud servers can access users' original embodied interaction data. This means that platform operators or cloud server providers can obtain users' most private interaction information, further exacerbating privacy concerns.

[0031] To at least address the aforementioned issues, this invention provides a localized processing solution for embodied interaction data based on robot edge computing. In this solution, data processing is offloaded to the edge computing unit of the embodied intelligent device, where data preprocessing, feature extraction, and user personality vector generation are performed locally. Only the user personality vector, whose privacy cannot be reversed, is uploaded to the cloud server, fundamentally protecting the privacy of the user's embodied interaction data.

[0032] The technical solutions according to the present invention will be described below with reference to specific embodiments and in conjunction with the accompanying drawings.

[0033] Figure 1 This is a system architecture diagram of a personalized interactive data processing system disclosed in an embodiment of the present invention. (Refer to...) Figure 1 The processing system includes an embodied intelligent device and a cloud server. The embodied intelligent device is equipped with a multi-source sensor array and an edge computing unit. The edge computing unit includes a preprocessing module, a personality vector generation module, and an encrypted upload module. The multi-source sensor array is used to acquire embodied interaction data with the target user and send the embodied interaction data to the edge computing unit for local storage. The preprocessing module is used to preprocess the embodied interaction data to generate preprocessed data, and to store the preprocessed data locally in the edge computing unit; The personality vector generation module is used to determine the user personality vector of the target user based on the preprocessed data; The encrypted upload module is used to encrypt and upload the user personality vector to the cloud server.

[0034] As described above, the beneficial effects of this invention are as follows: By setting up a corresponding edge computing module in the embodied intelligent device, when the multi-source sensor array acquires the corresponding embodied interaction data, the embodied interaction data is directly sent to the edge computing module for preprocessing and user personality vector construction. Furthermore, the embodied interaction data, preprocessed data, and other privacy data are stored locally in the edge computing unit, while only the user personality vector, whose privacy cannot be reversed, is uploaded to the cloud server, fundamentally protecting user privacy.

[0035] The following combination Figure 1 The above-mentioned embodied intelligent devices, cloud servers, and other optional structures are described in detail.

[0036] The embodied intelligent device includes an embodied intelligent body and modules such as an edge computing unit, a multi-source sensor array, and a communication unit disposed on the embodied intelligent body.

[0037] The edge computing unit can be any one of an embedded edge computing module, a system-on-a-chip (SoC), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In practical applications, the edge computing unit communicates with the cloud server via a communication unit (e.g., encrypted communication), which supports multiple communication methods such as Wi-Fi, 5G, and Bluetooth.

[0038] In some embodiments, the edge computing unit may further include a local storage module for storing local data such as the original embodied interaction data, the first-level processing data, and the local feature data. It should be noted that the local data stored in the local storage module is not uploaded to the cloud server. In practical applications, the local storage module can also be used to process the local data stored in the local storage according to a preset data lifecycle management strategy when the local storage of the edge computing unit reaches a preset capacity threshold. It should be understood that the preprocessing module, the personality vector generation module, the encrypted upload module, and the local storage module can be further configured to perform corresponding steps or actions in the embodied interaction data processing method, which will not be elaborated further here.

[0039] The multi-source sensor array refers to a sensing system composed of various types of sensors deployed on an embodied intelligent entity. In practical applications, when the embodied intelligent device interacts with the target user in physical space, the multi-source sensor array can collect and capture multimodal sensor data (i.e., embodied interaction data). After collecting the corresponding embodied interaction data, the multi-source sensor array will send the embodied interaction data to the edge computing unit connected to it. It should be noted that the sensor settings in the multi-source sensor array can be selected according to the actual situation. For example, the multi-source sensor array may include at least three of the following sensors: visual sensors (such as at least one of RGB cameras, depth cameras, infrared cameras, and event cameras), auditory sensors (such as at least one of single microphones, microphone arrays, and bone conduction sensors), tactile sensors (such as at least one of pressure sensor arrays, capacitive tactile sensors, resistive tactile sensors, and six-dimensional force sensors), motion sensors (such as at least one of inertial measurement units, accelerometers, gyroscopes, and magnetometers), and environmental sensors (such as at least one of temperature sensors, humidity sensors, light sensors, distance sensors, and ultra-wideband positioning sensors).

[0040] The cloud server refers to a cluster of virtualized or physical servers deployed in a remote data center, providing computing resources and services to users via the internet. In practical applications, this cloud server connects to edge computing units via encrypted communication. The cloud server can receive user personality vectors uploaded by at least one embodied intelligent device and perform corresponding cloud-based social matching. Specifically, when the cloud server receives multiple user personality vectors uploaded by multiple embodied intelligent devices, it can construct a user personality vector database based on these vectors to support social matching between users. It should be noted that when the cloud server performs social matching based on the user personality vectors, it can only access the user's personality vectors and cannot obtain the original embodied interaction data, first-level processing data, and local feature data corresponding to the user personality vectors. This fundamentally protects user privacy and reduces the risk of data leakage. Simultaneously, this "usable but invisible" privacy protection paradigm prevents platform operators and cloud server providers from prying into users' private interaction information, fundamentally establishing user trust in the platform.

[0041] Figure 2 This is a flowchart illustrating the steps of a method for processing embodied interactive data according to an embodiment of the present invention. This processing method is applied to an embodied smart device, which is equipped with a multi-source sensor array and an edge computing unit. (Refer to...) Figure 2 The processing method includes the following steps 100 to 106.

[0042] Step 100: Acquire embodied interaction data with the target user through the multi-source sensor array, and send the embodied interaction data to the edge computing unit for local storage; Step 102: The embodied interaction data is preprocessed by the edge computing unit to generate preprocessed data, and the preprocessed data is stored locally in the edge computing unit. Step 104: The edge computing unit determines the user personality vector of the target user based on the preprocessed data; Step 106: The user personality vector is encrypted and uploaded to the cloud server through the edge computing unit.

[0043] As described above, the beneficial effects of this invention are as follows: by offloading all processing operations for generating user personality vectors based on embodied interaction data to the edge computing unit of the embodied smart device, and storing privacy data such as embodied interaction data and preprocessed data locally on the embodied smart device, only the user personality vectors, which cannot be reversed to reconstruct the original user interaction information, are encrypted and uploaded to the cloud server. Compared with the processing mode of uploading all data to the cloud, this invention effectively mitigates the risk of privacy leakage during data transmission and cloud storage, allowing users to confidently allow embodied smart devices to collect embodied interaction data in private environments such as homes, reducing the risk of data leakage. Furthermore, compared to uploading massive amounts of original embodied interaction data (such as high-definition video streams, high-fidelity audio streams, etc.), this invention reduces the data transmission volume by several orders of magnitude by uploading only low-dimensional user personality vectors, thereby significantly reducing network bandwidth usage and effectively alleviating the storage pressure on the cloud server.

[0044] The following combination Figure 2 The steps in steps 100 to 106, as well as other optional steps, are described in detail.

[0045] Step 100: Acquire embodied interaction data with the target user through the multi-source sensor array, and send the embodied interaction data to the edge computing unit for local storage.

[0046] The embodied interaction data refers to the multimodal sensor data continuously collected by the embodied intelligent device through its multi-source sensor array during physical interaction with the target user. Specifically, when the embodied intelligent device interacts with the target user in physical space, the multi-source sensor array on the embodied intelligent device collects the embodied interaction data (i.e., multimodal sensor data) generated during the interaction and sends it to the edge computing unit. In practical applications, the embodied interaction data can include sensor data of at least three modalities: visual modal data (such as image frame sequences, depth map sequences), auditory modal data (such as audio streams), tactile modal data (such as pressure distribution sequences), motion modal data (such as IMU data sequences), and environmental modal data (such as temperature, humidity, light, and location data).

[0047] Step 102: The embodied interaction data is preprocessed by the edge computing unit to generate preprocessed data, and the preprocessed data is stored locally in the edge computing unit.

[0048] The preprocessing refers to the initial processing operations performed on the raw embodied interaction data on the edge computing unit of the embodied intelligent device itself, which provides a clean and consistent data foundation for subsequent feature extraction. In optional embodiments, the preprocessing may include at least one of data cleaning, data alignment, and noise suppression. Of course, preprocessing may also include other operations, which are not limited here. An optional embodiment is provided below.

[0049] In an optional embodiment, in step 102, as Figure 3 As shown, preprocessing the embodied interaction data through the edge computing unit to generate preprocessed data may include the following steps: Step 200: Perform time-series alignment processing based on hardware timestamps on the data of each modal sensor in the embodied interaction data; Step 202: Filter and denoise the data from each modal sensor after time-series alignment processing; Step 204: Normalize the sensor data of each mode after filtering and denoising.

[0050] In practical applications, after receiving embodied interaction data, the edge computing unit can first perform time-series alignment processing on the embodied interaction data. Specifically, it can perform time-series alignment processing based on hardware timestamps on the data for each modality of the embodied interaction, aligning sensor data with different sampling frequencies on a unified time axis. Next, it can perform filtering and denoising processing on the time-series aligned data to remove environmental and sensor noise. Finally, it can perform normalization processing on the filtered and denoised data to map each modality to a unified numerical range, facilitating subsequent feature extraction and multimodal fusion processing.

[0051] Step 104: The edge computing unit determines the user personality vector of the target user based on the preprocessed data.

[0052] The user personality vector refers to a high-dimensional mathematical vector used to characterize user personality traits. Unlike simple classification labels, the user personality vector forms a fine-grained profile of a user's personality by quantitatively characterizing the user's multi-dimensional personality traits (such as openness, conscientiousness, extraversion, agreeableness, and neuroticism in the Big Five personality traits). At the same time, the user personality vector supports dynamic incremental updates based on the user's continuous interactive behavior, and can reflect the user's current personality state, rather than a fixed static description.

[0053] In an optional embodiment, in step 104, as Figure 4As shown, generating a user personality vector based on the preprocessed data through the edge computing unit may include the following steps: Step 300: Extract the specific interaction features of the target user from the preprocessed data; Step 304: Generate local feature data based on the embodied interaction features, and store the local feature data locally in the edge computing unit; Step 306: Generate the user personality vector of the target user based on the local feature data.

[0054] Specifically, a local lightweight feature extraction model can be deployed on the edge computing unit to extract the embodied interaction features of the target user from the first-level processed data, and then generate local feature data based on these embodied interaction features. The local lightweight feature extraction model is a lightweight neural network model that has undergone model compression (e.g., pruning, quantization) and / or knowledge distillation, with fewer parameters than the full model deployed on the cloud server. Therefore, by deploying the local lightweight feature extraction model in the trusted execution environment of the edge computing unit, the confidentiality and integrity of the model itself and the data processing are ensured. Based on this, by deploying the local lightweight feature extraction model on the edge computing unit and the full model on the cloud server, the collaboration between rapid inference at the edge and high-precision modeling in the cloud is achieved. It should be noted that after the first-level processed data is input into the local lightweight feature extraction model, the output of the local lightweight feature extraction model is the embodied interaction features of the target user. Subsequently, the embodied interaction features of all modalities corresponding to the target user need to be fused to generate the local feature data of the target user.

[0055] Similarly, a corresponding personality vector generation model can be deployed in the edge computing unit. After inputting local feature data into the personality vector generation model, a user personality vector corresponding to the target user can be directly generated through the model. The personality vector generation model is a neural network model that encodes user personality vectors based on local feature data. It should be noted that the personality vector generation model has the same output vector dimension (e.g., 128-dimensional, 256-dimensional, or 512-dimensional) as the full-scale personality vector generation model deployed on the cloud server, ensuring that the cloud server can perform subsequent processing based on user personality vectors of a unified dimension. Therefore, by ensuring that the user personality vectors generated by the edge computing unit have the same dimension as those generated by the full-scale model in the cloud, data consistency and availability are ensured.

[0056] As described above, by completing preprocessing, feature extraction of embodied interaction characteristics, and user personality vector generation all within the edge computing unit of the embodied intelligent device, there is no need to wait for data to be uploaded to a cloud server before processing. Compared to models that heavily rely on cloud computing, this invention can significantly reduce dependence on external networks, improve processing real-time performance, and even under conditions of unstable network connections or limited bandwidth, user personality vector generation can still proceed normally, effectively improving the system's real-time performance and reliability.

[0057] Step 106: The user personality vector is encrypted and uploaded to the cloud server through the edge computing unit.

[0058] It should be noted that the user personality vector is the updated data of the target user's cloud-based digital twin, uploaded to the cloud server via an encrypted channel. The embodied interaction data, the first-level processing data, and the local feature data are all stored in the local storage module of the edge computing unit and are not uploaded to the cloud server.

[0059] In an optional embodiment, in step 106, the user personality vector is encrypted and uploaded to the cloud server through the edge computing unit, including: uploading the user personality vector to the cloud server through an encrypted channel; wherein, the encrypted channel is an encrypted transmission channel based on a symmetric encryption algorithm, an encrypted transmission channel based on an asymmetric encryption algorithm, or an encrypted transmission channel based on the Secure Sockets Layer protocol.

[0060] In practical applications, the encrypted channel can adopt any of the following encryption methods: (1) an encrypted transmission channel based on symmetric encryption algorithms, such as AES (Advanced Encryption Standard); (2) an encrypted transmission channel based on asymmetric encryption algorithms, such as RSA (Rivest-Shamir-Adleman encryption algorithm) and ECC (Elliptic Curve Cryptography); (3) an encrypted transmission channel based on Secure Sockets Layer protocols, such as TLS (Transport Layer Security) and SSL (Secure Sockets Layer).

[0061] As described above, encrypted transmission ensures the confidentiality and integrity of local personality vectors during the upload process to the cloud server, preventing data from being stolen or tampered with during transmission and improving security.

[0062] In an optional embodiment, in step 106, as Figure 5As shown, the process of encrypting and uploading the user personality vector to the cloud server through the edge computing unit may further include the following steps: Step 400: When the network connection between the edge computing unit and the cloud server is interrupted, the edge computing unit records the current upload progress and caches the user personality vector data that has not yet been uploaded; Step 402: When the edge computing unit detects that the network connection has been restored, it continues to upload the cached but not yet uploaded user personality vector data to the cloud server according to the current upload progress.

[0063] In practical applications, a breakpoint resume mechanism can be used between the edge computing unit and the cloud server. Specifically, when the network connection between the edge computing unit and the cloud server is interrupted, the edge computing unit caches the user personality vector data that has not yet been uploaded and records the current upload progress. After the network connection is restored, the upload continues from the breakpoint based on the current upload progress, ensuring the reliability and integrity of the data upload.

[0064] As mentioned above, the local data, such as the embodied interaction data, preprocessed data, and local feature data, are all stored in the local storage module of the edge computing unit. To ensure storage security, when the storage utilization rate in the local storage module exceeds a certain threshold, the edge computing unit can perform local data lifecycle management to ensure data security, as follows.

[0065] In an optional embodiment, the processing method may further include: when the local storage of the edge computing unit reaches a preset capacity threshold, processing the local data stored in the local storage according to a preset data lifecycle management strategy; wherein the local data includes at least one of the embodied interaction data, the preprocessed data, and the local feature data.

[0066] The data lifecycle management strategy includes at least one of the following: a data timeliness-based cleanup strategy, a data importance-based hierarchical storage strategy, and an incremental update-based differential storage strategy.

[0067] Specifically, when the local storage of the local storage module in the edge computing unit reaches a preset capacity threshold, the edge computing unit will clean up or compress and archive the local data stored in the local storage, such as embodied interaction data, first-level processing data, and local feature data, according to a preset data lifecycle management strategy. In practical applications, there can be a variety of data lifecycle management strategies. For example, (1) a cleaning strategy based on data timeliness: cleaning up local data that exceeds the preset retention period (such as 30 days or 90 days). (2) a hierarchical storage strategy based on data importance: storing local data of different importance levels in different storage media (such as storing high-frequency access data in high-speed storage media and low-frequency access data in large-capacity storage media); (3) a differential storage strategy based on incremental updates: only retaining incrementally changing local data to avoid duplicate storage.

[0068] For example, the local storage capacity of the local storage module in the edge computing unit is 256GB. When the local storage utilization rate reaches 80%, the execution mechanism of the data lifecycle management strategy is triggered. Among them, the data lifecycle management strategy includes at least one of the following strategies: (1) Cleaning strategy based on data timeliness: By checking the timeliness, the retention time of each data in the local storage is checked, and local data (such as embodied interaction data) that exceeds 90 days is marked for subsequent deletion and cleanup. (2) Hierarchical storage strategy based on data importance: Local data (such as embodied interaction data) is divided into three levels by classifying importance. The first level data is embodied interaction data of the last 30 days, which is retained; the second level data is embodied interaction data of 30 to 90 days, which is compressed and archived; and the third level data is embodied interaction data of more than 90 days, which is cleaned up and deleted. (3) Differential storage strategy based on incremental update: For local data with high repetition (such as data with slow changes in ambient temperature, humidity, etc.), only the incremental change part is retained to avoid duplicate storage. Based on this, the aforementioned data lifecycle management strategy ensures that the local storage of the edge computing unit always maintains a reasonable available capacity and guarantees the complete availability of recent high-value data.

[0069] As described above, based on the local data lifecycle management mechanism, the embodied interactive data, preprocessed data, and local feature data in local storage are managed hierarchically and cleaned up periodically. Compared with indiscriminate data storage, the data management strategy of this invention is more efficient and reasonable, ensuring data availability while avoiding the unlimited expansion of local storage resources.

[0070] To make this application easier to understand, the following is combined with... Figure 6An exemplary application is provided. This exemplary application is applied in an embodied intelligent social platform, which uses a bipedal humanoid robot as an embodied intelligent device (i.e., an embodied intelligent robot companion) to achieve localized processing of user embodied interaction data in a home environment. The system architecture and embodied interaction data processing method of this exemplary application are described in detail below.

[0071] I. System Architecture The embodied interactive data processing system includes embodied smart devices and cloud servers, such as Figure 1 As shown, the embodied intelligent device in this exemplary application is a bipedal humanoid robot, on which modules such as a multi-source sensor array, an edge computing unit, and a communication unit are deployed.

[0072] (1) Multi-source sensor array, which includes a variety of sensors such as vision sensor, hearing sensor, touch sensor, motion sensor and environmental sensor.

[0073] Visual sensors: Composed of a binocular RGB camera and a depth camera, both operating at 30fps, deployed in the head of the robot.

[0074] Hearing sensor: A four-microphone array is used to collect audio at a sampling rate of 16kHz and is deployed in the head of the robot body.

[0075] Tactile sensors: including pressure sensor arrays and capacitive tactile sensors, wherein the pressure sensor array samples at a frequency of 100Hz and is deployed on the surface of the robot's torso and arms.

[0076] Motion sensor: It adopts an inertial measurement unit, operates at a sampling frequency of 200Hz, and is deployed inside the torso of the robot body.

[0077] Environmental sensors include ultra-wideband positioning sensors, temperature sensors, humidity sensors, and light sensors, with the ultra-wideband positioning sensors operating at a frequency of 10Hz.

[0078] (2) Edge computing unit Deployed inside the robot's body, the edge computing unit includes a multi-core processor (such as an ARM architecture processor), a neural network processing unit or a graphics processor, and a local storage device (such as a solid-state drive or an embedded multimedia card). The edge computing unit is connected to each sensor in the multi-source sensor array via an internal bus (such as I2C, SPI, MIPI-CSI, etc.) to achieve low-latency data acquisition.

[0079] (3) Communication unit It supports multiple communication methods such as Wi-Fi, 5G and Bluetooth, and is used for encrypted communication between edge computing units and cloud servers.

[0080] II. Localization methods for embodied interactive data Step S1, Embodied Interaction Data Collection When a target user (such as user A) interacts with a bipedal humanoid robot at home, the robot continuously collects embodied interaction data with user A through its multi-source sensor array. At this time, the edge computing unit can obtain the raw embodied interaction data from each sensor in the multi-source sensor array through the internal bus for storage and subsequent processing.

[0081] The embodied interaction data may specifically include the following: (1) Visual data: user A's facial expression image sequence, body posture image sequence, gesture action image sequence, and physical distance data between user A and the bipedal humanoid robot; (2) Auditory data: user A's voice audio stream and ambient sound; (3) Tactile data: user A's contact position, contact force, and contact frequency when touching the bipedal humanoid robot; (4) Motion data: robot's posture changes and motion trajectory; (5) Environmental data: ambient temperature, humidity, light intensity, and relative spatial position between user A and the bipedal humanoid robot when the interaction occurs. It should be noted that all the above modal sensor data have hardware-level timestamps, which are uniformly provided by the robot's own clock source.

[0082] Step S2: Preprocess the embodied interaction data. (1) Timing alignment processing Because different sensors have different sampling frequencies, the edge computing unit first performs time-series alignment processing based on hardware timestamps on the data from each modality of the embodied interaction data. Specifically, using the finest time granularity (such as the frame-level time granularity corresponding to 16kHz for the auditory sensor) as the reference time axis, visual data, tactile data, motion data, and environmental data are interpolated onto the reference time axis through linear interpolation, so that all sensor data are aligned on a unified time axis.

[0083] (2) Filtering and noise reduction After time-aligned processing, the sensor data of each modality were filtered and denoised. Specifically: (1) For visual data, median filtering or Gaussian filtering was used to remove image noise; (2) For auditory data, spectral subtraction or Wiener filtering was used to remove environmental noise; (3) For tactile data, low-pass filtering was used to remove high-frequency noise; (4) For motion data, Kalman filtering was used to remove sensor drift noise; (5) For environmental data, moving average filtering was used to remove transient noise.

[0084] (3) Normalization The filtered and denoised sensor data of each mode are mapped to a uniform numerical range (such as [0, 1] or [-1, 1]) to eliminate the influence of the difference in the dimensions of different sensors on subsequent processing.

[0085] It should be noted that after preprocessing, first-level processed data will be generated. This first-level processed data is stored in the local storage module of the edge computing unit and is not uploaded to the cloud server.

[0086] Step S3, Local Feature Extraction (1) Input the first-level processing data into the local lightweight feature extraction model for processing.

[0087] This local lightweight feature extraction model is a lightweight neural network model that has undergone knowledge distillation and is deployed in a trusted execution environment of an edge computing unit to ensure the confidentiality and integrity of model parameters and the inference process. Specifically, the teacher model is a large-scale multimodal feature extraction model trained in the cloud (such as a fusion model based on Vision Transformer and a large speech model), with approximately 1 billion parameters; the student model, i.e., the local lightweight model, is obtained by knowledge distillation from the teacher model, with approximately 10 million parameters, and is deployed on the NPU of the edge computing unit.

[0088] (2) Feature extraction is performed using a local lightweight feature extraction model to generate local feature data.

[0089] Specifically, after the first-level processing data is input into the local lightweight feature extraction model, the local lightweight feature extraction model will output the embodied interaction features of user A. Then, the embodied interaction features of all modalities corresponding to user A are fused to generate local feature data of user A. This local feature data is stored in the local storage module of the edge computing unit and is not uploaded to the cloud server.

[0090] Among them, the above-mentioned embodied interaction features may include: (1) visual features: facial expression feature vector, body posture feature vector, gesture feature vector; (2) auditory features: speech content feature vector, speech rhythm feature vector, speech emotion feature vector; (3) tactile features: contact pattern feature vector, contact force feature vector; (4) motion features: relative motion feature vector; (5) environmental features: spatial position feature vector.

[0091] Step S4: Generate user personality vector A personality vector generation model is deployed in an edge computing unit. Local feature data is input into the personality vector generation model to generate a user personality vector for user A. Specifically, after inputting local feature data into the personality vector generation model, the model outputs a user personality vector P_A∈ ² 56 The user personality vector P_A represents the distribution of user A's traits across the five dimensions of personality traits (such as openness, conscientiousness, extraversion, agreeableness, and neuroticism) in the user personality vector space.

[0092] The aforementioned personality vector generation model is a neural network model based on a multilayer perceptron. Its input is local feature data, and its output is a user personality vector. This personality vector generation model has the same output vector dimension (e.g., 256 dimensions) as the full-scale personality vector generation model deployed on the cloud server, ensuring that the cloud server can perform subsequent processing based on user personality vectors of a unified dimension.

[0093] Step S5: Encrypt and upload to the cloud server. The edge computing unit uploads user A's personality vector to the cloud server via an encrypted channel. It should be noted that this exemplary application uses the TLS 1.3 protocol to establish an encrypted transmission channel for encrypted transmission of the user personality vector data. Specifically: the edge computing unit first performs a TLS handshake with the cloud server to negotiate a session key; then, it uses the session key to encrypt the user personality vector using AES-256; finally, it uploads the encrypted user personality vector to the cloud server via the HTTPS protocol.

[0094] It should be noted that during encrypted uploads, a resume mechanism can be used between the edge computing unit and the cloud server. Specifically, when the network connection between the edge computing unit and the cloud server is interrupted, the edge computing unit caches the local personality vector data that has not yet been uploaded and records the current upload progress. Once the network is restored, the upload resumes from the point of interruption according to the current upload progress, ensuring the reliability and integrity of the data upload.

[0095] S6, Cloud-based Social Matching The cloud server receives user personality vectors uploaded by multiple embodied intelligent machine partners, constructing a user personality vector database. When user A initiates a social matching request, the cloud server calculates the similarity (e.g., cosine similarity) between user A's user personality vector and other user personality vectors in the database, filters out candidate users with high personality similarity, and pushes matching recommendation results to user A. These matching recommendation results include multiple candidate users with high personality similarity, allowing the target user to choose from the recommended results to determine whether to engage in social interaction with the corresponding candidate user.

[0096] Therefore, throughout the entire social matching process, the cloud server cannot obtain user A's original embodied interaction data, first-level processing data, and local feature data; it can only access user A's user personality vector. This privacy protection paradigm of "usable but invisible" prevents platform operators and cloud server providers from snooping on users' private interaction information.

[0097] In addition, it should be noted that the edge computing unit will perform corresponding local data lifecycle management. Through data lifecycle management, the local storage of the edge computing unit will always maintain a reasonable available capacity, while ensuring the complete availability of recent high-value data.

[0098] In summary, throughout the entire processing, embodied interaction data, preprocessed data, and local feature data are all stored locally in the edge computing unit and are not uploaded to the cloud server. Only the user personality vector is uploaded to the cloud server through an encrypted channel. Because the user personality vector is a high-dimensional abstract representation, it is impossible to reverse-engineer the user's original interaction data, facial images, voice content, and other private information from it, thus fundamentally protecting user privacy.

[0099] It should be understood that the terms "first," "second," etc., are used in this invention to describe various types of information, but these terms are not limited to them and are only used to distinguish information of the same type from each other. For example, without departing from the scope of this invention, "first" information can also be referred to as "second" information, and similarly, "second" information can also be referred to as "first" information.

[0100] It should be noted that all data collected in this application (such as embodied interaction data and other related data) was collected with the consent and authorization of the data owner (such as user, organization or enterprise), and the collection, use and processing of related data comply with the relevant laws, regulations and standards of the relevant regions.

[0101] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for processing embodied interactive data, characterized in that, Applied to an embodied intelligent device, the embodied intelligent device is equipped with a multi-source sensor array and an edge computing unit, the processing method includes: The multi-source sensor array acquires embodied interaction data with the target user, and sends the embodied interaction data to the edge computing unit for local storage. The edge computing unit preprocesses the embodied interaction data to generate preprocessed data, and stores the preprocessed data locally in the edge computing unit. The edge computing unit determines the user personality vector of the target user based on the preprocessed data. The user personality vector is encrypted and uploaded to the cloud server through the edge computing unit.

2. The processing method according to claim 1, characterized in that, The edge computing unit determines the user personality vector of the target user based on the preprocessed data, including: Extract the specific interaction features of the target user from the preprocessed data; Local feature data is generated based on the embodied interaction features, and the local feature data is stored locally in the edge computing unit. The user personality vector of the target user is generated based on the local feature data.

3. The processing method according to claim 1, characterized in that, The user personality vector is encrypted and uploaded to the cloud server via the edge computing unit, including: The user personality vector is uploaded to the cloud server through an encrypted channel; wherein the encrypted channel is an encrypted transmission channel based on a symmetric encryption algorithm, an encrypted transmission channel based on an asymmetric encryption algorithm, or an encrypted transmission channel based on the Secure Sockets Layer protocol.

4. The processing method according to claim 1, characterized in that, The user personality vector is encrypted and uploaded to the cloud server via the edge computing unit, including: When the network connection between the edge computing unit and the cloud server is interrupted, the edge computing unit records the current upload progress and caches user personality vector data that has not yet been uploaded; Once the edge computing unit detects that the network connection has been restored, it continues to upload the cached but not yet uploaded user personality vector data to the cloud server according to the current upload progress.

5. The processing method according to claim 2, characterized in that, Also includes: When the local storage of the edge computing unit reaches a preset capacity threshold, the local data stored in the local storage is processed according to a preset data lifecycle management strategy. The local data includes at least one of the embodied interaction data, the preprocessed data, and the local feature data.

6. The processing method according to claim 5, characterized in that, The data lifecycle management strategy includes at least one of the following: a data timeliness-based cleanup strategy, a data importance-based hierarchical storage strategy, and an incremental update-based differential storage strategy.

7. The processing method according to claim 1, characterized in that, The preprocessing includes at least one of data cleaning, data alignment, and noise suppression.

8. The processing method according to claim 7, characterized in that, The edge computing unit preprocesses the embodied interaction data to generate preprocessed data, including: The data from each modal sensor in the embodied interaction data are subjected to time-series alignment processing based on hardware timestamps. The sensor data of each mode after time alignment processing are filtered and denoised. Normalization is performed on the sensor data of each mode after filtering and denoising.

9. The processing method according to claim 1, characterized in that, The edge computing unit can be any one of an embedded edge computing module, a system-on-a-chip, a field-programmable gate array, or an application-specific integrated circuit.

10. A system for processing embodied interactive data, characterized in that, The processing system includes an embodied intelligent device, which is equipped with a multi-source sensor array and an edge computing unit. The edge computing unit includes a preprocessing module, a personality vector generation module, and an encrypted upload module. The multi-source sensor array is used to acquire embodied interaction data with the target user and send the embodied interaction data to the edge computing unit for local storage. The preprocessing module is used to preprocess the embodied interaction data to generate preprocessed data, and to store the preprocessed data locally in the edge computing unit; The personality vector generation module is used to determine the user personality vector of the target user based on the preprocessed data; The encrypted upload module is used to encrypt and upload the user personality vector to the cloud server.

Images