Electronic skin-based semantic interaction method and system, semantic interaction device and computer readable storage medium

Abstract

This application relates to the field of human-computer interaction technology, specifically to a semantic interaction method, a semantic interaction system, a semantic interaction device, and a computer-readable storage medium based on electronic skin. The semantic interaction method based on electronic skin is executed by an edge device and includes: receiving sensor data from the electronic skin; identifying the user's original actions based on the sensor data using an action recognition model deployed on the edge device; and if the original action is successfully identified, generating natural language interaction content based on the identified original action using a semantic generation model deployed on the edge device. This solution achieves network-independent semantic interaction by locally integrating action recognition and semantic generation functions on the edge device, which helps reduce interaction latency and improve data privacy protection.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, specifically to a semantic interaction method based on electronic skin, a semantic interaction system based on electronic skin, a semantic interaction device, and a computer-readable storage medium. Background Technology

[0002] In the field of human-computer interaction technology, especially in interaction scenarios based on flexible electronic skin sensing devices, achieving a natural, smooth, and low-latency interactive experience is a core requirement. With the popularization of wearable devices and mobile computing platforms, users expect to obtain instant and private human-computer interaction feedback in various environments, including mobile scenarios without a stable network connection.

[0003] In related technologies, a common approach to achieving complex action intent understanding and natural language feedback is to upload sensor data collected by electronic skin to a cloud server for processing. The cloud, with its powerful computing capabilities, deploys complex action recognition models and large language models to analyze user actions and generate corresponding interactive content. However, this architecture, which relies on network communication and remote computing power, has inherent limitations due to its fundamental principles. First, the uploading of action data, cloud processing, and result feedback inevitably introduce communication latency, making it difficult to meet the demands of real-time interactive scenarios. Second, this solution will be completely ineffective in aircraft, remote areas, or mobile environments without network coverage. Furthermore, the continuous transmission of sensor data containing user physiological characteristics and behavioral habits to the cloud also poses a potential risk of user privacy breaches. Summary of the Invention

[0004] To overcome the above-mentioned shortcomings, this application provides a semantic interaction method based on electronic skin, a semantic interaction system based on electronic skin, a semantic interaction device, and a computer-readable storage medium.

[0005] A first aspect of this application provides a semantic interaction method based on electronic skin, comprising: receiving sensor data from electronic skin; identifying a user's original action based on the sensor data using an action recognition model deployed on an edge device; and if the original action is successfully identified, generating natural language interaction content based on the identified original action using a semantic generation model deployed on the edge device.

[0006] A second aspect of this application provides a semantic interaction system based on electronic skin. The system includes: a data receiving module for receiving sensor data from the electronic skin; an action recognition module for recognizing the user's original actions based on the sensor data using an action recognition model deployed on the edge device; and a semantic generation module for generating natural language interaction content based on the recognized original actions using a semantic generation model deployed on the edge device when the original actions are successfully recognized.

[0007] A third aspect of the embodiments of this application provides a semantic interaction device, characterized in that it includes: an electronic skin for collecting user touch and pressure interaction data and generating sensor data; a semantic interaction system based on electronic skin as provided in the second aspect of the embodiments of this application; and an output unit for outputting natural language interaction content generated by the semantic interaction system based on electronic skin.

[0008] A fourth aspect of the present application provides a computer-readable storage medium, characterized in that it stores a computer program thereon, which, when executed by a processor, implements the semantic interaction method based on electronic skin as provided in the first aspect of the present application.

[0009] This application provides a semantic interaction method, a semantic interaction system, a semantic interaction device, and a computer-readable storage medium based on electronic skin. The aforementioned semantic interaction method based on electronic skin integrates action recognition and semantic generation functions on the edge device, enabling action recognition and semantic generation locally based directly on the sensor data of the electronic skin. This achieves real-time human-computer interaction in environments without network connectivity, fundamentally eliminating the network transmission link, effectively reducing interaction latency, avoiding the privacy leakage risks associated with uploading sensitive user data to the cloud, and improving system availability and reliability in mobile or offline scenarios. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 The diagram shown is a schematic representation of the implementation environment of a semantic interaction system based on electronic skin according to an embodiment of this application.

[0012] Figure 2 The diagram shown is a flowchart illustrating a semantic interaction method based on electronic skin according to an embodiment of this application.

[0013] Figure 3 The diagram shown is a flowchart illustrating a method for recognizing a user's original actions according to an embodiment of this application.

[0014] Figure 4 The diagram shown is a flowchart illustrating a method for generating natural language interactive content based on original actions, according to an embodiment of this application.

[0015] Figure 5 The diagram shown is a schematic diagram of a semantic interaction system based on electronic skin provided in an embodiment of this application.

[0016] Figure 6 The diagram shown is a structural schematic of a semantic interaction device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without creative effort falls within the scope of protection of this application.

[0018] In the field of human-computer interaction based on electronic skin sensing technology, a common technical solution for achieving intelligent recognition of user touch gestures and natural language feedback is to upload sensor data to a cloud server for processing. Specifically, this solution collects touch data using sensors deployed on the device side and sends the raw data or pre-processed features to the cloud via a network connection. Its basic working principle is to utilize the powerful computing resources of the cloud server to run complex action recognition algorithms and large language models, completing the entire chain from raw data to semantic understanding and response generation. Its widespread application stems primarily from its ability to effectively utilize the near-limitless computing power of the cloud, supporting the operation of high-precision, multi-functional, and complex models, thereby ensuring the accuracy and richness of the interaction.

[0019] However, when applied to offline or mobile scenarios with strict requirements for real-time performance, privacy, and deployment environment, the solution's performance is less than ideal. A fundamental contradiction lies in the fact that, in order to optimize the accuracy of its recognition and the richness of its semantic generation, the inherent design of relying on cloud computing inevitably compromises the system's response speed, data privacy, and environmental adaptability, and may even cause interruptions in the interaction process. Specifically, in medical rehabilitation monitoring, outdoor assistive devices, or data-sensitive personal interaction scenarios, network instability or absence will directly lead to service unavailability; even with a smooth network, the round-trip latency of data upload and result delivery significantly increases interaction delays, affecting user experience; furthermore, uploading raw stress data containing user biometrics and behavioral patterns to the cloud poses a risk of privacy leakage, limiting the application of this technology in sensitive areas.

[0020] Through in-depth analysis, the inventors discovered that the root causes of the aforementioned contradictions are multifaceted: From a system architecture perspective, the cloud-based processing model introduces an additional network transmission link between the edge and the cloud, which becomes a performance bottleneck and a single point of failure for the entire system. Its latency and availability are beyond the control of the edge. From a data processing perspective, uploading raw or shallow feature data in pursuit of high accuracy increases the amount of data transmitted and exposes sensitive information, contradicting the requirements for offline operation and privacy protection. From a deployment flexibility perspective, the strong dependence on cloud infrastructure prevents the system from operating independently in environments without or with limited network access (such as remote areas, aircraft, and secure isolated networks), significantly limiting the breadth of its application scenarios.

[0021] To overcome the aforementioned contradictions, this application proposes a different technical approach. Its core concept lies in: by locally integrating a lightweight action recognition model and a semantic generation model on the edge device, a self-contained offline processing flow is constructed. This allows for the local reconstruction of the entire "sensor data-action-semantics" conversion process, thereby effectively eliminating network transmission latency, ensuring user data privacy and security, and achieving network-free, anytime, anywhere interaction without significantly sacrificing recognition accuracy and interaction naturalness. In other words, it provides an offline human-computer interaction method based on a complete edge-side model deployment, addressing the problems of high latency, privacy leaks, and inability to operate in offline scenarios caused by cloud reliance in related technologies. This achieves low latency, high privacy and security, and strong environmental adaptability.

[0022] To facilitate understanding, let's first combine Figure 1 This paper introduces an exemplary implementation environment for the semantic interaction method based on electronic skin provided in the embodiments of this application.

[0023] Figure 1The diagram shown illustrates the implementation environment of a semantic interaction system based on electronic skin, according to an embodiment of this application. The system mainly comprises two core components: an electronic skin sensor and an edge processing unit.

[0024] In this context, electronic skin sensors refer to any flexible or non-flexible sensing device capable of acquiring two-dimensional spatially distributed pressure signals generated by a user's physical contact methods such as touching, stroking, or patting. Examples include, but are not limited to, thin-film sensors composed of pressure sensor arrays, flexible piezoelectric sensor arrays, or capacitive touch sensor matrices. They are configured to acquire touch data generated during real-time user touch interactions. This touch data typically reflects the magnitude and / or distribution of pressure during the touch interaction and can form a sensor data stream with temporal information.

[0025] An edge processing unit is a hardware platform with local processor, memory, storage resources, and input / output capabilities, capable of operating independently without external network connectivity. Edge processing units can include, but are not limited to, embedded microcontroller units, single-board computers, smartphones, tablets, or dedicated semantic interaction devices, typically containing a local processor, memory, non-volatile storage, and necessary input / output interfaces. Specifically, edge processing units can be smart wearable devices (such as interactive gloves and smartwatches), plush toys, and intelligent interactive robots equipped with MCUs or embedded Linux chips.

[0026] The electronic skin connects to the edge processing unit via a wired (e.g., Ethernet, integrated circuit bus, or serial peripheral interface) or wireless (e.g., Wi-Fi or Bluetooth) interface, directly transmitting the collected sensor data to the latter. The edge processing unit can independently run the action recognition model, semantic generation model, and related software modules described in subsequent embodiments in an environment without external network connection. It completes all calculations from data reception to the generation of natural language interactive content and directly controls the local output unit (e.g., a speaker or screen) for feedback. As can be seen from the above, this environment can constitute a complete, closed-loop offline human-computer interaction system, where the data flow can complete the closed loop from perception to feedback without leaving the local device.

[0027] Figure 2 The diagram shown is a schematic flowchart of a semantic interaction method based on electronic skin according to an embodiment of this application. The method is executed by an edge device. Figure 2 As shown, the method includes the following steps: Step S210: Receive sensing data from the electronic skin.

[0028] Among them, electronic skin can provide Figure 1In the context of electronic skin sensors, sensing data generally refers to any form of data generated by the electronic skin that reflects user interaction actions. In one embodiment of this application, this sensing data can be touch data. Specifically, the electronic skin can collect pressure values ​​from each pressure sensing unit on its surface at fixed time intervals, forming a series of two-dimensional pressure matrices arranged in chronological order. Since the two-dimensional distribution of pressure on the sensor array corresponds to the shape and force of the contact surface, and the time series records the trajectory, speed, and rhythm of the action, this method can directly and effectively characterize the spatial distribution pattern and dynamic change process of touch pressure actions.

[0029] For example, when the electronic skin device collects two-dimensional pressure data generated by human hand gestures in real time, the pressure matrix size can be set to 18 rows × 29 columns, and the acquisition interval can be configured to 15 milliseconds, which can be adjusted as needed within the range of 10 to 20 milliseconds. After each acquisition, the electronic skin outputs a raw data packet containing a millisecond-level timestamp and the two-dimensional pressure matrix, and transmits it to the edge device. The data receiving module of the edge device caches these raw data in timestamp order to ensure their temporal continuity. Those skilled in the art will understand that the size of the pressure matrix and the acquisition frequency can be adjusted according to the specific hardware design and interaction accuracy requirements of the electronic skin, such as using sensor arrays with different resolutions such as 8×8 or 32×32, and the acquisition interval can also be set in a wider range (such as 5ms-50ms), as long as meaningful changes in movement can be captured.

[0030] Step S220: Identify the user's original actions based on sensor data using the action recognition model deployed on the terminal device.

[0031] End-side devices can Figure 1 The edge processing unit in the model receives the raw sensing data generated by the electronic skin. Before inputting the sensing data into the action recognition model, the edge device usually needs to preprocess the sensing data. Preprocessing can be a series of calculations or transformations performed on the sensing data to improve data quality, unify data format, or extract effective features. The preprocessed data can provide a reliable data foundation for subsequent action recognition.

[0032] Specifically, when the sensing data is touch data, the edge device can preprocess the touch data and generate a first input sample, wherein the format of the first input sample conforms to the standard input format of the action recognition model. The preprocessing steps can be as follows: invalid data cleaning, pressure offset correction, normalization processing, and timing stitching.

[0033] The invalid data cleaning process filters out abnormal samples, such as those with all pressure values ​​of 0 (no action input) or pressure distributions that do not conform to common gesture characteristics (non-standard actions), retaining only the valid pressure matrix to reduce invalid calculations. Pressure offset correction can eliminate static errors in the electronic skin sensor and improve data accuracy by performing offset correction on the cleaned pressure matrix. For example, the correction formula can be: Correction value = |Original pressure value - Sensor reference value|, where the reference value can be adjusted according to the calibration results of the electronic skin device, with a default value of 0.1 to ensure the accuracy of the pressure data. Normalization can map the pressure value after pressure offset correction to the [0,1] interval, eliminating dimensional differences caused by different users or different forces, making model training and inference more stable. For example, the normalization formula can be: Normalized value = (Corrected value - Minimum value) / (Maximum value - Minimum value). If the maximum value is equal to the minimum value (i.e., the pressure has not changed), the original correction value remains unchanged. Temporal stitching can concatenate multiple consecutive normalized stress matrices according to their timestamps, integrating independent frames into temporal samples that reflect the action process, and finally converting them into the standard input format of the action recognition model. For example, if 10 consecutive stress matrices are temporally stitched, a "10×18×29" temporal sample can be formed, and the final converted standard input format is (N, 10, 1, 18, 29), where N is the number of samples, 1 is the number of channels, and each sample corresponds to one action recognition, allowing the model to extract temporal features. Those skilled in the art will understand that the preprocessing flow is not limited to the above fixed order and all steps. For example, only normalization and stitching can be performed, or a filtering and denoising step can be added. The threshold, offset correction baseline, normalization range, and number of consecutive frames stitched in the cleaning rules can all be adjusted according to the actual sensor characteristics and application scenario.

[0034] Figure 3 The diagram shown is a flowchart illustrating a method for recognizing a user's original actions according to an embodiment of this application. Figure 3 As shown, identifying a user's original action may include the following steps: Step S310: Input the first input sample into the action recognition model to identify valid action primitives.

[0035] The action recognition model, deployed on the edge device, broadly refers to any machine learning model or algorithm module capable of automatically recognizing a user's original actions based on sensor data from the electronic skin. Examples include, but are not limited to: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), fusion models of the aforementioned networks (such as CNN-LSTM), or Transformer models. This model is trained and optimized to adapt to the computational and storage resource constraints of the edge device.

[0036] In one embodiment of this application, the action recognition model is a fusion model of convolutional neural network and long short-term memory network, that is, a CNN+LSTM fusion model.

[0037] Specifically, the structure of the CNN+LSTM fusion model may include the following modules: The input layer is used to receive preprocessed standard input samples; for example, its format is (N, 10, 1, 18, 29), where N is the number of samples, 10 is the time step (10 consecutive frames of data), 1 is the number of channels, and 18×29 is the size of the stress matrix; The CNN submodule is used to extract spatial features of stress data (such as stress distribution and gesture contours). For example, this CNN submodule can contain two convolutional layers and two pooling layers. The first convolutional layer uses eight 3×3 convolutional kernels with a stride of 1 and a ReLU activation function to initially extract spatial features. The first pooling layer uses 2×2 max pooling with a stride of 2 to reduce dimensionality and computation. The second convolutional layer uses sixteen 3×3 convolutional kernels with a stride of 1 and a ReLU activation function to further enhance spatial feature extraction. The second pooling layer uses 2×2 max pooling with a stride of 2 to further reduce dimensionality and model complexity. The flattening layer is used to convert the two-dimensional feature map output by the CNN submodule into a one-dimensional feature vector, which is convenient for subsequent processing by the LSTM submodule. The LSTM submodule is used to extract temporal features of stress data (such as the sequence of actions and speed changes). For example, in the LSTM submodule, the hidden layer dimension can be set to 64, the returned sequence is False, and only the temporal features of the last frame are output to avoid redundant calculations and improve inference speed. A fully connected layer is used to fuse spatial features extracted by CNN and temporal features extracted by LSTM to improve action recognition accuracy; for example, the fully connected layer contains 32 neurons and the activation function is ReLU.

[0038] The output layer is used to output each identified action primitive and its confidence level. Here, an action primitive refers to the most basic and indivisible unit that constitutes a complex action or behavior. For example, the output layer may contain 8 neurons, corresponding to 8 types of action primitives, and the activation function is Softmax.

[0039] In one embodiment of this application, step S310 can identify a valid action primitive by the following steps: inputting the first input sample into the action recognition model to obtain the action primitive and the confidence level of the action primitive; if the confidence level of the action primitive is greater than or equal to the preset confidence level, then the action primitive is determined to be a valid action primitive; otherwise, the action primitive is determined to be an invalid action primitive.

[0040] In this context, a valid action primitive refers to an action primitive for which the action recognition model provides a sufficiently high level of confidence in its prediction of a particular action during the recognition process. Conversely, an invalid action primitive refers to a situation where the action recognition model fails to provide the expected level of confidence in its prediction of a particular action during the recognition process. It should be noted that the specific value of the preset confidence level can be adjusted according to the actual situation, and this application does not impose any restrictions on it.

[0041] In one embodiment of this application, valid action primitives can be input to subsequent step S320 to participate in subsequent processing, while invalid action primitives can be directly discarded or stored in a buffer to wait for confidence accumulation.

[0042] For example, taking a CNN+LSTM fusion model as the action recognition model, and with the 8 neurons in the output layer of the CNN+LSTM fusion model corresponding to 8 types of action primitives, the 8 types of action primitives output by the output layer of the CNN+LSTM fusion model can be set as: tap, press, stroke, pat, quick pat, soothing pat, slow pat, and quick pat. Each action primitive corresponds to a unique output label. If the preset confidence level is set to 80%, then when the confidence level of an action primitive is greater than or equal to 80%, the action primitive is considered a valid action primitive; otherwise, it is considered an invalid action primitive.

[0043] User interactions via electronic skin can range from simple basic actions to complex combinations of actions expressing intentions. For example, expressing "urge" often involves first pressing down on the other person (a long press), followed by several quick taps; the sequence and combination of these actions themselves carry semantic information. Similarly, the same act of "tapping" can have vastly different meanings depending on the force and rhythm of the tap. To cover richer interactive semantics, in a preferred implementation, user actions can be categorized into basic actions and combinations of at least two basic actions within a preset time window. This limitation expands the definition of action types, enabling the system to recognize and understand more complex and expressive user behaviors, thus elevating simple touch operations to an interactive language closer to natural communication. The effect is to significantly enrich the semantic space of offline interaction, making human-computer interaction more refined and human-centered.

[0044] Therefore, the user's action type can be determined through the following step S320.

[0045] Step S320: If at least two valid action primitives appear consecutively within a preset time window, the combination of the at least two valid action primitives appearing consecutively within the preset time window will be used as the original action; otherwise, the valid action primitives will be used as the original action.

[0046] Specifically, in one example, suppose the system defines eight basic action primitives, such as tapping, pressing, stroking, and patting. During the recognition process, the device continuously caches all successfully recognized basic actions (i.e., valid action primitives) within the most recent one-second time window. For example, if two successfully recognized basic actions, "quick pat" and "slow stroking," are identified within one second, the system will determine that the user has performed a combined action and will use this combined action as the original action. Conversely, if only one "stroking" action is identified within one second, the user's action type is the basic action "stroking," and the corresponding original action is "stroking."

[0047] This implementation uses temporal proximity rules to link discrete basic actions, forming combined units with new semantics. Its advantages include clear logic, low computational overhead, no need for dedicated training of a combined action recognition model, and ease of implementation on the device side. Those skilled in the art will understand that the rules for determining combined actions are not limited to "continuous appearance" but can also be "appearance in any order within a window"; the window length can be adjusted according to action habits, such as 0.5 seconds, 1 second, or 2 seconds, and this application does not limit this; the combined mapping relationship can also be configured and adjusted according to the actual interaction design, and this application does not limit this either.

[0048] Step S230: If the original action is successfully recognized, then natural language interactive content is generated based on the recognized original action through the semantic generation model deployed on the terminal device.

[0049] The semantic generation model refers to any generative artificial intelligence model deployed on edge devices that can generate natural language text that conforms to the context based on specific input. To achieve powerful natural language generation capabilities on edge devices while overcoming storage and computing power limitations, in one embodiment, the semantic generation model can be a large language model that has undergone quantization and compression. For example, semantic generation models may include, but are not limited to: Alibaba's Qwen-1.8B, Baidu's ERNIE 3.0 Mini, and Microsoft's Phi-2; after 4-bit quantization, the size of the semantic generation model can be controlled to 1-2GB, which can adapt to the storage and computing power requirements of edge embedded chips, and can run in a completely offline environment without a network, with an inference latency of less than 1 second. Furthermore, since this semantic generation model can run entirely offline, it will also be referred to as an offline large language model below.

[0050] Natural language interaction content, also known as natural language text, response text, or response content, is the final output of the entire interaction process. For example, if the user's initial action is a quick tap followed by a slow pat, the corresponding natural language interaction content could be: "You're in a great mood today!" Figure 4 The diagram shown is a flowchart illustrating a method for generating natural language interactive content based on original actions, according to an embodiment of this application. Figure 4 As shown, in order to make the generated interactive content more emotional and human-like, rather than just a mechanical response, in an optimized implementation, after recognizing the user's original action and before generating the corresponding natural language interactive content through the semantic generation model, the method provided in this application further includes step S410: mapping the recognized original action to the corresponding emotional semantic label according to the preset action-emotional semantic label mapping relationship.

[0051] Accordingly, generating natural language interactive content based on the original action may include the following step S420: generating natural language interactive content based on the sentiment semantic tag through a semantic generation model. This implementation introduces a sentiment semantic abstraction layer between action recognition and semantic generation. This sentiment semantic abstraction layer can transform specific physical action types into abstract descriptions of emotions or intentions, thereby enabling the subsequent language generation process to focus on emotional expression rather than the specific action itself, making the final generated response more natural, appropriate, and emotional.

[0052] The preset action-emotion semantic tag mapping relationship can be a mapping table pre-stored on the edge device. This implementation method is simple and efficient, and the mapping can be completed by looking up the table, with almost no computational delay. The mapping table can be a simple list of key-value pairs, where the key is the original action (the original action can be a basic action or a combination of actions), and the value is the corresponding emotional semantic tag (such as "curiosity", "confirmation", "happiness", "urging", etc.). Those skilled in the art will understand that the mapping relationship can also be achieved without a static table, but through a lightweight neural network model to learn the mapping function from action type to emotional semantics, in order to handle more complex or variable correspondences.

[0053] Specifically, the device stores an action type-emotion semantic label mapping table. When the original action identified is a combination of "quick pat + slow stroking," the system queries the table and obtains the corresponding emotion semantic label as "happy." This "happy" label will serve as the core input for subsequent processes. The contents of the mapping table are configurable; for example, "tap" can be mapped to "curiosity," "press" to "confirm," and "long press + quick pat" to "urge," etc. This design allows the system's interaction style and semantic correspondence to be adjusted by updating the mapping table without modifying the core model.

[0054] After obtaining the sentiment semantic tag, if the offline large language model is frequently called for inference based on the sentiment semantic tag, it will still be the most computationally demanding part on the edge device. In order to reduce the average response latency, reduce power consumption, and improve the smoothness of the system in high-frequency interaction scenarios, one implementation can generate natural language interaction content through the following steps: query the local cache library based on the sentiment semantic tag; if the sentiment semantic tag matches the local cache library, the corresponding natural language interaction content is directly read.

[0055] The local cache stores the mapping between sentiment semantic tags and natural language interaction content. A sentiment semantic tag hitting the local cache means that a record corresponding to that sentiment semantic tag can be found in the local cache; conversely, a miss means that no record corresponding to that sentiment semantic tag can be found in the local cache.

[0056] This constraint introduces a caching mechanism, which aims to intercept repeated or common interaction requests and directly return historical results, thereby completely avoiding the complex calculations of offline large language models and is the key to achieving low-latency, high-frequency interaction.

[0057] In one embodiment of this application, the local cache can be updated according to preset triggering conditions. Specifically, this includes: responding to a sentiment semantic tag not being found in the local cache, calling a semantic generation model to generate corresponding natural language interaction content, and adding the correspondence between the sentiment semantic tag and the corresponding natural language interaction content to the local cache; or responding to the natural language interaction content corresponding to the sentiment semantic tag meeting preset update conditions, updating the natural language interaction content corresponding to the sentiment semantic tag; or responding to the local cache reaching its storage limit, deleting the least frequently accessed correspondence. This update mechanism ensures the effectiveness of the cache and the manageability of storage space.

[0058] In this application, the update condition is triggered when the natural language interaction content corresponding to the sentiment semantic tag meets the preset update conditions. Specifically, for a given correspondence, when the natural language interaction content corresponding to its sentiment semantic tag meets the preset update conditions, the system automatically skips the step of reading the corresponding natural language interaction content from the cache, forcibly sends the sentiment semantic tag into the offline large language model to generate new natural language interaction content, and uses this new natural language interaction content to overwrite the original natural language interaction content in the correspondence. In one embodiment of this application, the preset update conditions include at least one of the cache expiration condition and the cache activity condition. The cache expiration condition refers to the cache duration of the correspondence meeting the preset expiration requirement; for example, if the cache time of a correspondence exceeds 24 hours, an update is triggered. The cache activity condition is triggered when the activity of the sentiment semantic tag in a correspondence meets the preset activity requirement; for example, if a sentiment tag is triggered three times consecutively within one minute (indicating that the user has been performing the same action), an update is triggered. In human-computer interaction scenarios, the above-mentioned methods for triggering natural language interaction content updates can, on the one hand, prevent devices from repeatedly outputting completely identical response content under the same or similar input, thereby affecting the naturalness and experience of the interaction; on the other hand, it can ensure the freshness and effectiveness of cached content in the local cache library. That is, when the cache exists for a longer period than a preset threshold, it needs to be updated to reflect the latest status or data.

[0059] The design of a local cache library can be very flexible. Those skilled in the art will understand that the cache library can be a small instance of a relational database or a simple file storage (such as a key-value pair storage method based on JSON files). This application does not limit this. Since the key-value pair format is simple and efficient and facilitates fast querying, the key-value pair storage method based on JSON files can be a preferred implementation method.

[0060] Specifically, in one example, the edge device maintains a JSON-formatted cache file in local storage. Its content is in the form of: {“Happy”: “You’re in a great mood today!”, “Curious”: “Anything you want to know?”}. If the user’s original action is “quick pat + slow pat”, its corresponding sentiment semantic tag is “Happy”. To obtain the natural language interaction content corresponding to this sentiment semantic tag, the method provided in this application first queries the cache. If an entry with the key “Happy” is found in the local cache (i.e., a match), its value “You’re in a great mood today!” is immediately output as the response for this interaction. The entire process may take less than 100 milliseconds. For another example, if the user’s original action is “long press + quick pat”, its corresponding sentiment semantic tag is “urge”. If no entry with the key “urge” is found in the local cache (i.e., a miss), then the offline large language model is called. The responses generated by the offline large language model are added to the local cache as new key-value pairs (e.g., {"urging": "Don't rush, take your time."}) as they are output to the user. The local cache can be set with an upper limit on the number of entries (e.g., 100). When the cache is full, the least used entry will be automatically removed.

[0061] Specifically, in one example, after the edge device obtains the sentiment semantic tag, if the cache library is not hit, a fixed-format prompt is generated. For example, the prompt generation rule can be: substitute the sentiment semantic tag into a fixed template to generate a prompt that reads, "The user performs the action corresponding to [sentiment semantic tag], please reply in a natural and gentle tone, with a word count of 20 or less." For instance, if the sentiment semantic tag is "happy," the generated prompt would be, "The user performs the action corresponding to 'happy,' please reply in a natural and gentle tone, with a word count of 20 or less." Subsequently, the offline large language model can generate a compliant reply text based on this prompt (such as "You're in a really good mood today!").

[0062] In one embodiment of this application, the semantic interaction method provided by this application can also optimize memory and storage; specifically, the local cache library and the preset action sentiment semantic tag mapping relationship can be stored in the chip's local storage, and the weights of large models can be loaded into memory to avoid frequent reading and writing of local storage, thereby improving the running speed; some storage space can also be reserved for cache library updates and temporary inference data of large models, thereby ensuring hardware operation stability.

[0063] In offline environments, it is impossible to rely on cloud-based backup services, making the system's self-fault tolerance crucial. To ensure uninterrupted interaction and thus guarantee user experience and system robustness, in one embodiment of this application, the semantic interaction method further includes: if the action primitive is an invalid action primitive or if the time taken to generate the corresponding natural language interaction content using the semantic generation model exceeds a preset timeout, using a preset prompt message as the natural language interaction content.

[0064] If the action primitive is invalid, the above method can avoid subsequent interactions based on unreliable action recognition results. Furthermore, in the event of an offline large language model response timeout, this method can prevent prolonged system unresponsiveness due to model inference lag. This approach essentially provides a fallback response for human-computer interaction; triggering any of the above conditions activates this fallback service, ensuring that the user always receives a clear, even generic, feedback, thus maintaining the continuity of the interaction.

[0065] In one embodiment of this application, the semantic interaction method provided by this application further includes: real-time monitoring of the running time of the action recognition model and the semantic generation model; if the running time of the action recognition model is greater than the preset delay of the action recognition model or the running time of the semantic generation model is greater than the preset delay of the semantic generation model, triggering a lightweight degradation of the action recognition model or the semantic generation model.

[0066] For example, the preset latency of the action recognition model can be 50ms, and the preset latency of the semantic generation model can be set to 1s. The specific data can be set according to actual needs, and this application does not limit it. Lightweight degradation can reduce the number of CNN convolution kernels or further quantize the relevant actions of the model. Lightweight degradation can be automatically triggered or manually triggered, and this application does not limit it. The above performance monitoring can ensure that the real-time requirements of edge interaction are met.

[0067] It should be noted that before implementing the above semantic interaction methods, model adaptation and software compilation and deployment can be performed first. Model adaptation refers to lightweight deployment of the model in a heterogeneous hardware environment. Specifically, model adaptation can convert the trained action recognition model (such as an ONNX format CNN+LSTM fusion model) into an inference format supported by the edge chip (such as TFLite / NCNN), perform INT8 / FP16 quantization and prune redundant layers to ensure that the model size is ≤5MB and the inference latency is ≤50ms; and perform 4-bit quantization on the semantic generation model, converting it into a chip-supported format (such as GGUF / MLC-LLM) to ensure that the model size is 1-2GB and the inference latency is ≤1 second. Software compilation and deployment refers to compiling all software modules corresponding to the methods into executable files supported by the edge chip to adapt to the chip's inference framework (such as NCNN / TFLite), thereby ensuring that the modules can be called collaboratively and that data transmission is smooth.

[0068] The above describes a semantic interaction method based on electronic skin provided in the embodiments of this application. This method can be implemented by... Figure 5 The illustration shows the execution of a semantic interaction system based on electronic skin.

[0069] Figure 5 The diagram shown is a schematic representation of a semantic interaction system based on electronic skin according to an embodiment of this application. Figure 5 As shown, the system 500 includes: The data receiving module 510 is used to receive sensing data from the electronic skin; Action recognition module 520 is used to recognize the user's original actions based on sensor data using an action recognition model deployed on the edge device; and The semantic generation module 530 is used to generate natural language interactive content based on the recognized original action when the original action is successfully recognized, through the semantic generation model deployed on the edge device.

[0070] In one embodiment of this application, the sensing data is touch data.

[0071] In one embodiment of this application, the system 500 further includes a data preprocessing module 540, which is used to preprocess the touch data and generate a first input sample after receiving the sensing data from the electronic skin and before recognizing the user's original action based on the sensing data through the action recognition model deployed on the edge device, wherein the format of the first input sample conforms to the standard input format of the action recognition model.

[0072] In one embodiment of this application, the action recognition module 520 is further configured to input the first input sample into the action recognition model to identify valid action primitives; if at least two valid action primitives appear consecutively within a preset time window, the combination of at least two valid action primitives appearing consecutively within the preset time window will be used as the original action; otherwise, the valid action primitives will be used as the original action.

[0073] In one embodiment of this application, the action recognition module 520 is further configured to input the first input sample into the action recognition model to obtain the action primitive and the confidence level of the action primitive; if the confidence level of the action primitive is greater than or equal to the preset confidence level, the action primitive is determined to be a valid action primitive; otherwise, the action primitive is determined to be an invalid action primitive.

[0074] In one embodiment of this application, the system 500 further includes a fault-tolerant processing module 550. If the action primitive is an invalid action primitive, the fault-tolerant processing module 550 is used to use the preset prompt information as natural language interaction content.

[0075] In one embodiment of this application, the action recognition model is a fusion of a convolutional neural network and a long short-term memory network.

[0076] In one embodiment of this application, the system 500 further includes an emotion semantic tag generation module 560, which is used to map the identified original action to the corresponding emotion semantic tag according to the preset action emotion semantic tag mapping relationship; wherein, the semantic generation module 530 is also used to generate natural language interactive content based on the emotion semantic tag through a semantic generation model.

[0077] In one embodiment of this application, the preset action emotion semantic tag mapping relationship is a mapping table pre-stored in the end device.

[0078] In one embodiment of this application, the semantic generation module 530 is further configured to query a local cache library based on sentiment semantic tags, wherein the local cache library stores the mapping relationship between sentiment semantic tags and natural language interaction content; if the sentiment semantic tag matches the local cache library, the corresponding natural language interaction content is directly read.

[0079] In one embodiment of this application, the semantic generation module 530 is further configured to call the semantic generation model to generate corresponding natural language interactive content if the sentiment semantic tag does not match the local cache library.

[0080] In one embodiment of this application, the semantic generation model is a large language model that has undergone quantization and compression.

[0081] In one embodiment of this application, the fault-tolerant processing module 550 is further configured to use a preset prompt message as the natural language interaction content if the time taken to call the semantic generation model to generate the corresponding natural language interaction content exceeds a preset timeout period.

[0082] In one embodiment of this application, the semantic generation module 530 is further configured to update the local cache library according to preset triggering conditions.

[0083] In one embodiment of this application, the semantic generation module 530 is further configured to, in response to the sentiment semantic tag not being found in the local cache, call the semantic generation model to generate the corresponding natural language interaction content and add the correspondence between the sentiment semantic tag and the corresponding natural language interaction content to the local cache; or, in response to the natural language interaction content corresponding to the sentiment semantic tag meeting the preset update conditions, update the natural language interaction content corresponding to the sentiment semantic tag; or, in response to the storage capacity of the local cache reaching the preset upper limit, delete the correspondence with the lowest access frequency.

[0084] In one embodiment of this application, the system 500 further includes an edge software adaptation module 570, used to adapt the various software modules of the system 500 (i.e., the data receiving module 510, the action recognition module 520, the semantic generation module 530, the data preprocessing module 540, the fault-tolerant processing module 550, and the sentiment semantic tag generation module 560) to the edge embedded chip, ensuring that the modules operate collaboratively and meeting the requirements of low latency and high stability. Specifically, the edge software adaptation module 570 is used for model adaptation, software compilation and deployment, memory and storage optimization, and performance monitoring.

[0085] The semantic interaction system based on electronic skin provided in this embodiment belongs to the same application concept as the semantic interaction method based on electronic skin provided in the above embodiments of this application. It can execute the semantic interaction method based on electronic skin provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the specific processing content of the semantic interaction method based on electronic skin provided in the above embodiments of this application, and will not be repeated here.

[0086] It should be understood that the modules in the above system can be implemented by a processor calling software. For example, the system includes a processor connected to memory, which stores instructions. The processor calls the instructions stored in memory to implement any of the above methods or to implement the functions of each unit in the system. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the system. Alternatively, the units in the system can be implemented as hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented by a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above units. All units in the above system can be implemented entirely by a processor calling software, entirely by hardware circuits, or partially by a processor calling software with the remaining parts implemented by hardware circuits.

[0087] This application also provides a semantic interaction device. Figure 6 The diagram shown is a structural schematic of a semantic interaction device according to an embodiment of this application. (Refer to...) Figure 6 The semantic interaction device 600 includes an electronic skin 610 for collecting user touch and pressure interaction data and generating sensor data; an electronic skin-based semantic interaction system 500; and an output unit 620 for outputting natural language interaction content generated by the electronic skin-based semantic interaction system.

[0088] The electronic skin 610 and the semantic interaction system 500 based on the electronic skin can be described in the preceding text and will not be repeated here. The output unit 620 can be a display screen, a printer, a speaker, or any combination of the three, and is used to output the natural language interactive content generated by the semantic generation module to complete the interaction loop with the user.

[0089] It should be noted that the embodiment of the device 600 provides a complete product form, such as a smart companion toy or rehabilitation training device with built-in electronic skin, which can provide intelligent and emotional verbal feedback to the user's touch in any environment without the need for network connection.

[0090] To better understand the technical effects of the various embodiments of this application, a specific application scenario example is provided below. Consider an embedded smart companion device for children or the elderly, which is shaped like a plush toy and integrates electronic skin (distributed on the toy's surface) and an edge processing unit (such as a low-power embedded chip). This device is designed for use in various environments without stable networks, such as at home and outdoors.

[0091] When a user touches the toy, the electronic skin collects pressure data at 15ms intervals and sends it to the edge chip. The preprocessing software on the chip cleans, corrects, and normalizes 10 consecutive frames of data, stitching them together into a single sample. A locally deployed lightweight CNN+LSTM model (4MB in size) infers from this sample, recognizing the "touching" action with 85% confidence within 30ms. The system queries the built-in mapping table, mapping "touching" to the emotional semantic tag "gentle." Subsequently, the system queries the local JSON cache library and finds a cached response corresponding to the "gentle" tag, "Your touch is so warm." Therefore, within 150ms (50ms for recognition + 100ms for cache lookup), this voice is played through the toy's speaker. The entire interaction process has a total latency of approximately 180ms, providing the user with immediate, human-like feedback, and all pressure data never leaves the device, protecting privacy.

[0092] If a user performs a combination of "quickly patting and then gently stroking," the system identifies these two consecutive basic actions within a 1-second time window and classifies them as the "happy" combination. Assuming this is a new action and the cache is not found, the system calls its local 4-bit quantized Qwen-1.8B model, inputting the prompt "The user has performed an action expressing happiness; please generate a reply of no more than 20 characters in a natural and gentle tone." The model generates the reply "You had a really fun day!" within 1 second. Simultaneously, the key-value pair "happy": "You had a really fun day!" is stored in the cache. The next time the user performs the same action, the reply will be retrieved directly from the cache, reducing latency to less than 100ms.

[0093] In this scenario, this application enables the companion device to completely eliminate its reliance on Wi-Fi or cellular networks, allowing for use in any environment, including outdoors and in vehicles. Interaction latency is extremely low, resulting in a smooth user experience. All interaction data (including sensitive tactile data) is processed internally, eliminating the risk of privacy leaks. Furthermore, through emotional semantic mapping and natural language generation, interaction is no longer a cold, impersonal command execution, but a natural exchange imbued with emotional warmth, significantly enhancing the product's appeal and practicality.

[0094] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to implement the semantic interaction method based on electronic skin provided in any of the foregoing embodiments.

[0095] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, can implement the semantic interaction method based on electronic skin provided in any of the foregoing embodiments.

[0096] This application also provides a computer program product, which includes instructions that, when executed by a processor, can implement the semantic interaction method based on electronic skin provided in any of the foregoing embodiments.

[0097] It is understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0098] It is understood that the memory in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Specifically, non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0099] The prompting method in this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in this application are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM, or other programmable device.

[0100] It is understood that the specific examples provided in this application are only intended to help those skilled in the art better understand the embodiments of this application, and are not intended to limit the scope of this application.

[0101] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0102] It is understood that the various embodiments described in this application can be implemented individually or in combination, and the embodiments of this application are not limited in this respect.

[0103] It is understood that the embodiments described in this application are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0104] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0105] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0106] The modules and sub-modules in the devices and terminals in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0107] Unless otherwise stated, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0109] For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be indirect couplings or communication connections between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0112] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A semantic interaction method based on electronic skin, characterized in that, Performed by an end-side device, the method includes: Receive sensor data from the electronic skin; The user's original actions are identified based on the sensor data using an action recognition model deployed on the terminal device. If the original action is successfully recognized, natural language interactive content is generated based on the recognized original action using a semantic generation model deployed on the edge device.

2. The semantic interaction method based on electronic skin according to claim 1, characterized in that, The sensing data is touch data.

3. The semantic interaction method based on electronic skin according to claim 2, characterized in that, After receiving sensor data from the electronic skin, and before identifying the user's original actions based on the sensor data using a motion recognition model deployed on the edge device, the method further includes: The touch data is preprocessed to generate a first input sample, wherein the format of the first input sample conforms to the standard input format of the action recognition model.

4. The semantic interaction method based on electronic skin according to claim 3, characterized in that, The action recognition model deployed on the edge device, based on the sensor data, identifies the user's original actions, including: The first input sample is input into the action recognition model to identify valid action primitives, wherein the action recognition model is a model that integrates a convolutional neural network and a long short-term memory network; If at least two of the valid action primitives appear consecutively within a preset time window, the combination of the at least two valid action primitives appearing consecutively within the preset time window will be used as the original action; otherwise, the valid action primitives will be used as the original action.

5. The semantic interaction method based on electronic skin according to claim 4, characterized in that, The step of inputting the first input sample into the action recognition model to identify valid action primitives includes: The first input sample is input into the action recognition model to obtain the action primitive and the confidence level of the action primitive; If the confidence level of the action primitive is greater than or equal to the preset confidence level, then the action primitive is determined to be a valid action primitive; otherwise, the action primitive is determined to be an invalid action primitive.

6. The semantic interaction method based on electronic skin according to claim 5, characterized in that, The method further includes: If the action primitive is an invalid action primitive, the preset prompt information will be used as the natural language interaction content.

7. The semantic interaction method based on electronic skin according to claim 1, characterized in that, After the original action is successfully recognized, and before generating natural language interactive content based on the recognized original action using a semantic generation model deployed on the edge device, the method further includes: Based on a preset action-emotion semantic tag mapping relationship, the identified original action is mapped to a corresponding emotion semantic tag, wherein the preset action-emotion semantic tag mapping relationship is a mapping table pre-stored in the terminal device; wherein... The step of generating natural language interactive content based on the identified original action using a semantic generation model deployed on the edge device includes: Based on the aforementioned sentiment semantic tags, the natural language interactive content is generated through the aforementioned semantic generation model.

8. The semantic interaction method based on electronic skin according to claim 7, characterized in that, The process of generating the natural language interactive content based on the sentiment semantic tags and through the semantic generation model includes: Based on the sentiment semantic tags, a local cache is queried, wherein the local cache stores the correspondence between the sentiment semantic tags and the natural language interaction content; If the sentiment semantic tag matches the local cache library, the corresponding natural language interaction content is read directly.

9. The semantic interaction method based on electronic skin according to claim 8, characterized in that, The process of generating the natural language interactive content based on the sentiment semantic tags further includes: If the sentiment semantic tag does not match the local cache library, the semantic generation model is invoked to generate the corresponding natural language interaction content.

10. The semantic interaction method based on electronic skin according to claim 9, characterized in that, The method further includes: If the time taken to generate the corresponding natural language interactive content by calling the semantic generation model exceeds a preset timeout period, then the preset prompt message will be used as the natural language interactive content.

11. The semantic interaction method based on electronic skin according to claim 8, characterized in that, The method further includes: The local cache is updated according to preset trigger conditions, wherein updating the local cache according to preset trigger conditions includes: In response to the sentiment semantic tag not being found in the local cache, the semantic generation model is invoked to generate the corresponding natural language interaction content, and the correspondence between the sentiment semantic tag and the corresponding natural language interaction content is added to the local cache; or In response to the natural language interaction content corresponding to the sentiment semantic tag meeting preset update conditions, the natural language interaction content corresponding to the sentiment semantic tag is updated; or In response to the local cache library reaching its storage limit, the least frequently accessed correspondence is deleted.

12. A semantic interaction system based on electronic skin, characterized in that, Deployed on an end-side device, the system includes: The data receiving module is used to receive sensor data from the electronic skin; The action recognition module is used to identify the user's original actions based on the sensor data using an action recognition model deployed on the terminal device; The semantic generation module is used to generate natural language interactive content based on the identified original action when the original action is successfully recognized, using a semantic generation model deployed on the edge device.

13. A semantic interaction device, characterized in that, include: Electronic skin is used to collect users' touch and pressure interaction data and generate sensor data; The semantic interaction system based on electronic skin as described in claim 12; The output unit is used to output the natural language interaction content generated by the electronic skin-based semantic interaction system.

14. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the semantic interaction method based on electronic skin as described in any one of claims 1 to 11.

Images