Human-computer interaction system and method based on flexible electronic perception and display
By integrating multimodal sensor arrays and deep learning technology into a flexible electronic sensing and display human-computer interaction system, the problems of stiff interaction experience and low system integration on curved devices are solved, achieving adaptive and highly integrated natural interaction and improving recognition accuracy and feedback response.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- RES & DEV INST OF NORTHWESTERN POLYTECHNICAL UNIV IN SHENZHEN
- Filing Date
- 2026-03-13
- Publication Date
- 2026-06-02
Smart Images
Figure CN121832779B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of data processing and human-computer interaction, and in particular to a human-computer interaction system and method based on flexible electronic sensing and display. Background Technology
[0002] As the demands for human-computer interaction continue to evolve, its application scenarios have expanded from traditional flat terminals to complex interfaces with curved and deformable characteristics, such as wearable devices, intelligent robots, augmented reality (AR), and aerospace. In these scenarios, achieving natural, efficient, and reliable interactive control is crucial. Existing technologies primarily rely on rigid or contactless solutions such as touchscreens, physical buttons, gesture recognition, and voice interaction.
[0003] However, such traditional interactive architectures reveal a series of inherent defects when facing new application demands: First, there is a fundamental contradiction between rigid interactive interfaces and flexible application requirements. Hard screens or buttons cannot conformally fit with human joints, robot curved surfaces, or irregular structures inside aircraft cabins, resulting in a stiff interactive experience and limited applicability. Second, currently, each module is independent and usually uses discrete hardware splicing, resulting in low system integration, long signal paths, and overall sluggish response, and it is difficult to achieve direct feedback of "what you see is what you touch". Finally, they generally lack context awareness and adaptive capabilities. Their interaction logic is fixed and cannot be self-optimized according to user habits, ambient light, or device deformation, resulting in insufficient intelligence and a long-term decline in user experience. Summary of the Invention
[0004] The purpose of this application is to provide a human-computer interaction system and method based on flexible electronic sensing and display, which can improve the naturalness and reliability of human-computer interaction.
[0005] To achieve the above objectives, this application provides the following solution:
[0006] In a first aspect, this application provides a human-computer interaction system based on flexible electronic sensing and display, comprising:
[0007] Flexible sensing module, flexible display module, execution module, and signal processing and decision-making unit;
[0008] The flexible sensing module, the flexible display module, and the execution module are all connected to the signal processing and decision-making unit; the flexible sensing module includes a flexible substrate and a multimodal sensing array integrated on the flexible substrate;
[0009] The flexible sensing module is used to acquire physical interaction information of users on the user end in real time and collect the generated multimodal interaction signals;
[0010] The signal processing and decision-making unit is used for:
[0011] The multimodal interaction signals are preprocessed and feature analyzed to extract feature parameters; these feature parameters characterize the user's interaction intent; the feature parameters include: pressure gradient, contact area change rate, and deformation rate.
[0012] An action recognition model is used to output a recognition result based on the feature parameters; the recognition result includes user gestures or actions; the action recognition model is obtained by training a convolutional neural network using deep learning methods;
[0013] Control commands are generated based on the recognition results; the control commands include display drive commands and execution control commands.
[0014] The flexible display module is used to drive the display pixel unit at the corresponding position according to the display driving instruction and based on the multimodal interaction signal to change the brightness, color or display pattern and perform dynamic visual feedback display.
[0015] The execution module is used to drive external execution devices to perform operations according to execution control instructions.
[0016] In one embodiment, the multimodal sensing array includes pressure sensor units, capacitance sensor units, temperature sensor units, and deformation sensor units arranged in a distributed array.
[0017] In one embodiment, the human-computer interaction system based on flexible electronic sensing and display further includes: a communication module;
[0018] The communication module is connected to the execution module and the signal processing and decision-making unit, respectively.
[0019] The communication module is used to send the execution control instructions to the execution module.
[0020] In one embodiment, the communication module employs wireless communication; the communication module is a Bluetooth module.
[0021] In one embodiment, the multimodal interaction signal includes a touch position signal, a pressure distribution signal, a temperature signal, and an interface deformation signal.
[0022] In one embodiment, the flexible sensing module and the flexible display module are conformally fabricated on a single flexible substrate through a shared conductive path.
[0023] In one embodiment, the weight of the human-computer interaction system based on flexible electronic sensing and display is less than 10g / 100cm. 2 It can be applied to curved surfaces or the surface of wearable devices.
[0024] Secondly, this application provides an interaction method based on a flexible electronic sensing and display human-computer interaction system, wherein the interaction method is implemented using the aforementioned flexible electronic sensing and display human-computer interaction system; the interaction method based on the flexible electronic sensing and display human-computer interaction system includes:
[0025] Acquire multimodal interaction signals;
[0026] The multimodal interaction signals are preprocessed and feature analyzed to extract feature parameters; these feature parameters characterize the user's interaction intent; the feature parameters include: pressure gradient, contact area change rate, and deformation rate.
[0027] An action recognition model is used to output a recognition result based on the feature parameters; the recognition result includes user gestures or actions; the action recognition model is obtained by training a convolutional neural network using deep learning methods;
[0028] Control commands are generated based on the recognition results; the control commands include display drive commands and execution control commands.
[0029] The flexible display module controls the display pixel units at the corresponding positions according to the display driving instructions and based on multimodal interaction signals to change the brightness, color or display pattern and perform dynamic visual feedback display.
[0030] The control execution module drives external execution devices to perform operations according to execution control commands.
[0031] In one embodiment, the action recognition model is determined by using a deep learning method, training a convolutional neural network based on the historical feature parameters of known recognition results, to achieve nonlinear mapping.
[0032] In one embodiment, the interaction method of the human-computer interaction system based on flexible electronic sensing and display further includes:
[0033] Based on the information displayed by the visual feedback and the information corresponding to the executed operation, the preset extraction threshold of the feature parameters is dynamically adjusted.
[0034] According to the specific embodiments provided in this application, the following technical effects are disclosed:
[0035] This application provides a human-computer interaction system and method based on flexible electronic sensing and display. The flexible sensing module includes a flexible substrate and a multimodal sensing array integrated on the flexible substrate. The flexible sensing module acquires physical interaction information from the user in real time and collects the generated multimodal interaction signals. The signal processing and decision-making unit preprocesses and performs feature analysis on the multimodal interaction signals to extract feature parameters, and uses an action recognition model to output recognition results based on the feature parameters, and generates control commands based on the recognition results. The flexible display module drives the display pixel units at corresponding positions based on the multimodal interaction signals according to the display driving commands, to change brightness, color, or display patterns, providing dynamic visual feedback display. The execution module drives external execution devices to perform operations according to execution control commands. This application, based on the collaboration of the flexible sensing module, flexible display module, execution module, and signal processing and decision-making unit, realizes signal acquisition, processing, and execution feedback, which can improve the naturalness and reliability of human-computer interaction. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0037] Figure 1 This is a structural diagram of a human-computer interaction system based on flexible electronic sensing and display.
[0038] Figure 2 This is a flowchart of the interaction method for a human-computer interaction system based on flexible electronic sensing and display. Detailed Implementation
[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0040] This application constructs an adaptive, highly integrated intelligent interactive interface by synergistically integrating multimodal perception, dynamic visual feedback, and intelligent decision control onto a single flexible substrate, thereby significantly improving the naturalness, intuitiveness, and system reliability of the interaction.
[0041] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0042] In one exemplary embodiment, such as Figure 1 As shown, a human-computer interaction system based on flexible electronic sensing and display is provided, including: a flexible sensing module, a flexible display module, an execution module, and a signal processing and decision-making unit. The flexible sensing module and the flexible display module are conformally fabricated on a single flexible substrate through a shared conductive path.
[0043] The human-computer interaction system based on flexible electronic sensing and display weighs less than 10g / 100cm². 2 It can be applied to curved surfaces or the surface of wearable devices.
[0044] The flexible sensing module, flexible display module, and execution module are all connected to the signal processing and decision-making unit; the flexible sensing module includes a flexible substrate and a multimodal sensing array integrated on the flexible substrate.
[0045] The flexible sensing module is used to acquire physical interaction information from users in real time and collect the generated multimodal interaction signals. These multimodal interaction signals include touch position signals, pressure distribution signals, temperature signals, and interface deformation signals.
[0046] The signal processing and decision unit is used to preprocess and perform feature analysis on multimodal interaction signals to extract feature parameters. The feature parameters are used to characterize the user's interaction intent. The feature parameters include: pressure gradient, contact area change rate, and deformation rate.
[0047] The signal processing and decision unit is used to output recognition results based on the feature parameters using an action recognition model; the recognition results include user gestures or actions; the action recognition model is obtained by training a convolutional neural network using deep learning methods.
[0048] The signal processing and decision unit is also used to generate control instructions based on the recognition results; the control instructions include display drive instructions and execution control instructions.
[0049] The flexible display module is used to drive the display pixel unit at the corresponding position according to the display driving instruction and based on the multimodal interaction signal to change the brightness, color or display pattern and perform dynamic visual feedback display.
[0050] The execution module is used to drive external execution devices to perform operations according to execution control instructions.
[0051] The multimodal sensor array includes pressure sensor units, capacitance sensor units, temperature sensor units, and deformation sensor units arranged in a distributed array.
[0052] As an optional implementation, the human-computer interaction system based on flexible electronic sensing and display further includes: a communication module; the communication module is connected to the execution module and the signal processing and decision-making unit respectively; the communication module is used to send execution control commands to the execution module.
[0053] The communication module uses wireless communication; the communication module is a Bluetooth module.
[0054] In one exemplary embodiment, an interaction method based on a flexible electronic sensing and display human-computer interaction system is provided, wherein the interaction method based on the flexible electronic sensing and display human-computer interaction system is implemented using the aforementioned flexible electronic sensing and display human-computer interaction system.
[0055] like Figure 2 As shown, the interaction method of the human-computer interaction system based on flexible electronic sensing and display includes:
[0056] Step 100: Acquire multimodal interaction signals.
[0057] Step 200: Preprocess and perform feature analysis on the multimodal interaction signals to extract feature parameters. These feature parameters characterize the user's interaction intent; they include pressure gradient, contact area change rate, and deformation rate.
[0058] Step 300: The action recognition model outputs the recognition result based on the feature parameters. The recognition result includes the user's gestures or actions; the action recognition model is obtained by training a convolutional neural network using deep learning methods.
[0059] Step 400: Generate control instructions based on the recognition results. Control instructions include display drive instructions and execution control instructions.
[0060] Step 500: Control the flexible display module to drive the display pixel unit at the corresponding position according to the display driving instruction and based on the multimodal interaction signal, so as to change the brightness, color or display pattern and perform dynamic visual feedback display.
[0061] Step 600: The control execution module drives the external execution device to perform operations according to the execution control instructions.
[0062] Specifically, the action recognition model uses deep learning methods to train a convolutional neural network based on the historical feature parameters of known recognition results, in order to achieve a non-linear mapping and determine the result.
[0063] As an optional implementation method, the interaction method of the human-computer interaction system based on flexible electronic sensing and display also includes:
[0064] Based on the information displayed by the visual feedback and the information corresponding to the executed operation, the preset extraction threshold of the feature parameters is dynamically adjusted.
[0065] This application first acquires multimodal interaction signals such as touch pressure and deformation from the user in real time through a multimodal sensor array integrated on a flexible substrate. Then, the signal processing and decision-making unit extracts signal features and uses a trained convolutional neural network to perform gesture or action recognition. Based on the recognition results, on the one hand, it generates display driving instructions to immediately drive the display layer (flexible display module) integrated on the same flexible substrate to generate dynamic visual feedback, realizing the interactive effect of display upon contact. On the other hand, it generates execution control instructions and sends them to external execution devices, such as robotic arms, via wireless communication (communication module) to complete the corresponding operations. Finally, through a reinforcement learning mechanism, the perception threshold and feedback strategy can be dynamically optimized based on historical interaction data, thereby forming an adaptive closed-loop human-computer interaction process that integrates real-time perception, intelligent recognition, dynamic feedback, and precise control.
[0066] The interaction method of a human-computer interaction system based on flexible electronic sensing and display, in practical applications, specifically includes the following steps:
[0067] S101, Multimodal Interactive Signal Acquisition Steps:
[0068] The flexible sensing layer (flexible sensing module) is specifically composed of pressure sensing units, capacitance sensing units, temperature sensing units and deformation sensing units arranged in a distributed array. It is used to capture physical interaction information between the user and the flexible interactive interface in real time and collect the generated multimodal raw signals (multimodal interaction signals). The multimodal interaction signals include at least one or more of the following: touch position signal, pressure distribution signal, temperature signal and interface deformation signal.
[0069] S102, Steps for extracting interactive feature parameters:
[0070] The signal processing and decision-making unit receives multimodal interaction signals from the flexible sensing module and performs preprocessing and feature analysis on these signals to extract feature parameters representing the user's interaction intent. These feature parameters include, but are not limited to, pressure gradient, contact area change rate, and deformation rate. This step aims to extract stable and discriminative features from the raw, potentially noisy signals, laying the foundation for subsequent accurate identification.
[0071] Specifically, after receiving the multimodal interaction signals from the flexible sensing module, the signal processing and decision-making unit first preprocesses the multimodal interaction signals to reduce noise interference and improve signal stability. Preprocessing methods include, but are not limited to, filtering, signal smoothing, normalization, and drift compensation. Filtering is used to suppress environmental noise and transient interference, normalization is used to eliminate dimensional differences between different modal signals, and drift compensation is used to reduce the zero-point drift effect generated by the flexible sensor during long-term use.
[0072] After preprocessing, feature analysis is performed on the multimodal interaction signals to extract feature parameters representing the user's interaction intent. Feature analysis can employ time-domain analysis, time-frequency analysis, or multi-scale analysis methods, such as, but not limited to, wavelet transform or short-time Fourier transform, to characterize the variation characteristics of the interaction signals at different time and frequency scales, thereby extracting discriminative feature parameters such as pressure gradient, contact area change rate, and deformation rate. Through the above processing, stable and reliable features can be extracted from the original, potentially noisy signal, providing a foundation for subsequent action recognition and decision control.
[0073] S103, Action recognition steps based on neural networks:
[0074] The signal processing and decision-making unit inputs the extracted feature parameters into a pre-trained convolutional neural network model. This model (i.e., the action recognition model) is configured to calculate and classify the input feature parameters, outputting the corresponding user gesture or action recognition result. The purpose of this step is to leverage the powerful non-linear mapping capability of the action recognition model trained on the convolutional neural network using deep learning methods to map low-level physical signal features into high-level, semantically meaningful user interaction commands.
[0075] S104. Steps for generating and presenting dynamic visual feedback:
[0076] The signal processing and decision-making unit generates corresponding display driving instructions based on the recognition results; the flexible display layer receives the display driving instructions and drives the display pixel unit corresponding to the touch position to change its brightness, color or display pattern, thereby generating dynamic visual feedback on the flexible interactive interface.
[0077] S105. External Device Control Command Mapping and Execution Steps:
[0078] The signal processing and decision-making unit realizes a closed-loop control system from the human-machine interface to external devices. The unit maps the recognition results to predefined control commands for external devices. The communication module (such as a Bluetooth Low Energy module) sends the control commands to designated external execution devices (such as robotic arms, drones, or augmented reality systems), driving them to perform corresponding operations.
[0079] S106. Adaptive optimization steps:
[0080] This step endows the system with self-learning and continuous optimization capabilities. The signal processing and decision-making unit is configured to continuously record historical interaction data and its corresponding results, and based on reinforcement learning algorithms, dynamically optimize and calibrate the feature extraction sensitivity threshold in step S102 and / or the feedback intensity parameter in step S104, thereby improving the accuracy of subsequent interaction recognition and the suitability of feedback.
[0081] Reinforcement learning algorithms update a system's policy based on its state, parameter adjustment actions, and reward signals, causing it to tend to choose parameter adjustment actions that yield higher rewards in subsequent interactions. Through continuous interactive iteration, the system gradually converges to parameter configurations that adapt to different user interaction habits and usage scenarios, achieving dynamic optimization and online calibration of feature extraction sensitivity thresholds and feedback strength parameters.
[0082] This application has the following advantages:
[0083] 1. The naturalness and intuitiveness of the interaction have been greatly improved.
[0084] When users interact with the interface, their touch and pressing actions immediately receive dynamic visual feedback (such as lighting up or changing color) at the operation location, achieving a "what you touch is what you get" intuitive experience and reducing the learning cost. This advantage directly stems from the closed-loop linkage between steps S101 and S104 in the technical solution. Step S101 ensures that any user interaction action can be perceived and captured in real time, while step S104 guarantees that the perceived signal can drive the flexible display layer to produce corresponding visual changes without delay. This tightly coupled "perception-feedback" design breaks the feedback lag caused by the separation of perception and display modules in traditional systems, forming the basis for highly natural and intuitive interaction.
[0085] 2. High recognition accuracy and robustness.
[0086] It can accurately distinguish and recognize complex gestures and actions (such as swiping, rotating, and multi-finger pressing), with strong anti-interference capabilities and a low false trigger rate. This advantage is mainly attributed to steps S102 and S103 in the technical solution. Step S102 provides rich and stable information input for recognition by extracting multi-dimensional feature parameters such as "pressure gradient" and "contact area change rate," rather than relying on a single trigger signal. Step S103 utilizes a pre-trained convolutional neural network (CNN) model. This powerful deep learning tool can learn and recognize complex nonlinear patterns from multi-dimensional features, thereby achieving high-precision and robust recognition of various gestures and actions.
[0087] 3. High integration and strong environmental adaptability.
[0088] The overall structure is thin, flexible, and can conform to curved surfaces or wearable device surfaces, with a light weight (less than 10g / 100cm). 2 It boasts a long battery life (over 24 hours on a single charge), making it suitable for complex scenarios with limited space or varied forms. This advantage stems from the integrated concept and manufacturing method of this application. Specifically, the flexible sensing layer (flexible sensing module) and the flexible display layer (flexible display module) are conformally fabricated on a single flexible substrate through a shared conductive path, fundamentally avoiding the volume, weight, and rigidity constraints imposed by traditional discrete module splicing. Simultaneously, the low-power wireless communication control in step S105 and the overall architecture's compatibility with the flexible energy storage unit jointly ensure the system's lightweight and long-lasting characteristics.
[0089] 4. It possesses self-learning and self-adaptive capabilities and has a high degree of intelligence.
[0090] The system can automatically optimize its recognition sensitivity and feedback intensity based on users' historical usage habits and environmental changes, achieving a personalized interactive experience and maintaining optimal performance over the long term. This is the core intelligent advantage, directly achieved by optional step S106 in the technical solution. This step introduces a reinforcement learning mechanism, enabling the system (signal processing and decision-making unit) to continuously self-calibrate and optimize its strategies based on historical interaction data, dynamically adjusting the feature extraction threshold in step S102 and the feedback-driven parameters in step S104. This makes the system no longer static, but an adaptive intelligent system that can evolve with users and the environment.
[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0092] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A human-computer interaction system based on flexible electronic sensing and display, characterized in that, include: Flexible sensing module, flexible display module, execution module, and signal processing and decision-making unit; The flexible sensing module, the flexible display module, and the execution module are all connected to the signal processing and decision-making unit; the flexible sensing module includes a flexible substrate and a multimodal sensing array integrated on the flexible substrate; The flexible sensing module is used to acquire physical interaction information of users on the user end in real time and collect the generated multimodal interaction signals; The signal processing and decision-making unit is used for: The multimodal interaction signals are preprocessed and feature analyzed to extract feature parameters; The feature parameters are used to characterize the user's interaction intent on the user side; The characteristic parameters include: pressure gradient, contact area change rate, and deformation rate; An action recognition model is used to output a recognition result based on the feature parameters; the recognition result includes user gestures or actions; the action recognition model is obtained by training a convolutional neural network using deep learning methods; Control commands are generated based on the recognition results; the control commands include display drive commands and execution control commands. The flexible display module is used to drive the display pixel unit at the corresponding position according to the display driving instruction and based on the multimodal interaction signal to change the brightness, color or display pattern and perform dynamic visual feedback display. The execution module is used to drive external execution devices to perform operations according to execution control instructions; The flexible sensing module and the flexible display module are conformally fabricated on a single flexible substrate through a shared conductive path; By using a reinforcement learning mechanism, the perception threshold and feedback strategy are dynamically optimized based on historical interaction data, thereby forming an adaptive closed-loop human-computer interaction process that integrates real-time perception, intelligent recognition, dynamic feedback and precise control.
2. The human-computer interaction system based on flexible electronic sensing and display according to claim 1, characterized in that, The multimodal sensing array includes pressure sensor units, capacitance sensor units, temperature sensor units, and deformation sensor units arranged in a distributed array.
3. The human-computer interaction system based on flexible electronic sensing and display according to claim 1, characterized in that, The human-computer interaction system based on flexible electronic sensing and display also includes: a communication module; The communication module is connected to the execution module and the signal processing and decision-making unit, respectively. The communication module is used to send the execution control instructions to the execution module.
4. The human-computer interaction system based on flexible electronic sensing and display according to claim 3, characterized in that, The communication module uses wireless communication; the communication module is a Bluetooth module.
5. The human-computer interaction system based on flexible electronic sensing and display according to claim 1, characterized in that, The multimodal interaction signals include touch position signals, pressure distribution signals, temperature signals, and interface deformation signals.
6. The human-computer interaction system based on flexible electronic sensing and display according to claim 1, characterized in that, The human-computer interaction system based on flexible electronic sensing and display weighs less than 10g / 100cm². 2 It can be applied to curved surfaces or the surface of wearable devices.
7. An interaction method for a human-computer interaction system based on flexible electronic sensing and display, characterized in that, The interaction method of the human-computer interaction system based on flexible electronic sensing and display is implemented using the human-computer interaction system based on flexible electronic sensing and display as described in any one of claims 1-6; the interaction method of the human-computer interaction system based on flexible electronic sensing and display includes: Acquire multimodal interaction signals; The multimodal interaction signals are preprocessed and feature analyzed to extract feature parameters; these feature parameters characterize the user's interaction intent; the feature parameters include: pressure gradient, contact area change rate, and deformation rate. An action recognition model is used to output a recognition result based on the feature parameters; the recognition result includes user gestures or actions; the action recognition model is obtained by training a convolutional neural network using deep learning methods; Control commands are generated based on the recognition results; the control commands include display drive commands and execution control commands. The flexible display module controls the display pixel units at the corresponding positions according to the display driving instructions and based on multimodal interaction signals to change the brightness, color or display pattern and perform dynamic visual feedback display. The control execution module drives external execution devices to perform operations according to execution control commands.
8. The interaction method of the human-computer interaction system based on flexible electronic sensing and display according to claim 7, characterized in that, The action recognition model is determined by using deep learning methods, training a convolutional neural network based on the historical feature parameters of known recognition results, and then performing a nonlinear mapping.
9. The interaction method of the human-computer interaction system based on flexible electronic sensing and display according to claim 7, characterized in that, The interaction method of the human-computer interaction system based on flexible electronic sensing and display further includes: Based on the information displayed by the visual feedback and the information corresponding to the executed operation, the preset extraction threshold of the feature parameters is dynamically adjusted.