A signal flow dynamic mapping interaction system based on human body sensing

Abstract

The application relates to the technical field of human-computer interaction, and discloses a signal flow dynamic mapping interaction system based on human body sensing. The application collects various data of a human body in a movement process, inputs the collected data into a lightweight neural network, obtains a multi-dimensional intermediate representation vector aligned with a time stamp, obtains a corresponding trigger score based on the vector, generates a trigger signal including an external target device when the trigger score reaches a trigger threshold, and sends the trigger signal to an external device structure module. The application analyzes a human body posture by collecting multi-dimensional data of the human body, improves the personalization and intelligentization of subsequent triggering, converts the trigger signal into a data packet by the module, sends the data packet to a target device, achieves the effect of controlling the external device, and enables a user to edit control sensitivity and a threshold value through a visual interface, so that the personalization control of the external device is realized.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a dynamic mapping interaction system for signal flow based on human body sensing. Background Technology

[0002] With the development of human-computer interaction technology, users increasingly expect to be able to directly interact with electronic devices, lighting systems, sound synthesizers, and even robots through natural human movements in an immersive way. Traditional human-computer interaction methods, such as mice, keyboards, touchscreens, or handheld remote controls, require users to touch fixed physical interfaces, have limited interaction dimensions, and cannot fully utilize the rich expressive power of the human body's full-body movements.

[0003] Existing interactive technologies suffer from numerous technical shortcomings. Most existing systems rely on only one sensor, making it difficult to simultaneously acquire multi-dimensional information such as joint angles, center of gravity distribution, fine gestures, and muscle activity. Single sensors exhibit poor robustness against occlusion, electromagnetic interference, or signal drift, failing to meet the continuous and stable control requirements of complex interactive scenarios. More intelligent motion-sensing interactive systems often employ preset action command trigger tables (e.g., waving a hand to turn a page), but these trigger rules are fixed in the software code, preventing users from dynamically adjusting them during interaction. Furthermore, ordinary users cannot intuitively understand the intrinsic relationship between raw sensor data and external device responses, and manually setting trigger thresholds, sensitivity curves, etc., lacking a visual signal flow feedback interface and trigger rule editor. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a signal flow dynamic mapping interaction system based on human body sensing. This system has advantages such as collecting various data of the human body during movement, obtaining multi-dimensional intermediate representation vectors at each moment based on a lightweight neural network, and then generating trigger signals to send to external devices to achieve the effect of controlling external devices through human body posture data, thus solving the aforementioned technical problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a signal flow dynamic mapping and interaction system based on human body sensing, comprising a front-end multimodal acquisition module, an attitude signal mapping module, a dynamic control module, an external device structure module, and a visualization interaction and rule customization module; The front-end multimodal acquisition module is used to collect various data of the human body during movement; The attitude signal mapping module is used to input the collected data into a lightweight neural network to obtain a multi-dimensional intermediate representation vector aligned with the timestamp. The dynamic control module generates trigger signals based on a multi-dimensional intermediate representation vector. The external device structure module is used to convert the trigger signal into a data packet that conforms to the target external device communication protocol, and send the data packet to the target external device; The visualization interaction and rule customization module is used to provide users with a visual interface.

[0006] As a preferred technical solution of the present invention, the front-end multimodal acquisition module includes a depth camera, an inertial measurement unit, an array of pressure sensors, and electromyography (EMG) electrodes on the human body surface.

[0007] As a preferred technical solution of the present invention, the depth camera is used to collect human skeleton sequence and extract three-dimensional angle data of joints; the inertial measurement unit is used to collect limb movement trajectory, acceleration and spatial orientation; the array pressure sensor is used to collect the real-time dynamic distribution and displacement trajectory of the human body's center of gravity; and the electromyography signal electrode on the human body surface is used to collect the human body's electromyography signals.

[0008] As a preferred embodiment of the present invention, the expression for the multidimensional intermediate representation vector is as follows:

[0009] in, Indicates the timestamp Multidimensional intermediate representation vector at time. , For the set of real numbers, For vectors Total dimensions; Indicates the length of the time window; For neural network functions; This is the current timestamp; Indicates the current timestamp Input tensor at time; It is a separator; These are the network weight parameters.

[0010] As a preferred technical solution of the present invention, the input tensor The expression is as follows: in, Indicates the current timestamp Input tensor at time; For human joints in time stamp Three-dimensional Euler angles; For timestamp Two-dimensional coordinates of the human body's center of gravity; For timestamp Human posture quaternions; For timestamp The multidimensional feature vector of electromyographic signals.

[0011] As a preferred embodiment of the present invention, the step of generating the trigger signal based on the multidimensional intermediate representation vector includes the following steps: Step A1: Calculate the trigger score based on the multidimensional intermediate representation vector corresponding to each timestamp. ; Step A2: Determine whether to generate a trigger signal based on the trigger score.

[0012] As a preferred embodiment of the present invention, step A1 triggers the score. The expression is as follows: in, Indicates the timestamp Scoring is triggered at the time; The total number of triggering conditions; For vectors The total dimension Indicates the timestamp Multidimensional intermediate representation vector; Indicates the timestamp Time vector The dimensional vector; Indicates the first The triggering rules of the external device are in the first One trigger condition; For indicator functions, when When, the value is 1, when When the value is 0; Representing vectors The duration; Indicates the first The specified duration corresponding to the triggering rule of an external device; For indicator functions, when When, the value is 1, when When the value is 0; For summation operations; and These are the weighting coefficients.

[0013] As a preferred embodiment of the present invention, step A2, which determines whether to generate a trigger signal based on the trigger score, specifically involves: when the trigger score... When the trigger threshold is reached, the generation includes external devices. The trigger signal is sent to the external device structure module, and the trigger score is obtained. When the threshold is less than the trigger threshold, no trigger signal is generated.

[0014] As a preferred embodiment of the present invention, the external devices include a lighting matrix device, a digital audio workstation, and a robot joint actuator.

[0015] As a preferred embodiment of the present invention, the visualization interface includes a real-time signal flow visualization panel, a draggable mapping rule editing area, and sensitivity and threshold adjustment controls for each trigger channel.

[0016] Compared with existing technologies, this invention provides a signal flow dynamic mapping interaction system based on human body sensing, which has the following beneficial effects: This invention collects various data of the human body during movement and inputs the collected data into a lightweight neural network to obtain a multi-dimensional intermediate representation vector aligned with a timestamp. Based on this vector, a corresponding trigger score is obtained. When the trigger score reaches the trigger threshold, a trigger signal including an external target device is generated and sent to the external device structure module. By collecting multi-dimensional data of the human body and analyzing the human posture, the personalization and intelligence of subsequent triggering are improved. The trigger signal is converted into a data packet by this module and sent to the target device to achieve the effect of controlling the external device. At the same time, users can edit the control sensitivity and threshold through a visual interface to realize personalized control of the external device. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system framework of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 A signal flow dynamic mapping and interaction system based on human body sensing includes: a front-end multimodal acquisition module, an attitude signal mapping module, a dynamic control module, an external device structure module, and a visualization interaction and rule customization module; The front-end multimodal acquisition module is used to collect various data of the human body during movement. The front-end multimodal acquisition module includes a depth camera, an inertial measurement unit, an array of pressure sensors, and electromyography (EMG) electrodes on the human body surface. The depth camera is used to acquire the human skeleton sequence and extract the three-dimensional angle data of the joints. The inertial measurement unit is worn at the end of the user's limbs to acquire limb movement trajectory, acceleration, and spatial orientation. The array of pressure sensors is deployed on the standing plane to acquire the real-time dynamic distribution and displacement trajectory of the human body's center of gravity. The EMG electrodes on the human body surface are attached to the target muscle group to acquire the human body's EMG signals. The attitude signal mapping module is used to input the acquired data into a lightweight neural network to obtain a multi-dimensional intermediate representation vector aligned with the timestamp. The expression for the multi-dimensional intermediate representation vector is as follows:

[0020] in, Indicates the timestamp Multidimensional intermediate representation vector at time. , For the set of real numbers, For vectors The total dimension of the vector. Include Each real component, such as the overall height of the left arm raised and the angle of the torso leaning forward; Indicates the length of the time window; For neural network functions; This is the current timestamp; Indicates the current timestamp Input tensor at time; It is a separator; These are network weight parameters; At any discrete timestamp The system then uses a lightweight neural network. The past time window (length is) Multimodal human body sensing data continuously collected within the system As input, the fixed weight parameters obtained during network training are used The mapping generates a value related to the current time. Aligned multidimensional intermediate representation vector The vector It is the real number space one of the A dimensional vector, where each dimension represents a human motion feature with clear semantics automatically abstracted from raw sensor data (such as the height of the left arm raised, the angle of the torso leaning forward, the left and right shift of the center of gravity, etc.). Input tensor The expression is as follows:

[0021] in, Indicates the current timestamp Input tensor at time; For human joints in time stamp Three-dimensional Euler angles; For timestamp Two-dimensional coordinates of the human body's center of gravity; For timestamp Human posture quaternions; For timestamp The multidimensional feature vector of electromyographic signals; The dynamic control module generates trigger signals based on a multi-dimensional intermediate representation vector, including the following steps: Step A1: Calculate the trigger score based on the multidimensional intermediate representation vector corresponding to each timestamp. Its expression is as follows:

[0022] in, Indicates the timestamp Scoring is triggered at the time; The total number of triggering conditions; For vectors The total dimension Indicates the timestamp Multidimensional intermediate representation vector; Indicates the timestamp Time vector The dimensional vector; Indicates the first The triggering rules of the external device are in the first One trigger condition; For indicator functions, when When, the value is 1, when When the value is 0; Representing vectors The duration; Indicates the first The specified duration corresponding to the triggering rule of an external device; For indicator functions, when When, the value is 1, when When the value is 0; For summation operations; and The weighting coefficient can be set in the visualization interaction and rule customization module; Formula Part 1 Reflecting feature matching degree, through indicator functions Statistics on timestamps Below, multidimensional intermediate representation vector The number of times each dimension value matches a condition in the preset trigger condition set is calculated, and then divided by the total number of trigger conditions. The matching ratio is obtained, and finally multiplied by the weight. Formula Part 2 This reflects duration matching, which determines the duration of the current human posture through another indicator function. Has the duration specified by the triggering rule been reached? If the target is reached, the contribution weight will be calculated. Otherwise, the score is 0; a score is triggered when both feature matching and time matching meet the criteria. If the trigger threshold is reached, the corresponding triggering event is determined to be allowed to occur. Step A2: Determine whether to generate a trigger signal based on the trigger score, specifically: when the trigger score... When the trigger threshold is reached, the generation includes external devices. The trigger signal is sent to the external device structure module, and the trigger score is obtained. When the threshold is less than the trigger threshold, no trigger signal is generated; The external device structure module is used to convert the trigger signal into a data packet that conforms to the target external device's communication protocol and send the data packet to the target external device; the external devices include a lighting matrix device, a digital audio workstation, and a robot joint actuator; The visualization and rule customization module provides users with a visual interface, which includes a real-time signal flow visualization panel, a draggable mapping rule editing area, and sensitivity and threshold adjustment controls for each trigger channel.

[0023] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A signal flow dynamic mapping and interaction system based on human body sensing, characterized in that: It includes a front-end multimodal acquisition module, an attitude signal mapping module, a dynamic control module, an external device structure module, and a visualization interaction and rule customization module; The front-end multimodal acquisition module is used to collect various data of the human body during movement; The attitude signal mapping module is used to input the collected data into a lightweight neural network to obtain a multi-dimensional intermediate representation vector aligned with the timestamp. The dynamic control module generates trigger signals based on a multi-dimensional intermediate representation vector. The external device structure module is used to convert the trigger signal into a data packet that conforms to the target external device communication protocol, and send the data packet to the target external device; The visualization interaction and rule customization module is used to provide users with a visual interface.

2. The signal flow dynamic mapping and interaction system based on human body sensing according to claim 1, characterized in that: The front-end multimodal acquisition module includes a depth camera, an inertial measurement unit, an array of pressure sensors, and electromyography (EMG) electrodes on the human body surface.

3. The signal flow dynamic mapping and interaction system based on human body sensing according to claim 2, characterized in that: The depth camera is used to acquire human skeleton sequences and extract three-dimensional angle data of joints. The inertial measurement unit is used to acquire limb movement trajectories, accelerations, and spatial orientations. The array-type pressure sensor is used to acquire the real-time dynamic distribution and displacement trajectory of the human body's center of gravity. The electromyography (EMG) signal electrodes on the human body surface are used to acquire the human body's EMG signals.

4. The signal flow dynamic mapping and interaction system based on human body sensing according to claim 2, characterized in that: The expression for the multidimensional intermediate representation vector is as follows: in, Indicates the timestamp Multidimensional intermediate representation vector at time. , For the set of real numbers, For vectors Total dimensions; Indicates the length of the time window; For neural network functions; This is the current timestamp; Indicates the current timestamp Input tensor at time; It is a separator; These are the network weight parameters.

5. The signal flow dynamic mapping and interaction system based on human body sensing according to claim 4, characterized in that: The input tensor The expression is as follows:

6. Among them, Indicates the current timestamp Input tensor at time; For human joints in time stamp Three-dimensional Euler angles; For timestamp Two-dimensional coordinates of the human body's center of gravity; For timestamp Human posture quaternions; For timestamp The multidimensional feature vector of electromyographic signals.

7. The signal flow dynamic mapping and interaction system based on human body sensing according to claim 4, characterized in that: The generation of the trigger signal based on the multidimensional intermediate representation vector includes the following steps: Step A1: Calculate the trigger score based on the multidimensional intermediate representation vector corresponding to each timestamp. ; Step A2: Determine whether to generate a trigger signal based on the trigger score.

8. The signal flow dynamic mapping and interaction system based on human body sensing according to claim 6, characterized in that: Step A1 triggers the score. The expression is as follows:

9. Among them, Indicates the timestamp Scoring is triggered at the time; The total number of triggering conditions; For vectors The total dimension Indicates the timestamp Multidimensional intermediate representation vector; Indicates the timestamp Time vector The dimensional vector; Indicates the first The triggering rules of the external device are in the first One trigger condition; For indicator functions, when When, the value is 1, when When the value is 0; Representing vectors The duration; Indicates the first The specified duration corresponding to the triggering rule of an external device; For indicator functions, when When, the value is 1, when When the value is 0; For summation operations; and These are the weighting coefficients.

10. The signal flow dynamic mapping and interaction system based on human body sensing according to claim 6, characterized in that: Step A2 determines whether to generate a trigger signal based on the trigger score, specifically: when the trigger score... When the trigger threshold is reached, the generation includes external devices. The trigger signal is sent to the external device structure module, and the trigger score is obtained. When the threshold is less than the trigger threshold, no trigger signal is generated.

11. The signal flow dynamic mapping and interaction system based on human body sensing according to claim 6, characterized in that: The external devices include a light matrix device, a digital audio workstation, and a robot joint actuator.

12. The signal flow dynamic mapping and interaction system based on human body sensing according to claim 9, characterized in that: The visualization interface includes a real-time signal flow visualization panel, a draggable mapping rule editing area, and sensitivity and threshold adjustment controls for each trigger channel.

Images