An active space intelligent interaction method and system based on multi-modal perception
By constructing a digital twin foundation and a multimodal perception network, the passive nature and insufficient privacy protection of existing spatial intelligent interaction systems are solved, enabling proactive and seamless services and efficient responses to device collaboration, and adapting to high-precision intent recognition in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-05-12
- Publication Date
- 2026-07-14
AI Technical Summary
Existing spatial intelligent interaction systems suffer from passive interaction modes, insufficient privacy protection, and rigid device collaboration, resulting in delayed service response, cumbersome operation, and difficulty in achieving high-precision and full-coverage intent quantification in complex scenarios.
By constructing a digital twin foundation for physical space, using a multimodal perception network to acquire user behavior feature data, performing edge processing and calculating behavior anomaly indexes, and combining device election strategies to achieve proactive interaction, a dynamic device scheduling and end-to-end privacy protection architecture are adopted to ensure data compliance and response consistency.
It enables proactive and seamless services without explicit voice or touch commands, improving the naturalness of interaction and responsiveness, ensuring user privacy protection and device collaboration flexibility, and adapting to high-precision intent recognition in complex application scenarios.
Smart Images
Figure CN122387318A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence interaction technology, specifically, it relates to an active spatial intelligent interaction method and system based on multimodal perception. Background Technology
[0002] With the deep integration of artificial intelligence and the Internet of Things (IoT) technologies, spatial intelligence has become a core support for building smart living and working environments. By integrating sensor networks and sensing algorithms, it enables real-time monitoring and digital mapping of people, machines, and objects within physical spaces. The field of spatial intelligence encompasses multiple technological branches, including context awareness, semantic understanding, and automated control, and plays a crucial role in improving human-computer interaction efficiency, optimizing resource allocation, and enhancing environmental adaptability.
[0003] Among them, proactive spatial intelligent interaction, as a cutting-edge development direction in this field, aims to acquire environmental information and user characteristics through multimodal perception technology, and use decision-making algorithms to accurately predict and respond to users' potential needs in real time. The core of this technology lies in establishing a dynamic interactive relationship between users and physical entities through the deep integration of diverse and heterogeneous data such as posture, gaze, and position, thereby realizing a transformation from the traditional passive command-triggered interaction paradigm to proactive and seamless service.
[0004] Existing technologies face multiple challenges in achieving spatial intelligent interaction. First, the interaction mode is inherently passive, typically relying heavily on explicit voice commands or physical touch operations. It cannot effectively predict user intent as they approach the target device, resulting in delayed service response and an unnatural interaction process. Second, data compliance and privacy protection capabilities are insufficient. Existing visual perception solutions often rely on uploading raw image data to the cloud for processing, posing serious risks of privacy leaks and compliance issues. Third, the device collaboration architecture is too rigid, often employing fixed, centralized management logic. It cannot flexibly elect master / slave devices and schedule tasks based on real-time user dynamics, device capabilities, and load status, leading to high response latency and poor consistency among multiple devices. Finally, in complex application scenarios, existing solutions struggle to achieve high-precision, comprehensive intent quantification while balancing privacy desensitization and system performance. These issues collectively hinder the widespread application and technological evolution of proactive spatial intelligent interaction systems. Summary of the Invention
[0005] The purpose of this invention is to provide an active spatial intelligent interaction method and system based on multimodal perception, which mainly solves the technical problems of passive interaction mode, insufficient privacy protection and rigid device collaboration in the existing interaction mode.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] A proactive spatial intelligent interaction method based on multimodal perception includes the following steps:
[0008] S1, constructs a digital twin foundation for the physical space, creates digital identity models for each functional area and device within the space, and configures parameters;
[0009] S2, continuously acquire user behavior feature data through a distributed sensing network. The user behavior feature data includes collecting user head micro-motion data and calculating angular velocity variance through at least one type of micro-motion detection sensor, and / or collecting eye gaze data and calculating gaze drift frequency through a visual sensor. The micro-motion detection sensor is configured to output radio frequency signals or sound wave signals containing phase information, and is processed into structured feature vectors at the network edge.
[0010] S3, based on the angular velocity variance of head micro-motion data and the gaze drift frequency of eye fixation data, a behavior abnormality index is obtained through weighted calculation; when the behavior abnormality index exceeds a preset threshold, a suppression command is generated to block or reduce the probability of erroneous active interaction through a hardware interrupt signal; when no suppression command is generated and the interaction tendency quantification value meets the preset triggering conditions, a device control command is generated.
[0011] S4, based on the device election strategy, schedule one or more devices to execute the device control commands.
[0012] Furthermore, in this invention, the specific implementation process of S1 is as follows:
[0013] S11, using 3D scanning equipment to acquire full 3D point cloud data of the physical space;
[0014] S12, Based on the semantic segmentation algorithm, the physical entities in the physical space are classified and their boundaries are defined;
[0015] S13. Assign a globally unique digital identity identifier to each labeled physical entity through a digital mapping protocol, and encapsulate it into an entity digital identity model containing geometric attributes and interaction logic.
[0016] Furthermore, in S13, the metadata information associated with the digital identity identifier includes a three-dimensional spatial geometric description, physical occupant boundary coordinates, entity type attribute tags, and interaction logic parameters; the three-dimensional spatial geometric description records the fine outer contour of the physical entity in the form of a polygonal mesh; the physical occupant boundary coordinates define the multi-degree-of-freedom pose and minimum bounding box of the physical entity in the global spatial coordinate system; the entity type attribute tags are used to distinguish between static environment entities and dynamic controlled devices at the logical level, the static environment entities participate in spatial collision detection, and the dynamic controlled devices are associated with controlled interfaces and function functions.
[0017] Furthermore, in this invention, the interaction logic parameters include a response potential energy threshold, an intent triggering condition sequence, and an associated execution strategy set; the response potential energy threshold sets the minimum energy level required to trigger the interaction action; the intent triggering condition sequence specifies the user's posture combination or preset gaze duration; the associated execution strategy set defines the differentiated response actions taken by the dynamically controlled device in different scene modes, such as adjusting the color temperature to a preset target value in office mode and turning it off in movie-watching mode.
[0018] Furthermore, in this invention, the specific implementation process of S2 is as follows:
[0019] S21, using a distributed multimodal sensor cluster to capture environmental changes in the physical space in real time, and performing feature processing on the original sensing signals within the edge computing node;
[0020] S22, extract anonymized behavioral feature data vectors that reflect user behavior patterns, and at the same time execute an immediate destruction mechanism for the original privacy data to ensure that the original image and biometric information are erased immediately after feature extraction is completed in the memory buffer.
[0021] Furthermore, in this invention, in S2, the sensing network consists of a visual sensing unit, a passive sensing unit, and an environmental sensing unit.
[0022] The multimodal sensor cluster includes at least one type of micro-motion detection sensor and / or vision sensor; the micro-motion detection sensor is configured to output a radio frequency signal or acoustic signal containing phase information.
[0023] The visual perception unit is configured to run a lightweight human key point extraction algorithm in the edge processor and only output skeleton topology data that does not contain biometric information.
[0024] The passive sensing unit uses channel state information of wireless local area network signals or frequency-modulated continuous wave signals of millimeter-wave radar to obtain the user's respiratory rate, heart rate and minor body deformation characteristics.
[0025] The environmental sensing unit collects data on ambient brightness, ambient temperature, ambient humidity, and sound pressure level within the physical space.
[0026] The sensing network uses a hardware-triggered synchronization mechanism to ensure that the data collected by the visual sensing unit, the passive sensing unit, and the environmental sensing unit are aligned with a preset precision on the time axis, forming a multimodal fusion feature frame.
[0027] Furthermore, in this invention, the specific implementation process of S3 is as follows:
[0028] S31, Real-time monitoring of the user's position vector, posture features, and gaze vector and their relative spatial relationship with the bounding boxes in each entity's digital identity model;
[0029] S32, use a quantitative evaluation model based on interaction potential to calculate the quantitative value of the interaction tendency between the user and the physical entity, and compare the quantitative value of the interaction tendency with the preset activation threshold in the entity's digital identity model;
[0030] S33, when the calculated interaction tendency quantification value meets the preset triggering conditions, the scenario logic model is used to analyze the user's potential interaction needs, and a specific set of control instructions is generated in combination with the current scenario context semantics to realize the transformation from passive response to proactive service.
[0031] Furthermore, in this invention, the interaction tendency quantification value is calculated using the following formula:
[0032]
[0033] Where P represents the interaction tendency quantification value, For pose matching degree, For the sake of attracting attention, The abnormal behavior index is represented by α, β, and γ, which are preset weight coefficients corresponding to the current interaction scenario, and α+β+γ=1.
[0034] Furthermore, in step S4, the election strategy is as follows: calculate a comprehensive weight value for each candidate device. ,choose The device with the highest value is designated as the primary execution device; the comprehensive weight value The calculation formula is:
[0035]
[0036] Where Dist(U, Di) is the Euclidean distance between the user and device i, f() is the distance decreasing function, and cosθ is the cosine similarity between the user's gaze vector and the direction vector pointing to device i. Score the degree of matching between the function and instructions of device i. , , This is the preset adjustment coefficient.
[0037] This invention also provides an active spatial intelligent interaction system based on multimodal perception, used to implement the above-mentioned interaction method, comprising:
[0038] The entity modeling module is responsible for the storage, updating, and maintenance of spatial topology relationships of digital archives;
[0039] The multimodal sensing module contains multiple sensor units and is responsible for collecting environmental data, aligning multi-source heterogeneous data, and desensitizing privacy data.
[0040] The intent decision module is responsible for the real-time calculation of the interaction tendency quantification value, the analysis of temporal characteristics, and the determination of trigger logic.
[0041] The dynamic scheduling module is responsible for implementing the device election strategy, distributing device control commands, and coordinating control between device clusters.
[0042] Compared with the prior art, the present invention has the following beneficial effects:
[0043] (1) This invention uses an interaction tendency quantification algorithm to perform fusion calculations on three types of features: user posture matching degree, eye attention degree, and behavior abnormality index. It can predict potential user interaction needs without requiring users to issue explicit voice or touch commands, thus realizing an upgrade of the interaction paradigm from passive command triggering to proactive seamless service. This significantly improves the naturalness and responsiveness of spatial interaction and effectively solves the problems of service lag and cumbersome operation in the traditional interaction mode.
[0044] (2) The present invention adopts a full-link privacy protection architecture of edge feature extraction + immediate destruction of original data. After all multimodal perception data is characterized at the edge, the original image, original radio frequency signal and other carrier data that may involve user privacy are immediately erased. Only the anonymized structured feature vector is transmitted to participate in the subsequent decision-making logic. There is no need to upload the original privacy data to the cloud. The risk of privacy leakage is completely avoided from the underlying logic level of data flow, and the data compliance and supervision requirements are fully met.
[0045] (3) The present invention designs a dynamic device election and scheduling strategy. By comprehensively considering three dimensions of factors, namely the spatial distance between the user and the device, the matching degree of the user's line of sight, and the adaptability of the device function, the device execution weight is calculated. The optimal main execution device can be flexibly selected and the timing of multi-device collaborative actions can be accurately aligned. This breaks through the limitations of the rigid scheduling of the traditional central control management architecture, effectively reduces the response delay of multiple devices, and improves the consistency of multi-device cluster collaboration.
[0046] (4) This invention constructs a digital twin base covering all elements of physical space, assigns a unified global unique digital identity model and spatiotemporal benchmark to all static entities and dynamic controlled devices, realizes high-precision spatiotemporal alignment of multimodal perception data, and can simultaneously take into account privacy protection capabilities and intent recognition accuracy in complex application scenarios, providing a complete and feasible technical solution for the large-scale application of proactive spatial intelligent interaction systems. Attached Figure Description
[0047] Figure 1This is a schematic diagram of the overall architecture of the present invention.
[0048] Figure 2 This is a flowchart illustrating the method of the present invention.
[0049] Figure 3 This is a schematic diagram of the quantitative evaluation process in this invention. Detailed Implementation
[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.
[0051] like Figure 1 As shown, this invention discloses an active spatial intelligent interaction system based on multimodal perception. The physical architecture and logical components of this system work together to accurately capture user intentions within the physical space and achieve automated device collaboration. In the system initialization phase, a 3D laser scanning device is first used to perform a full scan of the target physical space. The 3D laser scanning device establishes a data connection with the entity modeling module via a USB interface or industrial network interface. The 3D laser scanning device rotates and scans at the center of the space or multiple measuring stations, emitting laser pulses and receiving reflected signals to acquire high-density point cloud data of the surfaces of all objects within the space. This raw point cloud data is transmitted to the entity modeling module, which executes a point cloud filtering algorithm to remove noise points caused by glass reflection or airborne dust, and uses a surface reconstruction algorithm to transform the scattered point cloud into a continuous geometric mesh model. The entity modeling module further calls a pre-stored semantic segmentation model to identify and classify objects in the mesh model, distinguishing between static environmental entities such as walls and floors, and controlled devices such as lights, air conditioners, and display terminals.
[0052] After semantic segmentation, the entity modeling module defines the physical boundaries of each identified physical entity, typically using axis-aligned bounding boxes or oriented bounding boxes to define the entity's spatial occupancy. Each entity is assigned a globally unique digital identifier, and the metadata associated with this identifier is stored in the entity modeling module's database, including the entity's center coordinates, 3D dimensions, controlled interface protocol, and interaction logic parameters. Through this digital mapping protocol, the system constructs a digital twin foundation that fully corresponds to the physical space, providing a unified spatiotemporal reference for subsequent interactive computations.
[0053] The digital twin base includes an AABB axis-aligned bounding box or an OBB oriented bounding box for physical space collision detection.
[0054] refer to Figure 2The system's perception layer consists of a multimodal perception module, which includes visual perception units, passive perception units, and environmental sensing units distributed throughout the space. The visual perception units are typically ceiling-mounted, positioned at the four corners and center of the room's ceiling to ensure the perception field of view covers the entire interior space. Each visual perception unit integrates a CMOS image sensor and an embedded SoC chip, connected to edge computing nodes via a PoE-powered industrial Ethernet switch. During operation, the visual perception unit does not transmit raw images; instead, it extracts the coordinates of key points on the human skeleton using a lightweight neural inference engine at the edge. These coordinate sequences are anonymized to form structured feature vectors.
[0055] The passive sensing unit is embedded inside the wall or behind decorative panels. Its core components include an antenna array and a radio frequency processing chip. The passive sensing unit collects channel state information of wireless signals and uses multipath reflection characteristics to identify subtle deformation movements of the user, such as chest rise and fall caused by breathing or gait frequency. Environmental sensing units are distributed in functional areas such as work areas and rest areas, and are connected to the system via RS485 bus or wireless sensor network to monitor indoor temperature, humidity, illuminance, and carbon dioxide concentration. All sensing nodes in the multimodal sensing module are connected through a hardware-triggered synchronization bus. The central control unit sends synchronization pulse signals to each node through this bus to ensure that visual, passive, and environmental data are aligned on the time axis with a synchronization error of less than 10ms.
[0056] The micro-motion detection sensor module and the vision sensor module use hardware synchronous sampling, with a sampling clock synchronization error of less than 10ms.
[0057] The acquired multimodal feature frames are aggregated to edge computing nodes for preliminary processing. These edge computing nodes are equipped with high-performance memory buffers. After feature fusion and dimensionality reduction, they immediately execute memory erasure commands to destroy all original carrier data that may involve biometric privacy, blocking the path of privacy information to the storage medium. The processed feature vectors are then transmitted to the intent decision module via the PCIe bus. The intent decision module employs a heterogeneous architecture consisting of an FPGA and a high-performance embedded processor, with the FPGA responsible for performing high-concurrency geometric operations and spatial collision detection.
[0058] refer to Figure 3 During the intent quantification and assessment phase, the intent decision module monitors the user's displacement vector in the three-dimensional coordinate system in real time. When the user enters the preset interaction area, the system calculates the user's interaction potential energy relative to the bounding box of each entity's digital identity model. This interaction potential energy is a virtual field quantity reflecting the degree of association between the user and the device, and its value depends on the user's posture matching degree, line-of-sight overlap degree, and behavioral anomaly index. The specific formula for calculating the interaction tendency quantification value P is as follows:
[0059]
[0060] Among them, S pose The posture matching degree is represented by the cosine distance between the user's real-time skeleton vector and the system's preset standard action vector, calculated by the FPGA. It is used to determine whether the user has made a directional or preparatory action. gaze Gaze attention intensity is the probability of intersection between the gaze vector captured by the visual perception unit and the bounding box of the target entity. The system uses spatial geometry algorithms to identify the duration of gaze on the entity surface. The user's behavioral anomaly index is calculated by analyzing the variance of the user's head rotation angular velocity and the frequency of gaze drift. Parameters α, β, and γ are dynamically adjusted according to the current scene mode (such as meeting mode or rest mode) to change the weight of different sensory modalities in decision-making.
[0061] The behavioral abnormality index can be calculated using the formula: D anomaly = γ1 × variance of head micro-motion angular velocity + γ2 × line-of-sight drift frequency, where γ1 and γ2 are weighting coefficients, ranging from 0.1 to 0.9, and γ1 + γ2 = 1. The weighted calculation is performed by the hardware logic circuit of the edge computing node or FPGA, and the calculation result is directly used to control the power supply or signal triggering of devices in the physical space.
[0062] When the interaction tendency quantification value P exceeds the preset response potential threshold (no action occurs if it is not reached), and falls within a preset time window, the intent decision module determines that the user has a clear interaction intent and identifies the target entity. Subsequently, the dynamic scheduling module takes over the control flow. The dynamic scheduling module establishes a logical connection with the controlled devices in the physical space via a wireless control link (such as Wi-Fi, Zigbee, or Bluetooth Mesh protocol). To select the most suitable executor from multiple potentially responding devices, the dynamic scheduling module calculates the comprehensive weight value of each controlled device. The calculation formula is:
[0063]
[0064] In the formula, Dist(U, Di) represents the Euclidean distance between the user and the device. The distance-decreasing function f() gives higher weights to devices that are closer to the user. cosθ is the cosine similarity between the user's gaze vector and the direction vector pointing to device i, ensuring that the system prioritizes responding to the device that the user is currently looking at. Score the degree of matching between the function and instructions of device i. , , This is the preset adjustment coefficient.
[0065] The dynamic scheduling module allocates execution priorities based on calculated weight values and generates a set of collaborative control instructions that includes response timing. For example, when it is predicted that a user is preparing to give a presentation, the system will automatically dim the lights, close the curtains, and start the projection terminal. The execution latency of these actions is precisely compensated to offset the impact of network bandwidth fluctuations and ensure the visual synchronization of multiple device actions.
[0066] The FPGA plays a crucial role in parallel acceleration during this process, offloading complex matrix operations and point cloud collision detection tasks from the central control unit, ensuring extremely low end-to-end latency from intent capture to device response. This architecture enables the system to support real-time scheduling of large-scale controlled device clusters, maintaining stable interactive performance even in high-density office environments.
[0067] In real-world applications, such as a smart office, when a user walks to the conference table and pulls out a chair, the visual perception unit captures the user's displacement vector and skeletal features. Edge computing nodes extract key point data and transmit it to the intent decision module. At this point, the system detects collisions between the user's gaze and the digital twin model of the conference table, finding that the gaze lingers in the center of the table for more than a preset duration. The FPGA quickly calculates the interaction potential energy P, determining that the user is about to start a meeting. The dynamic scheduling module then calculates the combined weights of surrounding lighting, air conditioning, and displays, issuing a pre-activation command. The moment the user sits down, all the environmental parameters required for the meeting mode are already adjusted. The entire process requires no voice or touch commands from the user; the physical space provides proactive service through a multimodal perception network.
[0068] Furthermore, the system's passive sensing unit provides continuous physiological characteristic monitoring when the user is at rest. If the environmental sensing unit detects an increase in indoor carbon dioxide concentration and the passive sensing unit identifies abnormal fluctuations in the user's breathing rate, the central control unit will automatically adjust the air conditioning's ventilation power through the dynamic scheduling module, based on the current interaction weights, and may issue health alerts through the display terminals in the controlled devices. This deep fusion of multimodal data enables the system to not only respond to explicit physical movements but also proactively respond to implicit physiological needs.
[0069] When the ratio of respiratory harmonic energy to limb micro-movement energy is less than 0.3 and the duration is ≥500ms, it is judged as an abnormal behavioral state.
[0070] In another practical application scenario, for example a 20m 2Inside the independent intelligent fitness cabin, the system has pre-built a digital twin base through 3D laser scanning. Static entities such as walls and floor mirrors, as well as controlled devices such as intelligent treadmills, adjustable dumbbell sets, ambient lighting, fresh air systems, air conditioners, intelligent interactive mirrors, audio equipment, and emergency call devices, have all completed semantic segmentation and unique identity binding. The spatial occupancy, interface protocols, and interaction parameters of all entities are stored in the database of the entity modeling module.
[0071] When a user scans a code to open the door and enter the fitness pod, the multimodal module of the perception layer immediately starts collecting data synchronously: the ceiling-mounted visual perception unit captures the user's displacement vector and the coordinates of key points on their entire skeletal frame; the passive perception unit embedded in the wall identifies the user's gait frequency and initial heart rate through wireless channel status information; and the environmental sensing units distributed in the aerobic and strength zones synchronously collect the current temperature, humidity, illuminance, and carbon dioxide concentration inside the pod. All data is synchronized via a hardware synchronization bus with a synchronization error of less than 10ms before being aggregated to the edge computing node. After completing feature fusion and dimensionality reduction, the edge node immediately erases potentially privacy-involving data such as original images and signals, transmitting only the structured feature vector to the intent decision module.
[0072] If a user enters the cabin and first walks towards the adjustable dumbbell set in the strength area, the visual perception unit recognizes that the user's gaze intersects with the bounding box of the digital twin model of the dumbbell set, and the duration of the gaze exceeds the preset 1.5 seconds (gaze attention score reaches 0.89). Simultaneously, the cosine distance between the user's raised arm posture and the system's preset "ready to retrieve training equipment" standard motion vector is only 0.07 (posture matching score reaches 0.93), and the variance of the user's head rotation angular velocity is less than 0.2 (behavioral anomaly index is only 0.12). At this point, the FPGA accelerates the calculation and obtains an interaction tendency quantification value, which exceeds the preset response threshold of 0.7, determining that the user has a clear intention to interact with the dumbbells, and the target entity is the adjustable dumbbell set.
[0073] The dynamic scheduling module then calculates the overall weight value of all associated controlled devices: the adjustable dumbbell set is only 0.8 meters away from the user's Euclidean distance, and the user's line of sight is completely aligned with the device, resulting in a perfect functional match and the highest overall weight. It is given priority to issue instructions to automatically adjust to the 15kg weight corresponding to the user's historical training record. The ambient light next to the strength training area has the second highest weight and automatically switches to a high-brightness cool white light mode suitable for strength training. The smart interactive mirror facing the training area has the third highest weight and automatically starts and pushes a standard biceps training movement demonstration corresponding to the weight. The air conditioning and audio equipment are simultaneously adjusted to the user's preset strength training preference parameters. The entire response process has a latency of less than 80ms. The moment the user raises their hand to touch the dumbbell, all supporting devices are already in place, without requiring any touch or voice operation from the user.
[0074] During user training, the passive sensing unit continuously monitors the user's physiological characteristics non-contactly. After completing four sets of dumbbell exercises, the passive sensing unit detects that the user's breathing rate has increased by 60% compared to the initial value, and the heart rate has reached 140 beats per minute. At the same time, the environmental sensing unit detects that the carbon dioxide concentration near the strength training area has risen to 1200 ppm. Without requiring explicit user action, the system directly determines through the intention decision module that the user has an implicit need for ventilation and state adjustment. The dynamic scheduling module then issues coordinated instructions: the fresh air system temporarily increases its power by 30% to accelerate air circulation in the cabin, the smart interactive mirror pops up a relaxation prompt, and the audio equipment automatically switches to soothing rhythmic music to help the user quickly adjust their state.
[0075] If the user then moves to the treadmill in the aerobic zone, the system determines the user's running intention through the same interactive potential energy calculation logic. The treadmill then automatically loads the user's preset 30-minute jogging plan and adjusts the running parameters to an initial incline of 3% and a pace of 6km / h. The ambient lighting switches to a warm orange dynamic light effect exclusive to aerobic training. The smart interactive mirror simultaneously displays the user's real-time heart rate, cadence, calorie consumption, and other data. At the same time, it captures the user's skeletal key points through the visual perception unit and compares them with standard running movements in real time. If there are problems such as incorrect foot placement or torso tilt, a visual correction prompt will pop up on the mirror immediately.
[0076] When a user finishes training and walks towards the cabin door, the system determines that the training session has ended, automatically stops all training equipment, switches the ambient lighting to a soft natural light mode, restores the air conditioning to its normal standby temperature, and displays a complete training report on the smart interactive mirror. This report includes data such as total training time of 42 minutes, total calories burned of 387 kcal, completion rate of biceps / quadriceps muscle group training, and heart rate fluctuation range. At the same time, the fresh air system operates at high power for 5 minutes to replace the air in the cabin, preparing a suitable training environment for the next user.
[0077] In terms of installation and maintenance, this system demonstrates extremely high flexibility. Thanks to its digital twin-based logical architecture, when controlled devices are added, removed, or relocated within the physical space, only the corresponding digital identity model and spatial coordinates need to be updated through the entity modeling module, without rewriting the underlying perception and decision-making logic. The sensing nodes in the distributed sensing network support plug-and-play functionality, and edge computing nodes can automatically identify newly connected visual sensing units or environmental sensing units and incorporate them into the global synchronization system via a hardware-triggered synchronization bus.
[0078] Frequent data exchange between the central control unit and the FPGA via high-speed caching ensures seamless integration between high-level logic decisions and low-level hardware acceleration. This hardware-software co-design pattern addresses the shortcomings of low intent recognition accuracy and slow response in traditional solutions. Through continuous iteration using a recursive least squares algorithm, the system develops personalized spatial interaction strategies over long-term operation, truly achieving deep integration and proactive interaction between physical space and humans.
[0079] In other embodiments, the micro-motion detection sensor includes, but is not limited to, millimeter-wave radar, ultrasonic sensors, ultra-wideband radar, terahertz radar, or WiFi sensing modules; the visual sensor can be replaced with an electrooculogram sensor or an infrared photoelectric sensor. All equivalent data acquisition methods employed by those skilled in the art based on the concept of this invention should fall within the protection scope of this invention.
[0080] The above embodiments are merely one of the preferred embodiments of the present invention and should not be used to limit the scope of protection of the present invention. Any modifications or refinements made to the main design concept and spirit of the present invention that are not of substantial significance, but solve the same technical problem as the present invention, should be included within the scope of protection of the present invention.
Claims
1. A proactive spatial intelligent interaction method based on multimodal perception, characterized in that, Includes the following steps: S1, constructs a digital twin foundation for the physical space, creates digital identity models for each functional area and device within the space, and configures parameters; S2, continuously acquire user behavior feature data through a distributed sensing network. The user behavior feature data includes collecting user head micro-motion data and calculating angular velocity variance through at least one type of micro-motion detection sensor, and / or collecting eye gaze data and calculating gaze drift frequency through a visual sensor. The micro-motion detection sensor is configured to output radio frequency signals or sound wave signals containing phase information, and is processed into structured feature vectors at the network edge. S3 calculates the behavioral abnormality index by weighting the angular velocity variance of head micro-motion data and the gaze drift frequency of eye fixation data; when the behavioral abnormality index exceeds a preset threshold, a suppression command is generated to block or reduce the probability of false triggering of active interaction through a hardware interrupt signal. When no suppression instruction is generated and the interaction tendency quantization value meets the preset triggering conditions, a device control instruction is generated; S4, based on the device election strategy, schedule one or more devices to execute the device control commands.
2. The active spatial intelligent interaction method based on multimodal perception according to claim 1, characterized in that, The specific implementation process of S1 is as follows: S11, using 3D scanning equipment to acquire full 3D point cloud data of the physical space; S12, Based on the semantic segmentation algorithm, the physical entities in the physical space are classified and their boundaries are defined; S13. Assign a globally unique digital identity identifier to each labeled physical entity through a digital mapping protocol, and encapsulate it into an entity digital identity model containing geometric attributes and interaction logic.
3. The active spatial intelligent interaction method based on multimodal perception according to claim 2, characterized in that, In step S13, the metadata information associated with the digital identity identifier includes a three-dimensional spatial geometric description, physical occupant boundary coordinates, entity type attribute tags, and interaction logic parameters. The three-dimensional spatial geometric description records the fine outer contour of the physical entity in the form of a polygonal mesh. The physical occupant boundary coordinates define the multi-degree-of-freedom pose and minimum bounding box of the physical entity in the global spatial coordinate system. The entity type attribute tags are used to distinguish between static environmental entities and dynamic controlled devices at the logical level. The static environmental entities participate in spatial collision detection, and the dynamic controlled devices are associated with controlled interfaces and function calls.
4. The active spatial intelligent interaction method based on multimodal perception according to claim 3, characterized in that, The interaction logic parameters include a response potential energy threshold, an intent triggering condition sequence, and an associated execution strategy set; the response potential energy threshold sets the minimum energy level required to trigger the interaction action; the intent triggering condition sequence specifies the user's posture combination or preset gaze duration. The associated execution strategy set defines the differentiated response actions taken by dynamically controlled devices in different scenario modes. In office mode, the response strategy of the lighting entity is to adjust the color temperature to a preset target value, and in movie-watching mode, it is adjusted to the off state.
5. The active spatial intelligent interaction method based on multimodal perception according to claim 1, characterized in that, The specific implementation process of S2 is as follows: S21, using a distributed multimodal sensor cluster to capture environmental changes in the physical space in real time, and performing feature processing on the original sensing signals within the edge computing node; S22, extract anonymized behavioral feature data vectors that reflect user behavior patterns, and at the same time execute an immediate destruction mechanism for the original privacy data to ensure that the original image and biometric information are erased immediately after feature extraction is completed in the memory buffer.
6. The active spatial intelligent interaction method based on multimodal perception according to claim 5, characterized in that, In S2, the sensing network consists of a visual sensing unit, a passive sensing unit, and an environmental sensing unit. The multimodal sensor cluster includes at least one type of micro-motion detection sensor and / or vision sensor; the micro-motion detection sensor is configured to output a radio frequency signal or acoustic signal containing phase information. The visual perception unit is configured to run a lightweight human key point extraction algorithm in the edge processor and only output skeleton topology data that does not contain biometric information. The passive sensing unit uses channel state information of wireless local area network signals or frequency-modulated continuous wave signals of millimeter-wave radar to obtain the user's respiratory rate, heart rate and minor body deformation characteristics. The environmental sensing unit collects data on ambient brightness, ambient temperature, ambient humidity, and sound pressure level within the physical space. The sensing network uses a hardware-triggered synchronization mechanism to ensure that the data collected by the visual sensing unit, the passive sensing unit, and the environmental sensing unit are aligned with a preset precision on the time axis, forming a multimodal fusion feature frame.
7. The active spatial intelligent interaction method based on multimodal perception according to claim 1, characterized in that, The specific implementation process of S3 is as follows: S31, Real-time monitoring of the user's position vector, posture features, and gaze vector and their relative spatial relationship with the bounding boxes in each entity's digital identity model; S32, use a quantitative evaluation model based on interaction potential to calculate the quantitative value of the interaction tendency between the user and the physical entity, and compare the quantitative value of the interaction tendency with the preset activation threshold in the entity's digital identity model; S33, when the calculated interaction tendency quantification value meets the preset triggering conditions, the potential interaction needs of the user are analyzed using the scene logic model, and a specific set of control instructions is generated in combination with the semantic context of the current scene, so as to realize the transformation from passive response to active service.
8. The active spatial intelligent interaction method based on multimodal perception according to claim 7, characterized in that, The interaction tendency quantification value is calculated using the following formula: Where P represents the interaction tendency quantification value, For pose matching degree, For the sake of attracting attention, The abnormal behavior index is represented by α, β, and γ, which are preset weight coefficients corresponding to the current interaction scenario, and α+β+γ=1.
9. The active spatial intelligent interaction method based on multimodal perception according to claim 1, characterized in that, In step S4, the election strategy is as follows: calculate a comprehensive weight value for each candidate device. ,choose The device with the highest value is designated as the primary execution device; the comprehensive weight value The calculation formula is: Where Dist(U, Di) is the Euclidean distance between the user and device i, f() is the distance decreasing function, and cosθ is the cosine similarity between the user's gaze vector and the direction vector pointing to device i. Score the degree of matching between the function and instructions of device i. , , This is the preset adjustment coefficient.
10. A proactive spatial intelligent interaction system based on multimodal perception, characterized in that, To implement the interaction method as described in any one of claims 1 to 9, comprising: The entity modeling module is responsible for the storage, updating, and maintenance of spatial topology relationships of digital archives; The multimodal sensing module contains multiple sensor units and is responsible for collecting environmental data, aligning multi-source heterogeneous data, and desensitizing privacy data. The intent decision module is responsible for the real-time calculation of the interaction tendency quantification value, the analysis of temporal characteristics, and the determination of trigger logic. The dynamic scheduling module is responsible for implementing the device election strategy, distributing device control commands, and coordinating control between device clusters.