An interactive product experience terminal system that integrates sound, light, and electricity.
Through a multi-module collaborative audio-visual linkage terminal system, environmental and user information is collected, user emotions and needs are identified, and directional sound fields and contactless haptic interaction are output. This solves the problems of difficulty in personalizing user adjustments and inconsistent interaction in existing technologies, and enables efficient product display in an open environment.
Patent Information
- Application Number
- CN202611337145.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
The audio-visual displays of existing offline retail terminals cannot be personalized according to the user's real-time status, making it difficult for users to clearly obtain product information. The haptic interaction is disconnected from the product function, and the interaction process is not coherent in an open environment.
Through multi-module collaborative work, it collects environmental and user information, identifies emotional tendencies and interaction needs, outputs directional immersive sound fields and contactless haptic interaction, calibrates the multimodal output timing, and achieves personalized display.
Providing clear product information displays in noisy environments can extend user dwell time, improve information delivery efficiency, reduce interactive learning costs, and ensure the synchronization of multi-sensory feedback.
Smart Images

Figure CN123119783A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of interactive display technology, specifically to an interactive product experience terminal system that integrates sound, light, electricity, and electricity. Background Technology
[0002] Offline retail terminals are crucial touchpoints for products to directly reach consumers, and the interactivity and experiential aspects of these displays directly impact user awareness of the products. As consumer demands for enhanced experiences continue to rise, interactive display terminals integrating sound, light, and electricity are gradually replacing traditional static displays. Voice interaction, motion-activated triggers, and other interactive methods are increasingly being applied in various offline retail scenarios to attract user attention and convey product information.
[0003] Existing interactive terminals still have significant limitations in practical use. Most terminals display audio-visual content using pre-set, fixed programs, unable to be personalized based on the user's real-time status. This makes it difficult to provide targeted guidance for users with different levels of interest, limiting improvements in user dwell time and information reception. Furthermore, the complex noise environment of open retail spaces and the lack of directional sound from conventional speakers make it difficult for users to clearly grasp product information and create an immersive experience. In addition, most motion-sensing interactions are only used to trigger general operations such as page switching and content playback, lacking deep integration with the product's functional parameters. This disconnect between the interaction process and product understanding, coupled with potential timing misalignments in multi-module outputs, affects the overall continuity of the experience. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a product experience terminal interactive system that integrates sound, light, and electricity. This system constructs a complete terminal interactive experience chain through multi-module collaboration. It can collect surrounding environmental parameters and multi-dimensional user behavior information, identify user emotional tendencies and core interaction needs, output a directional immersive sound field that follows the user's head position, support contactless haptic interaction that maps product function parameters, and simultaneously calibrate the temporal consistency of multimodal outputs. The system can provide users with a personalized product display experience in open and noisy environments, possesses interactive data collection and offline operation capabilities, and can adapt to the display needs of various offline retail terminals.
[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an interactive product experience terminal system that integrates sound, light, and electricity, the system comprising: Multimodal acquisition module: used to collect ambient noise and ambient light parameters around the terminal, and to obtain facial images, voice signals and head spatial position information of users in the interaction area, providing accurate raw input data for subsequent processing by various modules; Emotion and Intent Determination Module: Used to extract facial and voice features, and through multimodal feature fusion calculation, identify user emotional tendencies and core interaction needs, generate corresponding experience adjustment strategies, and provide decision-making basis for personalized display output; Follow-up sound field control module: Based on head position information, it calculates and controls the output of a directional speaker array to create a follow-up, enveloping sound field through multi-point sound pressure superposition. It also dynamically adjusts the sound field parameters in combination with the ambient noise level to ensure that users can clearly receive audio information in noisy environments. The motion-sensing interaction mapping module is used to recognize motion-sensing gestures within a preset area. After completing the validity verification, it adjusts the demonstration parameters according to the mapping relationship between the gestures and product functions, and synchronously generates sound and light linkage control signals to realize intuitive operation of product functions without contact. Sound, light, electricity and electricity linkage output module: It is used to adjust the visual, lighting and sound field output synchronously according to the experience adjustment strategy and combined with the ambient light parameters. Before the output, multi-modal time delay calibration is performed to ensure the timing consistency of multi-sensory feedback.
[0006] Furthermore, the emotion and intent determination module performs multimodal feature fusion calculation using a weighted fusion formula, specifically: In the formula, For the total score of emotional integration, These are preset weighting coefficients for facial expression features, which were obtained through training and calibration using thousands of interaction samples in retail scenarios. The score for facial expression feature recognition. The preset weighting coefficients for speech features are set to a higher proportion because speech contains explicit semantic information. This represents the individual score for speech feature recognition. The emotion and intention determination module is pre-set with score ranges for three types of emotion tendencies. The range thresholds are statistically divided based on the emotion labels of the sample annotations. The module outputs the classification results according to the range in which the total score is located, and at the same time, it combines voice semantics and gesture-triggered actions to determine the category of core interaction requirements.
[0007] Furthermore, the follow-up sound field control module performs multi-point sound pressure superposition calculations using a synthetic sound pressure calculation formula, specifically: In the formula, The synthesized sound pressure level at the user's head position. This represents the total number of directional sound-emitting units. For the first The amplitude gain coefficient of the directional sound unit. For the first The initial output sound pressure of the directional sound unit. For the first The phase difference between the output sound wave of the directional sound unit and the reference sound wave; When the user's head movement distance exceeds the preset displacement threshold, which is determined based on the measured amplitude of the natural slight shaking of the human head, the follow-up sound field control module completes the update calculation of the amplitude gain and phase difference of each unit within a preset time. The preset time matches the physiological threshold of human hearing retention, and controls the synthetic sound pressure fluctuation of the head position within the preset range.
[0008] Furthermore, the somatosensory interaction mapping module has pre-stored gesture mapping relationships configured according to product categories: swiping gestures correspond to continuous parameter adjustment, pinching gestures correspond to function mode switching, and pointing gestures correspond to single function point triggering. The mapping relationships can be updated through backend configuration to adapt to the product display needs of different categories. When the motion-sensing interaction mapping module performs action validity verification, it extracts the trajectory continuity and dwell time parameters of the gesture. Actions with a dwell time exceeding a preset duration threshold and trajectory smoothness meeting preset requirements are judged as valid interaction actions, excluding erroneous actions that are not intended for interaction, thereby improving the stability of terminal interaction.
[0009] Furthermore, the audio-visual linkage output module has built-in linkage execution rules corresponding to the three types of emotional tendencies: When the product is deemed to be of high interest, the display of the product's core selling points is increased, the lighting is switched to a product area focus mode, the sound field is enhanced to increase the frequency of the narration, and the display brightness and light intensity are adjusted in conjunction with real-time ambient light parameters to avoid visual discomfort. When the user is hesitant, the visual display switches to a comparison interface of parameters of multiple products in the same category. The lighting uses independent lighting in different areas to correspond to different products, and the sound field outputs the parameter comparison explanations of each product in sequence to help the user complete the selection decision. When low attention is detected, the visual display runs a dynamic product display sequence, the lighting starts a gradual flow effect, and the sound field outputs preset background audio to attract the user's attention and guide the user to initiate active interaction.
[0010] Furthermore, the acoustic-optical-electric linkage output module performs multimodal output delay calibration using a delay difference calibration formula, specifically: In the formula, For the first Trigger adjustment time for each output module To achieve the preset target time point for multimodal synchronous presentation, For the first The inherent transmission delay of each output module is calibrated and stored in local storage for each terminal before it leaves the factory. Each output module starts outputting in advance according to its corresponding trigger adjustment time, so that the presentation time difference of multimodal content is controlled within a preset threshold, avoiding sensory misalignment that would disrupt the immersive experience.
[0011] Furthermore, the audio-visual linkage output module also has an interactive data acquisition function, which synchronously records the total interaction time, gesture operation frequency, emotional change time points, and product function items that the user focuses on during the user interaction process. After being summarized at a preset period, the data is uploaded to the backend management terminal to provide data reference for optimizing the display strategy.
[0012] Furthermore, when the multimodal acquisition module detects that multiple users are simultaneously in the preset interaction area, the user closest to the terminal is taken as the main interaction object. The gestures of the main interaction object can adjust the core parameters of the product demonstration; the gestures of other users only trigger the preset collaborative display effect without changing the core parameters of the product demonstration, thus avoiding logical confusion caused by multiple people operating at the same time.
[0013] Furthermore, the system supports offline operation mode; in offline mode, it retains the functions of motion interaction recognition, follow-up sound field output and basic sound and light linkage. The data related to emotion recognition in-depth analysis and experience data upload is temporarily stored locally. The local storage can hold no less than 7 days of interaction data. After the network is restored, it will be automatically completed and synchronized to the backend management terminal.
[0014] Compared with existing technologies, this interactive product experience terminal system that integrates sound, light, and electricity has the following advantages: I. This invention identifies users' emotional tendencies and interaction needs through multimodal feature fusion calculation. Combined with head-position-following package sound field control and environmental parameter adaptive adjustment of sound, light and electricity output strategies, it can output matching display content for users in different states in open retail scenarios. It ensures that users can clearly receive product information in noisy environments and obtain a suitable experience without manual operation by the user. It effectively extends the user's terminal dwell time and improves the efficiency of product information transmission.
[0015] Second, this invention establishes a mapping relationship between somatosensory gestures and product function parameters, and combines it with a multimodal output delay calibration mechanism to deeply integrate the somatosensory interaction process with product function demonstration. Users can intuitively operate various functions of the product through natural gestures, while ensuring the synchronization of visual, light, and sound multi-sensory feedback. This not only reduces the user's interaction learning cost, but also allows the user to more intuitively understand the product characteristics. The experience data collected during the interaction process can also provide practical reference for subsequent adjustment of display strategies.
[0016] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0018] Figure 1 This is a flowchart illustrating the overall module interaction of the product experience terminal interactive system that integrates sound, light, electricity, and electricity, according to the present invention. Figure 2 This is a flowchart illustrating the execution steps of the emotion and intention determination process in this invention. Figure 3 This is a flowchart of the computational processing of the follow-up sound field control stage of the present invention. Detailed Implementation
[0019] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.
[0020] This embodiment provides an interactive product experience terminal system that integrates sound, light, and electricity, designed for offline retail product display scenarios. The system comprises a multimodal acquisition module, an emotion and intention determination module, a follow-up sound field control module, a motion-sensing interaction mapping module, and a sound, light, and electricity linkage output module that operate collaboratively. For example... Figure 1 As shown, the multimodal acquisition module serves as the system's input, transmitting collected environmental and user data to each processing module. After completing their respective calculations, each processing module aggregates the control signals to the audio-visual linkage output module, ultimately outputting a synchronized multimodal display effect. The system collects environmental parameters and multi-dimensional user information to determine user emotional tendencies and interaction needs in real time. It then synchronously outputs a directional immersive sound field that follows the user's position. Combined with a motion-sensory interaction method that maps product functions, it adaptively adjusts the visual, lighting, and sound display effects while ensuring the timing synchronization of multimodal output, achieving a personalized product experience in open scenarios.
[0021] After powering on, the terminal first performs a system self-test, loads locally stored configuration parameters, and completes the reset and calibration of each module before entering a low-power standby state. The multimodal acquisition module, serving as the system's input unit, is divided into environmental perception and user perception components. In standby mode, only the millimeter-wave presence detection unit is activated, scanning the area in front of the terminal with low power. Once a valid human target is detected entering a preset range, the visual and audio acquisition components are activated and enter full-function operation. The environmental perception section uses four array-type microphones to collect ambient noise at a sampling frequency of 16kHz, a standard commonly used in speech recognition that balances recognition accuracy and local computational load. The module performs root mean square calculations on the collected audio data every second, outputting the real-time noise average in dB. Simultaneously, photosensitive elements are placed at the top and bottom of the terminal's front to collect two ambient light values, averaging them as an ambient light reference value in lux, used for subsequent adjustments to display brightness and light intensity. The user perception component employs a depth vision module in conjunction with millimeter-wave radar to achieve head spatial positioning. A three-dimensional Cartesian coordinate system is established with the center of the bottom of the terminal as the origin. The head center coordinates are calculated by fusing the skeletal key points of the depth image with the distance information from the radar, achieving positioning accuracy at the centimeter level. Voice signals are also picked up using an array of microphones, and beamforming algorithms are used to enhance the voice signal in the user's direction, suppress ambient noise, and improve the accuracy of subsequent speech recognition. The preset interaction area is a rectangular space directly in front of the terminal. Boundary parameters can be adjusted via a configuration file according to the installation site conditions. The system only generates interactive responses for users within this area.
[0022] After receiving the collected facial images and voice signals, the emotion and intention determination module completes user state recognition according to a preset process, such as... Figure 2 As shown, feature extraction and individual score calculation are performed on the two types of signals separately. Facial image feature extraction focuses on the movement unit changes of the eyebrows, eyes, and mouth. Based on a pre-trained expression recognition model, a facial expression score in the range of 0 to 1 is output, with higher scores indicating greater user interest in the product. For the speech signal, three types of features are extracted simultaneously: speech rate, intonation, and semantics. Speech rate and intonation features are obtained through audio time-domain and frequency-domain analysis, while semantic features are converted into text by a local speech recognition model and then matched with keywords. After fusing the three types of features, an individual speech feature score in the range of 0 to 1 is output, with higher scores indicating a stronger user willingness to interact.
[0023] After calculating the individual scores for the two categories, the module uses a weighted fusion formula to calculate the total emotion fusion score, which is expressed as follows: In the formula, For the total score of emotional integration, Preset weighting coefficients for facial expression features. The score for facial expression feature recognition. Preset weighting coefficients for speech features. This represents the individual score for speech feature recognition.
[0024] In this embodiment, the facial expression feature weight coefficient is set to 0.45, and the voice feature weight coefficient is set to 0.55. Both weight coefficients were obtained through training and calibration using thousands of interaction samples in a retail scenario. Because voice signals contain clear semantic information, they have higher reference value for the user's true intentions, hence a higher weight is assigned. In practical applications, the weight ratio can be adjusted according to the noise level of the scene and the characteristics of the user group. The module pre-sets three score ranges for different emotional tendencies: a total score greater than 0.7 indicates high interest, a score between 0.4 and 0.7 indicates hesitation, and a score less than 0.4 indicates low attention. The range thresholds are also based on statistical division of emotion labels from the sample annotations, covering the emotional expression characteristics of most users. The module outputs the corresponding emotional tendency classification result based on the range of the total score. Simultaneously, it combines keywords extracted from the voice semantics with gesture-triggered action recognition of core interaction needs. When keywords such as "parameter" or "function" appear in the voice, it is determined to be a parameter query need; when keywords such as "comparison" or "difference" appear, it is determined to be a product comparison need; and when accompanied by clear function operation gestures, it is determined to be a function demonstration need. Finally, an experience adjustment strategy incorporating emotional tendencies and interaction needs is generated and synchronously transmitted to the audio-visual linkage output module.
[0025] After receiving the head's spatial position information, the follow-up sound field control module controls the directional speaker array to output an enveloping sound field, such as... Figure 3 As shown, the sound field center point moves synchronously with the user's head position. This embodiment uses four sets of directional sound units, arranged along the four corners of the front of the terminal. The output amplitude and output phase of each set of units can be independently adjusted. Based on the head center coordinates, the module calculates the straight-line propagation distance from each set of sound units to the user's ears, calculates the output phase difference of each set of units based on the propagation distance difference, and then calculates the corresponding amplitude gain coefficient based on the distance attenuation law. Finally, the synthesized sound pressure value at the user's head position is obtained through the synthesized sound pressure calculation formula, which is expressed as: In the formula, The synthesized sound pressure level at the user's head position. This represents the total number of directional sound-emitting units. For the first The amplitude gain coefficient of the directional sound unit. For the first The initial output sound pressure of the directional sound unit. For the first The phase difference between the output sound wave of the directional sound unit and the reference sound wave.
[0026] The module adjusts the amplitude gain and phase difference of each unit in real time to keep the synthesized sound pressure at the user's head position within a preset comfortable range. When the module detects that the user's head movement exceeds a preset displacement threshold, it completes parameter update calculations for all sound-emitting units within a preset time period, controlling the synthesized sound pressure fluctuation at the head position within the preset range. In this embodiment, the preset displacement threshold is 3cm, which is determined based on actual measurements of the amplitude of natural slight head movements, avoiding sound field fluctuations caused by frequent adjustments. The preset parameter update time is 200ms, which is determined based on the physiological threshold of auditory persistence. Switching within 200ms does not result in noticeable changes perceived by the human ear, maintaining auditory continuity. The module also receives the average ambient noise parameter and uses dynamic gain adjustment logic. For every 10dB increase in ambient noise, the gain in the mid-frequency band of the sound field is increased by 3dB. This gain ratio is based on a speech intelligibility compensation curve combined with actual measurements in retail scenarios, which can enhance the clarity of human narration without increasing the overall volume, ensuring information intelligibility in noisy environments.
[0027] The motion-sensing interaction mapping module collects hand movement information through depth vision components and identifies user gestures within a preset interaction area. The module completes gesture classification and recognition by tracking the movement trajectory of 21 key skeletal points in the hand. It internally stores gesture mapping relationships configured according to product categories: swiping gestures correspond to continuous parameter adjustments, pinch gestures correspond to function mode switching, and pointing gestures correspond to single function triggers. Different product categories can be configured with different mapping logics. For example, audio products map left and right swipes to volume adjustment, while wearable products map up and down swipes to mode switching. The mapping relationships are imported through a backend configuration file and automatically loaded when the terminal starts, supporting remote updates to adapt to new products.
[0028] To eliminate false triggers due to non-interactive intent, the module incorporates action validity verification logic, simultaneously extracting two parameters during gesture recognition: trajectory continuity and dwell time. Trajectory continuity is determined by calculating the variance of adjacent offsets of trajectory points over 10 consecutive frames; if the variance is less than a preset value, the trajectory is considered smooth. Dwell time is counted from the moment the gesture becomes stable; if it exceeds a preset threshold, it is considered a valid interactive action. In this embodiment, the preset threshold is 0.3 seconds, a value based on statistics of the dwell time of natural, unintentional hand movements; actions shorter than this duration are considered false triggers. After confirming a valid gesture, the module adjusts the corresponding parameters of the product demonstration according to the mapping relationship, and simultaneously generates an audio-visual linkage control signal, which is transmitted to the audio-visual linkage output module to drive the matching feedback effect of the light and sound field output, achieving synchronous response between operation and feedback.
[0029] After receiving the experience adjustment strategy, the audio-visual-electric linkage output module, combined with real-time ambient light parameters, synchronously adjusts the visual display, lighting, and sound field outputs. The module has built-in linkage execution rules corresponding to the three emotional tendencies. When high interest is detected, the screen automatically jumps to the product's core parameter details page, enlarging the display area of the core parameter text, narrowing the lighting range to concentrate on illuminating the physical product area, reducing the background sound effects to highlight the human voice explanation, and simultaneously adjusting the screen brightness and light intensity according to the real-time ambient light values to avoid visual discomfort from excessive brightness or darkness. When hesitant, the screen automatically brings up a parameter comparison interface for two products in the same price range, displaying the differences in core parameters in left and right columns. The lighting is split into two independent outputs, corresponding to the display positions of the two products respectively, and the sound field explains the parameter differences of the two products in sequence to assist the user in making a selection decision. When low attention is detected, the screen loops a dynamic demonstration video of the product's appearance and core functions, the lighting performs a slow color gradient flow, and the sound field plays soft background audio, attracting the user's attention through dynamic changes and guiding them to initiate active interaction.
[0030] To avoid timing misalignment in multimodal outputs, the module performs multimodal delay calibration before each output adjustment. The trigger timing of each output unit is calculated using a delay difference calibration formula, expressed as follows: In the formula, For the first Trigger adjustment time for each output module To achieve the preset target time point for multimodal synchronous presentation, For the first The inherent transmission delay of each output module.
[0031] The inherent transmission latency of each output module is calibrated for each terminal before shipment. The calibration method involves sending a trigger signal to the module, detecting the response time at the output end using a high-speed acquisition device, recording this as the inherent latency of the corresponding module, and storing it locally. The latency of the visual display module includes image rendering time and screen response time; the latency of the lighting module includes signal transmission time and LED response time; and the latency of the sound field module includes audio decoding time and sound wave propagation time. The module calculates the trigger adjustment time for each output unit according to a formula, and triggers the corresponding module in advance according to the time difference, ensuring that the presentation time difference of the three types of outputs—visual, lighting, and sound—is controlled within a preset threshold. In this embodiment, the preset threshold is 15ms. This value is determined based on the human multi-sensory synchronous perception threshold; below this value, the human eye and ear cannot distinguish the order of outputs, ensuring the consistency of the multi-sensory experience.
[0032] The interactive data acquisition function is integrated into the audio-visual linkage output module. During user interaction, the module synchronously records data such as the start and end time of the interaction, the type and frequency of triggered gestures, the time points of changes in emotional state, and the product functions viewed by the user. The data is packaged and stored daily and automatically uploaded to the backend management terminal during off-peak hours. The backend can use the collected data to statistically analyze indicators such as user attractiveness and function attention of the terminal, providing data reference for adjusting display strategies and optimizing the ranking of product selling points.
[0033] When the multimodal acquisition module detects multiple users simultaneously within the preset interaction area, the system refreshes the distance between each user and the terminal every 0.5 seconds. The user closest to the device becomes the primary interaction target. The gestures of the primary user can adjust the core parameters of the product demonstration, while the gestures of other users only trigger preset collaborative display effects without changing the core parameters of the product demonstration, thus avoiding logical confusion caused by multiple users operating simultaneously. If the primary user leaves the interaction area for more than 2 seconds, the system automatically switches the next closest user to become the primary user, ensuring consistent interaction logic.
[0034] The system supports offline operation mode, with all core calculations performed locally on the processor, eliminating the need for a network. In offline mode, it retains all functions including motion-sensing interaction recognition, dynamic sound field output, and basic audio-visual linkage. In-depth analysis of emotion recognition and related experience data uploads are temporarily stored in the local flash memory. Local storage can hold at least 7 days of interaction data; this capacity is determined based on the regular store maintenance and inspection cycle, ensuring no data loss during network outages. Once the network connection is restored, the system automatically detects the connection status and resumes the data transmission from the local storage to the backend management terminal. Network outages will not affect the terminal's basic user experience.
[0035] The system described in this embodiment forms a complete interactive experience loop through the collaboration of multiple modules. It optimizes the interaction logic and sound field effects for the open environment and short user dwell time of offline retail terminals. It can be adapted to offline retail display terminals of various categories such as consumer electronics, daily chemicals, dairy products, and fast-moving consumer goods. It can be put into use by adjusting the gesture mapping rules and display content according to the characteristics of the corresponding products.
[0036] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A product experience terminal interactive system that integrates sound, light, electricity, and electricity, characterized in that, The system includes: Multimodal acquisition module: used to collect ambient noise values and ambient light parameters around the terminal, and to acquire facial images, voice signals and head spatial position information of users in the interaction area; Emotion and Intent Determination Module: Used to extract facial and voice features, and through multimodal feature fusion calculation, identify user emotional tendencies and core interaction needs, and generate corresponding experience adjustment strategies; Follow-up sound field control module: It is used to control the output of a follow-up immersive sound field of the directional speaker array based on the head position information and multi-point sound pressure superposition calculation, and dynamically adjust the sound field parameters in combination with the ambient noise value; The motion-sensing interaction mapping module is used to recognize motion-sensing gestures within a preset area. After completing the validity verification, it adjusts the demonstration parameters according to the mapping relationship between the gestures and product functions, and synchronously generates sound and light linkage control signals. Sound, light, electricity and linkage output module: It is used to adjust the visual, lighting and sound field output synchronously according to the experience adjustment strategy and combined with the ambient light parameters, and perform multimodal time delay calibration before output.
2. The product experience terminal interactive system with sound, light, and electricity linkage according to claim 1, characterized in that, The emotion and intent determination module performs multimodal feature fusion calculation using a weighted fusion formula, specifically: In the formula, For the total score of emotional integration, Preset weighting coefficients for facial expression features. The score for facial expression feature recognition. Preset weighting coefficients for speech features. This represents the individual score for speech feature recognition. The emotion and intention determination module is pre-set with score ranges for three types of emotion tendencies. It outputs classification results based on the range in which the total score falls, and combines voice semantics and gesture-triggered actions to determine the category of core interaction needs.
3. The product experience terminal interactive system with sound, light, and electricity linkage according to claim 1, characterized in that, The follow-up sound field control module performs multi-point sound pressure superposition calculations using a synthetic sound pressure calculation formula, specifically: In the formula, The synthesized sound pressure level at the user's head position. This represents the total number of directional sound-emitting units. For the first The amplitude gain coefficient of the directional sound unit. For the first The initial output sound pressure of the directional sound unit. For the first The phase difference between the output sound wave of the directional sound unit and the reference sound wave; When the user's head movement distance exceeds the preset displacement threshold, the follow-up sound field control module completes the update calculation of the amplitude gain and phase difference of each unit within a preset time period, and controls the synthetic sound pressure fluctuation of the head position within the preset range.
4. The product experience terminal interactive system with sound, light, and electricity linkage according to claim 1, characterized in that, The somatosensory interaction mapping module has pre-stored gesture mapping relationships configured according to product categories: swiping gestures correspond to continuous parameter adjustment, pinching gestures correspond to function mode switching, and pointing gestures correspond to single function point triggering. When the motion-sensing interaction mapping module performs action validity verification, it extracts the trajectory continuity and dwell time parameters of the gesture, and judges actions with dwell time exceeding a preset duration threshold and trajectory smoothness meeting preset requirements as valid interaction actions, excluding erroneous actions that are not intended for interaction.
5. The product experience terminal interactive system with sound, light, and electricity linkage according to claim 1, characterized in that, The audio-visual-electric linkage output module has built-in linkage execution rules corresponding to the three types of emotional tendencies: When the product is deemed to be of high interest, the display of the product's core selling points is enlarged, the lighting is switched to a product area focus mode, the sound field is enhanced to increase the frequency of the narration, and the display brightness and lighting intensity are adjusted in conjunction with real-time ambient light parameters. When the user is deemed hesitant, the visual display switches to a comparison interface of parameters of multiple products in the same category. The lighting uses independent lighting in different areas to correspond to different products, and the sound field outputs the parameter comparison explanations of each product in sequence. When low attention is detected, the visual display runs a dynamic product display sequence, the lighting activates a gradual flowing effect, and the sound field outputs preset background audio to attract the user's attention.
6. The product experience terminal interactive system with sound, light, and electricity linkage according to claim 1, characterized in that, The acoustic-optical-electric linkage output module performs multimodal output delay calibration using a delay difference calibration formula, specifically: In the formula, For the first Trigger adjustment time for each output module To achieve the preset target time point for multimodal synchronous presentation, For the first The inherent transmission delay of each output module; Each output module starts outputting in advance according to its corresponding trigger adjustment time, so that the presentation time difference of multimodal content is controlled within a preset threshold.
7. The product experience terminal interactive system with sound, light, and electricity linkage according to claim 1, characterized in that, The audio-visual-electric linkage output module also has an interactive data acquisition function, which synchronously records the total interaction time, frequency of gesture operations, time nodes of emotional changes, and data on product function items that the user focuses on during the user interaction process, and uploads the data to the backend management terminal after summarizing it according to a preset period.
8. The product experience terminal interactive system with sound, light, and electricity linkage according to claim 1, characterized in that, When the multimodal acquisition module detects that multiple users are simultaneously in the preset interaction area, the user closest to the terminal is taken as the main interaction object. The gestures of the main interaction object can adjust the core parameters of the product demonstration; the gestures of the other users only trigger the preset collaborative display effect and do not change the core parameters of the product demonstration.
9. The product experience terminal interactive system with sound, light, and electricity linkage according to claim 1, characterized in that, The system supports offline operation mode; in offline mode, it retains the functions of motion interaction recognition, follow-up sound field output and basic sound and light linkage. The data related to emotion recognition in-depth analysis and experience data upload is temporarily stored locally and will be automatically completed and synchronized to the backend management terminal after the network is restored.
