Digital exhibition hall multimedia equipment interaction control method and system in multi-mode

Abstract

The invention relates to the technical field of digital exhibition hall intelligent control, and discloses a multi-mode digital exhibition hall multimedia equipment interaction control method and system. The method comprises the following steps: collecting and cutting a continuous behavior data stream of a user, and analyzing spatial-temporal characteristics of behavior fragments to establish dynamic mapping between behavior intentions and equipment responses; constructing a semantic deconstruction model, and decomposing the complex interaction scene into atomic interaction units which can be independently processed; generating an interaction quality evaluation index and dynamically correcting the processing priority of each atomic unit by analyzing closed-loop association of equipment response and user feedback; and finally, fusing the corrected unit and the real-time environment state to generate an equipment operation instruction sequence. According to the method, refined analysis and self-adaptive intelligent control of concurrent and complex interaction scenes are realized, and the system response accuracy and the natural fluency of multi-user interaction experience are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for digital exhibition halls, specifically to a multi-mode interactive control method and system for multimedia equipment in digital exhibition halls. Background Technology

[0002] In current digital exhibition halls, the interactive control of multimedia devices largely relies on preset, discrete triggering rules. These systems typically use a single sensor signal, such as infrared sensing, camera gesture recognition, or touch events, to directly drive the device's response. Their control logic is linear, statically binding a specific user action to a fixed device operation. This approach treats the entire interactive scenario as a black box, focusing only on isolated input and output signals.

[0003] Existing technical solutions struggle to handle the complex interactive scenarios involving multiple users and concurrent behaviors common in exhibition environments. When multiple users interact with different devices simultaneously, or a single user generates a series of complex behaviors, systems based on fixed rules are prone to response conflicts, false triggers, or response delays. Their fundamental flaw lies in the lack of a holistic semantic understanding of the interactive scenario, failing to decompose complex on-site situations into computationally achievable parallel processing units. Furthermore, static rules cannot self-adjust based on the effectiveness of historical interactions; once set, the system's response strategy remains fixed, resulting in a rigid interactive experience that fails to adapt to the behavioral habits of different user groups.

[0004] Digital exhibition halls urgently need an interactive control method capable of understanding scene semantics and adaptively optimizing. This method needs to break through the rigid "event-response" model and solve two key problems: how to dynamically parse the inherent relationships between concurrent interactive behaviors, and how to continuously optimize control strategies based on real-time feedback. Existing technical solutions have significant shortcomings in terms of the depth of scene understanding and the intelligence of system behavior. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-mode interactive control method and system for multimedia equipment in digital exhibition halls to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides a multi-mode interactive control method for multimedia equipment in digital exhibition halls, the method comprising: Collect continuous behavioral data streams of users within the exhibition area, and extract interactive behavior segments corresponding to preset interactive nodes from the data stream; Perform multi-granularity spatiotemporal feature analysis on each interactive behavior segment to establish a dynamic mapping relationship between behavioral intent and device response; Based on environmental status parameters and device operation logs, a semantic deconstruction model of the interaction scenario is constructed, decomposing complex interaction scenarios into atomic interaction units that can be processed independently. By analyzing the closed-loop correlation between device response and user behavior feedback, interaction quality evaluation indicators are generated, and the processing priority of the atomic interaction units is dynamically adjusted accordingly. By integrating the corrected atomic interaction units with the real-time environment state, a specific sequence of operation instructions is generated to drive the target multimedia device to execute.

[0007] Preferably, the collection and segmentation of interactive behavior fragments includes the following: The system continuously captures the user's location coordinates through a spatial positioning system and synchronously records the timing data of the user's actions. Mark the moment when the device is triggered to start or stop in the data stream as an interaction node; Extend the interaction behavior segment forward and backward for a preset time period centered on the interaction node, and capture the interaction behavior segment containing the complete interaction context; The interaction behavior segments are timestamped and sensor data is fused to form a standardized data package of interaction behavior segments.

[0008] Preferably, performing multi-granularity spatiotemporal feature parsing includes the following: The standardized interactive behavior fragment data packets are sliced ​​in the time domain to extract macroscopic motion trajectory features and microscopic operation posture features respectively; Construct a spatiotemporal relationship graph between user behavior and device response, where nodes represent key interactive actions and edges represent the transition probabilities between actions; Calculate the centrality measure of each node in the spatiotemporal correlation graph and identify the core behavioral patterns that have a critical impact on the interaction process; Cross-modal matching is performed between the feature vectors of the core behavioral patterns and the device state codes to establish a dynamic mapping table from behavioral intent to device response.

[0009] Preferably, the semantic deconstruction model for constructing interactive scenarios includes the following: Ambient light intensity, sound decibel value, and personnel density parameters are collected as environmental state parameters; Analyze the operating mode switching records and abnormal status reports in the equipment operation log; Graph convolutional networks are used to perform hierarchical semantic parsing of interactive scenarios, decomposing complex interactive scenarios into atomic interactive units; Semantic labels and execution weights are assigned to each atomic interaction unit to form a scene semantic deconstruction graph.

[0010] Preferably, the generated interaction quality evaluation metrics include the following: Monitor the status changes of the device after it executes commands, and record the user's subsequent behavioral feedback; The basic evaluation dimensions are: device response latency, operation success rate, and user dwell time. By analyzing the correlation coefficient between device response and user feedback through time-series pattern matching, and combining the basic evaluation dimensions and correlation coefficients, a comprehensive interaction quality evaluation index is generated.

[0011] Preferably, the processing priority of dynamically modified atomic interaction units includes the following: Establish a correlation function between interaction quality evaluation indicators and the execution weights of atomic interaction units; When the overall interaction quality assessment index falls below the preset baseline, the priority adjustment mechanism is activated. The priority scores of each atomic interaction unit are recalculated based on the association function, and the atomic interaction units in the scene semantic deconstruction graph are reordered according to the updated priority scores.

[0012] Preferably, the fused and corrected atomic interaction unit and the real-time environment state include the following: Continuously monitor real-time readings from environmental sensors to generate an environmental state vector; Select the set of atomic interaction units with the highest priority from the reordered scene semantic deconstruction graph; Perform tensor multiplication operation between the environment state vector and the feature representation of the atomic interaction unit; An enhanced representation of interactive units is generated by weighted fusion of environmental context information through an attention mechanism.

[0013] Preferably, the generated operation instruction sequence includes the following: Parse the device operation semantics in the enhanced interaction unit representation; Verify the operability constraints of the target device based on the device status database; Convert the operational semantics that meet the constraints into standard instructions supported by the device control protocol; The execution order of instructions is arranged according to the temporal dependencies of the interaction logic to generate the final operation instruction sequence.

[0014] Preferably, the constraints for verifying operability include the following: Query the current operating mode and resource usage of the target device in the device status database; Check whether the command parameters meet the equipment's safe operating threshold range; Verify whether there are device resource conflicts or timing discrepancies in the instruction sequence; The instruction will be included in the final operation instruction sequence only if all verification conditions are met.

[0015] Preferably, the present invention also includes a multi-mode digital exhibition hall multimedia equipment interactive control system, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements the steps of the multi-mode digital exhibition hall multimedia equipment interactive control method described above.

[0016] Compared with the prior art, the beneficial effects of the present invention are: By constructing a semantic deconstruction model of interactive scenarios, complex interactive scenarios are broken down into independently processable atomic interaction units. This technology enables the system to perceive and understand the environmental state, device state, and user behavior distribution within the exhibition area as a whole, and to structurally analyze the originally chaotic interactive information flow. The deconstruction of scene semantics enables refined identification and parallel processing of multimodal, multi-target concurrent interactive behaviors, effectively avoiding command conflicts and response blind spots. The system is no longer limited to responding to single events, but can comprehensively manage a dynamically changing interactive environment, improving the control accuracy and robustness in complex scenarios.

[0017] By analyzing the closed-loop correlation between device response and user behavior feedback, an interaction quality evaluation index is generated, and the processing priority of atomic interaction units is dynamically adjusted accordingly. This technology establishes a continuously optimizing feedback loop, enabling the system to automatically evaluate the actual effect of each interaction. Based on objective evaluation indicators, the system dynamically adjusts the response weights and order of different interaction units, giving the control strategy self-learning and adaptive capabilities. The interaction logic is no longer static and preset, but evolves continuously based on actual operating results, thereby continuously improving the smoothness and naturalness of the user interaction experience. The overall performance of the system continuously improves itself over time, exhibiting true intelligent characteristics. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the working principle of the multi-mode digital exhibition hall multimedia equipment interactive control method described in this invention. Figure 2 A flowchart for collecting and segmenting interactive behavior fragments; Figure 3 A flowchart for multi-granularity spatiotemporal feature analysis; Figure 4 Comparison chart of multi-granularity spatiotemporal feature analysis results; Figure 5 Comparison chart of weight dynamic optimization for atomic interaction units. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 This invention provides a method and system for interactive control of multimedia devices in digital exhibition halls under multiple modes. The method includes: collecting continuous behavioral data streams of users within the exhibition area and segmenting interactive behavior fragments corresponding to preset interactive nodes from the data streams; performing multi-granularity spatiotemporal feature analysis on each interactive behavior fragment to establish a dynamic mapping relationship between behavioral intent and device response; constructing a semantic deconstruction model of the interactive scenario based on environmental state parameters and device operation logs, decomposing complex interactive scenarios into independently processable atomic interactive units; generating interaction quality evaluation indicators through closed-loop correlation analysis of device response and user behavior feedback, and dynamically correcting the processing priority of atomic interactive units accordingly; and fusing the corrected atomic interactive units with real-time environmental states to generate a sequence of specific operation instructions that drive the target multimedia device to execute.

[0021] Example 1: See Figure 2 In specific implementation, the spatial positioning system employs an ultra-wideband base station array deployed on the ceiling of the exhibition area to continuously capture the coordinates of positioning tags attached to users' shoes and hats. The sampling frequency is set to 100 Hz, and the system simultaneously records the timing data of user arm waving and finger tapping actions acquired by inertial measurement unit (IMU) sensors. The moment when the device is triggered to start or stop is marked in the data stream as an interaction node. The interaction node identification logic relies on the level signal transitions fed back from the multimedia device control terminal. For example, when the projection device switches from standby to playback mode, a high-level pulse is generated, and the rising edge of the pulse is marked as the start time of the interaction node. In some embodiments, the spatial positioning system can integrate infrared camera skeleton tracking data with wireless signal strength indicator fingerprint positioning to improve coordinate accuracy. Optionally, a nine-axis IMU sensor is introduced in the operation action timing data recording stage to collect raw readings of acceleration, angular velocity, and geomagnetic field vector.

[0022] The system extracts interactive behavior segments containing the complete interactive context by extending the interaction node forward and backward for a preset duration. The preset duration is dynamically adjusted based on historical interaction data analysis. The default forward extension duration is 2 seconds to cover the user's preparatory actions when approaching the device, and the default backward extension duration is 3 seconds to include the user's feedback behavior after the device responds. The extraction operation is implemented using a sliding time window algorithm. The window width is determined by the sum of the forward and backward extension durations, and the center of the window is aligned with the timestamp of the interaction node. It is understood that the extraction of interactive behavior segments must ensure data continuity to avoid segment breaks due to sensor packet loss. In some embodiments, the preset duration adjustment mechanism introduces a scene complexity factor, automatically extending the extension duration to 5 seconds when the personnel density exceeds a threshold.

[0023] The interactive behavior segments undergo timestamp alignment and sensor data fusion. Timestamp alignment uses a network time protocol to synchronize the clocks of all sensors, and deviation compensation is corrected using an interpolation algorithm. Sensor data fusion uses a Kalman filter to spatially register the coordinates of the spatial positioning system with the attitude data of the inertial measurement unit sensors. Standardization processing includes coordinate system 1 transformation to the exhibition area's world coordinate system and normalization of motion data to the [-1,1] interval. Finally, it is encapsulated into an interactive behavior segment data packet containing a timestamp sequence, coordinate sequence, and motion identifier. In specific implementation, the data packet structure definition adopts the JavaScript object representation format, including a header metadata field recording the sensor type and sampling rate, and a main data field storing the aligned multimodal time series array. Optionally, data integrity verification is implemented using a cyclic redundancy check code, discarding data segments that fail verification.

[0024] Example 2: See Figure 3In the specific implementation, the time-domain slicing of standardized interactive behavior fragment data packets is achieved using a fixed-length sliding window with a window width of 500 milliseconds to capture the phased features of the action. Macroscopic action trajectory features are extracted from the position coordinate sequence within the window, including the mean of the motion speed, the variance of the change in the motion direction, and the radius of curvature of the trajectory. Microscopic operation posture features are extracted from the sensor data of the inertial measurement unit, involving the spectral energy of the wrist joint angular velocity and the zero-crossing rate of the finger acceleration. In specific implementations, the feature extraction module utilizes digital signal processing algorithms to calculate instantaneous velocity using first-order difference on the coordinate sequence and to obtain frequency domain characteristics from the attitude data using short-time Fourier transform. Specifically, when processing the coordinate sequence using digital signal processing algorithms, the feature extraction module employs discrete difference operations to calculate instantaneous velocity, i.e., calculating the difference between adjacent points in the position coordinate sequence under continuous timestamps and then dividing by the sampling time interval to obtain the time-series data of the user's movement speed. For attitude data, the module uses a short-time Fourier transform algorithm, first dividing the time-series data collected by the inertial measurement unit sensor into fixed-length overlapping time windows, applying the Hanning window function to reduce spectral leakage, and then performing a fast Fourier transform on the data within each window to extract frequency amplitude features as frequency domain characteristics for subsequent microscopic operation attitude analysis. In some embodiments, the time-domain slicing can employ an adaptive window mechanism, dynamically adjusting the window length to the range of 300 milliseconds to 800 milliseconds based on the action amplitude.

[0025] A spatiotemporal correlation graph of user behavior and device response is constructed and stored in a graph database as a directed graph structure. Nodes in the graph represent key interaction actions, which are identified from historical interaction behavior segments using the DBSCAN clustering algorithm. Examples of key interaction actions include "raising an arm for more than 2 seconds" or "staying 0.5 meters in front of the device." Edges in the graph represent transition probabilities between actions, which are calculated by statistically analyzing the transition frequency between action nodes in historical interaction sequences. It can be understood that the construction of the spatiotemporal correlation graph relies on training with a large amount of historical interaction data, and the edge weights of the graph are dynamically updated as new interaction behavior segments are processed. In the specific implementation, the graph structure uses an adjacency list storage method, with each node recording its outgoing edges and corresponding conditional probabilities.

[0026] The centrality metric of each node in the spatiotemporal correlation graph is calculated iteratively using the PageRank algorithm. The algorithm formula is as follows: in: Represents a node PageRank value, The damping coefficient is set to 0.85 by default. This represents the total number of nodes in the graph. Represents all pointer nodes The set of nodes, Represents a node The set of all nodes pointed to. Core behavioral patterns that have a critical impact on the interaction flow are identified by sorting them according to PageRank values, and the top 10% of nodes are selected as core behavioral patterns. Optionally, the centrality metric calculation can be replaced by an eigenvector centrality algorithm, which determines node importance based on the principal eigenvectors of the adjacency matrix; in some embodiments, the identification threshold for core behavioral patterns can be configured to be dynamically adjusted, fluctuating according to the complexity of the current interaction scenario.

[0027] Cross-modal matching is performed between the feature vectors of the core behavioral patterns and the device status codes. The feature vectors of the core behavioral patterns are 64-dimensional floating-point arrays containing time-domain and frequency-domain features, while the device status codes are 8-bit binary numbers representing the device's on / off state, mode, and load status. The matching process calculates the cosine similarity between the feature vectors and the device status codes in the embedding space, establishing a dynamic mapping table from behavioral intent to device response. In specific implementations, the dynamic mapping table is stored in an in-memory database as key-value pairs, where the key is the hash value of the feature vector, and the value is the device status code and the matching confidence score. In some embodiments, cross-modal matching can employ a deep neural network model to project the two types of heterogeneous data onto a unified semantic space for alignment.

[0028] See Figure 4 This chart, a grouped bar chart, visualizes the multi-granularity spatiotemporal feature analysis process for interactive behavior segments: the horizontal axis represents five feature types, and the vertical axis represents the feature normalization value; the two colors correspond to macroscopic action trajectory features and microscopic operation posture features, respectively. The chart clearly distinguishes the logic of extracting different granular features from temporal slices: macroscopic features cover the first three types, corresponding to the overall trajectory characteristics of user interaction; microscopic features cover the last two types, corresponding to the fine-grained posture characteristics of user operations. This multi-granularity analysis method captures both the overall behavioral trend of user interaction and refines the microscopic posture details of the operation, providing hierarchical feature support for constructing a spatiotemporal correlation graph and establishing a mapping between behavioral intent and device response. This reflects the design goal of the technical solution: multi-granularity analysis improves the accuracy of interactive intent recognition.

[0029] Example 3: In this implementation, environmental status parameters are collected through an environmental sensor network deployed at the four corners of the exhibition area. Illumination intensity is collected in lux by digital light sensors, sound decibels are collected by a condenser microphone array and the equivalent continuous sound level is calculated, and personnel density is estimated in real time using a top-mounted depth camera combined with a crowd counting algorithm. Equipment operation logs are analyzed by parsing status messages reported by the multimedia device controller. The mode switching record extracts the mode encoding field, and the abnormal status report parses error codes and timestamp information. In this implementation, the environmental status parameter sampling period is fixed at 1 second, and the equipment operation log uses a circular buffer to store the most recent 1000 records. In some embodiments, the personnel density parameter estimation can incorporate the number of terminals associated with the wireless access point as an auxiliary judgment criterion.

[0030] A hierarchical semantic parsing of interactive scenes is performed using graph convolutional networks (GCNNs). The input graph of the GCNN is a scene topology graph with interactive devices as nodes and spatial and logical relationships between devices as edges. Node feature vectors are fused with device type encoding and environmental state parameters. The GCNN contains three graph convolutional layers to aggregate neighborhood information, and finally outputs a hierarchical representation of the scene through graph pooling. Complex interactive scenes are decomposed into atomic interaction units, implemented through fully connected layers and a Softmax classifier. Each atomic interaction unit corresponds to a basic device control logic, such as "starting the projector and playing a specified video source" or "adjusting the spotlight angle." In the specific implementation, the GCNN uses an adjacency matrix and a node feature matrix as input, and the training data comes from historical scene annotation data. It can be understood that the granularity of the atomic interaction unit division depends on the scene complexity and the device control precision requirements.

[0031] Semantic labels and execution weights are assigned to each atomic interaction unit. Semantic labels use natural language to describe unit functions, such as "welcome mode initialization." Execution weights are initialized based on the frequency of the unit's occurrence in historical successful interactions, ranging from 0 to 1, forming a scene semantic deconstruction graph. The scene semantic deconstruction graph is stored in a graph structure, with nodes representing atomic interaction units and edges representing temporal dependencies or conditional triggering relationships between units. In specific implementations, semantic label assignment is based on a predefined interaction scene ontology library, and execution weights are dynamically updated using an exponential decay model. Optionally, the serialization format of the scene semantic deconstruction graph uses resource description framework triples.

[0032] Monitoring the status changes of the device after executing commands is achieved by polling the device status register. The status change data includes the device response code and execution completion timestamp. Recording subsequent user behavior feedback is achieved by continuously collecting user behavior data streams within the exhibition area, focusing on the user's movement trajectory and dwell time after device triggering. Device response latency is calculated as the time difference between sending a control command and receiving a successful response from the device. Operation success rate is calculated as the percentage of commands successfully executed within a preset time window. User dwell time is calculated as the duration the user remains within the device's effective sensing area after interacting with it. In specific implementations, device response latency is recorded in milliseconds, operation success rate is calculated as a percentage, and user dwell time is measured in seconds. In some embodiments, the calculation of user dwell time excludes cases where the user briefly passes by without any substantial interaction.

[0033] The correlation coefficient between device response and user feedback is analyzed by time-series pattern matching. The time-series pattern matching uses a dynamic time warping algorithm to align the device response time series and the user feedback time series. The correlation coefficient is calculated using the Pearson product-moment correlation coefficient formula. in: Represents the correlation coefficient. Indicates the length of the time series. The first element representing the device response delay sequence One value, This represents the mean of the device response delay sequence. Represents the first of the user dwell time sequence One value, This represents the mean of the user dwell time sequence. A weighted summation model is used to generate a comprehensive interaction quality assessment index, combining basic evaluation dimensions and correlation coefficients. The basic evaluation dimensions include device response latency, operation success rate, and user dwell time, with weight coefficients set based on expert experience. The comprehensive interaction quality assessment index is... The specific acquisition and calculation process is as follows: It is a quantitative result that integrates device response performance, user behavior feedback, and the correlation between the two. The value range is [0,100]. The higher the value, the better the interaction quality. It is used to determine whether the current interaction effect meets the preset requirements.

[0034] In the standardization process of basic assessment dimensions, the standardized value of equipment response delay is first determined. The maximum acceptable response latency for commonly used multimedia devices in the exhibition area is determined based on device hardware parameters and scene interaction requirements. For example, it references the official nominal response latency limits of mainstream projectors and lighting controllers, and incorporates actual test data from the devices deployed in the exhibition area. When the device response latency... hour, when hour, Other cases To ensure that the response latency is as short as possible, the standardization value should be as high as possible. Standardized operation success rate This is the ratio of the number of times the device successfully executes a command within a preset time window to the total number of attempts. The ratio is directly converted to a percentage value; the higher the success rate, the better. The higher. Determining the standardized value of user dwell time This reference dwell time for effective user interaction in the exhibition area is set based on statistical data of user interaction behavior in similar digital exhibition halls, combined with the average user dwell time measured during the system's development process. When the user dwell time... hour, ;when hour, Other cases The closer the dwell time is to or exceeds the reference value, the stronger the interactive appeal. The higher the correlation coefficient, the better. The correlation coefficient fusion process first aligns the device response time series and the user feedback time series using a dynamic time warping algorithm to obtain the linear correlation coefficient. (Value range [-1, 1]). To make Consistent with the range of standardized values ​​for the basic assessment dimensions, they are converted into standardized coefficients. At this time, the value of R is in the range of [0,100]. The larger the positive value of R, the stronger the positive correlation between the device response and the user feedback, and the higher the R value.

[0035] Weight allocation is determined based on scenario interaction requirements, including device response latency weight. Operation success rate weighting User dwell time weight Correlation coefficient weight The total weight is 1, and can be flexibly adjusted according to the type of exhibition area, such as technology or culture, based on user experience survey data. Final The value calculation logic is as follows: By integrating multi-dimensional indicators through this formula, a comprehensive interaction quality assessment result is obtained.

[0036] It is understandable that the correlation coefficient reflects the inherent connection between device performance and user experience; in specific implementation, the comprehensive interaction quality evaluation index is normalized to the range of 0 to 100 for easy comparison, and the index calculation cycle is configurable.

[0037] Example 4: In specific implementation, the correlation function between the interaction quality evaluation index and the execution weights of atomic interaction units is established using a multiple linear regression model. The interaction quality evaluation index is the dependent variable, and the execution weights of atomic interaction units are the independent variables. The coefficients of the correlation function are obtained through training on a historical interaction dataset. The historical interaction dataset contains thousands of labeled records, each recording the interaction quality evaluation index value and the corresponding atomic interaction unit execution weight value. In this example, the correlation function is... ( For the first Priority score of each atomic interaction unit To provide comprehensive interaction quality assessment indicators, For the first The execution weight of each atomic interaction unit), where the coefficient , , The acquisition process and basis are as follows: At least 1000 sets of historical interaction data were collected during data collection. The data sources were simulation tests and small-scale trial scenarios in pilot exhibition areas during the system development process. Each set of data contains 3 core elements: Value (Comprehensive Interaction Quality Assessment Results) (Initial execution weights of atomic interaction units), and manually labeled priority scores, which were independently scored by three professional testers based on the interaction effect, with the average score taken as the standard value, ranging from [0, 100]. Model training employed a linear regression algorithm, with the manually labeled priority scores as the dependent variable. Value and Using historical interaction datasets as independent variables, we train the model by minimizing the error between the predicted and manually labeled values ​​using the least squares method. The optimal solution.

[0038] The coefficients are set based on ensuring during the training process. ,Right now The value has a greater impact on the priority score than the initial execution weight. This is based on the design logic that interaction quality feedback takes precedence over initial weights, ensuring that the system prioritizes optimizing atomic interaction units corresponding to low interaction quality. In this embodiment, the value is obtained through training. , To correct constants and offset system errors, the priority scores are made to better reflect the needs of actual scenarios. The coefficients are stored in the system configuration file and can be dynamically updated based on subsequent actual running data.

[0039] Refer to Table 1 for the parameters of the correlation function.

[0040] Table 1: Correlation Function Coefficients Table When the overall interaction quality assessment index falls below the preset baseline, a priority adjustment mechanism is activated. The preset baseline is set as a fixed threshold of 70 points out of 100 points, based on the exhibition area operation requirements. After the priority adjustment mechanism is triggered, the system suspends the execution of new atomic interaction units and enters the priority recalculation process. In this embodiment, the preset baseline is denoted as... The setting method and basis are as follows: Fixed threshold setting: Default fixed threshold =70 (out of 100), set based on the interaction quality data of similar devices in 10 digital exhibition halls of similar scale. The industry average comprehensive interaction quality assessment value is 70. Combined with the design goals of this system, the interaction quality is better than the industry average. Therefore, a fixed baseline of 70 is set. When Q < If the current interaction quality is deemed unsatisfactory, priority adjustment is initiated. Dynamic threshold adjustment supports dynamic adjustment of the baseline based on real-time personnel density in the exhibition area. The adjustment factor is the current personnel density K (unit: people / square meter), where K_ref is the designed rated personnel density of the exhibition area, determined based on the exhibition area area, safety regulations, and equipment capacity. Dynamic baseline. = ×(1+min(0.3,K / K_ref-1)), when K≤K_ref, = When K > K_ref, Increase appropriately as population density increases, but not exceeding [a certain limit]. This is 1.3 times faster, to meet the higher interaction quality requirements in multi-user concurrent scenarios. When the threshold mechanism is active, the fixed threshold is the default configuration; the dynamic threshold needs to be manually enabled. Once enabled, the system updates every 5 minutes based on the latest personnel density data collected by environmental sensors. This ensures that the threshold matches the real-time scenario.

[0041] In practice, the preset baseline can be configured as a dynamic threshold and adjusted according to the flow of people in the exhibition area in different time periods. For example, the baseline for peak hours can be set to 60 minutes. In some embodiments, the triggering condition for the priority adjustment mechanism can be increased by judging the number of consecutive times below the baseline to avoid erroneous adjustments caused by instantaneous fluctuations.

[0042] The priority score of each atomic interaction unit is recalculated based on the correlation function. The priority score is calculated using the following formula: in: Indicates the first Priority score of each atomic interaction unit This indicates the current overall interaction quality assessment index value. Indicates the first The execution weight of each atomic interaction unit, , , These are the coefficient parameters obtained from training the association function. The atomic interaction units in the scene semantic deconstruction graph are reordered according to the updated priority scores. The sorting operation uses a quicksort algorithm to arrange the list of atomic interaction units in descending order, ensuring that high-priority units are at the front of the list. It can be understood that the recalculation of priority scores needs to guarantee the real-time performance of atomic interaction units, with computation latency controlled within milliseconds; in specific implementation, the coefficient parameters... , , Stored in a configuration file to support hot updates.

[0043] Continuous monitoring of real-time readings from environmental sensors generates an environmental state vector. These sensors include a light sensor, a noise sensor, and an infrared people counter. Real-time readings are sampled 10 times per second. The environmental state vector is constructed as a three-dimensional vector, with dimensions corresponding to light intensity, sound decibels, and people density, respectively. The set of atomic interaction units with the highest priority is selected from the reordered scene semantic deconstruction graph. The selection rule is to extract the top N atomic interaction units from the sorted list, where N is dynamically determined based on the number of available device resources, with a default setting of 5. In practice, the environmental state vector is normalized, scaling the values ​​of each dimension to the [0,1] range. Optionally, a diversity mechanism can be introduced in the selection of the atomic interaction unit set to avoid excessive concentration of similar units.

[0044] Tensor multiplication is performed between the environment state vector and the feature representation of the atomic interaction unit. The environment state vector is in row vector form, and the feature representation of the atomic interaction unit is in column vector form. The tensor multiplication operation uses an outer product operation to generate a two-dimensional interaction matrix, where the matrix elements reflect the coupling strength between environmental factors and unit features. An attention mechanism is used to weightedly fuse environmental context information. The attention mechanism calculates the attention score for each element in the interaction matrix, adjusting the score based on the current magnitude of the environment state vector. The weighted sum is then used to generate an enhanced interaction unit representation. In specific implementations, the attention score calculation uses the Softmax function to normalize the row vectors of the interaction matrix; in some embodiments, the tensor multiplication operation can be replaced with the Kronecker product to retain more interaction information.

[0045] See Figure 5This graph is a line graph comparing two different atomic interaction units, focusing on the core process of adjusting the weights of atomic interaction units based on interaction quality assessment metrics. The horizontal axis represents the five types of atomic interaction units, and the vertical axis represents the execution weight of the unit. The two lines correspond to the initial weight and the updated weight, respectively. The system establishes a correlation function between weights and assessment metrics through interaction quality assessment metrics, dynamically adjusting unit weights, increasing the execution priority of high-value interaction units, and decreasing the weight of low-priority units. This provides a basis for the reasonable sorting of subsequent instruction sequences, demonstrating the intelligent characteristics of the system's adaptive optimization interaction strategy.

[0046] Example 5: In a specific implementation, the device operation semantics in the enhanced interaction unit representation are parsed using a rule-based semantic parser. The enhanced interaction unit representation serves as the parser input. The parser performs recursive descent analysis based on a predefined device operation syntax rule tree, extracting atomic operation actions, the device identifier of the operation object, and operation parameter triples. For example, the parsing result might be: Start, Projector 001, Mode: HD or Adjust, Light Group A, Brightness: 80%. In a specific implementation, the semantic parser integrates a dictionary and a grammar analysis module. The dictionary stores a mapping table of device aliases and operation verbs, and the grammar analysis module processes the semantic tags of natural language descriptions. In some embodiments, the parsing process can employ a converter-based neural network model to improve the understanding of complex semantics.

[0047] The operability constraints of the target device are verified based on the device status database. This database uses a relational database to store real-time status snapshots of all networked multimedia devices, with snapshots updated 5 times per second. Operability constraint verification includes querying the target device's current operating mode and resource usage in the device status database. For example, verifying whether a projector is in standby mode rather than undergoing firmware upgrades, and checking whether command parameters meet the device's safe operating thresholds, such as verifying whether the light brightness parameter is within the device's allowed range of 0% to 100%. In practice, the device status database obtains real-time data through the application programming interface provided by the device driver layer.

[0048] The verification process checks for device resource conflicts or timing discrepancies within the instruction sequence. Resource conflict detection checks whether different instructions request exclusive resources from the same device; for example, two instructions simultaneously requesting control of the same projector's lens cover motor. Timing discrepancy detection analyzes the temporal logic between instructions, such as checking for illegal sequences requiring video playback after device shutdown. The conflict detection algorithm constructs an instruction-resource dependency graph and uses a depth-first search to traverse the graph for cycles. An instruction is included in the final operation instruction sequence only when all verification conditions are met. In practice, the verification process employs a transaction mechanism; if any condition fails verification, the entire instruction sequence is rolled back. Optionally, a priority preemption mechanism can be introduced for resource conflict detection, allowing high-priority instructions to queue and wait for resource release.

[0049] The operation semantics that meet the constraints are converted into standard instructions supported by the device control protocol. The conversion process establishes a mapping table based on the protocol document provided by the device manufacturer. In specific implementations, the instruction conversion module implements an encoder for each device protocol. In some embodiments, complex instruction conversion can be completed by calling the device-specific dynamic link library.

[0050] The execution order of instructions is arranged according to the temporal dependencies of the interaction logic. These dependencies are defined by directed edges between atomic interaction units in the scene semantic deconstruction graph. The instruction execution order is arranged using a topological sorting algorithm to process the dependency graph and generate a linear instruction sequence. For parallel instructions without dependencies, their execution order is optimized based on instruction execution latency prediction, with the goal of minimizing the total waiting time. In the specific implementation, the temporal dependency arrangement incorporates an instruction execution time prediction model, the formula of which is: in: This indicates the estimated total execution time. Indicates the number of instructions. Indicates the first The base execution time of an instruction is determined by measuring the minimum execution time of instructions corresponding to different atomic interaction units on the target device. Indicates the first The data load factor of each instruction is determined based on the ratio of the instruction data size to the standard data size, with the standard data size set to 1KB. This represents the network communication latency coefficient, determined by measuring the relationship between data transmission latency and data load under different network environments. The final operation instruction sequence is encapsulated in a structured data format, including metadata such as the instruction list, execution timestamp, and timeout retry policy. It is understood that optimized timing helps improve the smoothness of multi-device collaborative response; in practical implementation, the final operation instruction sequence is asynchronously sent to the device controller via a message queue. Optionally, the instruction sequence can be encoded in JSON or Protocol Buffers format to adapt to different transmission requirements.

[0051] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-mode interactive control method for multimedia equipment in a digital exhibition hall, characterized in that, The method includes: Collect continuous behavioral data streams of users within the exhibition area, and extract interactive behavior segments corresponding to preset interactive nodes from the data stream; Perform multi-granularity spatiotemporal feature analysis on each interactive behavior segment to establish a dynamic mapping relationship between behavioral intent and device response; Based on environmental status parameters and device operation logs, a semantic deconstruction model of the interaction scenario is constructed, decomposing complex interaction scenarios into atomic interaction units that can be processed independently. By analyzing the closed-loop correlation between device response and user behavior feedback, interaction quality evaluation indicators are generated, and the processing priority of the atomic interaction units is dynamically adjusted accordingly. By integrating the corrected atomic interaction units with the real-time environment state, a specific sequence of operation instructions is generated to drive the target multimedia device to execute.

2. The interactive control method for multimedia equipment in a digital exhibition hall under multiple modes according to claim 1, characterized in that, The collection and segmentation of interactive behavior fragments includes the following: The system continuously captures the user's location coordinates through a spatial positioning system and synchronously records the timing data of the user's actions. Mark the moment when the device is triggered to start or stop in the data stream as an interaction node; Extend the interaction behavior segment forward and backward for a preset time period centered on the interaction node, and capture the interaction behavior segment containing the complete interaction context; The interaction behavior segments are timestamped and sensor data is fused to form a standardized data package of interaction behavior segments.

3. The interactive control method for multimedia equipment in a digital exhibition hall under multiple modes according to claim 2, characterized in that, Performing multi-granularity spatiotemporal feature parsing includes the following: The standardized interactive behavior fragment data packets are sliced ​​in the time domain to extract macroscopic motion trajectory features and microscopic operation posture features respectively; Construct a spatiotemporal relationship graph between user behavior and device response, where nodes represent key interactive actions and edges represent the transition probabilities between actions; Calculate the centrality measure of each node in the spatiotemporal correlation graph and identify the core behavioral patterns that have a critical impact on the interaction process; Cross-modal matching is performed between the feature vectors of the core behavioral patterns and the device state codes to establish a dynamic mapping table from behavioral intent to device response.

4. The interactive control method for multimedia equipment in a digital exhibition hall under multiple modes according to claim 3, characterized in that, Constructing a semantic deconstruction model for interactive scenarios includes the following: Ambient light intensity, sound decibel value, and personnel density parameters are collected as environmental state parameters; Analyze the operating mode switching records and abnormal status reports in the equipment operation log; Graph convolutional networks are used to perform hierarchical semantic parsing of interactive scenarios, decomposing complex interactive scenarios into atomic interactive units; Semantic labels and execution weights are assigned to each atomic interaction unit to form a scene semantic deconstruction graph.

5. The interactive control method for multimedia equipment in a digital exhibition hall under multiple modes according to claim 4, characterized in that, The generated interaction quality assessment metrics include the following: Monitor the status changes of the device after it executes commands, and record the user's subsequent behavioral feedback; The basic evaluation dimensions are: device response latency, operation success rate, and user dwell time. By analyzing the correlation coefficient between device response and user feedback through time-series pattern matching, and combining the basic evaluation dimensions and correlation coefficients, a comprehensive interaction quality evaluation index is generated.

6. The interactive control method for multimedia equipment in a digital exhibition hall under multiple modes according to claim 5, characterized in that, The processing priority of dynamically adjusted atomic interaction units includes the following: Establish a correlation function between interaction quality evaluation indicators and the execution weights of atomic interaction units; When the overall interaction quality assessment index falls below the preset baseline, the priority adjustment mechanism is activated. The priority scores of each atomic interaction unit are recalculated based on the association function, and the atomic interaction units in the scene semantic deconstruction graph are reordered according to the updated priority scores.

7. The interactive control method for multimedia equipment in a digital exhibition hall under multiple modes according to claim 6, characterized in that, The fusion and correction of atomic interaction units and real-time environment states include the following: Continuously monitor real-time readings from environmental sensors to generate an environmental state vector; Select the set of atomic interaction units with the highest priority from the reordered scene semantic deconstruction graph; Perform tensor multiplication operation between the environment state vector and the feature representation of the atomic interaction unit; An enhanced representation of interactive units is generated by weighted fusion of environmental context information through an attention mechanism.

8. The interactive control method for multimedia equipment in a digital exhibition hall under multiple modes according to claim 7, characterized in that, The generated operation instruction sequence includes the following: Parse the device operation semantics in the enhanced interaction unit representation; Verify the operability constraints of the target device based on the device status database; Convert the operational semantics that meet the constraints into standard instructions supported by the device control protocol; The execution order of instructions is arranged according to the temporal dependencies of the interaction logic to generate the final operation instruction sequence.

9. The interactive control method for multimedia equipment in a digital exhibition hall under multiple modes according to claim 8, characterized in that, Verification of operability constraints includes the following: Query the current operating mode and resource usage of the target device in the device status database; Check whether the command parameters meet the equipment's safe operating threshold range; Verify whether there are device resource conflicts or timing discrepancies in the instruction sequence; The instruction will be included in the final operation instruction sequence only if all verification conditions are met.

10. A multi-mode digital exhibition hall multimedia equipment interactive control system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the interactive control method for multimedia equipment in a multi-mode digital exhibition hall as described in any one of claims 1 to 9.

Images