A real-time linkage control system for multimodal devices in a conference room

Abstract

This invention belongs to the field of automatic control technology and discloses a real-time linkage control system for multimodal devices in a conference room. It includes a multimodal device group, a control server, and an environmental sensor group. The system solves the problem of inter-device coordination conflicts in traditional conference systems by constructing a device execution conflict model. This model, based on a resource competition conflict matrix and action mutual exclusion conflict vectors, achieves accurate identification and prediction of device operation conflicts. Control parameters are optimized to give the system environmental adaptability. A closed-loop control loop enables coordinated device operation and dynamic parameter optimization, and closed-loop feedback adjustment can evaluate and correct linkage deviations in real time. During execution, the system achieves seamless collaboration among multiple devices through techniques such as calculating operating margins and adjusting time offsets, effectively improving the intelligence level of conference room environmental control.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic control, and more specifically, to a real-time linkage control system for multi-modal devices in a meeting room. Background Art

[0002] With the rapid development of intelligent buildings and office environments, modern meeting rooms have evolved from simple communication spaces into complex intelligent systems integrating functions such as audio-video display, voice interaction, environmental adjustment, and information processing.

[0003] However, the existing multi-modal device control systems in meeting rooms mainly lack effective closed-loop feedback control mechanisms and multi-device collaborative scheduling capabilities. In actual meeting scenarios, when a speaker starts to demonstrate video content, the traditional system cannot automatically coordinate the operating parameters of projection devices and lighting systems, resulting in a decrease in the clarity of the projected content due to excessive ambient light; at the same time, there is a lack of synchronous control between the audio system and the video system, causing audio-video asynchronization. The resource competition and action conflict problems between these devices are particularly prominent, such as the microphone array and the environmental noise suppression system competing for audio processing resources, resulting in system response delays; the control logics of intelligent lighting and air conditioning systems interfere with each other, causing the meeting environment to be alternately hot and cold and the brightness to change suddenly. In addition, existing control strategies generally adopt static preset parameters and cannot perform real-time adaptive adjustments according to dynamic changes in the environment during the meeting (such as changes in natural light, changes in personnel density, fluctuations in environmental noise, etc.), and the meeting experience highly depends on manual intervention. In multi-person video conferences, due to the lack of an accurate execution timing coordination mechanism, the device startup and switching processes are chaotic, resulting in interference such as signal interruption and video freezing; the system also lacks the ability to monitor and predict the operating status of devices and cannot identify potential performance bottlenecks and pre-adjust control parameters in advance, leading to sudden problems such as slow device response or function failure at critical moments of the meeting. These technical defects not only affect meeting efficiency but also hinder the application expansion of intelligent meeting systems in high-demand scenarios.

[0004] In view of this, the present invention proposes a real-time linkage control system for multi-modal devices in a meeting room to solve the above problems. Summary of the Invention

[0005] To overcome the above-mentioned defects of the prior art and to achieve the above object, the present invention provides the following technical solution: A real-time linkage control system for multi-modal devices in a meeting room, comprising:

[0006] A multi-modal device group, a control server, and an environmental sensor group, where the multi-modal device group includes at least two devices of different modalities, and the system realizes real-time linkage control through the following method:

[0007] Obtain the environmental status data and device operation data of the meeting room in each control cycle;

[0008] Based on the environmental status data and the preset meeting scenario requirements, an initial control parameter set for the multimodal device group is generated for each control cycle. The initial control parameter set includes the execution instructions and execution time of each device.

[0009] Based on the equipment operation data and the initial control parameter set, an execution conflict model for the multimodal equipment group is constructed. The execution conflict model detects whether there is an execution conflict in the multimodal equipment group through closed-loop feedback adjustment.

[0010] Based on the execution conflict model, the initial control parameter set is optimized by an adaptive adjustment algorithm to generate an optimized control parameter set. The adaptive adjustment algorithm includes dynamic weight allocation based on equipment operation data and disturbance compensation based on environmental state data.

[0011] Based on the feedback from the optimized control parameter set and the real-time collected equipment operation data, a closed-loop control loop is constructed to control the coordinated operation of the multimodal equipment group and dynamically update the initial control parameter set for the next control cycle.

[0012] The technical effects and advantages of the real-time linkage control system for multimodal equipment in a conference room according to the present invention are as follows:

[0013] This invention addresses the core pain point of multi-device collaborative control in traditional conference systems. In practical applications, when a speaker switches from presentation to video playback, the system seamlessly coordinates the operation of projectors, lights, and audio equipment, ensuring a smooth and natural transition that attendees perceive no change in technology. Common problems in traditional conferences, such as "audio-visual asynchrony," "brightness conflicts between lights and projectors," and "interference between environmental adjustments and the audio system," are comprehensively resolved, enhancing the professionalism and immersive experience of the conference. The system's intelligent perception and prediction capabilities regarding environmental changes make the conference process more manageable. Whether it's a sudden change in natural light or fluctuations in personnel density, adjustments can be made promptly without disrupting the conference proceedings. The system's proactive coordination and control capabilities ensure more consistent device operation, eliminating awkward situations caused by asynchronous device responses in traditional systems, such as the projector starting but the lights not dimming, or audio playing but the display not being ready. Simultaneously, the system's self-diagnosis and adaptive optimization capabilities significantly reduce maintenance difficulty and frequency, shifting from passive response to proactive prevention. With continuous operation, the system's understanding of various conference scenarios deepens, leading to increasingly precise control strategies. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a real-time linkage control system for multimodal equipment in a conference room according to the present invention;

[0015] Figure 2This is a schematic diagram illustrating the logic of real-time linkage control of multimodal devices in a conference room, as described in this invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 This invention provides a real-time linkage control system for multimodal devices in a conference room, comprising a multimodal device group, a control server, and an environmental sensor group. The multimodal device group includes devices with at least two different modes, such as a conference display screen, microphone array, intelligent lighting, and air conditioning equipment. The control server processes environmental status data and device operation data, generates control parameters, and executes linkage control. The environmental sensor group includes light sensors, sound sensors, and infrared sensors, etc., for collecting environmental status data of the conference room.

[0018] This invention achieves intelligent adjustment of the conference room environment through the collaborative control of multimodal devices. It constructs a device execution conflict model to provide a basis for subsequent optimization. Dynamic weight allocation based on device operating data and disturbance compensation based on environmental state data make the control adaptive. Closed-loop feedback adjustment can identify execution conflicts in different conference scenarios and dynamically adjust them. Trend prediction combined with preset conference scenario requirements improves the control's foresight. Optimized generation and distribution of control parameters ensure the system's real-time response capability. The resource competition conflict matrix and action mutual exclusion conflict vector solve the device collaboration conflict problem in traditional conference systems. Dynamic execution weight calculation and time offset adjustment ensure seamless collaboration between multimodal devices. The construction of the closed-loop control circuit realizes the dynamic optimization of linkage operation parameters.

[0019] Please see Figure 2 This is a logical diagram illustrating the real-time linkage control of multimodal devices in a conference room, as described in this application. In this embodiment of the invention, the system achieves real-time linkage control through the following steps:

[0020] Step 1: Acquire the environmental status data and equipment operation data of the conference room in each control cycle. The environmental status data is collected by the environmental sensor group, and the equipment operation data is generated by each device in the multimodal device group.

[0021] Step 2: Based on the environmental status data and the preset meeting scenario requirements, generate the initial control parameter set for the multimodal device group in each control cycle. The initial control parameter set includes the execution instructions and execution time of each device.

[0022] Step 3: Based on the equipment operation data and initial control parameter set, construct an execution conflict model for the multimodal equipment group. The execution conflict model uses closed-loop feedback adjustment to detect whether there is an execution conflict in the multimodal equipment group.

[0023] Step 4: Based on the execution conflict model, optimize the initial control parameter set through an adaptive adjustment algorithm to generate an optimized control parameter set. The adaptive adjustment algorithm includes dynamic weight allocation based on equipment operation data and disturbance compensation based on environmental state data.

[0024] Step 5: Based on the feedback from the optimized control parameter set and the real-time collected equipment operation data, construct a closed-loop control loop to control the coordinated operation of the multi-modal equipment group and dynamically update the initial control parameter set for the next control cycle.

[0025] In this embodiment, various environmental status data are first acquired from the conference room environment, including brightness values, color temperature values, and light distribution data from the light sensor; decibel values, sound source direction, and sound spectrum analysis data from the sound sensor; and personnel location data, headcount, and activity heat distribution data from the infrared sensor, forming an environmental status dataset. Simultaneously, operational data from each device in the multimodal device group is collected, including device operating status (on / off / standby), power level, response time, and execution quality indicators, forming an equipment operation dataset. The collected environmental status data and equipment operation data are preliminarily processed by the data processing unit built into the control server, including data normalization, outlier filtering, and signal smoothing, to ensure data quality. The processed data will serve as the basis for subsequent control decisions.

[0026] Then, based on the processed environmental status data and pre-configured meeting scenario requirements, an initial control parameter set is generated. Meeting scenario requirements include specific requirements for environmental conditions and equipment status for different meeting types (such as presentations, discussions, video conferences, training sessions, etc.), such as ideal lighting conditions, sound pickup sensitivity, and display content clarity. Based on the difference between the current environmental state and the target state, and combined with equipment capability parameters, the control server calculates the specific instructions and execution times that each device needs to execute, forming the initial control parameter set. The initial control parameter set is a structured dataset containing execution instructions for each device (such as light brightness adjustment values, air conditioning temperature setting values, display content switching commands, etc.) and corresponding execution time points or time windows, providing a preliminary execution plan for the collaborative work of the devices.

[0027] Based on equipment operating data and initial control parameter sets, an execution conflict model for a multimodal equipment group is constructed. Execution conflicts include competition for the same resource by at least two devices within the same control cycle, or mutual exclusion of execution actions by at least two devices. Resource competition conflicts include multiple devices simultaneously requiring the use of network bandwidth, power resources, or processor resources; mutual exclusion conflicts include the need to mute a microphone when switching content on a display screen, or to reduce air conditioning speed when dimming lights. The execution conflict model is constructed using resource demand vectors, resource competition conflict matrices, and mutual exclusion conflict vectors. It detects potential execution conflicts in real time and continuously updates and optimizes model parameters through closed-loop feedback, improving the accuracy of conflict prediction.

[0028] Based on the constructed execution conflict model, the system optimizes the initial control parameter set through an adaptive adjustment algorithm, generating an optimized control parameter set. The adaptive adjustment algorithm comprises two core mechanisms: dynamic weight allocation based on device operating data and disturbance compensation based on environmental state data. Dynamic weight allocation dynamically adjusts the execution priority of different devices according to their current operating margin (calculated from parameters such as task queue length, response latency, and power consumption fluctuation rate), ensuring the rational allocation of system resources. The disturbance compensation mechanism monitors sudden changes in the environment (such as sudden changes in light intensity, sudden increases in sound decibels, and changes in personnel density), and adjusts the control parameters in real time by calculating disturbance compensation coefficients, improving the system's adaptability to environmental changes. The optimized control parameter set, while ensuring system functionality, minimizes execution conflicts and improves the collaborative efficiency of multimodal devices.

[0029] Finally, based on the optimized control parameter set and real-time collected equipment operation data feedback, the system constructs a closed-loop control circuit to achieve coordinated operation control of the multimodal equipment group. The control server sends control signals to each device according to the execution time and instruction content specified in the optimized control parameter set, driving the devices to perform corresponding operations. Simultaneously, the system continuously collects actual operating status data of the devices, including instruction execution completion rate, execution latency, and resource usage, calculating the linkage deviation between the actual execution effect and the expected result. Based on the linkage deviation, the system dynamically adjusts the control parameters to form a closed-loop control, ensuring that the coordinated operation effect of the multimodal devices meets expectations. Furthermore, based on the execution status of the current control cycle, the system predicts the potential execution conflict risk in the next control cycle and adjusts the initial control parameter set for the next cycle in advance, achieving continuous optimization of the control process.

[0030] In this embodiment of the invention, constructing an execution conflict model for a multimodal device group includes:

[0031] Obtain the resource demand vector of each device in the multimodal device group during each control cycle. The resource demand vector includes the duration of the device's occupation of the shared resource, the occupation priority, and the resource type.

[0032] Based on the resource demand vector, a resource competition conflict matrix is ​​constructed. The element values ​​of the resource competition conflict matrix are calculated by weighting the overlap of the resource demand vectors of any two devices in the multimodal device group, the difference in occupancy priority, and the compatibility of resource types.

[0033] Based on the initial control parameter set, construct an action mutual exclusion conflict vector. The element value of the action mutual exclusion conflict vector is 1, which means that the execution actions of the corresponding two devices are mutually exclusive, and the value is 0, which means that they are not mutually exclusive. Action mutual exclusion is determined by the spatial occupancy range and time overlap of the execution instructions of the two devices.

[0034] Based on the resource contention conflict matrix and the action mutual exclusion conflict vector, an execution conflict model is constructed. The execution conflict model is updated in real time through closed-loop feedback adjustment. The closed-loop feedback adjustment includes dynamically adjusting the weights of the resource contention conflict matrix according to the execution delay and resource occupancy deviation in the equipment operation data.

[0035] In this embodiment, detailed resource requirement information is first obtained from each device in the multimodal device group. For the conference display screen, its network bandwidth requirement (Mbps), power consumption (W), and processor resource utilization (%) are collected. For the microphone array, its audio processing resource requirement (computational complexity), network transmission channel requirement (number of channels), and shared storage access frequency (times / second) are collected. For smart lighting, its power consumption (W), control channel utilization (number of channels), and feedback signal processing resource requirement (computational load) are collected. For air conditioning equipment, its power consumption (W), temperature sensor data processing requirement (data volume), and control accuracy requirement (accuracy level) are collected. These resource requirements are obtained in real time through the device's built-in resource monitoring module, and the sampling frequency is dynamically adjusted according to the resource change rate.

[0036] The acquired resource demand information is structured to form a resource demand vector. The resource demand vector is a multi-dimensional data structure containing three key dimensions: occupancy duration, representing the continuous time a device occupies a specific resource, in milliseconds or seconds; occupancy priority, representing the urgency of the device's resource demand, typically represented by a value from 1 to 10, with higher values ​​indicating higher priority; and resource type, representing the category of resource required by the device, such as network resources, power resources, computing resources, etc. The resource demand vector undergoes standardization to eliminate dimensional differences between different resource types, forming a comparable and unified format, providing foundational data for subsequent conflict analysis.

[0037] Based on the resource demand vectors of each device, a resource competition conflict matrix is ​​constructed. This matrix is ​​an n×n square matrix (n is the number of devices), and each element aij in the matrix represents the degree of resource competition conflict between device i and device j. The calculation of the element values ​​comprehensively considers three factors: the overlap of resource demand vectors, obtained by calculating the ratio of the intersection to the union of the resource occupancy of the two devices in the time dimension, reflecting the time range of resource competition; the difference in occupancy priority, obtained by calculating the absolute difference in the resource occupancy priority of the two devices, reflecting the degree of priority conflict; and the compatibility of resource types, obtained by determining whether the resources required by the two devices belong to the same resource pool, with a compatibility of 0 for completely identical resource types and a compatibility of 1 for completely different types. These three factors are weighted and summed, and after normalization, the final matrix element values ​​are formed, with a value range of [0,1]. The larger the value, the higher the degree of conflict.

[0038] Simultaneously, based on the execution instruction information in the initial control parameter set, an action mutual exclusion conflict vector is constructed. This vector describes the mutual exclusion relationship between actions executed by the device and is a... This is a 3D vector (where n is the number of devices), where each element corresponds to a pair of devices. Element values ​​are binary: 1 indicates that the actions performed by the two devices are mutually exclusive, and 0 indicates that they are not mutually exclusive. The determination of mutual exclusion is based on two key factors: the spatial occupancy of the execution command, referring to the physical space affected by the device's action, such as the area covered by lighting or the audio pickup range; and the temporal overlap, referring to the degree of overlap in the time windows of the two devices' execution commands. When the execution commands of two devices significantly overlap spatially and are executed synchronously in time, they are considered mutually exclusive, and the corresponding vector element value is set to 1; otherwise, it is set to 0. Typical mutually exclusive scenarios include: microphones needing to be muted to avoid echoes when a display is playing video; display brightness needing to be increased to ensure visibility when lights are dimmed; and light heat output needing to be reduced to maintain temperature balance when an air conditioner is cooling.

[0039] Based on the constructed resource contention conflict matrix and action mutual exclusion conflict vectors, the system integrates to form a complete execution conflict model. This model uses a graph structure, with devices as nodes and conflict relationships as edges, the weight of which is determined by the degree of conflict. The model supports two conflict detection methods: resource contention conflict detection, which identifies possible resource contention situations through the resource contention conflict matrix; and action mutual exclusion conflict detection, which identifies incompatible combinations of device operations through action mutual exclusion conflict vectors. The conflict detection results are used to guide subsequent control parameter optimization, avoiding resource contention or operational conflicts during system execution.

[0040] The execution conflict model is updated and optimized in real time through a closed-loop feedback adjustment mechanism. The system continuously monitors the actual operation of the equipment, collecting execution delay data (the difference between the actual execution time and the planned time) and resource utilization deviation (the difference between actual resource utilization and expected demand). This feedback data is used to dynamically adjust the weight coefficients of the resource competition conflict matrix. When the actual impact of competition for a certain type of resource is greater than expected, the corresponding weight is increased; conversely, the weight is decreased. Through this adaptive adjustment mechanism, the execution conflict model can continuously learn and adapt to the system's operating characteristics, improving the accuracy of conflict prediction and the rationality of control decisions.

[0041] In this embodiment of the invention, an optimized control parameter set is generated by optimizing the initial control parameter set using an adaptive adjustment algorithm, including:

[0042] Based on the execution conflict model, conflicting device pairs are identified, which include at least two devices in a multimodal device group that have an execution conflict.

[0043] For conflicting device pairs, the real-time operating margin of each device is obtained. The operating margin is calculated by weighting the task queue length, response latency, and power consumption fluctuation rate in the device's operating data.

[0044] Based on the operational margin, the dynamic execution weight of each device in the conflicting device pair is calculated. The dynamic execution weight is obtained by normalizing the product of the operational margin and the preset execution priority.

[0045] Based on dynamic execution weights, the execution instructions of the corresponding conflicting device pairs in the initial control parameter group are adjusted by time offset. The time offset adjustment is achieved through the following steps:

[0046] The minimum latency requirement for executing instructions from the lower-priority device in a conflicting device pair is obtained. The minimum latency requirement is determined by a weighted sum of the element values ​​of the resource contention conflict matrix and the element values ​​of the action mutual exclusion conflict vector.

[0047] Based on the operational margin, the execution margin compensation value of the conflicting equipment with lower priority is calculated. The execution margin compensation value is negatively correlated with the operational margin.

[0048] The larger of the sum of the minimum delay requirement and the execution margin compensation value and the preset minimum delay interval is used as the time offset;

[0049] An optimized set of control parameters is generated based on dynamic execution weights and time offsets.

[0050] In this embodiment, the system first identifies potential conflicting device pairs based on the analysis results of the execution conflict model. The determination of conflicting device pairs is based on two key indicators: device pairs whose corresponding element values ​​in the resource contention conflict matrix exceed a preset threshold (e.g., 0.7) indicate a significant risk of resource contention; and device pairs whose element value is 1 in the action mutual exclusion conflict vector indicate a definite action mutual exclusion relationship. The system integrates these two indicators to generate a list of conflicting device pairs. Each item in the list includes the identifiers of the two conflicting devices, the conflict type (resource contention / action mutual exclusion / both), the conflict severity (quantitative score based on matrix element values), and the conflict time window (calculated based on the execution time in the initial control parameter set). These conflicting device pairs will become the focus of subsequent parameter optimization.

[0051] For each pair of identified conflicting devices, the system acquires real-time operational margin data for each device. Operational margin is a comprehensive indicator measuring the current execution capability and adaptability of a device, calculated using three key parameters: task queue length, representing the number of instructions currently pending processing; a longer queue indicates a heavier load and lower margin; response latency, representing the time interval between receiving an instruction and starting execution; a larger latency indicates a slower response and lower margin; and power consumption fluctuation rate, representing the degree of variation in device power consumption; a larger fluctuation indicates a more unstable operating state and lower margin. The system obtains the real-time values ​​of these parameters through the device's internal monitoring interface and calculates the comprehensive operational margin value using a preset weighted calculation formula. The value range is typically normalized to [0,1], with higher values ​​indicating better device operating status and stronger adjustment capabilities.

[0052] Based on the calculated operational margin, the system further calculates the dynamic execution weight of each device in the conflicting device pair. The dynamic execution weight determines the priority and resource allocation ratio of each device during conflict resolution. The calculation process considers two factors: operational margin, reflecting the current execution and adjustment capabilities of the device; and preset execution priority, a basic priority pre-configured based on device type, functional importance, and business requirements. The dynamic execution weight is calculated by multiplying the operational margin by the preset execution priority, and then normalized within the conflicting device pair to ensure the total weight is 1. This calculation method considers both the inherent importance of the device and its current operational status, making resource allocation more reasonable and dynamic.

[0053] Based on dynamic execution weights, the system adjusts the time offset of execution commands for conflicting device pairs in the initial control parameter group. The core of time offset adjustment is to reduce or eliminate resource contention and action mutual exclusion conflicts by reasonably staggering the execution times of conflicting devices. The adjustment process first identifies the devices with lower priority (devices with smaller dynamic execution weights) and then calculates the minimum delay required for their execution commands. The minimum delay requirement is determined by the conflict level (a weighted sum of the element values ​​of the resource contention conflict matrix and the action mutual exclusion conflict vector); the higher the conflict level, the longer the required minimum delay. Simultaneously, the system considers the device's operational margin and calculates an execution margin compensation value. This compensation value is negatively correlated with the operational margin; the higher the operational margin, the smaller the compensation value, indicating that the device has a stronger ability to adapt to time adjustments; conversely, devices with low operational margins require a larger compensation value to alleviate the additional burden brought by time adjustments. The system adds the minimum delay requirement to the execution margin compensation value and compares it with a preset minimum delay interval, taking the larger value as the final time offset.

[0054] Finally, based on the determined dynamic execution weights and time offsets, the system generates an optimized control parameter set. The optimization process mainly involves two adjustments: time adjustment, which modifies the execution time of the lower-priority device among the conflicting devices based on the calculated time offset, delaying it to avoid conflict; and resource allocation adjustment, which adjusts the resource allocation ratio of the conflicting devices according to the dynamic execution weights, ensuring that the higher-priority device receives sufficient resources. The optimized control parameter set maintains the same data structure as the initial control parameter set, but includes the adjusted execution time and resource allocation parameters. The system verifies the optimization effect by comparing the conflict detection results before and after optimization, ensuring that the adjusted parameters can effectively reduce or eliminate execution conflicts and improve the overall collaborative efficiency of the system.

[0055] In this embodiment of the invention, the disturbance compensation in the adaptive adjustment algorithm includes:

[0056] Obtain disturbance factors from environmental status data, including the rate of change of light intensity, the fluctuation rate of sound decibel value, and the abrupt change value of population distribution density;

[0057] Based on the disturbance factor, the disturbance compensation coefficient is calculated. The disturbance compensation coefficient is determined by the ratio of the weighted sum of the disturbance factors to the preset disturbance threshold.

[0058] The disturbance compensation coefficient is applied to the execution time of the corresponding device in the optimized control parameter set to generate the disturbance-compensated optimized control parameter set. The execution time of the disturbance-compensated optimized control parameter set is adjusted by multiplying the execution time of the optimized control parameter set by the disturbance compensation coefficient.

[0059] In this embodiment, key disturbance factors are first extracted from environmental state data. These factors reflect sudden changes or abnormal fluctuations in the conference room environment. The rate of change in light intensity is calculated as the relative change in light intensity per unit time (e.g., 10 seconds), expressed as a percentage value. For example, a sudden change from 500 lux to 300 lux represents a rate of change of -40%. The rate of change in sound decibel levels is calculated as the ratio of the standard deviation to the mean of sound decibel levels per unit time, reflecting the stability of the sound environment; a larger value indicates more severe fluctuations. The abrupt change in personnel density is calculated as the magnitude of change in the number or distribution of people in the conference room per unit time, such as a sudden increase of 50% in the number of people or a change in the distribution pattern from concentrated to dispersed. These disturbance factors are collected and calculated in real time by an environmental sensor array. The sampling frequency is dynamically adjusted according to the rate of environmental change, typically 1-10 times per second.

[0060] Based on the extracted disturbance factors, the system calculates the disturbance compensation coefficient. The calculation process first standardizes each disturbance factor to eliminate dimensional differences between different types of disturbances. Then, weight coefficients are assigned according to the importance of different disturbances to equipment operation; for example, changes in illumination have a greater impact on displays and lights, so they are assigned higher weights; sound fluctuations have a greater impact on microphone arrays, so they are also assigned higher weights. A weighted summation is then performed to obtain the comprehensive disturbance index. The final calculation of the disturbance compensation coefficient involves comparing the comprehensive disturbance index with a preset disturbance threshold, calculating the ratio between the two, and then converting the ratio into a compensation coefficient within the range of [0.5, 2] using a mapping function (such as the sigmoid function). When the disturbance is small, the compensation coefficient is close to 1, indicating that almost no adjustment is needed; when the disturbance is significant, the compensation coefficient may be close to 0.5 or 2, indicating that a larger adjustment is needed to adapt to environmental changes.

[0061] The calculated disturbance compensation coefficients are applied to the execution time of the corresponding devices in the optimized control parameter set to generate the disturbance-compensated optimized control parameter set. The application process varies depending on the type of disturbance and device characteristics: For changes in light intensity, primarily affecting displays and lighting equipment, the system adjusts the response time of these devices according to the direction of the light change (increase or decrease). For example, when the light suddenly increases, the execution time of the display brightness adjustment command may need to be advanced to quickly reduce brightness and prevent glare. For fluctuations in the sound environment, primarily affecting microphone arrays and audio equipment, the system adjusts the gain adjustment speed and noise reduction parameter update frequency of these devices according to the degree of fluctuation. For changes in personnel distribution, potentially affecting all devices, the system adjusts the execution rhythm of the environmental control strategy according to the pattern and degree of personnel changes. The disturbance-compensated optimized control parameter set maintains the same data structure as the original parameter set, but the execution time is adjusted by multiplying the original execution time by the disturbance compensation coefficient. A compensation coefficient greater than 1 indicates delayed execution, and less than 1 indicates advanced execution.

[0062] The disturbance compensation mechanism enables the system to proactively adapt to environmental changes, improving control robustness and adaptability. Typical application scenarios include: sudden changes in external light during a meeting (such as clouds blocking sunlight or outdoor lights being turned on), the system quickly adjusts indoor lighting and display parameters to maintain a stable visual experience; sudden noise during a meeting (such as temporary construction or external disturbances), the system rapidly adjusts the microphone array's pickup mode and gain to ensure clear voice capture; and temporary changes in the number of attendees or layout, the system adjusts the air conditioning's airflow direction and temperature settings accordingly to maintain comfortable environmental conditions. Through this dynamic compensation mechanism, the system can maintain stable and efficient control in complex and ever-changing meeting environments, improving user experience and meeting efficiency.

[0063] In this embodiment of the invention, a closed-loop control circuit is constructed to control the coordinated operation of the multi-modal device group, including:

[0064] Within each control cycle, corresponding control signals are sent to each device in the multimodal device group according to the execution time and execution instructions of the optimized control parameter group;

[0065] Real-time acquisition of operational status data for each device in the multimodal device group, including the completion rate of device execution commands, execution latency, and resource usage deviation;

[0066] Based on the operating status data, the linkage deviation of the multimodal device group in the current control cycle is calculated. The linkage deviation is determined by the weighted sum of the variance of the execution delay of each device in the multimodal device group, the deviation of the execution instruction completion degree, and the resource usage deviation.

[0067] Based on the linkage deviation, feedback adjustment parameters for the closed-loop control circuit are constructed. The feedback adjustment parameters are obtained by normalizing the ratio of the linkage deviation to the preset deviation threshold.

[0068] The feedback adjustment parameters are applied to the optimized control parameter group for the next control cycle, adjusting the execution time and execution instructions of the corresponding equipment.

[0069] In this embodiment, at the beginning of each control cycle (typically 100-500 milliseconds), the control server first sends control signals to each device in the multimodal device group according to the specifications in the optimized control parameter group. The control signals are transmitted through a standardized communication interface and encapsulated using communication protocols supported by the devices (such as ModBus, KNX, HTTP, or proprietary protocols). The signal content includes two core parts: an execution instruction, which explicitly specifies the specific operation the device needs to perform, such as switching the display to specific content, adjusting the lighting to a specific brightness and color temperature, enabling a specific microphone pickup mode, or adjusting the air conditioner to a specific temperature and fan speed; and an execution time, which specifies the execution time or time window of the instruction, which can be an absolute time point (e.g., 12:05:30.500) or a relative time point (e.g., current time + 2 seconds). Control signals are sent in priority order, with signals from high-priority devices sent first to ensure timely response from critical devices. A signal integrity check and confirmation mechanism is implemented during transmission to ensure that the control signals reach the target devices accurately.

[0070] Simultaneously, the system collects real-time operational status data from each device in the multimodal device group. The collection frequency is dynamically adjusted based on device characteristics and monitoring needs. Key devices and key status parameters are collected at high frequency (e.g., 10 times per second), while general parameters are collected at low frequency (e.g., once per second). The collected operational status data includes three key types of information: completion rate of execution instructions, indicating the degree to which the actual execution of the device matches the instruction requirements, usually expressed as a percentage, such as 90% completion of display screen content switching or 95% achievement of target brightness adjustment; execution latency, indicating the time interval between receiving the instruction and starting execution, as well as the deviation from the planned timeline during execution, in milliseconds; and resource usage deviation, indicating the difference between the actual resource usage and the expected resource requirements, such as 20% higher actual network bandwidth usage or 15% lower processor load than expected. This data is collected through the device's status feedback interface, and after preliminary processing (e.g., outlier filtering and data smoothing), it is uploaded to the control server.

[0071] Based on the collected operational status data, the system calculates the linkage deviation of the multimodal device group within the current control cycle. Linkage deviation is a comprehensive indicator measuring the effectiveness of multi-device collaborative operation. It is calculated using three sub-indicators: 1) Execution delay variance: calculating the statistical variance of the execution delay of all devices, reflecting the consistency of time synchronization between devices; a larger variance indicates greater asynchrony in device execution time. 2) Execution instruction completion deviation: calculating the difference between the instruction completion rate of all devices and a preset completion rate threshold (usually 100%), and then calculating the root mean square of these differences, reflecting the degree of deviation in overall execution quality. 3) Weighted sum of resource utilization deviation: summing the resource utilization deviations of each device according to resource importance, reflecting the rationality of resource allocation. After normalization, these three sub-indicators are weighted and summed again (weights are dynamically adjusted according to the control objective) to obtain the final linkage deviation value, typically ranging from [0,1]. The closer to 0, the better the linkage effect.

[0072] Based on the calculated linkage deviation, the system constructs feedback adjustment parameters for the closed-loop control loop. These feedback adjustment parameters are the core mechanism for the system's self-adjustment and optimization, guiding the direction and magnitude of parameter adjustments in the next control cycle. The calculation process first compares the linkage deviation with a preset deviation threshold (determined according to the application scenario and control accuracy requirements), calculating their ratio. Then, a mapping function (such as linear or nonlinear mapping) converts this ratio into standardized feedback adjustment parameters, typically ranging from -1 to 1. Negative values ​​indicate a need for reduction or delay, while positive values ​​indicate a need for increase or advancement. The absolute value represents the adjustment magnitude. For example, when the execution delay variance is large, negative feedback parameters for time synchronization may be generated, guiding the system to enhance time coordination in the next cycle; when resource occupancy deviation is significant, adjustment parameters for resource allocation may be generated, guiding the system to rebalance resource allocation ratios.

[0073] Finally, the system applies the feedback adjustment parameters to the optimized control parameter set for the next control cycle, achieving dynamic adjustment and continuous optimization of the control process. The application process mainly involves two aspects: execution time adjustment, which adjusts the planned time of device execution commands based on time synchronization-related feedback parameters, such as advancing or delaying the operation time of specific devices or adjusting the time interval between devices; and execution command adjustment, which modifies the content or parameters of device execution commands based on execution quality and resource allocation-related feedback parameters, such as adjusting the fading time of light changes, microphone gain levels, or the temperature change rate of air conditioners. The adjustment process adopts a gradual strategy to avoid system instability that may result from large adjustments, typically limiting the magnitude of a single adjustment to a preset range (e.g., ±10%). Through this closed-loop feedback mechanism, the system can continuously learn and adapt to the operating environment and device characteristics, continuously optimize control effects, and improve the collaborative efficiency of multimodal devices and user experience.

[0074] In this embodiment of the invention, dynamically updating the initial control parameter set for the next control cycle includes:

[0075] Based on the equipment operation data and environmental status data of the current control cycle, the execution conflict risk of the next control cycle is predicted. The execution conflict risk is determined by the change rate of the element values ​​of the resource competition conflict matrix and the distribution of the element values ​​of the action mutual exclusion conflict vector.

[0076] Based on the risk of execution conflicts, adjust the priority and execution time of the execution instructions of the corresponding devices in the initial control parameter group of the next control cycle. The adjustments include increasing the execution time offset of lower priority devices or reducing the number of execution instructions of higher priority devices.

[0077] In this embodiment, the system first predicts the potential execution conflict risks in the next control cycle based on the equipment operation data and environmental status data of the current control cycle. The prediction process employs a trend analysis method, focusing on two key indicators: the rate of change of element values ​​in the resource competition conflict matrix, obtained by calculating the changing trend of each element in the matrix over several consecutive control cycles (positive rate of change indicates increasing conflict trend, negative rate of change indicates decreasing conflict trend); and the distribution of element values ​​in the action mutual exclusion conflict vector, analyzing the distribution pattern and concentration of elements with a value of 1 (indicating mutual exclusion), with a more concentrated distribution indicating a higher concentration of conflict risk on a specific equipment combination. Combining these two indicators, the system constructs an execution conflict risk assessment model, calculating the risk value of potential conflicts for each equipment pair in the next control cycle. Risk values ​​are typically represented by values ​​between 0 and 1, with larger values ​​indicating a higher probability of conflict.

[0078] The assessment model also considers the impact of changing environmental trends on conflict risk. For example, a continuous decrease in light intensity may increase the coordination requirements between displays and lights, thereby increasing the risk of conflict between them; rising noise levels in the conference room may increase the processing complexity of the microphone array, increasing its resource competition risk with other equipment; changes in personnel distribution may affect the operating parameters of multiple devices, indirectly increasing the coordination complexity and the likelihood of conflict between equipment. Through this comprehensive analysis, the system can identify potential conflict hotspots in advance, providing a basis for preventative adjustments.

[0079] Based on the predicted execution conflict risk, the system makes targeted adjustments to the initial control parameter set for the next control cycle. The adjustment strategy mainly includes two aspects: priority adjustment, which dynamically adjusts the priority of device execution instructions according to the distribution and severity of conflict risk, increasing the priority of critical devices with high conflict risk to ensure they receive sufficient system resources and execution guarantees; and time staggering, which reduces or avoids conflicts for identified high-conflict-risk device pairs through time staggering strategies. Specifically, this includes increasing the execution time offset of lower-priority devices (appropriately postponing their operation time) or reducing the number of execution instructions for higher-priority devices (reducing resource requirements through instruction merging or simplification).

[0080] The determination of time offsets considers multiple factors: conflict risk value (higher risk requires greater time separation); equipment operating characteristics (some equipment may be more or less sensitive to time adjustments); the urgency of the executed instructions (urgent instructions have less room for adjustment); and overall system load (lighter loads allow for more flexible time scheduling). The system calculates the optimal time offset scheme using optimization algorithms (such as greedy algorithms or heuristic search) to minimize conflict risk while ensuring functionality. Simultaneously, for equipment that may require instruction number adjustments, the system analyzes the dependencies and merging possibilities of its executed instructions. Without affecting functionality, it reduces the total number of instructions and alleviates resource pressure by merging similar instructions, simplifying non-critical instructions, or delaying non-urgent instructions.

[0081] This predictive adjustment mechanism enables the system to proactively identify and respond to potential conflicts, rather than passively reacting to conflicts that have already occurred. Through forward-looking parameter optimization, the system can make preventative adjustments before conflicts arise, significantly reducing conflict events in actual operation and improving the collaborative efficiency and control stability of multimodal devices. Simultaneously, predictive adjustment also reduces system response delays and user experience degradation caused by conflict handling, making the control of the conference room environment smoother and more natural.

[0082] In this embodiment of the invention, the method for calculating the element values ​​of the resource contention conflict matrix includes:

[0083] Obtain the overlap of the resource demand vectors of any two devices in the multimodal device group. The overlap is determined by the ratio of the intersection to the union of the occupancy durations of the resource demand vectors of the two devices in the same control cycle.

[0084] Obtain the difference in occupancy priority between two devices. The occupancy priority is determined by the task urgency and device type weight in the device operation data.

[0085] Get the resource type compatibility of two devices. The compatibility is determined by whether the resource types of the two devices belong to the same resource pool. A value of 1 indicates that they belong to the same resource pool, and a value of 0 indicates that they do not belong to the same resource pool.

[0086] The element values ​​of the resource contention conflict matrix are obtained by normalizing the weighted sum of overlap, occupancy priority difference, and compatibility.

[0087] In this embodiment, the overlap of resource demand vectors between any two devices is first calculated. The overlap reflects the degree of competition for resources between the two devices over time. The calculation process first determines the time intervals during which the two devices occupy various resources within the current control cycle, forming time sets. For example, a display screen requires network resources during the time interval [t1, t2], and a microphone array also requires network resources during the time interval [t3, t4]. Then, the intersection interval length (i.e., overlap time length) and union interval length (i.e., the total time occupied by the two devices) of these two time sets are calculated, and the ratio of these two is taken as the overlap degree PUL. The calculation of the overlap degree can be expressed as: Here, A and B are the resource occupancy time sets of two devices, |A∩B| represents the intersection size, and |A∪B| represents the union size. The overlap value range is [0,1], where a value of 0 indicates no overlap (no conflict) and a value of 1 indicates complete overlap (maximum conflict). For multiple resource types, the overlap value of each type of resource is calculated separately, and then a weighted average is performed according to the importance of the resources to obtain the comprehensive overlap value.

[0088] Then, the priority difference between the two devices is obtained. Priority is a quantitative representation of the urgency of a device's resource demand, determined by two factors: task urgency, reflecting the time sensitivity of the device's current task execution, calculated based on task deadlines, waiting times, and user-specified priorities, typically ranging from 1 to 10, with higher values ​​indicating greater urgency; and device type weight, reflecting the inherent importance of the device within the system, pre-configured based on device functionality, user experience impact, and system dependencies, also ranging from 1 to 10, with higher values ​​indicating greater importance. Priority is calculated by weighting the task urgency and device type weight, and then the absolute difference between the priorities of the two devices is calculated. A larger difference indicates a more significant difference in priorities between the two devices, making resource allocation decisions easier; a smaller difference indicates closer priorities, making conflict resolution more difficult.

[0089] Next, the resource type compatibility of the two devices is assessed. Compatibility is used to determine whether the resources required by the two devices belong to the same resource pool, directly affecting the likelihood and severity of conflicts. The determination process is based on resource type classification and resource pool configuration: First, the specific resource types required by each device are determined, such as network bandwidth, processor time, storage space, and power supply; then, it is determined whether these resource types belong to the same resource pool. A resource pool is a logical unit for resource management; resources within the same pool are shared and competitive, while resources in different pools are relatively independent. If the resource types of the two devices belong to the same resource pool, the compatibility value is 1, indicating a direct competitive relationship; if they belong to different resource pools, the compatibility value is 0, indicating no direct competition. For cases involving multiple resource types, the ratio of resource type matching is calculated as the overall compatibility value.

[0090] Finally, the weighted sum of overlap, priority difference ZY, and compatibility JR is normalized to obtain the element values ​​HJ of the resource contention conflict matrix. The calculation formula can be expressed as:

[0091]

[0092] Where w1, w2, and w3 are weighting coefficients, satisfying w1 + w2 + w3 = 1, dynamically adjusted according to the system's control priorities; MaxDiff is the maximum possible priority difference in the system, used for normalization. The calculation results are further processed through a nonlinear mapping function (such as the Sigmoid function) to ensure that the final value range is within [0,1] and to enhance the difference. The final matrix element values ​​intuitively reflect the degree of resource competition conflict between the two devices; the closer the value is to 1, the higher the conflict risk, and the closer the value is to 0, the lower the conflict risk.

[0093] This multi-factor weighted calculation method comprehensively considers key factors such as time overlap, priority differences, and resource types, accurately quantifying the resource competition among devices and providing a precise basis for conflict assessment for subsequent control parameter optimization. Furthermore, by dynamically adjusting the weighting coefficients, the system can flexibly change the focus of conflict assessment according to different application scenarios and control objectives, improving the adaptability and practicality of the conflict model.

[0094] In this embodiment of the invention, the method for calculating the running margin includes:

[0095] Obtain task queue length, response latency, and power consumption fluctuation rate from the device's operational data;

[0096] The task queue length, response latency, and power consumption fluctuation rate are normalized to obtain the corresponding normalized values;

[0097] Based on preset weights, the normalized values ​​are weighted and summed to obtain the operating margin. The preset weights are determined by the task execution success rate and resource utilization rate in the historical operating data of the equipment.

[0098] In this embodiment, three key parameters are first extracted from the device operation data: task queue length, which represents the number of instructions or tasks currently pending processing by the device, directly reflecting the device's workload; this data is obtained through the device's task management interface, and the unit is the number of tasks; response latency, which represents the time interval between receiving an instruction and starting execution, reflecting the device's response sensitivity and processing capability; this data is collected through the device's performance monitoring interface, and the unit is milliseconds; and power consumption fluctuation rate, which represents the degree of change in the device's power consumption within a certain time window, reflecting the stability of the device's operating state; this data is collected through the device's power monitoring interface and calculated as the ratio of the power standard deviation to the mean, dimensionless. These three parameters describe the current operating state of the device from different perspectives and together constitute the basic data for calculating the operating margin.

[0099] Since the three parameters have different dimensions and numerical ranges, they need to be normalized to make them comparable and calculable. The normalization method is selected based on the parameter characteristics: for the task queue length, maximum-min normalization is used, mapping it to the [0,1] interval. The closer the normalized value is to 1, the shorter the queue and the lighter the load. For response latency, exponential normalization is used, calculated as: Normalized value = exp(-current value / reference latency), where the reference latency is the system's preset baseline response time. The closer the normalized value is to 1, the smaller the latency and the faster the response. For power consumption fluctuation rate, threshold normalization is used, calculated as: Normalized value = 1 - min(current value / maximum allowable fluctuation rate, 1), where the maximum allowable fluctuation rate is the system's preset fluctuation tolerance. The closer the normalized value is to 1, the smaller the fluctuation and the more stable the state. Through these normalization processes, all three parameters are converted to values ​​within the [0,1] interval, with larger values ​​indicating better performance in that dimension.

[0100] Based on the normalized parameter values, the system performs a weighted summation according to preset weights to calculate the final operational margin. The weight coefficients reflect the degree of influence of different parameters on the equipment's operational capability and are determined through the equipment's historical operational data: Task execution success rate, representing the proportion of assigned tasks successfully completed by the equipment historically; a higher success rate indicates a more reliable equipment, and the corresponding parameter weight should be higher; and resource utilization rate, representing the equipment's resource utilization efficiency under different loads, calculated through the relationship between resource consumption and task completion rate in historical data; higher efficiency indicates better equipment performance, and the corresponding parameter weight should be higher. Based on these two indicators and the characteristics of the equipment type, the system dynamically calculates the weight coefficient for each parameter, ensuring that the total weight is 1.

[0101] The final formula for calculating runtime margin is: Runtime Margin = w1 × Normalized Task Queue Length + w2 × Normalized Response Latency + w3 × Normalized Power Consolidation Rate, where w1, w2, and w3 are the corresponding weighting coefficients. The value range of the calculated result is [0, 1]. The closer the value is to 1, the better the current operating state of the device, with greater adjustment capability and execution space; the closer the value is to 0, the heavier the current load or the unstable state of the device, with limited adjustment space. As a key input to the adaptive adjustment algorithm, runtime margin directly affects the selection of conflict resolution strategies and the magnitude of parameter adjustments, ensuring that the system can make reasonable control decisions based on the actual state of the device.

[0102] In this embodiment of the invention, the method for calculating the linkage deviation includes:

[0103] The execution delay of each device in the multimodal device group is obtained. The execution delay is determined by the difference between the actual completion time of the device's execution instruction and the planned time of the corresponding instruction in the optimized control parameter group.

[0104] Calculate the variance of the execution delay of all devices in the multimodal device group, and use it as the first component of the linkage deviation;

[0105] The execution instruction completion rate of each device in the multimodal device group is obtained. The execution instruction completion rate is determined by the matching degree between the actual actions executed by the device and the corresponding actions in the optimized control parameter group.

[0106] Calculate the deviation of the execution instruction completion degree of all devices in the multimodal device group. The deviation is determined by the root mean square of the difference between the execution instruction completion degree and the preset completion degree threshold, and is used as the second component of the linkage deviation.

[0107] The resource occupancy deviation of each device in the multimodal device group is obtained. The resource occupancy deviation is determined by the difference between the actual resource occupancy duration of the device and the planned occupancy duration in the resource demand vector.

[0108] Calculate the weighted sum of the resource occupancy deviations of all devices in the multimodal device group, and use it as the third component of the linkage deviation;

[0109] The weighted sum of the first, second, and third components is normalized to obtain the linkage deviation. The weights of the weighted sum are determined by the task urgency and equipment type weights in the equipment operation data.

[0110] In this embodiment, the execution delay data of each device in the multimodal device group is first acquired. Execution delay is a key indicator for measuring the accuracy of device execution time, calculated by comparing the actual completion time of the executed instruction with the planned time specified in the optimized control parameter set. The acquisition process includes: reading the actual start and completion times of instruction execution from the device's status feedback interface and recording the timestamps; extracting the planned execution time of the corresponding instruction from the optimized control parameter set, also using the timestamp format; calculating the difference between the actual completion time and the planned completion time to obtain the execution delay, in milliseconds. A positive value indicates that the actual execution lags behind the plan, while a negative value indicates that the actual execution is ahead of the plan. Execution delay directly reflects the timeliness and accuracy of the device's response to control instructions and is an important basis for evaluating the linkage effect.

[0111] Based on the acquired execution latency data of each device, the execution latency variance of the entire multimodal device group is calculated. The variance calculation formula is: Where Xi is the execution latency of the i-th device, μ is the average execution latency of all devices, and n is the total number of devices. The execution latency variance reflects the consistency and synchronization of execution times among devices. A smaller variance indicates that the execution times of each device are more consistent, resulting in better linkage effects; a larger variance indicates significant differences in execution times, potentially indicating coordination problems. The calculated variance is standardized (e.g., divided by a preset maximum allowable variance) to obtain the standardized variance value, which serves as the first component of the linkage deviation.

[0112] Then, the execution command completion rate of each device in the multimodal device group is obtained. Completion rate is an indicator of device execution quality, calculated by comparing the degree of matching between the actual actions performed by the device and the actions required by the control commands. The calculation method varies depending on the device type: for discrete-state devices (such as switch-type devices), completion rate is a binary value, with 100% for a perfect match and 0% for a mismatch; for continuous-state devices (such as dimmer lights, adjustable-speed fans, etc.), completion rate is calculated as: Completion Rate = 1 - |Actual Value - Target Value| / Allowable Deviation Range, where the allowable deviation range is a preset tolerance, and the value range of completion rate is [0,1], with a value closer to 1 indicating more accurate execution. The actual execution state of the device is obtained through the state feedback interface, and the target state is extracted from the optimized control parameter group.

[0113] Based on the execution instruction completion rate of each device, the completion rate deviation of the entire device group is calculated. The calculation method involves comparing the completion rate of each device with a preset completion rate threshold (usually 1 or 100%, representing perfect execution), calculating the difference, and then taking the root mean square (RMS) of all differences. The formula for calculating the root mean square is: Where Ci represents the completion rate of the i-th device, Cth is the preset completion rate threshold, and n is the total number of devices. A smaller root mean square (RMS) value indicates higher overall execution quality and closer to expectations; a larger value indicates significant deviations in execution quality, requiring adjustment. The calculated RMS value is also standardized to obtain the standardized completion rate deviation value, which serves as the second component of the linkage deviation.

[0114] Next, the resource utilization deviation of each device in the multimodal device group is obtained. Resource utilization deviation reflects the difference between the actual resource usage and planned demand, calculated by comparing the actual resource usage duration with the planned usage duration declared in the resource demand vector. The acquisition process includes: reading actual resource usage records from the device's resource monitoring interface, including start time, end time, and resource type; extracting the planned usage duration of the corresponding resource from the resource demand vector; calculating the difference between the actual usage duration and the planned usage duration to obtain the resource utilization deviation, in milliseconds. A positive value indicates that actual usage exceeds the plan, and a negative value indicates that actual usage is less than the plan. Resource utilization deviation reflects the rationality of resource allocation and the accuracy of resource demand prediction, serving as an important reference for optimizing resource utilization.

[0115] Based on the resource usage deviations of each device, a weighted sum of resource usage deviations for the entire device group is calculated. The weighted calculation considers the importance and scarcity of different resource types: deviations of critical resources (such as main processor time, main network bandwidth, etc.) are given higher weights, while deviations of secondary resources are given lower weights; timeout occupancy (positive deviation) typically has a higher weight than early release (negative deviation), because timeout occupancy is more likely to lead to resource conflicts and system performance degradation. The weighted sum calculation formula is: ∑(wi×Ri), where Ri is the resource usage deviation of the i-th device, and wi is the corresponding weight coefficient. The calculation result is standardized to obtain a standardized resource usage deviation value, which serves as the third component of the linkage deviation.

[0116] Finally, the weighted sum of the three components is normalized to obtain the final linkage deviation value. The weighted sum calculation formula is: Weighted Sum = α × First Component + β × Second Component + γ × Third Component, where α, β, and γ are weighting coefficients, satisfying α + β + γ = 1. The weighting coefficients are determined by two key indicators in the equipment operation data: task urgency, reflecting the time sensitivity of the current control task; urgent tasks usually focus more on the accuracy of execution time, corresponding to a higher weight for the first component; and equipment type weight, reflecting the importance of the equipment in the system; core equipment usually focuses more on execution quality, corresponding to a higher weight for the second component. The system dynamically adjusts these weighting coefficients according to the current control scenario and equipment combination characteristics, making the linkage deviation calculation more consistent with actual control needs.

[0117] The calculated weighted sum is converted into a standardized linkage deviation value through a mapping function (such as a linear mapping or a sigmoid function). The value ranges from [0,1]. The closer the value is to 0, the better the linkage effect, and the more the execution of each device matches expectations; the closer the value is to 1, the worse the linkage effect, requiring more significant control adjustments. As a key feedback indicator of the closed-loop control loop, the linkage deviation directly drives the system to adjust parameters and optimize control, ensuring continuous improvement in the collaborative operation of multimodal devices.

[0118] In this embodiment of the invention, the multimodal device group includes a conference display screen, a microphone array, intelligent lighting, and air conditioning equipment; the environmental sensor group includes a light sensor, a sound sensor, and an infrared sensor; the system also includes the following adaptive optimization steps:

[0119] Acquire historical operating data for each device in the multimodal device group. The historical operating data includes the distribution of execution latency, resource usage, and task execution success rate of the device within a preset historical period.

[0120] Based on historical operational data, operational characteristic models for each device are constructed. These models are determined by a weighted sum of the mean of the execution delay distribution, the variance of the resource usage distribution, and the task execution success rate.

[0121] Based on the operating characteristic model, the potential execution bottlenecks of the multimodal equipment group in the next control cycle are predicted. The potential execution bottlenecks are determined by the equipment in the operating characteristic model whose average execution delay is greater than a preset delay threshold or whose resource usage variance is greater than a preset variance threshold.

[0122] For devices corresponding to potential execution bottlenecks, adjust and optimize the execution time or execution priority of the corresponding device's execution instructions in the control parameter group. The adjustment includes moving the device's execution time forward or increasing the device's execution priority. The forward amount or priority increment is determined by the ratio of the average execution delay in the running characteristic model to the preset delay threshold.

[0123] In this embodiment, detailed historical operating data of each device in the multimodal device group is first obtained from the system's historical database. The data acquisition range is a preset historical period, typically the most recent 24 hours, 7 days, or 30 days, adjusted according to the system application scenario and data volume. The acquired historical operating data includes three key types of information: execution latency distribution, which records the execution latency values ​​of all control commands of the device within the historical period, forming a time series or statistical distribution to reflect the stability and trend of the device's response characteristics; resource occupancy distribution, which records the device's occupancy of various resources (such as processor, memory, network, power, etc.) within the historical period, including occupancy duration, occupancy intensity, and occupancy mode, reflecting the device's resource demand characteristics; and task execution success rate, which records the proportion of devices that successfully complete assigned tasks, statistically analyzed by time period, task type, or resource conditions, reflecting the device's reliability and performance stability. This historical data is continuously collected and stored through the system's log recording and performance monitoring modules, forming a historical archive of device operation.

[0124] Based on the acquired historical operational data, the system constructs an operational characteristic model for each device. This model is a mathematical description of the device's operational behavior, built using three key indicators: the mean of execution latency distribution (calculated as the arithmetic mean of all execution latency values ​​within a historical period, reflecting the device's average response speed; a smaller value indicates a faster response); the variance of resource usage distribution (calculated as the statistical variance of resource usage within a historical period, reflecting the fluctuation in the device's resource demand; a smaller value indicates more stable resource demand); and the task execution success rate (directly using success rate values ​​from historical data, reflecting the device's reliability and stability; a higher value indicates better reliability). These three indicators are standardized and then combined using a weighted sum to form the device's operational characteristic score. The score is typically normalized to the [0,1] interval, with higher values ​​indicating better overall operational characteristics.

[0125] The operational characteristic model includes not only static scoring but also trend analysis over time. The system performs time-series analysis on historical data to identify patterns in key indicators, such as linear trends (continuous improvement or deterioration), periodic fluctuations (related to time, load, or environmental conditions), or abrupt changes (potentially indicating changes in equipment condition or external disturbances). This trend information is captured and quantified using time-series models (such as autoregressive, moving average, or ARIMA models) as a dynamic component of the operational characteristic model, enhancing its predictive capabilities.

[0126] Based on the constructed operational characteristic model, the system predicts potential execution bottlenecks that may arise in the multimodal device group during the next control cycle. The prediction method is based on two key criteria: devices with a mean execution latency greater than a preset latency threshold indicate slow response speeds and may become bottlenecks in system response; devices with a resource utilization variance greater than a preset variance threshold indicate large fluctuations in resource demand, potentially leading to unstable resource allocation and becoming bottlenecks in system stability. The system marks devices that meet either criterion as potential bottleneck devices and further assesses their impact on overall system performance, including: bottleneck severity, calculated as the ratio of the difference between the indicator and the threshold; bottleneck persistence, determining whether the problem is temporary or persistent based on historical trends; and bottleneck propagation, assessing the potential scope of the problem through inter-device dependencies.

[0127] For identified potential bottleneck devices, the system makes targeted adjustments to the optimized control parameter set to proactively address potential performance issues. The adjustment strategy mainly includes two aspects: time advancement—for bottleneck devices with significant execution latency, the planned execution time of their instructions is appropriately advanced to provide sufficient execution buffer time and ensure that critical tasks can be completed on time; and priority enhancement—for bottleneck devices with large fluctuations in resource demand, the priority of their execution instructions is increased to ensure they receive stable and sufficient system resources, reducing performance fluctuations caused by resource contention.

[0128] The adjustment range is dynamically determined based on the severity of the bottleneck: the time advance is calculated by the ratio of the average execution latency to a preset latency threshold. A larger ratio indicates a more severe latency problem, requiring a larger advance. An upper limit is typically set to avoid excessive advancement. The priority increment is also calculated by the ratio of the average execution latency to a threshold or the ratio of resource usage variance to a threshold, ensuring the increment is proportional to the problem severity. The priority distribution of other devices in the system is also considered to avoid priority inflation. The adjusted parameters undergo conflict detection and coordination to ensure that adjustments to the parameters of a single device do not lead to new system conflicts or imbalances.

[0129] Through this adaptive optimization mechanism based on historical data and predictive models, the system can identify and address potential execution bottlenecks in advance, achieving preventative performance optimization, reducing operational anomalies and performance fluctuations, and improving the overall collaborative efficiency and user experience of multimodal equipment in the conference room. Simultaneously, as the system continues to operate and data accumulates, the operational characteristic model is continuously optimized and improved, leading to sustained increases in predictive accuracy and adjustment rationality, ultimately forming a self-evolving intelligent control system.

[0130] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0131] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0132] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A real-time linkage control system for multimodal equipment in a conference room, characterized in that, Including: A multi-modal device group, a control server, and an environmental sensor group. The multi-modal device group includes devices of at least two different modalities. The system achieves real-time linkage control through the following method: Obtain the environmental status data and device operation data of the meeting room in each control period; According to the environmental status data and the preset meeting scenario requirements, generate an initial control parameter group for the multi-modal device group in each control period. The initial control parameter group includes the execution instructions and execution times of each device; According to the device operation data and the initial control parameter group, construct an execution conflict model for the multi-modal device group. The execution conflict model detects whether there is an execution conflict in the multi-modal device group through closed-loop feedback regulation; Based on the execution conflict model, optimize the initial control parameter group through an adaptive adjustment algorithm to generate an optimized control parameter group. The adaptive adjustment algorithm includes dynamic weight allocation based on device operation data and disturbance compensation based on environmental status data; The optimization of the initial control parameter group through the adaptive adjustment algorithm to generate an optimized control parameter group includes: According to the execution conflict model, determine the conflict device pairs. The conflict device pairs include at least two devices in the multi-modal device group that have the execution conflict; For the conflict device pairs, obtain the real-time operation margins of each device; According to the operation margins, calculate the dynamic execution weights of each device in the conflict device pairs; Based on the dynamic execution weights, perform time offset adjustment on the execution instructions corresponding to the conflict device pairs in the initial control parameter group. The time offset adjustment is achieved through the following steps: Obtain the minimum delay requirement of the execution instruction of the device with a lower priority in the conflict device pair; According to the operation margins, calculate the execution margin compensation value of the device with a lower priority in the conflict device pair. The execution margin compensation value is negatively correlated with the operation margins; Take the larger value of the sum of the minimum delay requirement and the execution margin compensation value and the preset minimum delay interval as the time offset; Based on the dynamic execution weights and the time offset, generate the optimized control parameter group; According to the feedback of the optimized control parameter group and the real-time collected device operation data, construct a closed-loop control loop to control the linkage operation of the multi-modal device group and dynamically update the initial control parameter group for the next control period.

2. The real-time linkage control system for multimodal equipment in a conference room according to claim 1, characterized in that, The construction of the execution conflict model for the multi-modal device group includes: Obtain the resource demand vectors of each device in the multi-modal device group in each control period. The resource demand vector includes the occupation duration, occupation priority, and resource type of the device for shared resources; According to the resource demand vectors, construct a resource competition conflict matrix. The element values of the resource competition conflict matrix are weighted and calculated by the overlap degree of the resource demand vectors of any two devices in the multi-modal device group, the difference in occupation priorities, and the compatibility of resource types; According to the initial control parameter group, construct an action mutual exclusion conflict vector; Based on the resource contention conflict matrix and the action mutual exclusion conflict vector, the execution conflict model is constructed. The execution conflict model is updated in real time through closed-loop feedback adjustment. The closed-loop feedback adjustment includes dynamically adjusting the weights of the resource contention conflict matrix according to the execution delay and resource occupancy deviation in the device operation data.

3. The real-time linkage control system for multimodal equipment in a conference room according to claim 1, characterized in that, The disturbance compensation in the adaptive adjustment algorithm includes: The disturbance factors in the environmental state data are obtained, including the rate of change of light intensity, the fluctuation rate of sound decibel value, and the abrupt change value of personnel distribution density; Based on the disturbance factor, a disturbance compensation coefficient is calculated; the disturbance compensation coefficient is applied to the execution time of the corresponding device in the optimized control parameter group to generate a disturbance-compensated optimized control parameter group, and the execution time of the disturbance-compensated optimized control parameter group is adjusted by multiplying the execution time of the optimized control parameter group by the disturbance compensation coefficient.

4. The real-time linkage control system for multimodal equipment in a conference room according to claim 1, characterized in that, The construction of a closed-loop control circuit to control the coordinated operation of the multimodal device group includes: Within each control cycle, corresponding control signals are sent to each device in the multimodal device group according to the execution time and execution instructions of the optimized control parameter group; Real-time acquisition of operational status data of each device in the multimodal device group, including the completion rate of device execution instructions, execution delay, and resource usage deviation; Based on the operating status data, the linkage deviation of the multimodal device group in the current control cycle is calculated. The linkage deviation is determined by the weighted sum of the variance of the execution delay of each device in the multimodal device group, the deviation of the execution instruction completion degree, and the resource occupation deviation. Based on the aforementioned linkage deviation, feedback adjustment parameters for the closed-loop control circuit are constructed. The feedback adjustment parameters are applied to the optimized control parameter group in the next control cycle to adjust the execution time and execution instructions of the corresponding devices.

5. The real-time linkage control system for multimodal equipment in a conference room according to claim 1, characterized in that, The dynamic update of the initial control parameter set for the next control cycle includes: Based on the equipment operation data and environmental status data of the current control cycle, predict the execution conflict risk of the next control cycle; Based on the execution conflict risk, the priority and execution time of the execution instructions of the corresponding devices in the initial control parameter group of the next control cycle are adjusted. The adjustment includes increasing the execution time offset of the lower priority devices or reducing the number of execution instructions of the higher priority devices.

6. A real-time linkage control system for multimodal equipment in a conference room according to claim 2, characterized in that, The method for calculating the element values ​​of the resource competition conflict matrix includes: Obtain the overlap of the resource demand vectors of any two devices in the multimodal device group; Obtain the difference in occupancy priority between the two devices, wherein the occupancy priority is determined by the task urgency and device type weight in the device operation data; Obtain the compatibility of the resource types of the two devices; The element values ​​of the resource contention conflict matrix are obtained by normalizing the weighted sum of the overlap, the occupancy priority difference, and the compatibility.

7. The real-time linkage control system for multimodal equipment in a conference room according to claim 1, characterized in that, The method for calculating the operational margin includes: Obtain the task queue length, response latency, and power consumption fluctuation rate from the device's operating data; The task queue length, the response latency, and the power consumption fluctuation rate are normalized to obtain the corresponding normalized values; The normalized values ​​are weighted and summed according to preset weights to obtain the operating margin. The preset weights are determined by the task execution success rate and resource utilization rate in the historical operating data of the device.

8. A real-time linkage control system for multimodal equipment in a conference room according to claim 4, characterized in that, The method for calculating the linkage deviation includes: Obtain the execution latency of each device in the multimodal device group; Calculate the variance of the execution delay of all devices in the multimodal device group, and use it as the first component of the linkage deviation; obtain the execution instruction completion degree of each device in the multimodal device group; Calculate the deviation of the execution instruction completion rate of all devices in the multimodal device group, and use it as the second component of the linkage deviation; Obtain the resource occupancy deviation of each device in the multimodal device group; calculate the weighted sum of the resource occupancy deviations of all devices in the multimodal device group, and use it as the third component of the linkage deviation; The weighted sum of the first component, the second component, and the third component is normalized to obtain the linkage deviation.

9. A real-time linkage control system for multimodal equipment in a conference room according to claim 1, characterized in that, The multimodal device group includes a conference display screen, microphone array, intelligent lighting, and air conditioning equipment; the environmental sensor group includes a light sensor, a sound sensor, and an infrared sensor; the system also includes the following adaptive optimization steps: Obtain historical operating data for each device in the multimodal device group. The historical operating data includes the device's execution latency distribution, resource usage distribution, and task execution success rate within a preset historical period. Based on the historical operating data, construct an operating characteristic model for each device; Based on the operational characteristic model, predict the potential execution bottlenecks of the multimodal device group in the next control cycle; For the device corresponding to the potential execution bottleneck, adjust the execution time or execution priority of the execution instructions of the corresponding device in the optimization control parameter group. The adjustment includes moving the execution time of the device forward or increasing the execution priority of the device. The forward amount or priority increment is determined by the ratio of the average execution delay in the running characteristic model to a preset delay threshold.

Images