Conference room multi-mode equipment real-time linkage control system

Abstract

The invention belongs to the technical field of automatic control, and discloses a conference room multi-mode equipment real-time linkage control system. Comprising a multi-mode equipment group, a control server and an environment sensor group. The system solves the problem of collaborative conflicts among devices in a traditional conference system by constructing a device execution conflict model, and the model realizes accurate identification and prediction of device operation conflicts based on a resource competition conflict matrix and an action mutual exclusion conflict vector. And control parameters are optimized, so that the system has environmental adaptability. Equipment linkage operation and parameter dynamic optimization are achieved through a closed-loop control loop, and linkage deviation can be evaluated and corrected in real time through closed-loop feedback adjustment. In the execution process of the system, seamless cooperation of multiple devices is achieved through the technical means of operation margin calculation, time offset adjustment and the like, and the intelligent level of conference room environment control is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic control, more particularly, the present application relates to a conference room multi-modal equipment real-time linkage control system. BACKGROUND

[0002] With the rapid development of intelligent buildings and office environments, modern conference rooms have evolved from simple communication spaces into complex intelligent systems that integrate audio-video display, voice interaction, environmental regulation, information processing, and other functions.

[0003] However, existing conference room multi-modal equipment control systems mainly lack effective closed-loop feedback control mechanisms and multi-device coordination scheduling capabilities. In actual conference scenarios, when a presenter begins to demonstrate video content, traditional systems cannot automatically coordinate the operating parameters of the projection equipment and lighting systems, resulting in reduced clarity of the projected content due to excessive ambient light. At the same time, there is a lack of synchronization control between the audio system and the video system, causing sound and picture to be out of sync. Resource competition and action conflict between these devices are particularly prominent, such as the microphone array and the environmental noise suppression system competing for audio processing resources, causing system response delays; the control logic of intelligent lighting and air conditioning systems interfere with each other, causing the conference environment to be hot and cold, and the brightness to be bright and dark. In addition, existing control strategies generally use static preset parameters and cannot adapt in real time to dynamic changes in the environment during the conference (such as changes in natural light, changes in personnel density, fluctuations in environmental noise, etc.), and the conference experience is heavily dependent on manual intervention. In multi-person video conferences, due to the lack of precise execution timing coordination mechanisms, the device startup and switching process is chaotic, causing signal interruptions, picture freezing, and other disturbances; the system also lacks monitoring and prediction capabilities for device operating states, making it impossible to identify potential performance bottlenecks and adjust control parameters in advance, resulting in sudden problems such as device response delays or function failures at critical moments of the conference. These technical defects not only affect conference efficiency, but also hinder the application expansion of intelligent conference systems in high-demand scenarios.

[0004] In view of this, the present application proposes a conference room multi-modal equipment real-time linkage control system to solve the above problems. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purposes, the present application provides the following technical solutions: a conference room multi-modal equipment real-time linkage control system, comprising:

[0006] a multi-modal equipment group, a control server, and an environmental sensor group, the multi-modal equipment group including at least two different modal devices, and the system achieving real-time linkage control through the following methods:

[0007] obtaining environmental state data and device operating data of the conference room within each control cycle;

[0008] According to the environmental state data and the preset conference scene demand, an initial control parameter group for the multi-modal device group in each control period is generated, the initial control parameter group comprising execution instructions and execution time of each device;

[0009] According to the device operation data and the initial control parameter group, an execution conflict model of the multi-modal device group is constructed, the execution conflict model detecting whether there is an execution conflict in the multi-modal device group through closed-loop feedback adjustment;

[0010] Based on the execution conflict model, the initial control parameter group is optimized through an adaptive adjustment algorithm to generate an optimized control parameter group, the adaptive adjustment algorithm comprising dynamic weight distribution based on device operation data and disturbance compensation based on environmental state data;

[0011] According to the optimized control parameter group and the feedback of the real-time collected device operation data, a closed-loop control loop is constructed to control the linkage operation of the multi-modal device group and dynamically update the initial control parameter group of the next control period.

[0012] The technical effect and advantages of the conference room multi-modal device real-time linkage control system of the application are as follows:

[0013] The application solves the core pain points of multi-device collaborative control in traditional conference systems. In actual application, when the speaker switches from demonstration to video playback, the system can seamlessly coordinate the operation states of devices such as projection, lighting, and sound, making the environment transition natural and smooth, and participants cannot feel the traces of technical switching. The problems commonly seen in traditional conferences, such as "sound and picture out of sync", "lighting and projection brightness conflict", and "environment adjustment and audio system mutual interference", are comprehensively solved, improving the professionalism and immersion experience of the conference. The intelligent perception and prediction ability of the system to environmental changes makes the conference process more comfortable, and whether it is sudden natural light change or personnel density fluctuation, it can be adjusted in time without interfering with the conference process. The forward-looking coordination control ability of the system makes the device operation more coordinated and consistent, eliminating the embarrassing situation caused by different response speeds of devices in traditional systems, such as projection has started but lighting is not dark, audio has played but display is not ready, etc. At the same time, the self-diagnosis and adaptive optimization ability of the system greatly reduces the maintenance difficulty and frequency, and changes from passive response to active prevention. With continuous operation, the "understanding" of various conference scenes is continuously deepened, and the control strategy is more and more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a schematic diagram of a conference room multi-modal device real-time linkage control system of the application;

[0015] Figure 2A logic diagram for realizing real-time linkage control of multi-modal equipment in a conference room according to the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0017] Referring to Figure 1 The present application provides a real-time linkage control system for multi-modal equipment in a conference room, which comprises a multi-modal equipment group, a control server and an environment sensor group. The multi-modal equipment group comprises at least two different modal devices, such as a conference display screen, a microphone array, intelligent lighting and air conditioning equipment. The control server is used for processing environment state data and equipment operation data, generating control parameters and performing linkage control. The environment sensor group comprises light sensors, sound sensors and infrared sensors, etc., for collecting conference room environment state data.

[0018] The present application realizes intelligent adjustment of the conference room environment through cooperative control of multi-modal equipment, constructs an equipment execution conflict model to provide a basis for subsequent optimization, dynamically allocates weights based on equipment operation data and compensates for disturbances based on environment state data to make the control adaptive, a closed-loop feedback adjustment can identify execution conflicts in different conference scenarios and dynamically adjust, trend prediction is combined with preset conference scenario requirements to improve the forward-looking nature of the control, optimization of the generation and issuance of control parameters ensures the real-time response capability of the system, a resource competition conflict matrix and an action mutual exclusion conflict vector solve the conflict problem of equipment cooperation in traditional conference systems, dynamic execution weight calculation and time offset adjustment ensure seamless cooperation between multi-modal equipment, and the construction of a closed-loop control loop realizes dynamic optimization of linkage operation parameters.

[0019] Referring to Figure 2 A logic diagram for realizing real-time linkage control of multi-modal equipment in a conference room according to the present application. In the embodiments of the present application, the system realizes real-time linkage control through the following steps:

[0020] Step 1, obtaining environment state data and equipment operation data of the conference room in each control cycle, the environment state data is collected by the environment sensor group, and the equipment operation data is generated by each device in the multi-modal equipment group;

[0021] Step 2, generating an initial control parameter group for the multi-modal equipment group in each control cycle according to the environment state data and preset conference scenario requirements, the initial control parameter group comprises execution instructions and execution times of each device;

[0022] Step 3, based on the device operation data and the initial control parameter set, an execution conflict model of the multi-modal device group is constructed, and the execution conflict model detects whether there is an execution conflict in the multi-modal device group through closed-loop feedback adjustment;

[0023] Step 4, based on the execution conflict model, the initial control parameter set is optimized through an adaptive adjustment algorithm to generate an optimized control parameter set, and the adaptive adjustment algorithm includes dynamic weight distribution based on device operation data and disturbance compensation based on environment state data;

[0024] Step 5, according to the optimized control parameter set and the feedback of the real-time collected device operation data, a closed-loop control loop is constructed to control the linkage operation of the multi-modal device group, and the initial control parameter set of the next control cycle is dynamically updated.

[0025] In this embodiment, first, various types of environment state data are acquired from the conference room environment, including brightness values, color temperature values, and light distribution data of light sensors, decibel values, sound source directions, and sound spectrum analysis data of sound sensors, personnel position data, number of people statistics, and activity heat distribution of infrared sensors, forming an environment state data set. At the same time, the operation data of each device in the multi-modal device group is collected, including device working state (on / off / standby), power level, response time, and execution quality index, forming a device operation data set. Through the data processing unit built-in the control server, the collected environment state data and device operation data are preliminarily processed, including data normalization, outlier filtering, and signal smoothing operations, to ensure data quality. The processed data will be used as the basis for subsequent control decisions.

[0026] Then, according to the processed environment state data and the pre-configured conference scene requirements, an initial control parameter set is generated. Conference scene requirements include specific requirements for environmental conditions and device states for different conference types (such as speeches, discussions, video conferences, and training), such as ideal lighting conditions, sound pickup sensitivity, and display content clarity. Based on the difference between the current environment state and the target state, the control server combines device capability parameters to calculate the specific instruction content and execution time required for each device, forming an initial control parameter set. The initial control parameter set is a structured data set containing execution instructions for each device (such as light brightness adjustment values, air conditioner temperature set values, display screen content switching commands, etc.) and corresponding execution time points or time windows, providing a preliminary execution plan for the coordinated work of devices.

[0027] According to the device operation data and the initial control parameter set, an execution conflict model of the multi-modal device group is constructed. The execution conflict includes a competition conflict of at least two devices for the same resource in the same control cycle or an action exclusion conflict of at least two devices. The resource competition conflict is, for example, that multiple devices need to use network bandwidth, power resources or processor resources at the same time; the action exclusion conflict is, for example, that the microphone needs to be muted when the display screen switches content, or the air conditioner wind speed needs to be reduced when the light is dimmed, and the like. The execution conflict model is constructed by a resource demand vector, a resource competition conflict matrix and an action exclusion conflict vector, potential execution conflicts are detected in real time, and the model parameters are continuously updated and optimized through closed-loop feedback adjustment, so as to improve the accuracy of conflict prediction.

[0028] Based on the constructed execution conflict model, the system optimizes the initial control parameter set through an adaptive adjustment algorithm to generate an optimized control parameter set. The adaptive adjustment algorithm includes two core mechanisms: dynamic weight distribution based on device operation data and disturbance compensation based on environmental state data. The dynamic weight distribution dynamically adjusts the execution priority of different devices according to the current operation margin of the device (calculated from parameters such as task queue length, response delay and power consumption fluctuation rate), to ensure reasonable allocation of system resources. The disturbance compensation mechanism monitors sudden changes in the environment (such as sudden changes in light intensity, sudden increases in sound decibels, changes in personnel density, etc.), and adjusts the control parameters in real time by calculating the disturbance compensation coefficient, to improve the adaptability of the system to environmental changes. The optimized control parameter set maximally reduces execution conflicts while ensuring the realization of system functions, and improves the coordination efficiency of multi-modal devices.

[0029] Finally, according to the optimized control parameter set and the feedback of real-time collected device operation data, the system constructs a closed-loop control loop to realize the linkage operation control of the multi-modal device group. The control server sends control signals to each device according to the execution time and instruction content specified in the optimized control parameter set, to drive the device to execute corresponding operations. At the same time, the system continuously collects the actual operation state data of the device, including the instruction execution completion degree, the execution delay and the resource occupation situation, and calculates the linkage deviation between the actual execution effect and the expectation. Based on the linkage deviation, the system dynamically adjusts the control parameters to form a closed-loop control, to ensure that the coordination operation effect of the multi-modal device meets the expectation. In addition, the system also predicts the execution conflict risk that may occur in the next control cycle according to the execution situation of the current control cycle, to adjust the initial control parameter set of the next cycle in advance, to realize the continuous optimization of the control process.

[0030] In the embodiment of the application, the execution conflict model of the multi-modal device group is constructed, including:

[0031] The resource demand vector of each device in each control cycle in the multi-modal device group is obtained, and the resource demand vector includes the occupation time length, the occupation priority and the resource type of the device for the shared resource;

[0032] According to the resource demand vector, a resource competition conflict matrix is constructed, and the element value of the resource competition conflict matrix is weighted calculated by the overlap degree of the resource demand vectors of any two devices in the multi-modal device group, the difference of the occupation priority, and the compatibility of the resource type;

[0033] According to the initial control parameter group, an action mutual exclusion conflict vector is constructed, and the element value of the action mutual exclusion conflict vector is 1, indicating that the execution actions of the corresponding two devices are mutually exclusive, and the value is 0, indicating that they are not mutually exclusive. The action mutual exclusion is determined by the spatial occupation range and the time overlap degree of the execution instructions of the two devices.

[0034] Based on the resource competition conflict matrix and the action mutual exclusion conflict vector, an execution conflict model is constructed, and the execution conflict model is updated in real time through closed-loop feedback adjustment, which includes dynamically adjusting the weight of the resource competition conflict matrix according to the execution delay and resource occupation deviation in the device running data.

[0035] In this embodiment, first, detailed resource demand information is obtained from each device in the multi-modal device group. For the conference display screen, the amount of demand for network bandwidth (Mbps value), the consumption of power resources (W value), and the occupation rate of processor resources (%) value) are collected; for the microphone array, the demand for audio processing resources (computational complexity value), the demand for network transmission channels (channel number), and the access frequency to shared storage (times / second value) are collected; for the intelligent light, the demand for power resources (W value), the occupation of control channels (channel number), and the demand for feedback signal processing resources (computational amount) are collected; for the air conditioning device, the consumption of power resources (W value), the processing demand for temperature sensing data (data amount), and the requirement for control accuracy (accuracy level) are collected. These resource demands are obtained in real time through the built-in resource monitoring module of the device, and the sampling frequency is dynamically adjusted according to the resource change rate.

[0036] The obtained resource demand information is structured and processed to form a resource demand vector. The resource demand vector is a multi-dimensional data structure, which includes three key dimensions: occupation time, indicating the continuous occupation time of the device to a specific resource, in milliseconds or seconds; occupation priority, indicating the urgency of the device's demand for resources, usually represented by a value of 1-10, the higher the value, the higher the priority; resource type, indicating the category of the required resource, such as network resource, power resource, computing resource, etc. The resource demand vector is standardized to eliminate the dimensional differences between different resource types, forming a unified format for comparison, providing basic data for subsequent conflict analysis.

[0037] Based on the resource demand vectors of each device, a resource competition conflict matrix is ​​constructed. This matrix is ​​an n×n square matrix (n is the number of devices), and each element aij in the matrix represents the degree of resource competition conflict between device i and device j. The calculation of the element values ​​comprehensively considers three factors: the overlap of resource demand vectors, obtained by calculating the ratio of the intersection to the union of the resource occupancy of the two devices in the time dimension, reflecting the time range of resource competition; the difference in occupancy priority, obtained by calculating the absolute difference in the resource occupancy priority of the two devices, reflecting the degree of priority conflict; and the compatibility of resource types, obtained by determining whether the resources required by the two devices belong to the same resource pool, with a compatibility of 0 for completely identical resource types and a compatibility of 1 for completely different types. These three factors are weighted and summed, and after normalization, the final matrix element values ​​are formed, with a value range of [0,1]. The larger the value, the higher the degree of conflict.

[0038] Simultaneously, based on the execution instruction information in the initial control parameter set, an action mutual exclusion conflict vector is constructed. This vector describes the mutual exclusion relationship between the actions executed by the device and is a... This is a 3D vector (where n is the number of devices), where each element corresponds to a pair of devices. Element values ​​are binary: 1 indicates that the actions performed by the two devices are mutually exclusive, and 0 indicates that they are not mutually exclusive. The determination of mutual exclusion is based on two key factors: the spatial occupancy of the execution command, referring to the physical space affected by the device's action, such as the area covered by lighting or the audio pickup range; and the temporal overlap, referring to the degree of overlap in the time windows of the two devices' execution commands. When the execution commands of two devices significantly overlap spatially and are executed synchronously in time, they are considered mutually exclusive, and the corresponding vector element value is set to 1; otherwise, it is set to 0. Typical mutually exclusive scenarios include: microphones needing to be muted to avoid echoes when a display is playing video; display brightness needing to be increased to ensure visibility when lights are dimmed; and light heat output needing to be reduced to maintain temperature balance when an air conditioner is cooling.

[0039] Based on the constructed resource contention conflict matrix and action mutual exclusion conflict vectors, the system integrates to form a complete execution conflict model. This model uses a graph structure, with devices as nodes and conflict relationships as edges, the weight of which is determined by the degree of conflict. The model supports two conflict detection methods: resource contention conflict detection, which identifies possible resource contention situations through the resource contention conflict matrix; and action mutual exclusion conflict detection, which identifies incompatible combinations of device operations through action mutual exclusion conflict vectors. The conflict detection results are used to guide subsequent control parameter optimization, avoiding resource contention or operational conflicts during system execution.

[0040] The execution conflict model is updated and optimized in real time through a closed-loop feedback adjustment mechanism. The system continuously monitors the actual operation of the device, collects execution delay data (the difference between the actual time of instruction execution and the planned time) and resource occupation deviation (the difference between the actual resource occupation and the expected demand), and uses these feedback data to dynamically adjust the weight coefficients of the resource competition conflict matrix. When the actual impact of competition for a certain type of resource is greater than expected, the corresponding weight is increased; otherwise, the weight is decreased. Through this adaptive adjustment mechanism, the execution conflict model can continuously learn and adapt to the operating characteristics of the system, improving the accuracy of conflict prediction and the rationality of control decisions.

[0041] In the embodiments of the present application, the initial control parameter set is optimized by an adaptive adjustment algorithm to generate an optimized control parameter set, which includes:

[0042] According to the execution conflict model, a pair of conflicting devices is determined, the pair of conflicting devices including at least two devices in the multi-modal device group that have execution conflicts;

[0043] For the pair of conflicting devices, the real-time operating margin of each device is obtained, and the operating margin is calculated by the weighted sum of the task queue length, response delay and power fluctuation rate in the device operation data;

[0044] According to the operating margin, the dynamic execution weight of each device in the pair of conflicting devices is calculated, and the dynamic execution weight is obtained by normalizing the product of the operating margin and the preset execution priority;

[0045] Based on the dynamic execution weight, the execution instruction of the pair of conflicting devices in the initial control parameter set is adjusted by time offset, and the time offset is realized by the following steps:

[0046] The minimum delay requirement of the execution instruction of the device with lower priority in the pair of conflicting devices is obtained, and the minimum delay requirement is determined by the weighted sum of the element value of the resource competition conflict matrix and the element value of the action mutual exclusion conflict vector;

[0047] According to the operating margin, the execution margin compensation value of the device with lower priority in the pair of conflicting devices is calculated, and the execution margin compensation value is negatively related to the operating margin;

[0048] The larger value between the sum of the minimum delay requirement and the execution margin compensation value and the preset minimum delay interval is taken as the time offset;

[0049] Based on the dynamic execution weight and the time offset, an optimized control parameter set is generated.

[0050] In this embodiment, first, according to the analysis result of the execution conflict model, the device pairs that may have execution conflicts are identified. The determination of the conflict device pairs is based on two key indicators: the device pairs whose corresponding element values in the resource competition conflict matrix exceed the preset threshold (such as 0.7) indicate that there is a significant risk of resource competition; the device pairs whose element values in the action mutual exclusion conflict vector are 1 indicate that there is a certain action mutual exclusion relationship. The system generates a conflict device pair list by comprehensively considering these two indicators, each item in the list contains two device identifiers of the conflict, the conflict type (resource competition / action mutual exclusion / both), the conflict severity (quantitative score based on matrix element value) and the conflict time window (calculated according to the execution time in the initial control parameter set). These conflict device pairs will be the focus of subsequent parameter optimization.

[0051] For each pair of conflict devices identified, the system obtains the running margin data of each device in real time. The running margin is a comprehensive indicator to measure the current execution capability and adaptability of the device, which is calculated by three key parameters: task queue length, which represents the number of instructions currently to be processed by the device, the longer the queue, the heavier the load, and the lower the margin; response delay, which represents the time interval from receiving an instruction to starting execution, the larger the delay, the slower the response, and the lower the margin; power fluctuation rate, which represents the degree of change in the power consumption of the device, the greater the fluctuation, the more unstable the working state, and the lower the margin. The system obtains the real-time values of these parameters through the internal monitoring interface of the device, and calculates the comprehensive running margin value through a preset weighted calculation formula. The value range is usually normalized to [0, 1], and the higher the value, the better the device running state and the stronger the adjustment ability.

[0052] Based on the calculated running margin, the system further calculates the dynamic execution weight of each device in the conflict device pair. The dynamic execution weight determines the priority and resource allocation ratio of each device in the conflict resolution process. The calculation process considers two factors: running margin, which reflects the current execution capability and adjustment capability of the device; preset execution priority, which is the basic priority pre-configured according to the device type, functional importance and business demand. The dynamic execution weight is calculated by the product of the running margin and the preset execution priority, and then normalized within the conflict device pair to ensure that the weight sum is 1. This calculation method takes into account both the inherent importance of the device and its current running state, making resource allocation more reasonable and dynamic.

[0053] Based on the dynamic execution weight, the system adjusts the execution instruction of the conflicting device pair in the initial control parameter group in time offset. The core of the time offset adjustment is to reasonably stagger the execution time of the conflicting devices to reduce or eliminate resource competition and action mutual exclusion conflict. The adjustment process first determines the device with lower priority (the device with smaller dynamic execution weight), and then calculates the minimum time delay required for its execution instruction. The minimum delay requirement is determined by the conflict degree (the weighted sum of the resource competition conflict matrix element value and the action mutual exclusion conflict vector element value), the higher the conflict degree, the longer the minimum delay required. At the same time, the system considers the running margin of the device and calculates the execution margin compensation value. The compensation value is negatively related to the running margin, the higher the running margin, the smaller the compensation value, indicating that the device has stronger ability to adapt to time adjustment; otherwise, the device with low running margin needs a larger compensation value to reduce the additional burden brought by time adjustment. The system adds the minimum delay requirement and the execution margin compensation value, and compares it with the preset minimum delay interval, and takes the larger value as the final time offset.

[0054] Finally, the system generates an optimized control parameter group based on the determined dynamic execution weight and time offset. The optimization process mainly involves two aspects of adjustment: time adjustment, according to the calculated time offset, the execution time of the device with lower priority in the conflicting device is modified to avoid conflict; resource allocation adjustment, according to the dynamic execution weight, the allocation proportion of resources of the conflicting device is adjusted to ensure that the high-weight device obtains sufficient resources. The optimized control parameter group maintains the same data structure as the initial control parameter group, but contains the adjusted execution time and resource allocation parameters. The system verifies the optimization effect by comparing the conflict detection results before and after optimization, to ensure that the adjusted parameters can effectively reduce or eliminate execution conflicts and improve the overall coordination efficiency of the system.

[0055] In the embodiment of the application, the disturbance compensation in the adaptive adjustment algorithm comprises:

[0056] Obtain the disturbance factor in the environment state data, the disturbance factor including the change rate of light intensity, the fluctuation rate of sound decibel value and the mutation value of personnel distribution density;

[0057] According to the disturbance factor, calculate the disturbance compensation coefficient, the disturbance compensation coefficient being determined by the ratio of the weighted sum of the disturbance factor and the preset disturbance threshold value;

[0058] Apply the disturbance compensation coefficient to the execution time of the corresponding device in the optimized control parameter group to generate the optimized control parameter group after disturbance compensation, the execution time of the optimized control parameter group after disturbance compensation being adjusted by the product of the execution time of the optimized control parameter group and the disturbance compensation coefficient.

[0059] In this embodiment, first, the key disturbance factors are extracted from the environmental state data, which reflect the sudden changes or abnormal fluctuations in the conference room environment. The rate of change of light intensity, calculated as the relative change in light intensity per unit time (e.g., 10 seconds), is expressed as a percentage value, for example, from 500 lux to 300 lux suddenly, the change rate is -40%; the fluctuation rate of sound decibel value, calculated as the ratio of standard deviation to mean value of sound decibel value per unit time, reflects the stability of the sound environment, the larger the value, the more intense the sound environment fluctuation; the mutation value of personnel distribution density, calculated as the change amplitude of the number or distribution state of personnel in the conference room per unit time, such as a sudden increase of 50% in the number of people or a change from centralized to decentralized distribution, etc. These disturbance factors are collected and calculated in real time by the environmental sensor group, and the sampling frequency is dynamically adjusted according to the rate of environmental change, usually 1-10 times per second.

[0060] Based on the extracted disturbance factors, the system calculates the disturbance compensation coefficient. The calculation process first normalizes each disturbance factor to eliminate the dimensional differences between different types of disturbances. Then, according to the importance of different disturbances on the operation of the equipment, weight coefficients are assigned, such as the light change has a greater impact on the display screen and the light, and a higher weight is assigned; the sound fluctuation has a greater impact on the microphone array, and a higher weight is also assigned. Through weighted summation, a comprehensive disturbance index is obtained. The final calculation of the disturbance compensation coefficient is to compare the comprehensive disturbance index with the preset disturbance threshold, calculate the ratio of the two, and then convert the ratio to a compensation coefficient in the range of [0.5, 2] through a mapping function (such as sigmoid function). When the disturbance is small, the compensation coefficient is close to 1, indicating that almost no adjustment is needed; when the disturbance is significant, the compensation coefficient may be close to 0.5 or 2, indicating that a large adjustment is needed to adapt to the environmental change.

[0061] The calculated disturbance compensation coefficient is applied to the execution time of the corresponding device in the optimized control parameter group, generating the disturbance-compensated optimized control parameter group. The application process varies according to the type of disturbance and the characteristics of the device: for the change of light intensity, mainly affecting the display screen and the light device, the system adjusts the response time of these devices according to the direction of light change (enhancement or weakening), such as when the light suddenly increases, the execution time of the display screen brightness adjustment instruction may need to be advanced to quickly reduce the brightness to prevent glare; for sound environment fluctuation, mainly affecting the microphone array and audio devices, the system adjusts the gain adjustment speed and noise reduction parameter update frequency of these devices according to the fluctuation degree; for personnel distribution change, it may affect all devices, the system adjusts the execution rhythm of the environmental control strategy according to the mode and degree of personnel change. The disturbance-compensated optimized control parameter group maintains the same data structure as the original parameter group, but the execution time is adjusted by multiplying the original execution time by the disturbance compensation coefficient, and the compensation coefficient greater than 1 indicates delayed execution, and less than 1 indicates advanced execution.

[0062] The disturbance compensation mechanism enables the system to actively adapt to environmental changes, improving the robustness and adaptability of control. Typical application scenarios include: sudden changes in external light during a meeting (such as cloud cover blocking sunlight or turning on outdoor lights), the system quickly adjusts indoor light and display screen parameters to maintain a stable visual experience; sudden noise during a meeting (such as temporary construction or external commotion), the system quickly adjusts the pickup mode and gain of the microphone array to ensure clear voice capture; temporary changes in the number of people or layout in the meeting, the system adjusts the air conditioning outlet direction and temperature setting accordingly to maintain comfortable environmental conditions. Through this dynamic compensation mechanism, the system can maintain stable and efficient control effects in complex and variable meeting environments, improving user experience and meeting efficiency.

[0063] In the embodiments of the present application, a closed-loop control loop is constructed to control the linkage operation of the multi-modal device group, comprising:

[0064] In each control cycle, according to the execution time and execution instructions of the optimized control parameter group, corresponding control signals are sent to each device in the multi-modal device group;

[0065] Real-time acquisition of the running state data of each device in the multi-modal device group, the running state data including the completion degree, execution delay and resource occupation deviation of the device execution instructions;

[0066] According to the running state data, the linkage deviation of the multi-modal device group in the current control cycle is calculated, and the linkage deviation is determined by the weighted sum of the variance of the execution delay, the deviation of the execution instruction completion degree and the resource occupation deviation of each device in the multi-modal device group;

[0067] Based on the linkage deviation, the feedback adjustment parameter of the closed-loop control loop is constructed, and the feedback adjustment parameter is normalized by the ratio of the linkage deviation to the preset deviation threshold;

[0068] The feedback adjustment parameter is applied to the optimized control parameter group of the next control cycle to adjust the execution time and execution instructions of the corresponding device.

[0069] In this embodiment, at the beginning of each control cycle (usually 100-500 milliseconds), the control server sends control signals to each device in the multi-modal device group according to the specified rules in the optimization control parameter set. The control signals are transmitted through standardized communication interfaces and encapsulated using communication protocols supported by the devices (such as ModBus, KNX, HTTP, or proprietary protocols). The signal content includes two core parts: execution instructions, which explicitly specify the specific operations that the device needs to perform, such as switching the display screen to a specific content, adjusting the light to a specific brightness and color temperature, enabling a specific pickup mode for the microphone, adjusting the air conditioner to a specific temperature and wind speed, etc.; execution time, which specifies the execution time or time window of the instruction, which can be an absolute time point (such as 12:05:30.500) or a relative time point (such as the current time + 2 seconds). The control signals are sent in priority order, with high-priority device signals sent first to ensure that critical devices can respond in a timely manner. Signal integrity checks and confirmation mechanisms are implemented during the sending process to ensure that the control signals accurately reach the target devices.

[0070] At the same time, the system collects real-time operation state data of each device in the multi-modal device group. The collection frequency is dynamically adjusted according to device characteristics and monitoring requirements, with high-frequency collection (such as 10 times per second) for critical devices and key state parameters, and low-frequency collection (such as 1 time per second) for general parameters. The collected operation state data includes three types of key information: completion degree of execution instructions, indicating the matching degree between the actual execution of the device and the requirements of the instructions, usually expressed as a percentage, such as 90% completion of display screen content switching, 95% of light brightness adjustment reaching the target value, etc.; execution delay, indicating the time interval from receiving the instruction to starting execution, as well as the deviation from the planned timeline during execution, in milliseconds; resource occupation deviation, indicating the difference between the actual resource occupation of the device and the expected resource demand, such as 20% higher actual network bandwidth usage than expected, 15% lower processor load than expected, etc. These data are collected through the device's state feedback interface and uploaded to the control server after preliminary processing (such as outlier filtering, data smoothing).

[0071] Based on the collected operating state data, the system calculates the linkage deviation of the multi-modal device group in the current control period. The linkage deviation is a comprehensive indicator to measure the collaborative operation effect of multiple devices, which is calculated through three sub-indicators: variance of execution delay, statistical variance of all device execution delays, reflecting the consistency of time synchronization between devices, the larger the variance, the more out of sync the device execution time; execution instruction completion degree deviation, calculate the difference between the completion degree of all devices and the preset completion degree threshold (usually 100%), then calculate the root mean square of these differences, reflecting the deviation of the overall execution quality; weighted sum of resource occupation deviation, the resource occupation deviation of each device is weighted and summed according to the importance of resources, reflecting the rationality of resource allocation. After normalization processing of the three sub-indicators, they are weighted and summed again (weights are dynamically adjusted according to control targets) to get the final linkage deviation value, the numerical range is usually [0, 1], the closer to 0, the better the linkage effect.

[0072] Based on the calculated linkage deviation, the system constructs the feedback adjustment parameters of the closed-loop control loop. Feedback adjustment parameters are the core mechanism of system self-adjustment and optimization, used to guide the direction and amplitude of parameter adjustment in the next control period. The calculation process first compares the linkage deviation with the preset deviation threshold (determined according to application scenarios and control accuracy requirements), calculates the ratio of the two, and then converts the ratio to a standardized feedback adjustment parameter through a mapping function (such as linear mapping or nonlinear mapping), the numerical range is usually [-1, 1], negative value means to reduce or delay, positive value means to increase or advance, absolute value represents the adjustment amplitude. For example, when the execution delay variance is large, a negative feedback parameter for time synchronization may be generated to guide the system to enhance time coordination in the next period; when the resource occupation deviation is significant, an adjustment parameter for resource allocation may be generated to guide the system to rebalance the resource allocation ratio.

[0073] Finally, the system applies the feedback adjustment parameters to the optimization control parameter set of the next control period, realizing dynamic adjustment and continuous optimization of the control process. The application process mainly involves two aspects: execution time adjustment, according to the feedback parameters related to time synchronization, adjust the planned time of device execution instructions, such as advancing or delaying the operation time of specific devices, or adjusting the time interval between devices; execution instruction adjustment, according to the feedback parameters related to execution quality and resource allocation, modify the execution instruction content or parameters of the device, such as adjusting the gradient time of light change, the gain level of microphone or the temperature change rate of air conditioner, etc. The adjustment process adopts a gradual strategy to avoid system instability caused by large-scale adjustment, usually limiting the single adjustment amplitude within a preset range (such as ±10%). Through this closed-loop feedback mechanism, the system can continuously learn and adapt to the operating environment and device characteristics, continuously optimize the control effect, and improve the collaborative efficiency of multi-modal devices and user experience.

[0074] In the embodiment of the present application, the initial control parameter set of the next control period is dynamically updated, including:

[0075] According to the device running data and environmental state data of the current control period, the execution conflict risk of the next control period is predicted, and the execution conflict risk is determined by the element value change rate of the resource competition conflict matrix and the element value distribution of the action mutual exclusion conflict vector.

[0076] According to the execution conflict risk, the priority and execution time of the execution instruction of the corresponding device in the initial control parameter set of the next control period are adjusted, and the adjustment includes increasing the execution time offset of the device with lower priority or reducing the number of execution instructions of the device with higher priority.

[0077] In the embodiment, first, based on the device running data and environmental state data of the current control period, the system predicts the execution conflict risk that may occur in the next control period. The prediction process uses a trend analysis method, focusing on two key indicators: the element value change rate of the resource competition conflict matrix, which is obtained by calculating the change trend of each element in the matrix in consecutive control periods, a positive change rate indicates that the conflict trend is increasing, and a negative change rate indicates that the conflict trend is decreasing; the element value distribution of the action mutual exclusion conflict vector, which analyzes the distribution pattern and concentration degree of the elements with a value of 1 (indicating mutual exclusion) in the vector, and the more concentrated the distribution, the more concentrated the conflict risk on a specific device combination. Combined with these two indicators, the system constructs an execution conflict risk assessment model to calculate the risk value of each device in the next control period that may occur conflict, the risk value is usually represented by a value between 0 and 1, and the larger the value, the higher the conflict possibility.

[0078] The assessment model also considers the influence of the change trend of the current environmental state on the conflict risk. For example, a continuous decrease in light intensity may lead to an increase in the coordination demand between the display screen and the light, thereby increasing the conflict risk between them; an increase in the noise level in the conference room may increase the processing complexity of the microphone array, thereby increasing the resource competition risk with other devices; changes in personnel distribution may affect the running parameters of multiple devices, indirectly increasing the coordination complexity and conflict possibility between devices. Through this comprehensive analysis, the system can identify potential conflict hotspots in advance to provide a basis for preventive adjustment.

[0079] Based on the predicted execution conflict risk, the system adjusts the initial control parameter set of the next control cycle. The adjustment strategy mainly includes two aspects: priority adjustment, according to the distribution and severity of the conflict risk, dynamically adjusting the priority of the device execution instruction, improving the priority of the key device with high conflict risk, and ensuring that it obtains sufficient system resources and execution guarantee; Time peak shifting, for the identified high conflict risk device pair, through time peak shifting strategy to reduce or avoid conflict, including increasing the execution time offset of the lower priority device (appropriately delaying its operation time), or reducing the execution instruction quantity of the higher priority device (reducing resource demand through instruction merging or simplification).

[0080] The determination of the time offset considers multiple factors: conflict risk value, the higher the risk, the greater the time separation needed; Device operation characteristics, such as some devices may be more sensitive or less sensitive to time adjustment; The urgency of the execution instruction, the adjustment space of the urgent instruction is smaller; System overall load, when the load is lighter, the time can be arranged more flexibly. The system calculates the optimal time offset scheme through optimization algorithm (such as greedy algorithm or heuristic search), minimizes the conflict risk under the premise of ensuring function implementation. At the same time, for the devices that may adjust the number of instructions, the system analyzes the dependency relationship and merging possibility of its execution instructions, reduces the total number of instructions and reduces resource pressure by merging similar instructions, simplifying non-critical instructions or delaying non-urgent instructions, etc. without affecting function implementation.

[0081] This predictive adjustment mechanism enables the system to actively identify and respond to potential conflicts, rather than passively responding to conflicts that have already occurred. Through forward-looking parameter optimization, the system can make preventive adjustments before conflicts occur, significantly reducing conflict events in actual operation, and improving the coordination efficiency and control stability of multi-modal devices. At the same time, predictive adjustment also reduces the system response delay and user experience decline caused by conflict processing, making the conference room environment control more smooth and natural.

[0082] In the embodiment of the application, the method for calculating the element value of the resource competition conflict matrix comprises:

[0083] Obtaining the overlap degree of the resource demand vectors of any two devices in the multi-modal device group, the overlap degree being determined by the ratio of the intersection and union of the occupation time of the resource demand vectors of the two devices in the same control cycle;

[0084] Obtaining the difference of the occupation priorities of the two devices, the occupation priority being determined by the task urgency in the device running data and the device type weight;

[0085] obtaining compatibility of resource types of the two devices, the compatibility being determined by whether the resource types of the two devices belong to the same resource pool, and a value of 1 indicating that the resource types belong to the same resource pool and a value of 0 indicating that the resource types do not belong to the same resource pool;

[0086] normalizing the weighted sum of the overlap, the priority difference and the compatibility to obtain an element value of the resource competition conflict matrix.

[0087] In this embodiment, first, the overlap of the resource demand vectors of any two devices is calculated. The overlap reflects the competition degree of the two devices on the resource in the time dimension. The calculation process first determines the occupation time intervals of the two devices on each type of resource in the current control period to form a time set. For example, the display screen needs network resources in the time period [t1, t2], and the microphone array also needs network resources in the time period [t3, t4]. Then, the intersection interval length (i.e., the overlap time length) and the union interval length (i.e., the total occupation time length of the two devices) of the two time sets are calculated, and the ratio of the two is taken as the overlap PUL. The calculation of the overlap can be expressed as: where A and B are the resource occupation time sets of the two devices, |A∩B| represents the intersection size, and |A∪B| represents the union size. The value range of the overlap is [0, 1], a value of 0 indicates complete non-overlapping (no conflict), and a value of 1 indicates complete overlapping (maximum conflict). For multiple resource types, the overlap of each type of resource is calculated, and then a weighted average is taken according to the importance of the resources to obtain the comprehensive overlap.

[0088] Then, the priority difference of the two devices is obtained. The occupation priority is a quantitative representation of the urgency of the device's demand for resources, which is determined by two factors: task urgency, which reflects the time sensitivity of the device's current task, and is calculated according to the task deadline, waiting time and user-specified priority, etc., and the value range is usually 1-10, and the larger the value, the more urgent; device type weight, which reflects the inherent importance of the device in the system, and is pre-configured according to factors such as device function, user experience impact and system dependency, and also has a value of 1-10, and the larger the value, the more important. The occupation priority is calculated by the weighted sum of the task urgency and the device type weight, and then the absolute difference of the occupation priorities of the two devices is calculated. The larger the difference, the more obvious the priority difference between the two devices, and the easier it is to make a decision in resource allocation; the smaller the difference, the closer the priority, and the more difficult it is to resolve the conflict.

[0089] Then the compatibility of the resource types of the two devices is obtained. The compatibility is used to determine whether the required resources of the two devices belong to the same resource pool, which directly affects the possibility and severity of the conflict. The determination process is based on resource type classification and resource pool configuration: first, determine the specific resource types required by each device, such as network bandwidth, processor time, storage space, power supply, etc.; then determine whether these resource types belong to the same resource pool. The resource pool is a logical unit of resource management, and the resources in the same pool are in a sharing and competing relationship, and the resources between different pools are relatively independent. If the resource types of the two devices belong to the same resource pool, the compatibility value is 1, indicating that there is a direct competition relationship; if they belong to different resource pools, the compatibility value is 0, indicating that there is no direct competition. For cases involving multiple resource types, the proportion of resource type matching is calculated as the comprehensive compatibility value.

[0090] Finally, the weighted sum of the overlap degree, the occupation priority difference ZY and the compatibility JR is normalized to obtain the element value HJ of the resource competition conflict matrix. The calculation formula can be represented as:

[0091]

[0092] where w1, w2, and w3 are weight coefficients, satisfying w1+w2+w3=1, which are dynamically adjusted according to the system control focus; MaxDiff is the maximum priority difference in the system, used for normalization processing. The calculation result is further processed by a nonlinear mapping function (such as a Sigmoid function) to ensure that the final value range is within [0, 1] and to enhance the difference. The final matrix element value intuitively reflects the degree of resource competition conflict between the two devices. The closer the value is to 1, the higher the conflict risk, and the closer the value is to 0, the lower the conflict risk.

[0093] This multi-factor weighted calculation method comprehensively considers key factors such as time overlap, priority difference and resource type, and can accurately quantify the resource competition between devices, providing accurate conflict evaluation basis for subsequent control parameter optimization. At the same time, through the dynamic adjustment of the weight coefficients, the system can flexibly change the focus of conflict evaluation according to different application scenarios and control targets, improving the adaptability and practicality of the conflict model.

[0094] In the embodiment of the application, the calculation method of the operation margin comprises:

[0095] Obtaining the task queue length, response delay and power consumption fluctuation rate in the device operation data;

[0096] Normalizing the task queue length, response delay and power consumption fluctuation rate to obtain the corresponding normalized values;

[0097] The normalized values are weighted and summed according to preset weights to obtain the operation margin, and the preset weights are determined by a task execution success rate and a resource occupation rate in historical operation data of the device.

[0098] In this embodiment, three key parameters are first extracted from the device operation data: a task queue length, indicating the number of instructions or tasks currently to be processed by the device, directly reflecting the workload of the device, the data being obtained through a task management interface of the device, and the unit being the number of tasks; a response delay, indicating the time interval from receiving an instruction to starting execution by the device, reflecting the response sensitivity and processing capacity of the device, the data being collected through a performance monitoring interface of the device, and the unit being milliseconds; and a power fluctuation rate, indicating the degree of change of power consumption of the device within a certain time window, reflecting the stability of the working state of the device, the data being collected through a power supply monitoring interface of the device, and being calculated as the ratio of the standard deviation to the mean of the power, being dimensionless. These three parameters describe the current operation state of the device from different angles and jointly constitute the basic data for calculating the operation margin.

[0099] Since the dimensions and numerical ranges of the three parameters are different, normalization processing is required to make them comparable and calculable. The normalization method is selected according to the characteristics of the parameters: for the task queue length, maximum-minimum normalization is used to map it to the [0, 1] interval, and the closer the normalized value is to 1, the shorter the queue and the lighter the load; for the response delay, exponential normalization is used, with the calculation formula being: normalized value = exp(-current value / reference delay), where the reference delay is a system preset baseline response time, and the closer the normalized value is to 1, the smaller the delay and the faster the response; and for the power fluctuation rate, threshold normalization is used, with the calculation formula being: normalized value = 1-min(current value / max allowed fluctuation rate, 1), where the max allowed fluctuation rate is a system preset fluctuation tolerance, and the closer the normalized value is to 1, the smaller the fluctuation and the more stable the state. Through these normalization processes, the three parameters are all converted to values in the [0, 1] interval, and the larger the value, the better the running state in that dimension.

[0100] Based on the normalized parameter values, the system performs weighted summation according to preset weights to calculate the final operation margin. The weight coefficients reflect the influence of different parameters on the operation capacity of the device, and are determined through historical operation data of the device: a task execution success rate, indicating the proportion of successfully completed assigned tasks in the history of the device, the higher the success rate, the more reliable the device, and the higher the weight of the corresponding parameter; and a resource occupation rate, indicating the resource utilization efficiency of the device under different loads, calculated through the relationship between resource consumption and task completion degree in the historical data, the higher the efficiency, the better the performance of the device, and the higher the weight of the corresponding parameter. The system dynamically calculates the weight coefficient of each parameter according to these two indexes and in combination with the type characteristics of the device, ensuring that the total weight is 1.

[0101] The final calculation formula of the operation margin is: operation margin=w1×task queue length normalization value+w2×response delay normalization value+w3×power consumption fluctuation rate normalization value, wherein w1, w2 and w3 are corresponding weight coefficients. The value range of the calculation result is [0, 1], and the value closer to 1 indicates that the current operation state of the device is better, and the device has greater adjustment ability and execution space; the value closer to 0 indicates that the current load of the device is heavy or the state is unstable, and the adjustment space is limited. The operation margin is a key input of the adaptive adjustment algorithm, directly affects the selection of the conflict resolution strategy and the adjustment range of the parameters, and ensures that the system can make reasonable control decisions according to the actual state of the device.

[0102] In the embodiment of the application, the calculation method of the linkage deviation comprises:

[0103] The execution delay of each device in the multi-modal device group is obtained, and the execution delay is determined by the difference between the actual execution time of the device and the planned time of the corresponding instruction in the optimization control parameter group;

[0104] The variance of the execution delay of all devices in the multi-modal device group is calculated as the first component of the linkage deviation;

[0105] The execution instruction completion degree of each device in the multi-modal device group is obtained, and the execution instruction completion degree is determined by the matching degree of the actual action of the device and the action of the corresponding instruction in the optimization control parameter group;

[0106] The deviation of the execution instruction completion degree of all devices in the multi-modal device group is calculated as the second component of the linkage deviation, and the deviation is determined by the root mean square of the difference between the execution instruction completion degree and the preset completion threshold;

[0107] The resource occupation deviation of each device in the multi-modal device group is obtained, and the resource occupation deviation is determined by the difference between the actual resource occupation time of the device and the planned occupation time in the resource demand vector;

[0108] The weighted sum of the resource occupation deviation of all devices in the multi-modal device group is calculated as the third component of the linkage deviation;

[0109] The weighted sum of the first component, the second component and the third component is normalized to obtain the linkage deviation, and the weight of the weighted sum is determined by the task urgency and the device type weight in the device operation data.

[0110] In this embodiment, first, the execution delay data of each device in the multi-modal device group is obtained. The execution delay is a key indicator to measure the accuracy of device execution time, which is calculated by comparing the actual completion time of device executing instructions with the planned time specified in the optimal control parameter set. The acquisition process includes: reading the actual start time and completion time of instruction execution from the device state feedback interface, recording the timestamp; extracting the planned execution time of the corresponding instruction from the optimal control parameter set, also using the timestamp format; calculating the difference between the actual completion time and the planned completion time to obtain the execution delay, which is in milliseconds, a positive value indicates that the actual execution lags behind the plan, and a negative value indicates that the actual execution is ahead of the plan. The execution delay directly reflects the timeliness and accuracy of the device's response to control instructions, and is an important basis for evaluating the linkage effect.

[0111] Based on the obtained execution delay data of each device, the execution delay variance of the entire multi-modal device group is calculated. The variance calculation formula is: Where Xi is the execution delay of the i-th device, μ is the average value of the execution delay of all devices, and n is the total number of devices. The execution delay variance reflects the consistency and synchronization of the execution time of the devices, the smaller the variance, the more consistent the execution time of each device, the better the linkage effect; the larger the variance, the more obvious the execution time difference, there may be a coordination problem. The calculated variance is standardized (e.g. divided by the maximum allowed variance) to obtain a standardized variance value, which is the first component of the linkage deviation.

[0112] Then the execution instruction completion degree of each device in the multi-modal device group is obtained. The completion degree is an indicator to measure the execution quality of the device, which is calculated by comparing the matching degree of the actual action executed by the device with the action required by the control instruction. The calculation method varies depending on the type of device: for discrete state devices (such as switch type devices), the completion degree is a binary value, a perfect match is 100%, and a mismatch is 0%; for continuous state devices (such as dimmable lights, adjustable speed fans, etc.), the completion degree is calculated as: completion degree = 1 - |actual value - target value| / allowable deviation range, where the allowable deviation range is the pre-set tolerance, and the value of the completion degree ranges from [0, 1], the closer the value to 1, the more accurate the execution. The actual execution state of the device is obtained through the state feedback interface, and the target state is extracted from the optimal control parameter set.

[0113] Based on the execution instruction completion degree of each device, the completion degree deviation of the entire device group is calculated. The calculation method is to compare the completion degree of each device with the pre-set completion degree threshold (usually 1 or 100%, indicating perfect execution), calculate the difference, and then take the root mean square (RMS) of all the differences. The root mean square calculation formula is: wherein Ci is the completion degree of the ith device, Cth is the preset completion degree threshold, and n is the total number of devices. The smaller the root mean square value is, the higher the overall execution quality is, and the closer to the expectation; the larger the value is, the more obvious the execution quality deviation is, and adjustment is needed. The root mean square value obtained is also subjected to standardization processing to obtain a standardized completion degree deviation value as a second component of the linkage deviation.

[0114] Then, a resource occupation deviation of each device in the multi-modal device group is obtained. The resource occupation deviation reflects the difference between the actual resource usage of the device and the planned demand, and is calculated by comparing the actual resource occupation time length with the planned occupation time length declared in the resource demand vector. The obtaining process includes: reading the actual resource occupation record from the resource monitoring interface of the device, including the start time, end time and resource type; extracting the planned occupation time length of the corresponding resource from the resource demand vector; calculating the difference between the actual occupation time length and the planned occupation time length to obtain the resource occupation deviation, which is in milliseconds, and a positive value indicates that the actual occupation exceeds the plan, and a negative value indicates that the actual occupation is less than the plan. The resource occupation deviation reflects the rationality of resource allocation and the accuracy of resource demand prediction, and is an important reference for optimizing resource utilization.

[0115] Based on the resource occupation deviation of each device, a resource occupation deviation weighted sum of the entire device group is calculated. The weighted calculation considers the importance and scarcity of different resource types: the deviation of key resources (such as main processor time, main network bandwidth, etc.) is given a higher weight, and the deviation of secondary resources is given a lower weight; the weight of overtime occupation (positive deviation) is usually higher than that of early release (negative deviation), because overtime occupation is more likely to cause resource conflict and system performance degradation. The weighted sum calculation formula is: ∑(wi×Ri), wherein Ri is the resource occupation deviation of the ith device, and wi is the corresponding weight coefficient. The calculation result is subjected to standardization processing to obtain a standardized resource occupation deviation value as a third component of the linkage deviation.

[0116] Finally, the weighted sum of the three components is normalized to obtain the final linkage deviation value. The weighted sum calculation formula is: weighted sum = a x first component + β x second component + γ x third component, wherein a, β, γ are weight coefficients, and a + β + γ = 1. The weight coefficients are determined by two key indicators in the device running data: task urgency, reflecting the time sensitivity of the current control task, and urgent tasks usually pay more attention to the accuracy of execution time, corresponding to increasing the weight of the first component; device type weight, reflecting the importance of the device in the system, and core devices usually pay more attention to the execution quality, corresponding to increasing the weight of the second component. The system dynamically adjusts these weight coefficients according to the current control scene and device combination characteristics, so that the linkage deviation calculation is more in line with the actual control demand.

[0117] The calculated weighted sum is converted into a normalized linkage deviation value by a mapping function (such as a linear mapping or a Sigmoid function), with a value range of [0, 1], and a value closer to 0 indicates better linkage effect and better performance of each device; a value closer to 1 indicates worse linkage effect and the need for more substantial control adjustment. As a key feedback indicator of the closed-loop control loop, the linkage deviation directly drives the system to perform parameter adjustment and control optimization, ensuring continuous improvement of the collaborative operation effect of the multi-modal devices.

[0118] In the embodiments of the present application, the multi-modal device group includes a conference display screen, a microphone array, intelligent lighting, and an air conditioning device, and the environmental sensor group includes an illumination sensor, a sound sensor, and an infrared sensor; the system further includes the following adaptive optimization steps:

[0119] The historical operation data of each device in the multi-modal device group is obtained, and the historical operation data includes the execution delay distribution, the resource occupation distribution, and the task execution success rate of the device in a preset historical period;

[0120] According to the historical operation data, a running characteristic model of each device is constructed, and the running characteristic model is determined by the weighted sum of the mean value of the execution delay distribution, the variance of the resource occupation distribution, and the task execution success rate;

[0121] According to the running characteristic model, potential execution bottlenecks of the multi-modal device group in the next control period are predicted, and the potential execution bottlenecks are determined by the devices whose mean execution delay is greater than a preset delay threshold or whose resource occupation variance is greater than a preset variance threshold in the running characteristic model;

[0122] For the device corresponding to the potential execution bottleneck, the execution time or the execution priority of the execution instruction of the corresponding device in the optimized control parameter group is adjusted, and the adjustment includes advancing the execution time of the device or increasing the execution priority of the device, and the advancing amount or the priority increment is determined by the ratio of the mean execution delay in the running characteristic model to the preset delay threshold.

[0123] In this embodiment, first, detailed historical running data of each device in the multi-modal device group is obtained from the system history database. The data acquisition range is a preset historical period, usually the last 24 hours, 7 days or 30 days, adjusted according to the system application scene and data volume. The obtained historical running data includes three types of key information: execution delay distribution, which records the execution delay values of all control instructions of the device in the historical period, forms a time series or statistical distribution, and reflects the stability and trend of the device response characteristics; resource occupation distribution, which records the occupation of various resources (such as processor, memory, network, power, etc.) of the device in the historical period, including occupation time, occupation intensity and occupation mode, etc. information, reflecting the resource demand characteristics of the device; task execution success rate, which records the proportion of successful completion of assigned tasks by the device, which is counted according to time period, task type or resource condition, etc. dimensions, reflecting the reliability and performance stability of the device. These historical data are continuously collected and stored through the log recording and performance monitoring module of the system, forming a historical archive of device operation.

[0124] Based on the obtained historical running data, the system constructs a running characteristic model for each device. The running characteristic model is a mathematical description of the device running behavior, which is constructed through three key indicators: the mean value of the execution delay distribution, which calculates the arithmetic mean value of all execution delay values in the historical period, reflecting the average response speed of the device, the smaller the value, the faster the response; the variance of the resource occupation distribution, which calculates the statistical variance of the resource occupation in the historical period, reflecting the fluctuation degree of the device resource demand, the smaller the value, the more stable the resource demand; the task execution success rate, which directly uses the success rate value in the historical data, reflecting the reliability and stability of the device, the higher the value, the better the reliability. After standardization, the three indicators are combined through weighted sum to form the running characteristic score of the device, which is usually normalized to the [0, 1] interval, the higher the value, the better the overall running characteristics of the device.

[0125] The running characteristic model not only contains static scores, but also contains time dimension trend analysis. The system performs time series analysis on the historical data to identify the change patterns of the key indicators, such as linear trend (continuous improvement or deterioration), periodic fluctuation (related to time, load or environmental conditions) or mutation point (may mark the change of device state or external interference). These trend information is captured and quantified through time series models (such as autoregressive model, moving average model or ARIMA model) as a dynamic component of the running characteristic model, enhancing the prediction ability of the model.

[0126] Based on the constructed operating characteristic model, the system predicts potential execution bottlenecks that may occur in the next control cycle for the multi-modal device group. The prediction method is based on two key criteria: devices with execution delay mean greater than the preset delay threshold, indicating that the response speed of the device is slower, which may become a bottleneck for system response; devices with resource occupation variance greater than the preset variance threshold, indicating that the resource demand fluctuation of the device is large, which may cause unstable resource allocation and become a bottleneck for system stability. The system marks the devices that meet any of the criteria as potential bottleneck devices and further evaluates their impact on the overall system performance, including: bottleneck severity, calculated by the proportion of the difference between the index and the threshold; bottleneck persistence, determined by the historical trend to judge whether the problem is temporary or persistent; bottleneck propagation, evaluated by the dependence between devices to assess the scope of the problem that may spread.

[0127] For the identified potential execution bottleneck devices, the system adjusts the optimization control parameter set in a targeted manner to address possible performance problems in advance. The adjustment strategy mainly includes two aspects: time advance, for bottleneck devices with large execution delay, the planned time of their execution instructions is appropriately advanced to give enough execution buffer time to ensure that critical tasks can be completed on time; priority enhancement, for bottleneck devices with large resource demand fluctuation, the priority of their execution instructions is increased to ensure that they can obtain stable and sufficient system resources, reducing performance fluctuations caused by resource competition.

[0128] The adjustment amplitude is dynamically determined according to the bottleneck severity: the time advance is calculated by the ratio of the execution delay mean to the preset delay threshold, the larger the ratio, the more serious the delay problem, the larger the advance amount, usually with an upper limit to avoid excessive advance; the priority increment is also calculated by the ratio of the execution delay mean to the threshold or the ratio of the resource occupation variance to the threshold, ensuring that the increment is proportional to the severity of the problem, while considering the priority distribution of other devices in the system to avoid priority inflation. The adjusted parameters are subjected to conflict detection and coordination processing to ensure that new system conflicts or imbalances will not be caused by parameter adjustment of a single device.

[0129] Through this adaptive optimization mechanism based on historical data and prediction model, the system can identify and address potential execution bottlenecks in advance, achieve preventive performance optimization, reduce abnormal situations and performance fluctuations during operation, and improve the overall collaborative efficiency and user experience of the multi-modal devices in the conference room. At the same time, as the system continues to run and data accumulates, the operating characteristic model is continuously optimized and improved, the prediction accuracy and adjustment rationality will also continuously improve, forming a self-evolving intelligent control system.

[0130] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can make modifications to the technical solutions described in the foregoing embodiments, or make equivalent replacements to some of the technical features, without departing from the spirit and principle of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0131] It should be noted that the formulas in the specification are dimensionless values calculated, the formulas are obtained by collecting a large amount of data to simulate a formula of the most recent real situation, and the preset parameters and threshold values in the formulas are set by those skilled in the art according to the actual situation.

[0132] Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A conference room multi-modal device real-time linkage control system, characterized in that, The system comprises a multi-modal device group, a control server and an environment sensor group, the multi-modal device group comprises devices of at least two different modalities, and the system realizes real-time linkage control through the following methods: acquiring environment state data and device operation data of a conference room in each control period; generating an initial control parameter group for the multi-modal device group in each control period according to the environment state data and preset conference scene requirements, the initial control parameter group comprising execution instructions and execution times of each device; constructing an execution conflict model of the multi-modal device group according to the device operation data and the initial control parameter group, the execution conflict model detecting whether an execution conflict exists in the multi-modal device group through closed-loop feedback adjustment; optimizing the initial control parameter group through an adaptive adjustment algorithm based on the execution conflict model to generate an optimized control parameter group, the adaptive adjustment algorithm comprising dynamic weight distribution based on device operation data and disturbance compensation based on environment state data; constructing a closed-loop control loop according to the optimized control parameter group and feedback of the real-time collected device operation data to control linkage operation of the multi-modal device group and dynamically update the initial control parameter group of the next control period. The construction of the execution conflict model of the multi-modal device group comprises:

2. The real-time linkage control system of a multi-modal device in a conference room according to claim 1, wherein, acquiring resource demand vectors of each device in the multi-modal device group in each control period, the resource demand vector comprising occupation time length, occupation priority and resource type of the device to shared resources; constructing a resource competition conflict matrix according to the resource demand vector, the element value of the resource competition conflict matrix being calculated by weighting the overlap degree, occupation priority difference and resource type compatibility of the resource demand vectors of any two devices in the multi-modal device group; constructing an action mutual exclusion conflict vector according to the initial control parameter group; constructing the execution conflict model based on the resource competition conflict matrix and the action mutual exclusion conflict vector, the execution conflict model being updated in real time through closed-loop feedback adjustment, the closed-loop feedback adjustment comprising dynamically adjusting the weight of the resource competition conflict matrix according to execution delay and resource occupation deviation in the device operation data. The optimization of the initial control parameter group through the adaptive adjustment algorithm to generate the optimized control parameter group comprises:

3. The real-time linkage control system of a multi-modal device in a conference room according to claim 1, wherein, determining a conflict device pair comprising at least two devices in the multi-modal device group that have the execution conflict according to the execution conflict model; acquiring real-time operation margins of each device in the conflict device pair; calculating dynamic execution weights of each device in the conflict device pair according to the operation margins; performing time offset adjustment on the execution instructions of the conflict device pair in the initial control parameter group based on the dynamic execution weights, the time offset adjustment being realized through the following steps: acquiring minimum delay requirements of the execution instructions of the device with lower priority in the conflict device pair; calculating execution margin compensation values of the device with lower priority in the conflict device pair according to the operation margins, the execution margin compensation values being negatively correlated with the operation margins; ​ a larger value between a sum of the minimum delay requirement and the execution margin compensation value and a preset minimum delay interval, as the time offset; generating the optimized control parameter group based on the dynamic execution weight and the time offset.

4. The real-time linkage control system of a multi-modal device in a conference room according to claim 1, wherein, The disturbance compensation in the adaptive adjustment algorithm comprises: obtaining a disturbance factor in the environment state data, the disturbance factor comprising a change rate of light intensity, a fluctuation rate of sound decibel value and a mutation value of personnel distribution density; calculating a disturbance compensation coefficient according to the disturbance factor; and applying the disturbance compensation coefficient to an execution time of a corresponding device in the optimized control parameter group to generate a disturbance-compensated optimized control parameter group, an execution time of the disturbance-compensated optimized control parameter group being adjusted by a product of the execution time of the optimized control parameter group and the disturbance compensation coefficient.

5. The real-time linkage control system of a multi-modal device in a conference room according to claim 1, wherein, The closed-loop control loop is constructed to control the linkage operation of the multi-modal device group, comprising: in each control period, sending a corresponding control signal to each device in the multi-modal device group according to the execution time and execution instruction of the optimized control parameter group; real-time collecting running state data of each device in the multi-modal device group, the running state data comprising a completion degree of device execution instruction, an execution delay and a resource occupation deviation; calculating a linkage deviation of the multi-modal device group in the current control period according to the running state data, the linkage deviation being determined by a weighted sum of a variance of the execution delay of each device in the multi-modal device group, a deviation of the execution instruction completion degree and the resource occupation deviation; based on the linkage deviation, constructing a feedback adjustment parameter of the closed-loop control loop; applying the feedback adjustment parameter to the optimized control parameter group of the next control period to adjust the execution time and execution instruction of the corresponding device.

6. The real-time linkage control system of a multi-modal device in a conference room according to claim 1, wherein, The dynamic updating of the initial control parameter group of the next control period comprises: predicting an execution conflict risk of the next control period according to the device running data and the environment state data of the current control period; adjusting a priority and an execution time of the execution instruction of the corresponding device in the initial control parameter group of the next control period according to the execution conflict risk, the adjustment comprising increasing an execution time offset of a device with a lower priority or reducing an execution instruction quantity of a device with a higher priority.

7. The real-time linkage control system of a multi-modal device in a conference room according to claim 2, wherein, The calculation method of the element value of the resource competition conflict matrix comprises: obtaining an overlap degree of resource requirement vectors of any two devices in the multi-modal device group; obtaining a difference value of occupation priority of the two devices, the occupation priority being determined by a task urgency degree and a device type weight in the device running data; obtaining a compatibility of resource types of the two devices; normalizing a weighted sum of the overlap degree, the occupation priority difference value and the compatibility to obtain the element value of the resource competition conflict matrix.

8. The real-time linkage control system of a multi-modal device in a conference room according to claim 3, wherein, The calculation method of the running margin comprises: obtaining a task queue length, a response delay and a power consumption fluctuation rate in the device running data; normalizing the task queue length, the response delay and the power consumption fluctuation rate to obtain corresponding normalized values; The normalized values are weighted and summed according to preset weights to obtain the operation margin, and the preset weights are determined by a task execution success rate and a resource occupation rate in historical operation data of the device.

9. The real-time linkage control system of a multi-modal device in a conference room according to claim 5, wherein, The linkage deviation calculation method comprises: Obtaining the execution delay of each device in the multi-modal device group; Calculating the variance of the execution delay of all devices in the multi-modal device group as the first component of the linkage deviation; Obtaining the execution instruction completion degree of each device in the multi-modal device group; Calculating the deviation of the execution instruction completion degree of all devices in the multi-modal device group as the second component of the linkage deviation; Obtaining the resource occupation deviation of each device in the multi-modal device group; 10. The real-time linkage control system of a multi-modal device in a conference room according to claim 1, wherein, Calculating the weighted sum of the resource occupation deviation of all devices in the multi-modal device group as the third component of the linkage deviation; The weighted sum of the first component, the second component and the third component is normalized to obtain the linkage deviation. The multi-modal device group comprises a conference display screen, a microphone array, intelligent lighting and an air conditioning device, and the environmental sensor group comprises an illumination sensor, a sound sensor and an infrared sensor; The system further comprises the following adaptive optimization steps: Obtaining historical operation data of each device in the multi-modal device group, the historical operation data comprising an execution delay distribution, a resource occupation distribution and a task execution success rate of the device within a preset historical period; According to the historical operation data, constructing an operation characteristic model of each device; According to the operation characteristic model, predicting potential execution bottlenecks of the multi-modal device group in the next control period; For the device corresponding to the potential execution bottleneck, adjusting the execution time or execution priority of the execution instruction of the corresponding device in the optimization control parameter group, the adjustment comprising advancing the execution time of the device or increasing the execution priority of the device, and the advance amount or priority increment being determined by the ratio of the mean execution delay in the operation characteristic model to a preset delay threshold.

Images