Closed-loop space management system and method based on multi-modal sensory feedback
By using a multimodal sensing module and a closed-loop feedback mechanism, the status of air conditioning, lighting, and audio equipment is adjusted in real time, solving the problems of poor adaptability and energy waste in existing systems, and achieving high efficiency, energy saving, and comfortable management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAYIN (ZHUHAI) DIGITAL TECH CO LTD
- Filing Date
- 2026-04-28
- Publication Date
- 2026-07-24
Smart Images

Figure CN122449974A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of closed-loop spatial scheduling technology, and in particular to a closed-loop spatial management system and method based on multimodal perception feedback. Background Technology
[0002] With the rapid development of smart office and smart building technologies, intelligent space management systems have been widely used in various scenarios such as conference rooms, office areas, exhibition halls, and hotel rooms. The core function is to realize centralized control of electrical equipment such as air conditioning, lighting, and audio equipment in the space, aiming to improve the convenience and comfort of space use. Existing intelligent space management systems generally adopt a one-way command issuance mode. Their working logic is as follows: the system pre-sets fixed equipment control parameters (such as a default air conditioning temperature of 26℃ and a default lighting illuminance of 500 lux). Users or administrators manually trigger control commands. The equipment can only execute corresponding actions according to the received commands and cannot adaptively adjust to changes in the actual environment within the space. This mode has the following significant technical defects: 1. Lack of environmental feedback capability: The system cannot perceive changes in dynamic environmental parameters such as the number of attendees, light intensity, temperature and humidity, ambient noise, and CO2 concentration in real time; it can only perform fixed operations based on preset parameters. 1. Inefficient control; 2. Rigid control methods and poor adaptability: When the spatial environment changes, such as the number of participants increasing from 5 to 20, leading to increased space temperature, excessive sunlight causing excessive light intensity, or increased environmental noise affecting the meeting experience, the equipment status cannot be automatically fine-tuned, requiring manual reset of control parameters, which is cumbersome; 3. Serious energy waste and poor user comfort: Fixed control parameters cannot match the real-time needs of the space, easily leading to problems such as air conditioning being too cold or too hot, lighting being too bright or too dim, and sound volume being too high or too low. This not only seriously affects the user experience but also causes a large amount of unnecessary energy waste, which is not in line with the development trend of green energy conservation; In light of the above, there is an urgent need to develop an intelligent space scheduling scheme that can perceive environmental changes in real time, has intelligent decision-making capabilities, and realizes closed-loop feedback regulation to overcome the shortcomings of existing technologies. To this end, this application proposes a closed-loop space management system and method based on multimodal perception feedback. Summary of the Invention
[0003] Based on the technical problems existing in the background technology, this invention proposes a closed-loop spatial management system and method based on multimodal perception feedback.
[0004] The proposed closed-loop spatial management system based on multimodal perception feedback includes a multimodal perception module, a data fusion and preprocessing module, a large-scale intelligent decision-making module, an equipment execution module, and a closed-loop feedback module. The multimodal sensing module is used to collect personnel data, environmental data, and equipment status data in the space in real time. It includes a personnel sensing unit, an environmental sensing unit, an equipment status sensing unit, and a data transmission unit. The data fusion and preprocessing module is used to standardize, reduce noise, align spatiotemporally, and extract features from multimodal data. The large-scale intelligent decision-making module is used to generate refined equipment control instructions based on fused data, combined with comfort thresholds, energy consumption rules, and user preferences. The device execution module is used to receive instructions and control the air conditioning, lighting and audio equipment to perform adjustments. It includes a protocol adaptation unit, an instruction parsing unit and a device control unit. The closed-loop feedback module is used to collect the device execution results and new environmental data, and transmit them back to the multimodal sensing module to form a closed loop.
[0005] Preferably, the personnel sensing unit consists of a high-definition camera, a millimeter-wave radar, and an infrared sensor. The high-definition camera is used to identify the number and distribution of personnel, the millimeter-wave radar is used to detect the movement trajectory and density of personnel, and the infrared sensor is used to assist in identifying the presence of personnel and avoid personnel counting errors caused by occlusion. The environmental sensing unit includes a temperature and humidity sensor, a light sensor, a CO2 sensor, and a sound sensor, which are used to collect real-time temperature, relative humidity, light intensity, CO2 concentration, and environmental noise decibels in the space, covering the core influencing factors of the space environment. The device status sensing unit establishes a communication connection with the controlled air conditioning, lighting, and audio equipment, and collects the operating parameters of the equipment in real time, including the current set temperature, actual operating temperature, and air volume level of the air conditioning, the current illuminance and power of the lighting, and the current volume and gain value of the audio equipment. The data transmission unit adopts one of the communication methods such as Wi-Fi, Bluetooth, LoRa or Industrial Ethernet to upload the raw data collected by each sensing unit to the data fusion and preprocessing module in real time, ensuring the real-time performance and reliability of data transmission, while supporting concurrent transmission of multiple devices.
[0006] Preferably, when performing data standardization, the data fusion and preprocessing module converts heterogeneous data collected by different sensors into a unified format and unit, and uses a min-max standardization algorithm to normalize the data, ensuring consistent data volume. The formula used is: ,in For standardized data, The raw data collected by the sensor, This is the minimum value of this type of data. The maximum value of this data type; When performing data denoising and cleaning, an outlier detection algorithm is used to remove outliers and missing values from the data. The judgment formula used is as follows: When data satisfies this formula, it is identified as an outlier and removed. A filtering algorithm is used to remove noise signals caused by environmental interference, improving data reliability. The formula used is: ,in The mean of the data. The standard deviation of the data. The filtered data, This is the raw data currently being collected. This is the data after the previous filtering. These are the filter coefficients; During spatiotemporal alignment, data collected from different sensing units and at different times are synchronized and aligned based on a unified timestamp, constructing a spatiotemporal environmental data matrix M to ensure data spatiotemporal consistency. The data matrix expression is as follows: ,in For the number of sensing units, Number of data collection points The data collected by the m-th sensing unit at time t and the n-th time point; During feature extraction, a feature extraction algorithm is used to extract key features from the preprocessed multi-source data, including the number of people, light intensity, temperature and humidity trends, and noise peaks, generating structured perception data, which is then output to the large-scale intelligent decision-making module. The extracted feature vector expression is as follows: Where N is the number of people and L is the light intensity. For temperature change trends, This shows the trend of humidity changes. This represents the peak noise level.
[0007] Preferably, the large-scale intelligent decision-making module includes a multimodal large-scale model, a decision rule base, and a dynamic decision engine; The aforementioned multimodal large model employs a pre-trained multimodal large model, possessing cross-modal understanding capabilities for image, text, numerical, and audio data. It can quickly fuse and analyze three types of data—personnel, environment, and equipment—to uncover the relationships between the data, employing a multimodal fusion function. To achieve multi-source data fusion, among which For image features, For text features, For numerical features, Audio features; The decision rule base includes built-in comfort thresholds, energy consumption optimization rules, and device coordination logic. Comfort thresholds include temperature 24-26℃, light intensity 300-700 lux, noise <50dB, and CO2 concentration <1000ppm. Energy consumption optimization rules include "turn off equipment when no one is present" and "reduce light intensity when there is sufficient light." The device coordination logic ensures that the adjustments of air conditioning, lighting, and audio equipment do not conflict. The dynamic decision engine's workflow is as follows: First, it receives structured perception data output from the data fusion and preprocessing module. Second, a multimodal large model comprehensively analyzes the data, generating preliminary decision suggestions based on data trends. Finally, combining the decision rule base and user-defined preferences, it optimizes and generates final refined control instructions. The instructions specify the equipment's parameters. Preliminary decision suggestions include: when the number of people increases, it is recommended to lower the air conditioning temperature and increase the airflow; when the light intensity exceeds the standard, it is recommended to reduce the light intensity; when the environmental noise increases, it is recommended to appropriately increase the audio gain. The preliminary decision recommendations can be represented by the following decision function: ,in For changes in the number of people, For air conditioning temperature adjustment amount, This refers to the air conditioning fan speed adjustment. This is the amount of light illuminance adjustment. This represents the change in noise level. This is the audio gain adjustment amount.
[0008] Preferably, the protocol adaptation unit supports mainstream industrial control protocols such as Modbus, BACnet, MQTT, and ZigBee, and is compatible with different brands and types of air conditioners, lighting, and audio equipment. This eliminates the need to modify existing equipment, reducing system deployment costs. Protocol adaptation utilizes a protocol conversion function. ,in The standard protocol commands output by the system. Protocols supported by the device; The instruction parsing unit is used to convert the natural language or structured control instructions output by the large model intelligent decision-making module into electrical or digital signals that can be recognized by the controlled device, ensuring that the instructions can be accurately recognized by the device. The instruction parsing formula is as follows: ,in For signals that the device can recognize. For the final control command, Equipment signal encoding rules; The device control unit establishes a connection with the controlled device through a drive circuit, sends the parsed control signal to the device, and controls the device to perform actions such as temperature adjustment, illuminance adjustment, and volume adjustment. Simultaneously, it provides real-time feedback on the device's execution status. The drive formula for device control is: ,in For the output signal of the drive circuit, The driving coefficient is denoted as .
[0009] This invention also proposes a closed-loop spatial scheduling method based on multimodal sensing feedback, comprising the following steps: S1: Perception Phase: Comprehensive Collection and Upload of Multi-Dimensional Raw Data: After the multi-modal perception module is activated, it collects personnel data, environmental data, and equipment status data in the space in real time, and uploads the raw data to the data fusion and preprocessing module through the data transmission unit; S2: Preprocessing stage: Optimize data quality and extract effective features: The data fusion and preprocessing module performs standardization, noise reduction, spatiotemporal alignment and feature extraction on the received multi-source raw data in sequence to generate structured perception data and output it to the large model intelligent decision-making module. S3: Decision-making stage: Generate precise and adapted equipment control commands based on preprocessed data: The large model intelligent decision-making module receives structured perception data, performs comprehensive analysis by the multimodal large model, combines the comfort threshold and energy consumption rules in the decision rule base with user-defined preferences, generates refined equipment control commands, and outputs them to the equipment execution module; S4: Execution phase: Transform decision commands into executable actions of the equipment to complete equipment adjustment operations: The equipment execution module converts control commands into signals that the equipment can recognize through the protocol adaptation unit and the command parsing unit, driving the air conditioning, lighting, and audio equipment to perform corresponding adjustment actions; S5: Feedback Phase: Collect execution results and new environmental data to provide feedback basis for closed-loop iteration: The closed-loop feedback module collects the actual operating status data after the device is executed, as well as the new environmental perception data in the space, and sends both types of data back to the multimodal perception module; S6: Iterative Loop: Closed-Loop Continuous Optimization to Ensure the Space Environment is Always in the Optimal State: Repeat S1-S5 to continuously perform the closed-loop cycle of "perception-decision-execution-re-perception". Based on environmental changes and equipment execution, dynamically optimize equipment control commands to ensure the space environment is always in the optimal state.
[0010] Preferably, the specific logical steps of S2 are as follows: S201: The min-max normalization algorithm is used to normalize the various types of raw data collected by S1 to the [0,1] interval, preserving the variation trend of the original data and achieving uniform data volume. The formula used is: ,in For standardized data, The raw data collected by the sensor, This is the minimum value of this type of data. The maximum value of this data type; S202: First, outliers are removed and missing values are added using the 3σ criterion. Then, environmental noise is eliminated using a first-order low-pass filtering algorithm, the formula of which is as follows: The formula used for outlier detection and removal is: ,in The mean of the data. The standard deviation of the data; The formula used for environmental noise filtering is: ,in The filtered data, This is the raw data currently being collected. This is the data after the previous filtering. These are the filter coefficients; S203: Using the system's unified timestamp t as a benchmark, synchronize and align all multi-source data after noise reduction and cleaning, and integrate data collected by different sensing units at the same timestamp to form a spatiotemporal data matrix M. ,in For the number of sensing units, Number of data collection points The data collected by the m-th sensing unit at time t and the n-th time point; S204: Employs a feature extraction algorithm combining statistical analysis and convolutional neural networks to extract three core key features: personnel, environment, and equipment. This extracts feature vectors to form structured perception data. The formula used is as follows: Where N is the number of people and L is the light intensity. For temperature change trends, This shows the trend of humidity changes. This represents the peak noise level. This indicates the current operating status of the air conditioner. This indicates the current operating status of the lights. The current operating status of the audio system is determined, and the feature vector F is encapsulated into a structured data format and output in real time to the large model intelligent decision-making module as the core data input for decision analysis.
[0011] Preferably, the specific logical steps of S3 are as follows: S301: Receives the feature vector F corresponding to the structured sensing data, verifies the value range of each feature parameter, and corrects and supplements parameters that exceed the reasonable range or are missing. Its data verification and judgment formula is as follows: ,in To accommodate the maximum number of people in the target space, This represents the maximum reasonable value for light intensity. , This represents the maximum reasonable variation in temperature and humidity. This represents the maximum reasonable noise level. If a feature parameter exceeds the above reasonable range, it is determined to be invalid data, and the valid feature value from the previous time step is used to replace it. If there are missing features, the historical mean of that feature is used to supplement them. After the verification is passed, the valid feature vector is output. ; S302: Employs a multimodal fusion algorithm to combine effective feature vectors The image and audio features of the multimodal large model are fused to generate a fused feature matrix. Its expression is: ,in ( () is a multimodal fusion function, which uses a weighted summation method to achieve multi-source feature fusion. For image feature vectors, For audio feature vectors, , , The fusion weights are preset to 0.6, 0.2, and 0.2 respectively, and satisfy the following conditions: ; S303: By analyzing the changing trends of fused features through a multimodal large model, preliminary adjustment suggestions are generated for different devices. The preliminary decision results are represented by the adjustment amount, and the formula is as follows: ; in For air conditioning temperature adjustment, This refers to the adjustment amount of light illuminance. This is the audio gain adjustment amount. For changes in the number of people, This is the standard value for light intensity. This represents the change in noise peak value. , , , For adjustment coefficients, ; When the number of people increases, the air conditioning will cool down; when the light intensity is higher than the standard value, the lights will be dimmed; when the noise peak increases, the sound gain will increase, and vice versa. S304: First, the adjustment range is constrained by the decision rule base, then the adjustment parameters are optimized by combining user preferences to generate the final refined control command. The formula used is as follows: Rule base constraints: ,in , , The constrained settings for air conditioning temperature, lighting illuminance, and sound gain. , , These are the current operating parameters of the equipment. 24-26℃, 300-700lux, and 0-10dB are the comfort thresholds for air conditioning temperature, lighting illuminance, and sound gain, respectively. Final setpoint optimization formula: ,in For the optimized equipment control parameters, These are the device settings constrained by the rule base. Allow users to customize preference parameters. For preference weights; S305: Will The corresponding air conditioning temperature, lighting illuminance, and audio gain settings are encapsulated into control commands conforming to the device communication protocol. These commands include the command type, device ID, control parameters, and key execution time information. The encapsulated expression is: ,in As a unique identifier for the device, For instruction type, For the optimized equipment control parameters, For instruction execution time, The data is categorized by device type and output to the device execution module accordingly.
[0012] Preferably, the specific logical steps of S4 are as follows: S401: The instruction parsing unit decodes the received structured control instructions, filters out the corresponding device control parameters, and converts them into standardized analog electrical signals to ensure a uniform signal format. (If the air conditioner's control parameters are optimized...) The normalized voltage signal after analysis is: If the lighting control parameters are optimized The normalized voltage signal after analysis is: ; S402: The protocol adaptation unit extracts the device ID from the instruction. It identifies the device type and supported communication protocols, and converts the standardized voltage signal into a device-specific protocol signal using a protocol conversion algorithm. The protocol conversion algorithm formula is as follows: ,in This is a device-specific communication protocol signal. ( This is a protocol adaptation conversion function that dynamically adjusts based on the protocol types supported by the device. The types of communication protocols supported by the device. These are protocol conversion coefficients, set according to the protocol type. This is the protocol baseline value, preset to 0; S403: The device control unit receives the protocol signal after protocol adaptation. Through a power amplification algorithm, a drive signal for the corresponding device is generated. The strength of the drive signal is positively correlated with the adjustment amount of the control parameters. The formula for generating the drive signal is as follows: ,in This is the device drive current signal. The amplification factor is set according to the device power. For device-specific communication protocol signals after protocol adaptation; S404: The drive signal is transmitted to the device through the drive circuit. The device performs corresponding adjustment actions based on the signal strength. At the same time, the device status acquisition module collects the actual operating parameters after adjustment, converts them into feedback signals, and sends them back. The execution feedback formula is: ,in This is a feedback signal for the device's execution status. The feedback coefficient is preset to 0.01. These are the actual operating status parameters of the equipment. The optimized control parameters for the decision-making stage; After receiving the drive signal, the device completes the adjustment according to the set parameters, and at the same time collects the actual operating parameters. Calculation and setting values The deviation is converted into a feedback signal. The data is then sent back to the closed-loop feedback module, completing the entire execution phase.
[0013] Compared with existing technologies, the beneficial effects of this invention are: 1. Through the multimodal sensing module, dynamic environmental parameters such as the number of people in the space, temperature and humidity, light intensity, CO2 concentration, and environmental noise are collected in real time. Combined with the closed-loop feedback module, a closed-loop scheduling of "perception-decision-execution-re-perception" is realized. The operating status of equipment such as air conditioning, lighting, and sound can be automatically adjusted without manual intervention. It has a strong adaptive adjustment capability and can accurately adapt to the real-time changes in the spatial environment. 2. By comprehensively analyzing the perceived data through a multimodal large model, and combining the comfort threshold in the decision rule base with user-defined preferences, refined equipment control instructions are generated. This effectively avoids problems such as air conditioning being too cold or too hot, lighting being too bright or too dim, and audio volume being unsuitable due to fixed control parameters in existing systems. At the same time, it eliminates the need for administrators to frequently manually set equipment parameters, achieving seamless intelligent management of space equipment, reducing manual management costs, and improving management efficiency. 3. The dynamic decision engine enables on-demand adjustment of equipment. Based on real-time parameters such as the number of people in the space and the light intensity, the operating power and operating status of the equipment are dynamically adjusted (e.g., turning off the equipment when no one is there, reducing the light intensity when there is sufficient light, and appropriately increasing the air conditioning temperature when there are few people). Actual verification has shown that this can effectively reduce energy consumption by 20%-40%, achieving the dual goals of energy saving and consumption reduction while ensuring user comfort.
[0014] This invention uses a multimodal sensing module to collect multi-dimensional dynamic parameters of people, environment, and equipment in a space in real time. Combined with a large-scale intelligent decision-making and closed-loop feedback mechanism, it achieves closed-loop adaptive scheduling of "perception-decision-execution-re-perception". It can automatically adjust the refined operating status of air conditioning, lighting, and audio equipment without human intervention, which not only significantly improves the comfort and management convenience of the space and reduces the cost of manual management, but also enables the equipment to be adjusted on demand according to real-time needs, effectively reducing energy consumption by 20%-40%. It practices the trend of green and energy-saving development and achieves the dual goals of comfortable experience and energy saving. Attached Figure Description
[0015] Figure 1 This is a block diagram of the closed-loop spatial management system based on multimodal sensing feedback proposed in this invention. Figure 2 This is a flowchart of the closed-loop spatial scheduling method based on multimodal sensing feedback proposed in this invention. Detailed Implementation
[0016] The present invention will be further explained below with reference to specific embodiments.
[0017] Example Reference Figure 1 This embodiment proposes a closed-loop spatial management system based on multimodal perception feedback, including a multimodal perception module, a data fusion and preprocessing module, a large model intelligent decision-making module, an equipment execution module, and a closed-loop feedback module; The multimodal sensing module is used to collect personnel data, environmental data, and equipment status data in the space in real time. It includes a personnel sensing unit, an environmental sensing unit, an equipment status sensing unit, and a data transmission unit. The personnel sensing unit consists of a high-definition camera, millimeter-wave radar, and infrared sensor. The high-definition camera is used to identify the number and distribution of personnel, the millimeter-wave radar is used to detect the movement trajectory and density of personnel, and the infrared sensor is used to assist in identifying the presence of personnel and avoid personnel counting errors caused by occlusion. The environmental sensing unit includes temperature and humidity sensors, light sensors, CO2 sensors, and sound sensors, which are used to collect real-time temperature, relative humidity, light intensity, CO2 concentration, and environmental noise decibel values in the space, covering the core influencing factors of the space environment. The equipment status sensing unit establishes a communication connection with the controlled equipment such as air conditioners, lights, and audio systems, and collects the operating parameters of the equipment in real time, including the current set temperature, actual operating temperature, and air volume level of the air conditioner, the current illuminance and power of the lights, and the current volume and gain value of the audio system. The data transmission unit adopts one of the communication methods such as Wi-Fi, Bluetooth, LoRa or industrial Ethernet to upload the raw data collected by each sensing unit to the data fusion and preprocessing module in real time, ensuring the real-time performance and reliability of data transmission, while supporting concurrent transmission of multiple devices. The data fusion and preprocessing module is used to standardize, denoise, align, and extract features from multimodal data. The data fusion and preprocessing module, during data standardization, converts heterogeneous data collected from different sensors into a unified format and units. It employs a min-max standardization algorithm to normalize the data, ensuring consistent data volume. The formula used is: ,in For standardized data, The raw data collected by the sensor, This is the minimum value of this type of data. The maximum value of this data type; When performing data denoising and cleaning, an outlier detection algorithm is used to remove outliers and missing values from the data. The judgment formula used is as follows: When data satisfies this formula, it is identified as an outlier and removed. A filtering algorithm is used to remove noise signals caused by environmental interference, improving data reliability. The formula used is: ,in The mean of the data. The standard deviation of the data. The filtered data, This is the raw data currently being collected. This is the data after the previous filtering. These are the filter coefficients; During spatiotemporal alignment, data collected from different sensing units and at different times are synchronized and aligned based on a unified timestamp, constructing a spatiotemporal environmental data matrix M to ensure data spatiotemporal consistency. The data matrix expression is as follows: ,in For the number of sensing units, Number of data collection points The data collected by the m-th sensing unit at time t and the n-th time point; During feature extraction, a feature extraction algorithm is used to extract key features from the preprocessed multi-source data, including the number of people, light intensity, temperature and humidity trends, and noise peaks, generating structured perception data, which is then output to the large-scale intelligent decision-making module. The extracted feature vector expression is as follows: Where N is the number of people and L is the light intensity. For temperature change trends, This shows the trend of humidity changes. This represents the peak noise level. The large-scale intelligent decision-making module is used to generate refined equipment control instructions based on fused data, combined with comfort thresholds, energy consumption rules, and user preferences; The large-scale intelligent decision-making module includes a multimodal large model, a decision rule base, and a dynamic decision engine; The multimodal large model employs a pre-trained multimodal large model, possessing cross-modal understanding capabilities for image, text, numerical, and audio data. It can quickly fuse and analyze three types of data—personnel, environment, and equipment—to uncover the relationships between data. A multimodal fusion function is used: To achieve multi-source data fusion, among which For image features, For text features, For numerical features, Audio features; The decision rule base includes comfort thresholds for space use, energy consumption optimization rules, and device coordination logic. Comfort thresholds include temperature 24-26℃, light intensity 300-700 lux, noise <50dB, and CO2 concentration <1000ppm. Energy consumption optimization rules include "turn off equipment when no one is present" and "reduce light intensity when there is sufficient light". Device coordination logic is used to ensure that the adjustment of air conditioning, lighting, and audio equipment does not conflict with each other. The dynamic decision engine's workflow is as follows: First, it receives structured perception data output from the data fusion and preprocessing module. Second, a multimodal large model comprehensively analyzes the data and generates preliminary decision suggestions based on data change trends. Finally, it combines the decision rule base and user-defined preferences to optimize and generate the final refined control instructions. The instructions are specific to the equipment's parameters. The preliminary decision suggestions include: when the number of people increases, it is recommended to lower the air conditioning temperature and increase the airflow; when the light intensity exceeds the standard, it is recommended to reduce the light intensity; when the environmental noise increases, it is recommended to appropriately increase the sound gain. The preliminary decision recommendations can be represented by the following decision function: ,in For changes in the number of people, For air conditioning temperature adjustment amount, This refers to the air conditioning fan speed adjustment. This is the amount of light illuminance adjustment. This represents the change in noise level. This is the audio gain adjustment amount; The device execution module is used to receive instructions and control the air conditioning, lighting and audio equipment to perform adjustments. It includes a protocol adaptation unit, an instruction parsing unit and a device control unit. The protocol adaptation unit supports mainstream industrial control protocols such as Modbus, BACnet, MQTT, and ZigBee, and is compatible with different brands and types of air conditioners, lighting, and audio equipment. It eliminates the need to modify existing equipment, reducing system deployment costs. Protocol adaptation utilizes protocol conversion functions. ,in The standard protocol commands output by the system. Protocols supported by the device; The instruction parsing unit converts the natural language or structured control instructions output by the large-scale intelligent decision-making module into electrical or digital signals that the controlled device can recognize, ensuring that the instructions can be accurately recognized by the device. The instruction parsing formula is as follows: , among which For signals that the device can recognize. For the final control command, Equipment signal encoding rules; The equipment control unit establishes a connection with the controlled equipment through a drive circuit, sends the parsed control signals to the equipment, and controls the equipment to perform actions such as temperature adjustment, illuminance adjustment, and volume adjustment. Simultaneously, it provides real-time feedback on the equipment's execution status. The drive formula for equipment control is: ,in For the output signal of the drive circuit, For driving coefficients; The closed-loop feedback module is used to collect the device execution results and new environmental data, and send them back to the multimodal sensing module to form a closed loop.
[0018] Reference Figure 2 This embodiment proposes a closed-loop spatial scheduling method based on multimodal sensing feedback, including the following steps: S1: Perception Phase: Comprehensive Collection and Upload of Multi-Dimensional Raw Data: After the multi-modal perception module is activated, it collects personnel data, environmental data, and equipment status data in the space in real time, and uploads the raw data to the data fusion and preprocessing module through the data transmission unit; S2: Preprocessing stage: Optimize data quality and extract effective features: The data fusion and preprocessing module performs standardization, noise reduction, spatiotemporal alignment and feature extraction on the received multi-source raw data in sequence to generate structured perception data and output it to the large model intelligent decision-making module. S3: Decision-making stage: Generate precise and adapted equipment control commands based on preprocessed data: The large model intelligent decision-making module receives structured perception data, performs comprehensive analysis by the multimodal large model, combines the comfort threshold and energy consumption rules in the decision rule base with user-defined preferences, generates refined equipment control commands, and outputs them to the equipment execution module; S4: Execution phase: Transform decision commands into executable actions of the equipment to complete equipment adjustment operations: The equipment execution module converts control commands into signals that the equipment can recognize through the protocol adaptation unit and the command parsing unit, driving the air conditioning, lighting, and audio equipment to perform corresponding adjustment actions; S5: Feedback Phase: Collect execution results and new environmental data to provide feedback basis for closed-loop iteration: The closed-loop feedback module collects the actual operating status data after the device is executed, as well as the new environmental perception data in the space, and sends both types of data back to the multimodal perception module; S6: Iterative Loop: Closed-Loop Continuous Optimization to Ensure the Space Environment is Always in the Optimal State: Repeat S1-S5 to continuously perform the closed-loop cycle of "perception-decision-execution-re-perception". Based on environmental changes and equipment execution, dynamically optimize equipment control commands to ensure the space environment is always in the optimal state.
[0019] In this implementation scheme, the specific logical steps of S2 are as follows: S201: The min-max normalization algorithm is used to normalize the various types of raw data collected by S1 to the [0,1] interval, preserving the variation trend of the original data and achieving uniform data volume. The formula used is: ,in For standardized data, The raw data collected by the sensor, This is the minimum value of this type of data. The maximum value of this data type; S202: First, outliers are removed and missing values are added using the 3σ criterion. Then, environmental noise is eliminated using a first-order low-pass filtering algorithm, the formula of which is as follows: The formula used for outlier detection and removal is: ,in The mean of the data. The standard deviation of the data; The formula used for environmental noise filtering is: ,in The filtered data, This is the raw data currently being collected. This is the data after the previous filtering. These are the filter coefficients; S203: Using the system's unified timestamp t as a benchmark, synchronize and align all multi-source data after noise reduction and cleaning, and integrate data collected by different sensing units at the same timestamp to form a spatiotemporal data matrix M. ,in For the number of sensing units, Number of data collection points The data collected by the m-th sensing unit at time t and the n-th time point; S204: Employs a feature extraction algorithm combining statistical analysis and convolutional neural networks to extract three core key features: personnel, environment, and equipment. This extracts feature vectors to form structured perception data. The formula used is as follows: Where N is the number of people and L is the light intensity. For temperature change trends, This shows the trend of humidity changes. This represents the peak noise level. This indicates the current operating status of the air conditioner. This indicates the current operating status of the lights. The current operating status of the audio system is determined, and the feature vector F is encapsulated into a structured data format and output in real time to the large model intelligent decision-making module as the core data input for decision analysis.
[0020] In this implementation scheme, the specific logical steps of S3 are as follows: S301: Receives the feature vector F corresponding to the structured sensing data, verifies the value range of each feature parameter, and corrects and supplements parameters that exceed the reasonable range or are missing. Its data verification and judgment formula is as follows: ,in To accommodate the maximum number of people in the target space, This represents the maximum reasonable value for light intensity. , This represents the maximum reasonable variation in temperature and humidity. This represents the maximum reasonable noise level. If a feature parameter exceeds the above reasonable range, it is determined to be invalid data, and the valid feature value from the previous time step is used to replace it. If there are missing features, the historical mean of that feature is used to supplement them. After the verification is passed, the valid feature vector is output. ; S302: Employs a multimodal fusion algorithm to combine effective feature vectors The image and audio features of the multimodal large model are fused to generate a fused feature matrix. Its expression is: ,in ( () is a multimodal fusion function, which uses a weighted summation method to achieve multi-source feature fusion. For image feature vectors, For audio feature vectors, , , The fusion weights are preset to 0.6, 0.2, and 0.2 respectively, and satisfy the following conditions: ; S303: By analyzing the changing trends of fused features through a multimodal large model, preliminary adjustment suggestions are generated for different devices. The preliminary decision results are represented by the adjustment amount, and the formula is as follows: ; in For air conditioning temperature adjustment, This refers to the adjustment amount of light illuminance. This is the audio gain adjustment amount. For changes in the number of people, This is the standard value for light intensity. This represents the change in noise peak value. , , , For adjustment coefficients, ; When the number of people increases, the air conditioning will cool down; when the light intensity is higher than the standard value, the lights will be dimmed; when the noise peak increases, the sound gain will increase, and vice versa. S304: First, the adjustment range is constrained by the decision rule base, then the adjustment parameters are optimized by combining user preferences to generate the final refined control command. The formula used is as follows: Rule base constraints: ,in , , The constrained settings for air conditioning temperature, lighting illuminance, and sound gain. , , These are the current operating parameters of the equipment. 24-26℃, 300-700lux, and 0-10dB are the comfort thresholds for air conditioning temperature, lighting illuminance, and sound gain, respectively. Final setpoint optimization formula: ,in For the optimized equipment control parameters, These are the device settings constrained by the rule base. Allow users to customize preference parameters. For preference weights; S305: Will The corresponding air conditioning temperature, lighting illuminance, and audio gain settings are encapsulated into control commands conforming to the device communication protocol. These commands include the command type, device ID, control parameters, and key execution time information. The encapsulated expression is: ,in As a unique identifier for the device, For instruction type, For the optimized equipment control parameters, For instruction execution time, The data is categorized by device type and output to the device execution module accordingly.
[0021] In this implementation scheme, the specific logical steps of S4 are as follows: S401: The instruction parsing unit decodes the received structured control instructions, filters out the corresponding device control parameters, and converts them into standardized analog electrical signals to ensure a uniform signal format. (If the air conditioner's control parameters are optimized...) The normalized voltage signal after analysis is: If the lighting control parameters are optimized The normalized voltage signal after analysis is: ; S402: The protocol adaptation unit extracts the device ID from the instruction. It identifies the device type and supported communication protocols, and converts the standardized voltage signal into a device-specific protocol signal using a protocol conversion algorithm. The protocol conversion algorithm formula is as follows: ,in This is a device-specific communication protocol signal. ( This is a protocol adaptation conversion function that dynamically adjusts based on the protocol types supported by the device. The types of communication protocols supported by the device. These are protocol conversion coefficients, set according to the protocol type. This is the protocol baseline value, preset to 0; S403: The device control unit receives the protocol signal after protocol adaptation. Through a power amplification algorithm, a drive signal for the corresponding device is generated. The strength of the drive signal is positively correlated with the adjustment amount of the control parameters. The formula for generating the drive signal is as follows: ,in This is the device drive current signal. The amplification factor is set according to the device power. For device-specific communication protocol signals after protocol adaptation; S404: The drive signal is transmitted to the device through the drive circuit. The device performs corresponding adjustment actions based on the signal strength. At the same time, the device status acquisition module collects the actual operating parameters after adjustment, converts them into feedback signals, and sends them back. The execution feedback formula is: ,in This is a feedback signal for the device's execution status. The feedback coefficient is preset to 0.01. These are the actual operating status parameters of the equipment. The optimized control parameters for the decision-making stage; After receiving the drive signal, the device completes the adjustment according to the set parameters, and at the same time collects the actual operating parameters. Calculation and setting values The deviation is converted into a feedback signal. The data is then sent back to the closed-loop feedback module, completing the entire process of the execution phase. This embodiment uses a multimodal sensing module to collect multi-dimensional dynamic parameters of people, environment, and equipment in the space in real time. Combined with a large-scale intelligent decision-making and closed-loop feedback mechanism, it realizes a closed-loop adaptive scheduling of "perception-decision-execution-re-perception". It can automatically adjust the fine-grained operating status of air conditioning, lighting, and audio equipment without manual intervention. This not only significantly improves the comfort and management convenience of the space and reduces manual management costs, but also enables the equipment to be adjusted on demand according to real-time needs, effectively reducing energy consumption by 20%-40%. It practices the trend of green and energy-saving development and achieves the dual goals of comfortable experience and energy saving.
[0022] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A closed-loop spatial management system based on multimodal sensing feedback, characterized in that, It includes a multimodal perception module, a data fusion and preprocessing module, a large-scale intelligent decision-making module, an equipment execution module, and a closed-loop feedback module; The multimodal sensing module is used to collect personnel data, environmental data, and equipment status data in the space in real time. It includes a personnel sensing unit, an environmental sensing unit, an equipment status sensing unit, and a data transmission unit. The data fusion and preprocessing module is used to standardize, reduce noise, align spatiotemporally, and extract features from multimodal data. The large-scale intelligent decision-making module is used to generate refined equipment control instructions based on fused data, combined with comfort thresholds, energy consumption rules, and user preferences. The device execution module is used to receive instructions and control the air conditioning, lighting and audio equipment to perform adjustments. It includes a protocol adaptation unit, an instruction parsing unit and a device control unit. The closed-loop feedback module is used to collect the device execution results and new environmental data, and transmit them back to the multimodal sensing module to form a closed loop.
2. The closed-loop spatial management system based on multimodal perception feedback according to claim 1, characterized in that, The personnel sensing unit consists of a high-definition camera, a millimeter-wave radar, and an infrared sensor. The high-definition camera is used to identify the number and distribution of personnel, the millimeter-wave radar is used to detect the movement trajectory and density of personnel, and the infrared sensor is used to assist in identifying the presence of personnel and avoid personnel counting errors caused by occlusion. The environmental sensing unit includes a temperature and humidity sensor, a light sensor, a CO2 sensor, and a sound sensor, which are used to collect real-time temperature, relative humidity, light intensity, CO2 concentration, and environmental noise decibels in the space, covering the core influencing factors of the space environment. The device status sensing unit establishes a communication connection with the controlled air conditioning, lighting, and audio equipment, and collects the operating parameters of the equipment in real time, including the current set temperature, actual operating temperature, and air volume level of the air conditioning, the current illuminance and power of the lighting, and the current volume and gain value of the audio equipment. The data transmission unit adopts one of the communication methods such as Wi-Fi, Bluetooth, LoRa or Industrial Ethernet to upload the raw data collected by each sensing unit to the data fusion and preprocessing module in real time, ensuring the real-time performance and reliability of data transmission, while supporting concurrent transmission of multiple devices.
3. The closed-loop spatial management system based on multimodal perception feedback according to claim 2, characterized in that, The data fusion and preprocessing module, during data standardization, converts heterogeneous data collected from different sensors into a unified format and unit, and uses a min-max standardization algorithm to normalize the data, ensuring consistent data volume. The formula used is: ,in For standardized data, The raw data collected by the sensor, This is the minimum value of this type of data. The maximum value of this data type; When performing data denoising and cleaning, an outlier detection algorithm is used to remove outliers and missing values from the data. The judgment formula used is as follows: When data satisfies this formula, it is identified as an outlier and removed. A filtering algorithm is used to remove noise signals caused by environmental interference, improving data reliability. The formula used is: ,in The mean of the data. The standard deviation of the data. The filtered data, This is the raw data currently being collected. This is the data after the previous filtering. These are the filter coefficients; During spatiotemporal alignment, data collected from different sensing units and at different times are synchronized and aligned based on a unified timestamp, constructing a spatiotemporal environmental data matrix M to ensure data spatiotemporal consistency. The data matrix expression is as follows: ,in The number of sensing units, Number of data collection points The data collected by the m-th sensing unit at time t and the n-th time point; During feature extraction, a feature extraction algorithm is used to extract key features from the preprocessed multi-source data, including the number of people, light intensity, temperature and humidity trends, and noise peaks, generating structured perception data, which is then output to the large-scale intelligent decision-making module. The extracted feature vector expression is as follows: Where N is the number of people and L is the light intensity. For temperature change trends, This shows the trend of humidity changes. This represents the peak noise level.
4. The closed-loop spatial management system based on multimodal perception feedback according to claim 3, characterized in that, The large-scale intelligent decision-making module includes a multimodal large-scale model, a decision rule base, and a dynamic decision engine; The aforementioned multimodal large model employs a pre-trained multimodal large model, possessing cross-modal understanding capabilities for image, text, numerical, and audio data. It can quickly fuse and analyze three types of data—personnel, environment, and equipment—to uncover the relationships between the data, employing a multimodal fusion function. To achieve multi-source data fusion, among which For image features, For text features, For numerical features, Audio features; The decision rule base includes comfort thresholds for space use, energy consumption optimization rules, and device coordination logic. The comfort thresholds include temperature 24-26℃, light intensity 300-700 lux, noise <50dB, and CO2 concentration <1000ppm. The energy consumption optimization rules include "turn off the equipment when no one is around" and "reduce the light intensity when there is sufficient light". The device coordination logic is used to ensure that the adjustment of air conditioning, lighting, and audio equipment does not conflict with each other. The dynamic decision engine operates as follows: First, it receives structured perception data output from the data fusion and preprocessing module. Second, a multimodal large model comprehensively analyzes the data and generates preliminary decision suggestions based on data trends. Finally, it combines the decision rule base and user-defined preferences to optimize and generate the final refined control instructions. The instructions specify the equipment's parameters. The preliminary decision suggestions include: when the number of people increases, it is recommended to lower the air conditioning temperature and increase the airflow; when the light intensity exceeds the standard, it is recommended to reduce the light intensity; when the environmental noise increases, it is recommended to appropriately increase the sound gain. The preliminary decision recommendations can be represented by the following decision function: ,in For changes in the number of people, For air conditioning temperature adjustment amount, This refers to the air conditioning fan speed adjustment. This is the amount of light illuminance adjustment. This represents the change in noise level. This is the audio gain adjustment amount.
5. The closed-loop spatial management system based on multimodal perception feedback according to claim 4, characterized in that, The protocol adaptation unit supports mainstream industrial control protocols such as Modbus, BACnet, MQTT, and ZigBee, and is compatible with different brands and types of air conditioners, lighting, and audio equipment. It eliminates the need to modify existing equipment, reducing system deployment costs. Protocol adaptation utilizes protocol conversion functions. ,in These are the standard protocol commands output by the system. Protocols supported by the device; The instruction parsing unit is used to convert the natural language or structured control instructions output by the large model intelligent decision-making module into electrical or digital signals that can be recognized by the controlled device, ensuring that the instructions can be accurately recognized by the device. The instruction parsing formula is as follows: ,in For signals that the device can recognize. For the final control command, Equipment signal encoding rules; The device control unit establishes a connection with the controlled device through a drive circuit, sends the parsed control signal to the device, and controls the device to perform actions such as temperature adjustment, illuminance adjustment, and volume adjustment. Simultaneously, it provides real-time feedback on the device's execution status. The drive formula for device control is: ,in For the output signal of the drive circuit, The driving coefficient is denoted as .
6. A closed-loop spatial scheduling method based on multimodal sensing feedback, based on the system described in any one of claims 1-5, characterized in that, Includes the following steps: S1: Perception Phase: Comprehensive Collection and Upload of Multi-Dimensional Raw Data: After the multi-modal perception module is activated, it collects personnel data, environmental data, and equipment status data in the space in real time, and uploads the raw data to the data fusion and preprocessing module through the data transmission unit; S2: Preprocessing stage: Optimize data quality and extract effective features: The data fusion and preprocessing module performs standardization, noise reduction, spatiotemporal alignment and feature extraction on the received multi-source raw data in sequence to generate structured perception data and output it to the large model intelligent decision-making module. S3: Decision-making stage: Generate precise and adapted equipment control commands based on preprocessed data: The large model intelligent decision-making module receives structured perception data, performs comprehensive analysis by the multimodal large model, combines the comfort threshold and energy consumption rules in the decision rule base with user-defined preferences, generates refined equipment control commands, and outputs them to the equipment execution module; S4: Execution phase: Transform decision commands into executable actions of the equipment to complete equipment adjustment operations: The equipment execution module converts control commands into signals that the equipment can recognize through the protocol adaptation unit and the command parsing unit, driving the air conditioning, lighting, and audio equipment to perform corresponding adjustment actions; S5: Feedback Phase: Collect execution results and new environmental data to provide feedback basis for closed-loop iteration: The closed-loop feedback module collects the actual operating status data after the device is executed, as well as the new environmental perception data in the space, and sends both types of data back to the multimodal perception module; S6: Iterative Loop: Closed-Loop Continuous Optimization to Ensure the Space Environment is Always in the Optimal State: Repeat S1-S5 to continuously perform the closed-loop cycle of "perception-decision-execution-re-perception". Based on environmental changes and equipment execution, dynamically optimize equipment control commands to ensure the space environment is always in the optimal state.
7. The closed-loop spatial scheduling method based on multimodal sensing feedback according to claim 6, characterized in that, The specific logical steps of S2 are as follows: S201: The min-max normalization algorithm is used to normalize the various types of raw data collected by S1 to the [0,1] interval, preserving the variation trend of the original data and achieving uniform data volume. The formula used is: ,in For standardized data, The raw data collected by the sensor, This is the minimum value of this type of data. The maximum value of this data type; S202: First, outliers are removed and missing values are added using the 3σ criterion. Then, environmental noise is eliminated using a first-order low-pass filtering algorithm, the formula of which is as follows: The formula used for outlier detection and removal is: ,in The mean of the data. The standard deviation of the data; The formula used for environmental noise filtering is: ,in The filtered data, This is the raw data currently being collected. This is the data after the previous filtering. These are the filter coefficients; S203: Using the system's unified timestamp t as a benchmark, synchronize and align all multi-source data after noise reduction and cleaning, and integrate data collected by different sensing units at the same timestamp to form a spatiotemporal data matrix M. ,in The number of sensing units, Number of data collection points The data collected by the m-th sensing unit at time t and the n-th time point; S204: Employs a feature extraction algorithm combining statistical analysis and convolutional neural networks to extract three core key features: personnel, environment, and equipment. This extracts feature vectors to form structured perception data. The formula used is as follows: Where N is the number of people and L is the light intensity. For temperature change trends, This shows the trend of humidity changes. This represents the peak noise level. This indicates the current operating status of the air conditioner. This indicates the current operating status of the lights. The current operating status of the audio system is determined, and the feature vector F is encapsulated into a structured data format and output in real time to the large model intelligent decision-making module as the core data input for decision analysis.
8. The closed-loop spatial scheduling method based on multimodal sensing feedback according to claim 7, characterized in that, The specific logical steps of S3 are as follows: S301: Receives the feature vector F corresponding to the structured sensing data, verifies the value range of each feature parameter, and corrects and supplements parameters that exceed the reasonable range or are missing. Its data verification and judgment formula is as follows: ,in To accommodate the maximum number of people in the target space, This represents the maximum reasonable value for light intensity. , This represents the maximum reasonable variation in temperature and humidity. This represents the maximum reasonable noise level. If a feature parameter exceeds the above reasonable range, it is determined to be invalid data, and the valid feature value from the previous time step is used to replace it. If there are missing features, the historical mean of that feature is used to supplement them. After the verification is passed, the valid feature vector is output. ; S302: Employs a multimodal fusion algorithm to combine effective feature vectors The image and audio features of the multimodal large model are fused to generate a fused feature matrix. Its expression is: ,in ( () is a multimodal fusion function, which uses a weighted summation method to achieve multi-source feature fusion. For image feature vectors, For audio feature vectors, , , The fusion weights are preset to 0.6, 0.2, and 0.2 respectively, and satisfy the following conditions: ; S303: By analyzing the changing trends of fused features through a multimodal large model, preliminary adjustment suggestions are generated for different devices. The preliminary decision results are represented by the adjustment amount, and the formula is as follows: ; in For air conditioning temperature adjustment, This refers to the adjustment amount of light illuminance. This is the audio gain adjustment amount. For changes in the number of people, This is the standard value for light intensity. This represents the change in noise peak value. , , , For adjustment coefficients, ; When the number of people increases, the air conditioning will cool down; when the light intensity is higher than the standard value, the lights will be dimmed; when the noise peak increases, the sound gain will increase, and vice versa. S304: First, the adjustment range is constrained by the decision rule base, then the adjustment parameters are optimized by combining user preferences to generate the final refined control command. The formula used is as follows: Rule base constraints: ,in , , The constrained settings for air conditioning temperature, lighting illuminance, and sound gain. , , These are the current operating parameters of the equipment. 24-26℃, 300-700lux, and 0-10dB are the comfort thresholds for air conditioning temperature, lighting illuminance, and sound gain, respectively. Final setpoint optimization formula: ,in For the optimized equipment control parameters, These are the device settings constrained by the rule base. Allow users to customize preference parameters. For preference weights; S305: Will The corresponding air conditioning temperature, lighting illuminance, and audio gain settings are encapsulated into control commands conforming to the device communication protocol. These commands include the command type, device ID, control parameters, and key execution time information. The encapsulated expression is: ,in As a unique identifier for the device, For instruction type, For the optimized equipment control parameters, For instruction execution time, The data is categorized by device type and output to the device execution module accordingly.
9. The closed-loop spatial scheduling method based on multimodal sensing feedback according to claim 8, characterized in that, The specific logical steps of S4 are as follows: S401: The instruction parsing unit decodes the received structured control instructions, filters out the corresponding device control parameters, and converts them into standardized analog electrical signals to ensure a uniform signal format. (If the air conditioner's control parameters are optimized...) The normalized voltage signal after analysis is: If the lighting control parameters are optimized The normalized voltage signal after analysis is: ; S402: The protocol adaptation unit extracts the device ID from the instruction. It identifies the device type and supported communication protocols, and converts the standardized voltage signal into a device-specific protocol signal using a protocol conversion algorithm. The protocol conversion algorithm formula is as follows: ,in This is a device-specific communication protocol signal. ( This is a protocol adaptation conversion function that dynamically adjusts based on the protocol types supported by the device. The types of communication protocols supported by the device. These are protocol conversion coefficients, set according to the protocol type. This is the protocol baseline value, preset to 0; S403: The device control unit receives the protocol signal after protocol adaptation. Through a power amplification algorithm, a drive signal for the corresponding device is generated. The strength of the drive signal is positively correlated with the adjustment amount of the control parameters. The formula for generating the drive signal is as follows: ,in This is the device drive current signal. The amplification factor is set according to the device power. For device-specific communication protocol signals after protocol adaptation; S404: The drive signal is transmitted to the device through the drive circuit. The device performs corresponding adjustment actions based on the signal strength. At the same time, the device status acquisition module collects the actual operating parameters after adjustment, converts them into feedback signals, and sends them back. The execution feedback formula is: ,in This is a feedback signal for the device's execution status. The feedback coefficient is preset to 0.
01. These are the actual operating status parameters of the equipment. The optimized control parameters for the decision-making stage; After receiving the drive signal, the device completes the adjustment according to the set parameters, and at the same time collects the actual operating parameters. Calculation and setting values The deviation is converted into a feedback signal. The data is then sent back to the closed-loop feedback module, completing the entire execution phase.