A home environment adjustment self-learning method and system based on multi-modal feedback and scene perception

Abstract

This invention relates to a self-learning method and system for home environment regulation based on multimodal feedback and scene awareness. The method collects the user's physiological characteristics and body movement signals through non-contact sensing devices to identify the user's current activity scene. The system integrates the user's explicit intervention operations with implicit comfort feedback based on physiological stability, dynamically assigning fusion weights for explicit and implicit feedback according to the activity scene to generate target feedback labels. An online incremental learning algorithm is used to update the local environmental control model, and a privacy-preserving model aggregation is performed in the cloud through a federated learning mechanism. This invention solves the problems of traditional home control relying on single commands and lacking physiological feedback mechanisms, achieving intelligent closed-loop control that balances user privacy and personalized comfort.

Description

Technical Field

[0001] This invention belongs to the field of smart home and artificial intelligence technology, and in particular relates to a self-learning method and system for home environment adjustment based on multimodal feedback and scene perception. Background Technology

[0002] With the development of IoT technology, smart home systems have evolved from simple remote control to intelligent automatic adjustment. However, existing technologies mainly rely on preset rules (such as timed on / off) or explicit commands (such as voice control), which presents the following pain points: 1. Lack of physiological feedback: The system cannot sense the user's actual temperature or sleep depth, resulting in "intelligent adjustment" often being uncomfortable.

[0003] 2. Poor scenario adaptability: It is normal for users to experience rapid breathing after exercise, but rapid breathing during sleep may indicate that the environment is too hot. Existing systems have difficulty distinguishing between these scenarios and are prone to misjudgment.

[0004] 3. Privacy concerns: Traditional cloud-based AI requires uploading user home videos or detailed data, posing a significant risk of privacy leaks. Summary of the Invention

[0005] The purpose of this invention is to provide a self-learning method and system for home environment regulation based on multimodal feedback and scene awareness. By integrating explicit instructions and implicit physiological signals and introducing a federated learning mechanism, personalized and scenario-based intelligent environmental regulation can be achieved while protecting privacy.

[0006] This invention provides a self-learning method for home environment regulation based on multimodal feedback and scene awareness, comprising the following steps: Step 1: Collect the user's vital signs signals in the home environment using non-contact sensing devices, extract the user's physiological characteristic data and body movement characteristics, and calculate the physiological stability index based on the changes of the physiological characteristic data over time. Step 2: In response to the user's control operation on the home environment devices, collect explicit intervention signals and convert the explicit intervention signals into explicit feedback tags; Step 3: Identify the user's current activity scenario based on the body movement features, and use a dynamic weighting strategy to perform multimodal fusion of the explicit feedback label and the implicit comfort feedback label generated based on physiological stability according to the activity scenario, so as to generate a target feedback label for model update. Step 4: Obtain the current environmental state features and user state features as input, use the local environmental control model to output recommended environmental control parameters, and update the parameters of the local environmental control model using an online incremental learning algorithm based on the target feedback label. Step 5: De-identify the updated model parameters and upload them to the cloud server, so that the cloud server can perform multi-terminal parameter aggregation based on the federated learning strategy and receive the globally optimized model parameters issued by the cloud.

[0007] Furthermore, the non-contact sensing device includes at least one of millimeter-wave radar, ultra-wideband radar, or infrared thermal imaging device; When millimeter-wave radar is used, step 1 includes: filtering and spectral analysis of the phase signal of the radar echo, extracting the respiratory frequency as the physiological characteristic data; and extracting the body motion amplitude as the body motion characteristic based on the Doppler feature.

[0008] Furthermore, the physiological stability index is the statistical dispersion of physiological characteristic data within multiple recent consecutive time windows; The statistical dispersion includes at least one of standard deviation, variance, or coefficient of variation; when the statistical dispersion is lower than a preset threshold, the user is determined to be in a physiologically comfortable state, and a positive implicit comfort feedback label is generated.

[0009] Furthermore, step 3 specifically includes: Identify the user's current activity scenario, which includes at least a resting scenario and an active scenario; When the user is identified as being in a resting environment, a first weight is assigned to the implicit comfort feedback label, and a second weight is assigned to the explicit feedback label, wherein the first weight is higher than the second weight; When a user is identified as being in an active scenario, a third weight is assigned to the implicit comfort feedback label, and a fourth weight is assigned to the explicit feedback label, wherein the fourth weight is higher than the third weight. The target feedback label is obtained by weighting the two labels based on the assigned weights.

[0010] Furthermore, the online incremental learning algorithm described in step 4 is configured to perform the following operations: The gradient of the loss function is calculated based on the difference between the target feedback label and the current model prediction value; Based on the gradient and the preset learning rate, the model weights are adjusted using an iterative optimization strategy to minimize the prediction error.

[0011] Furthermore, step 4 also includes a reverse intervention verification step: When a user is detected to have made an explicit intervention action that is opposite to the direction of the adjustment within a preset time after the environmental parameters have been automatically adjusted, the confidence of the model update weights is reduced, and the current state and intervention behavior are marked as negative samples and stored in the experience replay pool to suppress the model's erroneous predictions.

[0012] Further, the desensitization process described in step 5 includes: Removing the user identity information from the model parameters and adding differential privacy noise obeying a specific probability distribution to the uploaded model gradients.

[0013] Further, the cloud server in step 5 uses the federated averaging algorithm or its variant to perform weighted aggregation on the model parameters uploaded by multiple terminals.

[0014] The present invention also provides a home environment regulation system, including a non-contact perception module, an environmental parameter sensor group, a processing unit, a storage unit, and a communication unit. The processing unit is configured to execute the home environment regulation self-learning method based on multi-modal feedback and scenario perception as described above.

[0015] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method as described above is implemented.

[0016] By means of the above solution, through the home environment regulation self-learning method and system based on multi-modal feedback and scenario perception, by fusing explicit instructions and implicit physiological signals and introducing a federated learning mechanism, personalized and scenario-based environmental intelligent regulation can be achieved under the premise of protecting privacy.

[0017] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly and implement it according to the content of the specification, the following takes the preferred embodiments of the present invention and describes them in detail in conjunction with the drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of the home environment regulation self-learning method based on multi-modal feedback and scenario perception of the present invention; Figure 2 is a logic flowchart in an embodiment of the present invention; Figure 3 is an overall system architecture diagram in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The following will further describe in detail the specific embodiments of the present invention in conjunction with the drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0020] As shown in Figure 1 This embodiment provides a home environment regulation self-learning method based on multi-modal feedback and scenario perception, including the following steps: Step S1: Collect the user's vital signs signals in the home environment using a non-contact sensing device, extract the user's physiological characteristic data and body movement characteristics, and calculate the physiological stability index based on the changes of the physiological characteristic data in the time dimension. Step S2: In response to the user's control operation on the home environment equipment, collect explicit intervention signals and convert the explicit intervention signals into explicit feedback tags; Step S3: Identify the user's current activity scenario based on the body movement features, and perform multimodal fusion of the explicit feedback label and the implicit comfort feedback label generated based on physiological stability according to the activity scenario using a dynamic weighting strategy to generate a target feedback label for model update. Step S4: Obtain the current environmental state features and user state features as input, use the local environmental control model to output recommended environmental control parameters, and update the parameters of the local environmental control model using an online incremental learning algorithm based on the target feedback label. Step S5: The updated model parameters are anonymized and uploaded to the cloud server, so that the cloud server can perform multi-terminal parameter aggregation based on the federated learning strategy and receive the globally optimized model parameters sent from the cloud.

[0021] In this embodiment, the non-contact sensing device includes at least one of millimeter-wave radar, ultra-wideband (UWB) radar, or infrared thermal imaging device; When millimeter-wave radar is used, step S1 includes: filtering and spectral analysis of the phase signal of the radar echo, extracting the respiratory frequency as the physiological characteristic data; and extracting the body motion amplitude as the body motion characteristic based on Doppler features.

[0022] In this embodiment, the physiological stability index is the statistical dispersion of physiological characteristic data within multiple recent consecutive time windows; The statistical dispersion includes at least one of standard deviation, variance, or coefficient of variation; when the statistical dispersion is lower than a preset threshold, the user is determined to be in a physiologically comfortable state, and a positive implicit comfort feedback label is generated.

[0023] In this embodiment, step S3 specifically includes: Identify the user's current activity scenario, which includes at least a resting scenario and an active scenario; When the user is identified as being in a resting environment, a first weight is assigned to the implicit comfort feedback label, and a second weight is assigned to the explicit feedback label, wherein the first weight is higher than the second weight; When a user is identified as being in an active scenario, a third weight is assigned to the implicit comfort feedback label, and a fourth weight is assigned to the explicit feedback label, wherein the fourth weight is higher than the third weight. Perform weighted calculations on the two types of labels based on the assigned weights to obtain the target feedback label.

[0024] In this embodiment, the online incremental learning algorithm in step S4 is configured to perform the following operations: Calculate the gradient of the loss function based on the difference between the target feedback label and the current model prediction value; Based on the gradient and the preset learning rate, use an iterative optimization strategy to correct the model weights to minimize the prediction error.

[0025] In this embodiment, step S4 further includes a reverse intervention verification step: When an explicit intervention operation in the opposite direction of the adjustment is detected by the user within the preset time after the automatic adjustment of the environmental parameters, reduce the confidence of the weight updated by the model this time, and mark the current state and intervention behavior as negative samples and store them in the experience replay pool for suppressing the wrong predictions of the model.

[0026] In this embodiment, the desensitization processing in step S5 includes: Remove the user identity identification information in the model parameters, and add differential privacy noise that follows a specific probability distribution (such as Laplace distribution or Gaussian distribution) to the uploaded model gradient.

[0027] In this embodiment, the cloud server in step S5 uses the Federated Averaging algorithm (FedAvg) or its variant to perform weighted aggregation on the model parameters uploaded by multiple terminals.

[0028] Figure 2 To show the closed-loop process from perception, scene recognition, fusion to learning.

[0029] See Figure 3 The schematic diagram of the overall system architecture shown. A home environment regulation system includes a non-contact sensing module, an environmental parameter sensor group, a processing unit, a storage unit, and a communication unit. The processing unit is configured to execute the home environment regulation self-learning method based on multi-modal feedback and scene perception.

[0030] Embodiment 1: Perception and Scene Recognition Based on Millimeter Wave Radar In this embodiment, a 60GHz FMCW (Frequency Modulated Continuous Wave) millimeter wave radar is used as the sensing device.

[0031] Signal processing: After the radar echo signal is mixed and sampled by ADC, the phase change and micro-motion energy of the target are extracted through Range-Doppler Map.

[0032] Breathing extraction: The phase signal is bandpass filtered from 0.1Hz to 0.5Hz to extract the breathing waveform.

[0033] Scene recognition logic: The system calculates the integral of body kinetic energy per unit time.

[0034] If the energy score exceeds the set threshold (corresponding to walking, fitness, etc.), it is judged as an "active scene".

[0035] If the energy score is below the threshold and periodic breathing signals are detected, it is determined to be a "resting scenario" (including reading, watching TV, and sleeping).

[0036] Example 2: Multimodal Feedback Fusion with Dynamic Weights Traditional fusion methods typically use fixed ratios, which cannot adapt to diverse life scenarios. This invention employs a dynamic weighting strategy: Let y be the label for explicit feedback (voice / app adjustment). e The label for implicit feedback (physiological comfort) is y. i The calculation logic for the target label Y is as follows: Scenario A (Rest / Sleep): In this scenario, the user is unable to operate the device, and physiological signals best reflect thermal comfort. The system automatically increases the implicit feedback weight, for example, by setting an implicit weight W. i = 0.8, explicit weight W e = 0.2. If high respiratory stability (low standard deviation) is detected, then y i Marked as "comfortable", the model updates in the direction of maintaining the current parameters.

[0037] Scenario B (Active / Exercising): In this scenario, the user's rapid breathing stems from increased metabolism, not environmental discomfort. Physiological signals are less reliable, and the system relies on explicit user commands. Let W... i = 0.2, W e = 0.8.

[0038] Example 3: Online Learning and Reverse Intervention for Error Correction The local processing unit runs a lightweight neural network (such as an MLP or LSTM).

[0039] Forward propagation: Input current room temperature, humidity, time and user status, output air conditioner set temperature.

[0040] Feedback loop: The system calculates the loss function (Loss) based on the target label Y generated in Example 2, and updates the network weights using the gradient descent algorithm.

[0041] Reverse intervention mechanism: If the model raises the temperature to 26℃, and the user manually adjusts it back to 24℃ within 5 minutes, the system identifies this as "negative feedback." This sample will be stored in the "experience replay pool," and the penalty weight for this sample will be increased in subsequent training to prevent the model from repeating the same mistake.

[0042] Example 4: Privacy Protection Based on Federated Learning To address the issues of data silos and privacy, this system adopts a federated learning architecture.

[0043] Local update: The amount of gradient update performed locally on each home terminal. .

[0044] Differential privacy: Before uploading, the privacy settings are adjusted. Clipping and adding Laplace noise ensures that the user's daily routine cannot be inferred from the gradient.

[0045] Cloud Aggregation: The cloud server receives encrypted gradients from tens of thousands of households, calculates the average using the FedAvg algorithm, generates new global model parameters, and distributes them. This allows new users to immediately have a basic comfort model trained based on collective intelligence after installing the system, solving the "cold start" problem.

[0046] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0047] This invention collects users' physiological characteristics and body movement signals through non-contact sensing devices to identify the user's current activity scenario (such as resting or active). The system integrates the user's explicit intervention operations with implicit comfort feedback based on physiological stability, dynamically assigning fusion weights for explicit and implicit feedback according to the activity scenario to generate target feedback labels. It uses an online incremental learning algorithm to update the local environmental control model and performs privacy-preserving model aggregation in the cloud through a federated learning mechanism. This solves the problems of traditional home control relying on single commands and lacking physiological feedback mechanisms, achieving intelligent closed-loop control that balances user privacy and personalized comfort.

[0048] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A self-learning method for home environment regulation based on multimodal feedback and scene awareness, characterized in that, Includes the following steps: Step 1: Collect the user's vital signs signals in the home environment using non-contact sensing devices, extract the user's physiological characteristic data and body movement characteristics, and calculate the physiological stability index based on the changes of the physiological characteristic data over time. Step 2: In response to the user's control operation on the home environment devices, collect explicit intervention signals and convert the explicit intervention signals into explicit feedback tags; Step 3: Identify the user's current activity scenario based on the body movement features, and use a dynamic weighting strategy to perform multimodal fusion of the explicit feedback label and the implicit comfort feedback label generated based on physiological stability according to the activity scenario, so as to generate a target feedback label for model update. Step 4: Obtain the current environmental state features and user state features as input, use the local environmental control model to output recommended environmental control parameters, and update the parameters of the local environmental control model using an online incremental learning algorithm based on the target feedback label. Step 5: De-identify the updated model parameters and upload them to the cloud server, so that the cloud server can perform multi-terminal parameter aggregation based on the federated learning strategy and receive the globally optimized model parameters issued by the cloud.

2. The self-learning method for home environment regulation based on multimodal feedback and scene awareness as described in claim 1, characterized in that, The non-contact sensing device includes at least one of millimeter-wave radar, ultra-wideband radar, or infrared thermal imaging device. When millimeter-wave radar is used, step 1 includes: filtering and spectral analysis of the phase signal of the radar echo, and extracting the respiratory frequency as the physiological characteristic data; The amplitude of body motion is extracted based on Doppler features as the body motion feature.

3. The self-learning method for home environment regulation based on multimodal feedback and scene awareness as described in claim 1, characterized in that, The physiological stability index is the statistical dispersion of physiological characteristic data within multiple recent consecutive time windows. The statistical dispersion includes at least one of standard deviation, variance, or coefficient of variation; when the statistical dispersion is lower than a preset threshold, the user is determined to be in a physiologically comfortable state, and a positive implicit comfort feedback label is generated.

4. The self-learning method for home environment regulation based on multimodal feedback and scene awareness as described in claim 1, characterized in that, Step 3 specifically includes: Identify the user's current activity scenario, which includes at least a resting scenario and an active scenario; When the user is identified as being in a resting environment, a first weight is assigned to the implicit comfort feedback label, and a second weight is assigned to the explicit feedback label, wherein the first weight is higher than the second weight; When a user is identified as being in an active scenario, a third weight is assigned to the implicit comfort feedback label, and a fourth weight is assigned to the explicit feedback label, wherein the fourth weight is higher than the third weight. The target feedback label is obtained by weighting the two labels based on the assigned weights.

5. The self-learning method for home environment regulation based on multimodal feedback and scene awareness as described in claim 1, characterized in that, The online incremental learning algorithm described in step 4 is configured to perform the following operations: The gradient of the loss function is calculated based on the difference between the target feedback label and the current model prediction value; Based on the gradient and the preset learning rate, the model weights are adjusted using an iterative optimization strategy to minimize the prediction error.

6. The self-learning method for home environment regulation based on multimodal feedback and scene awareness as described in claim 5, characterized in that, Step 4 also includes a reverse intervention verification step: When a user is detected to have made an explicit intervention action that is opposite to the direction of the adjustment within a preset time after the environmental parameters have been automatically adjusted, the confidence of the model update weights is reduced, and the current state and intervention behavior are marked as negative samples and stored in the experience replay pool to suppress the model's erroneous predictions.

7. The self-learning method for home environment regulation based on multimodal feedback and scene awareness as described in claim 1, characterized in that, The desensitization process described in step 5 includes: Remove user identification information from the model parameters and add differential privacy noise that follows a specific probability distribution to the uploaded model gradient.

8. The self-learning method for home environment regulation based on multimodal feedback and scene awareness as described in claim 1, characterized in that, In step 5, the cloud server uses a federated average algorithm or a variant thereof to perform weighted aggregation of model parameters uploaded from multiple terminals.

9. A home environment regulation system, characterized in that, It includes a non-contact sensing module, an environmental parameter sensor group, a processing unit, a storage unit, and a communication unit, wherein the processing unit is configured to execute the home environment adjustment self-learning method based on multimodal feedback and scene perception as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method described in any one of claims 1 to 8.

Images