Smart home sensitive area dynamic covering system with multi-modal interaction

Abstract

This invention relates to the field of smart home technology, specifically to a multimodal interactive smart home sensitive area dynamic masking system. It includes a data acquisition unit for collecting user voice commands and hand gesture data, and for acquiring the user's gaze coordinates in real time using an eye-tracking device. An instruction acquisition unit performs semantic parsing of the voice commands based on a pre-trained speech recognition model to obtain sensitive area control instructions containing operation type and sensitive area information. It also acquires temporary masking state switching instructions based on the hand gesture data and performs coordinate mapping on the gaze coordinates to obtain detail enhancement instructions for the gaze area. Finally, an instruction operation unit performs global control of the sensitive area based on the sensitive area control instructions. This invention, by combining voice commands, hand gesture data, and eye-tracking data, achieves more comprehensive user input recognition, enabling a more accurate understanding of user needs and thus providing personalized services.

Description

Technical Field

[0001] This invention relates to the field of smart home technology, and more specifically to a dynamic masking system for sensitive areas of a smart home with multimodal interaction. Background Technology

[0002] Traditional methods typically rely on a single data source (e.g., timed control, simple motion sensors), failing to comprehensively capture user needs and environmental changes. This results in simplistic and inaccurate area occlusion adjustments. Traditional methods generally cannot respond to user behavior and environmental changes in real time. They depend on fixed programs or sensors, failing to perform sensitive area occlusion operations based on the user's gaze and movements. Traditional methods often do not consider changes in ambient light intensity caused by the occluded area, leading to wasted energy under varying lighting conditions. User interaction with traditional methods is relatively simple, typically relying on manual operation such as remote controls or buttons, lacking multimodal interaction (e.g., voice commands, motion, and eye tracking). This makes the user experience less intuitive and convenient. Traditional methods respond to user needs in a fixed manner, unable to personalize adjustments based on individual needs. They typically fail to consider specific user behaviors, thus limiting the adaptability of the occlusion effect. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a dynamic occlusion system for sensitive areas of smart homes with multimodal interaction.

[0004] The technical solution adopted to solve the above-mentioned technical problems is: a multimodal interactive smart home sensitive area dynamic occlusion system, including: The data acquisition unit is used to collect the user's voice commands and hand gesture data, and to obtain the user's gaze coordinates in real time based on the eye tracking device, so as to obtain a multimodal dataset containing the voice commands, hand gesture data and gaze coordinates; The instruction acquisition unit is used to perform semantic parsing on the voice instruction based on a pre-trained speech recognition model to obtain a sensitive area control instruction containing the operation type and sensitive area, to obtain a temporary occlusion state switching instruction based on the hand gesture data, and to perform coordinate mapping on the gaze point coordinates to obtain a detail enhancement instruction for the gaze area. The instruction operation unit is used to perform global control of the sensitive region based on the sensitive region control instruction, to perform temporary occlusion state switching of the sensitive region based on the temporary occlusion state switching instruction, and to adjust the pixel rendering accuracy of the gaze region based on the detail enhancement instruction, so as to obtain a dynamic occlusion control sequence. The illumination compensation unit is used to obtain the occlusion area based on the dynamic occlusion control sequence, calculate the ambient light compensation requirement based on the physical location coordinates of the occlusion area, and dim the smart lamps based on the ambient light compensation requirement so that the illuminance value of the unoccluded area is dynamically maintained within the visual comfort threshold range. The illumination correction unit is used to collect the ambient light intensity after the shading operation in real time based on the ambient light sensor, compare the ambient light intensity with the target light intensity to obtain the light intensity deviation value, and adjust the output power of the smart lamp based on the light intensity deviation value.

[0005] Preferably, the user's voice commands and hand gesture data are collected, including: The user's voice is acquired using a voice acquisition device to obtain the user's voice commands; The user's hand gesture data is obtained by capturing the user's hand gesture data using a camera integrated into the smart home system.

[0006] Preferably, the speech recognition model, trained on a large amount of speech data, is able to accurately recognize the user's speech commands and convert them into corresponding operation commands.

[0007] Preferably, the speech command is semantically parsed based on a pre-trained speech recognition model to obtain a sensitive region control command that includes the operation type and sensitive region, including: The voice command is segmented to obtain its keywords; Based on the speech recognition model, the keywords are matched with a lexicon of sensitive regions and operation types to obtain the control instructions for the sensitive regions; The sensitive area vocabulary includes common home areas such as bedrooms, living rooms, and kitchens, and the operation type vocabulary includes common operations such as opening, closing, and covering.

[0008] Preferably, coordinate mapping is performed on the gaze point coordinates to obtain detail enhancement instructions for the gaze region, including: The gaze point coordinates are mapped from the two-dimensional screen coordinate system to the three-dimensional space coordinate system to obtain the specific location of the gaze point coordinates in three-dimensional space. The gaze region is obtained based on the specific location, and pixel-level detail enhancement processing is performed on the gaze region based on a preset detail enhancement rule to obtain the detail enhancement instruction. The detail enhancement rules include increasing the resolution of the gaze area and enhancing the color contrast of the gaze area.

[0009] Preferably, global control of the sensitive area based on the sensitive area control command includes: Based on the operation type in the sensitive area control command, perform open and close cover operations on the sensitive area; Record the current status and operation history of the sensitive area so that the sensitive area can be monitored and managed in the future.

[0010] Preferably, adjusting the pixel rendering precision of the gaze region based on the detail enhancement instructions to obtain a dynamic occlusion control sequence includes: The pixel rendering of the gaze region is optimized based on the increased resolution and enhanced color contrast of the detail enhancement command to obtain the rendering effect of the gaze region; The rendering effect is updated in real time to obtain the dynamic masking control sequence; The dynamic occlusion control sequence includes occlusion status information of the sensitive area and dimming control information of the smart lamp.

[0011] Preferably, calculating the ambient light compensation requirement based on the physical location coordinates of the covered area includes: The ambient light intensity distribution of the uncovered area around the covered area is calculated based on the physical location coordinates of the covered area and the distribution of ambient light sensors in the smart home system. The degree of light impact of the shading operation on the uncovered area is obtained based on the ambient light intensity distribution. The ambient light compensation requirement is obtained based on the degree of light impact. The ambient light compensation requirements include the intensity and direction of the supplementary light.

[0012] Preferably, adjusting the output power of the smart lamp based on the light intensity deviation value includes: Obtain the positive or negative value of the light intensity deviation; If the light intensity deviation value is negative, then the output power of the smart lamp is increased; If the light intensity deviation value is positive, then the output power of the smart lamp is reduced; The output light intensity of the smart lamps is monitored in real time to ensure that the illuminance value of the uncovered area is stably maintained within the preset visual comfort threshold range.

[0013] The beneficial effects of the present invention are as follows: (1) The present invention achieves more comprehensive user input recognition by combining voice commands, hand gesture data and eye tracking data, and can more accurately understand user needs, thereby providing personalized services; (2) The present invention collects the light intensity data after covering in real time, compares it with the target light intensity, and dynamically adjusts the light output to ensure that the illuminance of the uncovered area is always within the visual comfort threshold range, avoiding the problem of local over-brightness or under-brightness; (3) The present invention dynamically adjusts the output power of the smart lamps according to the changes in ambient light and the covered area, optimizes energy use, avoids unnecessary light waste, and thus achieves more efficient energy management. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the overall system flow in one embodiment of the present invention; reference numerals: 1, data acquisition unit; 2, instruction acquisition unit; 3, instruction operation unit; 4, illumination compensation unit; 5, illumination correction unit. Detailed Implementation

[0015] Example 1, as Figure 1 As shown, the multimodal interactive smart home sensitive area dynamic masking system proposed in this invention includes: Data acquisition unit 1 is used to collect the user's voice commands and hand gesture data, and to obtain the user's gaze coordinates in real time based on the eye tracking device, so as to obtain a multimodal dataset containing voice commands, hand gesture data and gaze coordinates; The instruction acquisition unit 2 is used to perform semantic parsing of voice instructions based on a pre-trained speech recognition model to obtain sensitive area control instructions containing operation type and sensitive area, obtain temporary occlusion state switching instructions based on hand gesture data, and perform coordinate mapping on gaze point coordinates to obtain detail enhancement instructions for the gaze area. The instruction operation unit 3 is used to perform global control of the sensitive area based on the sensitive area control instruction, to perform temporary occlusion state switching of the sensitive area based on the temporary occlusion state switching instruction, and to adjust the pixel rendering accuracy of the gaze area based on the detail enhancement instruction, so as to obtain a dynamic occlusion control sequence. The illumination compensation unit 4 is used to obtain the occlusion area based on the dynamic occlusion control sequence, calculate the ambient light compensation requirement based on the physical location coordinates of the occlusion area, and dim the smart lamps based on the ambient light compensation requirement so that the illuminance value of the unoccluded area is dynamically maintained within the visual comfort threshold range. The illumination correction unit 5 is used to collect the ambient light intensity after the shading operation in real time based on the ambient light sensor, compare the ambient light intensity with the target light intensity to obtain the light intensity deviation value, and adjust the output power of the smart lamp based on the light intensity deviation value.

[0016] In this invention, a multimodal dataset refers to a dataset that simultaneously contains multiple types of data (such as voice commands, hand gesture data, gaze coordinates, etc.). Such a dataset can be used to train or analyze complex interaction methods and behavioral patterns. Voice commands and hand gesture data refer to a form of user input. Voice commands are usually converted into machine-understandable commands through speech recognition technology, while hand gesture data is user gestures captured by sensors. Both can serve as input signals for user interaction with the device. Gaze coordinates refer to the two-dimensional coordinates of the user's gaze focus obtained through eye-tracking technology. This helps the system know where the user's attention is focused, thereby making corresponding adjustments or enhancements in that area. Speech recognition models are models trained on large amounts of data that can convert user speech input into machine-understandable text. These models are based on large speech datasets and language rules, and through training, they learn how to recognize different speech commands and convert them into text information. Speech recognition models not only recognize speech content but also perform contextual understanding, thus more accurately interpreting the user's intent. Sensitive area control commands refer to operations such as displaying, occluding, or enhancing details in certain areas by analyzing speech commands, action data, etc., for example, controlling the display and occlusion of sensitive information on the screen. Temporary occlusion state switching commands refer to the system applying temporary occlusion operations to certain areas based on user commands or actions, for example... When a user makes a gesture, the system automatically hides or blurs the affected area; the detail enhancement command improves the image rendering accuracy within the area the user is looking at, making details clearer and more delicate, thus improving the visual experience; the ambient light compensation requirement means that when some areas are occluded, the system calculates the required light intensity for other uncovered areas to maintain overall lighting balance and comfort; intelligent lighting dimming means that intelligent lights automatically adjust their brightness according to ambient light compensation requirements, ensuring that users can obtain the best visual experience in different environments; the light intensity deviation value refers to the difference between the ambient light intensity measured by the system and the target light intensity. The system uses this deviation value to adjust the brightness of the lights to ensure that the ambient light intensity meets the preset standard.

[0017] Example 2: The multimodal interactive smart home sensitive area dynamic masking system proposed in this invention, compared with Example 1, further includes: Collect user voice commands and hand gesture data, including: The user's voice is acquired using a voice acquisition device to obtain the user's voice commands; The system uses cameras integrated into smart home systems to capture users' hand gestures and obtain waving gesture data.

[0018] In this embodiment, the voice acquisition device refers to hardware that captures user voice signals through sensor devices such as microphones. Typically, these devices can collect user voice commands in real time and convert them into machine-understandable commands or text through voice recognition technology. Cameras in smart home systems are usually integrated into smart home devices to capture images and videos of the surrounding environment in real time. In this scenario, they are mainly used to monitor and recognize user gestures.

[0019] In an optional embodiment, the speech recognition model, trained on a large amount of speech data, is able to accurately recognize the user's voice commands and convert them into corresponding operation commands.

[0020] In an optional embodiment, the speech command is semantically parsed based on a pre-trained speech recognition model to obtain a sensitive region control command that includes the operation type and sensitive region, including: The voice commands are segmented to obtain their keywords; Based on the speech recognition model, keywords are matched with sensitive areas and operation type vocabularies to obtain sensitive area control instructions; The sensitive area vocabulary includes common home areas such as bedrooms, living rooms, and kitchens, while the operation type vocabulary includes common operations such as opening, closing, and covering.

[0021] It's important to note that word segmentation refers to the process of converting continuous speech input into word units. Voice commands typically consist of a series of speech signals, which need to be broken down into specific words or phrases that can be recognized and understood. For example, if a user says "turn on the living room light," after word segmentation, the system will break it down into keywords such as "turn on," "living room," and "light" to extract the core information of the command. Keywords are the meaningful words or phrases extracted from the word segmentation results. They are the key parts of the voice command, carrying the user's actual needs. Keywords typically include action verbs (such as "turn on" and "turn off") and target objects or areas (such as "living room" and "bedroom"). These keywords help the system understand the user's commands and execute corresponding operations based on them. A sensitive area vocabulary refers to a list of words containing common area names in the home or workplace. For example, common areas such as bedrooms, living rooms, kitchens, and studies are included in the sensitive area vocabulary. This vocabulary serves to provide the speech recognition model with a reference for recognizing common areas, helping the system better understand user commands and locate the actual physical area. The operation type vocabulary contains keywords describing common home operations, such as "open," "close," "cover," and "adjust." These operation types are verbs or action terms frequently used by users when interacting with smart devices. By recognizing and matching these words, the system can determine the specific operation the user wants to perform. Matching the sensitive area and operation type vocabulary involves comparing the keywords in the voice command with the sensitive area and operation type vocabulary. Through matching, the speech recognition model can determine the sensitive area (e.g., "bedroom") and operation type (e.g., "open") in the user command, thereby generating accurate control commands.

[0022] In an optional embodiment, coordinate mapping is performed on the gaze point coordinates to obtain detail enhancement instructions for the gaze region, including: The gaze point coordinates are mapped from the two-dimensional screen coordinate system to the three-dimensional space coordinate system to obtain the specific location of the gaze point coordinates in three-dimensional space. The gaze region is obtained based on the specific location, and pixel-level detail enhancement processing is performed on the gaze region based on preset detail enhancement rules to obtain detail enhancement instructions; Among them, the detail enhancement rules include increasing the resolution of the gaze area and enhancing the color contrast of the gaze area.

[0023] It should be noted that detail enhancement rules refer to algorithms or strategies used to improve the details of an image or region. These rules typically involve improving aspects such as image sharpness, contrast, and color depth. The purpose of detail enhancement is to make the area the user is looking at more vivid, clear, or easily identifiable. Increasing the resolution of the gaze area is a method of detail enhancement, achieved through image enhancement techniques. This can increase the pixel density of the gaze area in the image, making the image in that area clearer and richer in detail, especially in the area the user is looking at. Enhancing the color contrast of the gaze area refers to adjusting the differences in colors in the image to make the visual effect of the gaze area more vivid. Enhancing color contrast makes the colors in the image more prominent and vivid, helping users to more easily identify and focus on specific details.

[0024] In an optional embodiment, global control of the sensitive area is performed based on sensitive area control commands, including: Based on the operation type in the sensitive area control command, perform open and close cover operations on the sensitive area; Record the current status and operation history of sensitive areas so that they can be monitored and managed in the future.

[0025] It should be noted that recording the current state of sensitive areas means that the system should monitor and record the visibility or occlusion status of sensitive areas at all times. For example, a sensitive area may be open at one time and covered at another. By recording the current state, the system can know the latest status of each sensitive area (such as whether it is covered or visible) so as to make appropriate responses or actions. Recording operation history means that the system should record all operations related to sensitive areas, including the type of operation (such as opening or closing the cover), the time of the operation, and the user who performed the operation. These records help to trace the operation history when needed, understand all operations on sensitive areas, and whether these operations comply with the specified requirements.

[0026] In an optional embodiment, the pixel rendering precision of the gaze region is adjusted based on detail enhancement instructions to obtain a dynamic occlusion control sequence, including: Pixel rendering optimization is performed on the gaze area based on the detail enhancement command to improve resolution and enhance color contrast, so as to obtain the rendering effect of the gaze area; The rendering effect is updated in real time to obtain a dynamic masking control sequence; The dynamic occlusion control sequence includes occlusion status information of sensitive areas and dimming control information of smart lighting fixtures.

[0027] It should be noted that real-time rendering updates refer to the system's ability to immediately adjust and optimize rendering effects when user actions or system environment change. For example, if the user moves their gaze point, the system will instantly update the rendering effect of the gaze area to maintain a high-quality visual experience. The dimming control information of smart lamps refers to the adjustment instructions for parameters such as brightness and color temperature of smart lamps. These instructions can automatically adjust the state of the lamps based on factors such as ambient light, user needs, and time of day. In visual optimization, lighting adjustment can also be combined with image rendering effects to provide a better visual experience.

[0028] In an optional embodiment, calculating the ambient light compensation requirement based on the physical location coordinates of the occluded area includes: The ambient light intensity distribution in the uncovered area around the covered area is calculated based on the physical location coordinates of the covered area and the distribution of ambient light sensors in the smart home system. The degree of light impact of the shading operation on the uncovered area is obtained based on the ambient light intensity distribution. Ambient light compensation requirements are determined based on the degree of light impact. Among them, the ambient light compensation requirements include the intensity and direction of the supplementary light.

[0029] It should be noted that the degree of illumination impact of the masking operation on the uncovered area refers to the effect of the masked area's shading on the illumination of the surrounding uncovered area. This can be calculated by determining the physical location, size, and shading angle of the masked area to estimate the changes in the light intensity and distribution of the uncovered area. The presence of the masked area may cause light refraction, reflection, or obstruction, altering the illumination conditions of the uncovered area. The supplementary light intensity refers to the increased light intensity required to compensate for insufficient illumination caused by the masking operation. The calculation of the supplementary light intensity is usually based on the readings of the ambient light sensor and the impact of the masked area on the surrounding illumination. The supplementary light direction refers to the direction of illumination from the supplementary light source. To ensure that the supplementary light can evenly illuminate the uncovered area, the supplementary light direction will be adjusted according to the position and shape of the masked area to ensure that the supplementary light can effectively fill the area with insufficient illumination.

[0030] In an optional embodiment, adjusting the output power of the smart luminaire based on the illuminance deviation value includes: Obtain the positive or negative value of the light intensity deviation; If the light intensity deviation value is negative, the output power of the smart lamp will be increased; If the light intensity deviation value is positive, the output power of the smart lamps will be reduced. The system monitors and adjusts the output light intensity of the smart lights in real time to ensure that the illuminance value of the uncovered area is kept within the preset visual comfort threshold range.

[0031] It should be noted that the visual comfort threshold range refers to the range of light intensity that the human eye feels comfortable under specific lighting conditions.

[0032] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A multimodal interactive smart home sensitive area dynamic masking system, characterized in that, include: The data acquisition unit (1) is used to collect the user's voice commands and waving action data, and to obtain the user's gaze coordinates in real time based on the eye tracking device, so as to obtain a multimodal dataset containing the voice commands, waving action data and gaze coordinates; The instruction acquisition unit (2) is used to perform semantic parsing of the speech instruction based on the pre-trained speech recognition model to obtain sensitive area control instructions containing operation type and sensitive area, obtain temporary occlusion state switching instructions based on the hand gesture data, and perform coordinate mapping on the gaze point coordinates to obtain detail enhancement instructions for the gaze area. The instruction operation unit (3) is used to perform global control of the sensitive area based on the sensitive area control instruction, to perform temporary occlusion state switching of the sensitive area based on the temporary occlusion state switching instruction, and to adjust the pixel rendering accuracy of the gaze area based on the detail enhancement instruction, so as to obtain a dynamic occlusion control sequence. The illumination compensation unit (4) is used to obtain the occlusion area based on the dynamic occlusion control sequence, calculate the ambient light compensation requirement based on the physical location coordinates of the occlusion area, and dim the smart lamps based on the ambient light compensation requirement so that the illuminance value of the unoccluded area is dynamically maintained within the visual comfort threshold range. The illumination correction unit (5) is used to collect the ambient light intensity after the occlusion operation in real time based on the ambient light sensor, compare the ambient light intensity with the target light intensity to obtain the light intensity deviation value, and adjust the output power of the smart lamp based on the light intensity deviation value.

2. The multimodal interactive smart home sensitive area dynamic occlusion system according to claim 1, characterized in that, Collect user voice commands and hand gesture data, including: The user's voice is acquired using a voice acquisition device to obtain the user's voice commands; The user's hand gesture data is obtained by capturing the user's hand gesture data using a camera integrated into the smart home system.

3. The multimodal interactive smart home sensitive area dynamic occlusion system according to claim 2, characterized in that, The speech recognition model, trained on a large amount of speech data, can accurately recognize users' voice commands and convert them into corresponding operation commands.

4. The multimodal interactive smart home sensitive area dynamic occlusion system according to claim 3, characterized in that, The speech commands are semantically parsed based on a pre-trained speech recognition model to obtain sensitive region control commands that include operation type and sensitive region, including: The voice command is segmented to obtain its keywords; Based on the speech recognition model, the keywords are matched with a lexicon of sensitive regions and operation types to obtain the control instructions for the sensitive regions; The sensitive area vocabulary includes common home areas such as bedrooms, living rooms, and kitchens, and the operation type vocabulary includes common operations such as opening, closing, and covering.

5. The multimodal interactive smart home sensitive area dynamic occlusion system according to claim 4, characterized in that, Coordinate mapping is performed on the gaze point coordinates to obtain detail enhancement instructions for the gaze region, including: The gaze point coordinates are mapped from the two-dimensional screen coordinate system to the three-dimensional space coordinate system to obtain the specific location of the gaze point coordinates in three-dimensional space. The gaze region is obtained based on the specific location, and pixel-level detail enhancement processing is performed on the gaze region based on a preset detail enhancement rule to obtain the detail enhancement instruction. The detail enhancement rules include increasing the resolution of the gaze area and enhancing the color contrast of the gaze area.

6. The multimodal interactive smart home sensitive area dynamic occlusion system according to claim 5, characterized in that, Global control of the sensitive area based on the sensitive area control command includes: Based on the operation type in the sensitive area control command, perform open and close cover operations on the sensitive area; Record the current status and operation history of the sensitive area so that the sensitive area can be monitored and managed in the future.

7. The multimodal interactive smart home sensitive area dynamic occlusion system according to claim 6, characterized in that, The pixel rendering precision of the gaze region is adjusted based on the aforementioned detail enhancement instructions to obtain a dynamic occlusion control sequence, including: The pixel rendering of the gaze region is optimized based on the increased resolution and enhanced color contrast of the detail enhancement command to obtain the rendering effect of the gaze region; The rendering effect is updated in real time to obtain the dynamic masking control sequence; The dynamic occlusion control sequence includes occlusion status information of the sensitive area and dimming control information of the smart lamp.

8. The multimodal interactive smart home sensitive area dynamic occlusion system according to claim 7, characterized in that, Calculate the ambient light compensation requirement based on the physical location coordinates of the covered area, including: The ambient light intensity distribution of the uncovered area around the covered area is calculated based on the physical location coordinates of the covered area and the distribution of ambient light sensors in the smart home system. The degree of light impact of the shading operation on the uncovered area is obtained based on the ambient light intensity distribution. The ambient light compensation requirement is obtained based on the degree of light impact. The ambient light compensation requirements include the intensity and direction of the supplementary light.

9. The multimodal interactive smart home sensitive area dynamic occlusion system according to claim 8, characterized in that, Adjusting the output power of the smart lamp based on the light intensity deviation value includes: Obtain the positive or negative value of the light intensity deviation; If the light intensity deviation value is negative, then the output power of the smart lamp is increased; If the light intensity deviation value is positive, then the output power of the smart lamp is reduced; The output light intensity of the smart lamps is monitored in real time to ensure that the illuminance value of the uncovered area is stably maintained within the preset visual comfort threshold range.

Images