Intelligent home control method and system based on visual control

Abstract

The application discloses a smart home control method and system based on visual control, relates to the technical field of smart home control, and comprises the following steps: collecting multi-source data of a smart home environment, constructing real-time visual data flow, and generating a space correlation matrix between a user and a device based on the real-time visual data flow; based on the real-time visual data flow, a head micro-pose detection method combining an extended Haar-like feature and a Boosted Cascade classifier is adopted, a continuous head micro-pose time sequence is constructed through the positional offset relationship of a nose region relative to a face region; head pose features are extracted based on the continuous head micro-pose time sequence, a cross-device visual attention transfer relationship is constructed in combination with user eye orientation features and the space correlation matrix, and the application has the advantages that through joint analysis of head micro-pose and eye orientation based on real-time visual data flow, and in combination with a multi-user visual game control strategy, the recognition accuracy of a smart home system for implicit user demands and the coordination of multi-user device control are improved.

Description

Technical Field

[0001] This invention relates to the field of smart home control technology, and in particular to a vision-based smart home control method and system. Background Technology

[0002] With the rapid development of smart home technology, the number of smart devices in the home environment is constantly increasing, evolving from the initial single control mode to a comprehensive interactive system involving multiple devices and users. In existing technologies, smart home control mainly relies on traditional human-computer interaction methods, such as voice recognition, touch operation, or application control based on mobile terminals. These technologies enable basic operations of devices such as lights, air conditioners, and security systems in the home environment, and to a certain extent, automate device behavior management through scene modes or rule engines. In recent years, researchers have gradually introduced visual perception technology into the field of smart home control, acquiring user behavior information through cameras or depth sensors to achieve a more natural and contactless interaction method.

[0003] However, existing vision-based smart home control technologies still have certain limitations when processing multi-source environmental information and multi-user behavioral dynamics. Traditional visual perception methods typically focus only on single features, such as gestures or head postures, lacking the ability to comprehensively analyze head micro-postures, eye orientation, and device spatial layout. Furthermore, in multi-user scenarios, existing technologies struggle to effectively integrate implicit needs information from different users and dynamically allocate control to optimize the device user experience. Therefore, there is room for improvement in areas such as device control intent prediction, user visual attention shift analysis, and multi-user control conflict resolution. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a vision-based smart home control method to solve the problems of insufficient accuracy in recognizing users' implicit control intentions and poor coordination of dynamic control rights of devices in multi-user smart home scenarios.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a vision-based smart home control method, which includes, Collect multi-source data of the smart home environment, construct a real-time visual data stream, and generate a spatial association matrix between users and devices based on the real-time visual data stream; Based on real-time visual data stream, a head micro-pose detection method combining extended Haar-like features and Boosted Cascade classifier is adopted. A continuous head micro-pose time series is constructed by the positional offset relationship between the nose region and the face region. Head posture features are extracted based on continuous head micro-pose time series, and user eye orientation features and spatial correlation matrix are combined to construct cross-device visual attention transfer relationship and generate user visual attention transfer trajectory. Based on the user's visual attention migration trajectory, the user's control intention is predicted, a predicted intention set is generated, and the predicted intention set is used as a constraint condition to identify the user's implicit demand information in combination with human micro-motion characteristics. When multiple users are detected, the implicit demand information of each user is merged to generate a set of family needs; Based on the set of family needs, combined with information on conflicting needs and the visual attention migration trajectory of multiple users, the control rights of multi-user visual games are allocated, a dynamic control right sequence of devices is generated, and a smart home control strategy is generated based on the dynamic control right sequence of devices.

[0007] As a preferred embodiment of the vision-based smart home control method of the present invention, the step of collecting multi-source data of the smart home environment, constructing a real-time visual data stream, and generating a spatial association matrix between the user and the device based on the real-time visual data stream includes the following steps: Collect environmental image data of the smart home environment, as well as spatial location data of users and devices; By continuously updating the location data of users and devices in the image through continuous frame target detection, a real-time visual data stream containing dynamic visual information of users and environmental spatial information is formed. Based on real-time visual data streams, correlation parameters are calculated according to the relative distance between the user and each device, the angle of facial orientation, and the interactive range. Users are represented as rows, devices as columns, and the association parameters between users and each device are used as matrix elements to construct a spatial association matrix that reflects the spatial relationship between users and devices.

[0008] As a preferred embodiment of the vision-based smart home control method of the present invention, the method for detecting head micro-pose based on real-time visual data streams, using a combination of extended Haar-like features and a Boosted Cascade classifier, constructs a continuous head micro-pose time series by means of the positional offset relationship between the nose region and the face region, including the following steps: Image frames are acquired based on real-time visual data streams and grayscale processing is performed. An integral image is then constructed based on the grayscale image. Sliding detection windows are set at different scales to scan the image region, and extended Haar-like features are used to extract head contour features, facial edge features, and local features of the eyes and nose within each window to obtain the feature response values ​​of each window. Based on the matching degree between the feature response values ​​of each detection window and the preset head feature template, image regions that meet the head feature conditions are selected as candidate user head regions. The candidate regions of the user's head are sequentially input into the pre-trained Boosted Cascade classifier model, and feature matching and classification are performed step by step. Candidate regions that do not meet the conditions are eliminated, and finally the user's facial regions are determined. Locate the nose region within the facial region, calculate the horizontal and vertical offsets of the nose center point relative to the geometric center of the face, and obtain the nose offset vector at consecutive time points. Based on the changes in the nose offset vector at continuous time intervals, the micro-pose features of the user's head in the pitch, yaw and roll directions are extracted. The continuous head micro-pose features are arranged in chronological order to generate a continuous head micro-pose time series.

[0009] As a preferred embodiment of the vision-based smart home control method of the present invention, the step of establishing cross-device visual attention transfer relationships and generating user visual attention transfer trajectories includes the following steps: Based on real-time visual data stream, the user's eye area is identified, the user's gaze direction and eye fixation direction features are extracted, and a sequence of the user's eye orientation is generated. The continuous head micro-pose feature sequence is time-aligned with the user's eye orientation feature sequence, and the head pose and eye gaze direction at each time step are weighted and fused to obtain a comprehensive visual attention direction vector. During the fusion process, the head pose fusion weights and eye gaze fusion weights are constructed as a weight vector to be optimized; Multiple weight particles are randomly generated within a preset weight range, and each weight particle represents a set of candidate fusion weight combinations. Based on the deviation between the fused visual attention direction and the historical attention direction, a visual attention continuity evaluation index is constructed, and the visual attention continuity evaluation index is used as the fitness value of the weight particles. Based on the fitness value of each weighted particle, determine the weight combination corresponding to the highest fitness obtained by each weighted particle in the historical iteration process, and use it as the individual optimal weighted particle; The weight particle with the highest fitness value among all individual optimal weight particles is selected as the global optimal weight particle. During the iterative update of the weighted particles, the inertial weights are dynamically adjusted based on the nonlinear perturbation parameters generated by the Logistic chaotic mapping, so that the weighted particles maintain a strong global search capability in the initial stage and gradually enhance their local optimization capability in the subsequent stage. A mapping perturbation factor generated based on the Tent chaotic mapping is introduced to dynamically adjust the individual learning coefficient and the group learning coefficient, so that the weighted particles can update their own positions according to the guiding directions of the individual optimal weighted particles and the global optimal weighted particles. Based on the updated position, a new candidate fusion weight combination is generated, and the corresponding fitness value is recalculated. The update process of individual optimal weight particles and global optimal weight particles is repeated. When the maximum number of iterations is reached, the head pose fusion weight and eye gaze fusion weight corresponding to the globally optimal weight particle are determined as the optimal fusion weight; Based on the optimal fusion weight, the head pose direction vector and the eye gaze direction vector are weighted and fused to generate visual attention direction information; The visual attention direction information is mapped to the devices in the spatial association matrix to determine the target attention device, and the change events of the target attention device at adjacent time moments are recorded to form the visual attention transfer relationship between devices; The number of times the target device changes per unit of time is counted to obtain the device switching frequency; Calculate the device dwell time corresponding to the target device based on the length of time the same target device is continuously monitored within consecutive time periods. Based on the number of times the same target device is refocused within a preset time window, extract the device refocusing feature; The device switching frequency, device dwell time, and device repetition attention characteristics are normalized to construct a visual attention transfer feature vector; A visual attention transfer graph is established based on the transfer relationship between the target attention devices at continuous time, where the target attention devices are graph nodes and the visual attention transfer relationship between devices is graph edge. The weights of graph edges are determined based on the device switching frequency, the weights of node attention are determined based on the device dwell time, and the weights of return visits are determined based on the repeated attention characteristics of devices. Visual attention events are encoded into a visual attention migration map in chronological order to generate a user visual attention migration trajectory that reflects changes in user visual attention across devices.

[0010] As a preferred embodiment of the vision-based smart home control method of the present invention, the step of predicting the user's control intention based on the user's visual attention migration trajectory, generating a predicted intention set, and using the predicted intention set as a constraint condition, combined with human micro-motion features to identify the user's implicit demand information, includes the following steps: Based on the user's visual attention migration trajectory, the user's visual attention sequence, device dwell time, device switching frequency and repeated device attention features are extracted to generate user visual attention behavior features. Based on user visual attention behavior characteristics and combined with historical device interaction records, the system predicts the user's possible device control direction and target operation category, and generates a set of predicted user intentions. Based on real-time visual data stream, the system identifies local areas of the user's limbs, extracts the user's hand movements, body posture changes, and local movement amplitude features, and generates a sequence of the user's human micro-movement features. The device control direction and target operation category in the predicted intent set are converted into intent constraint labels, and the human micro-motion feature sequence is filtered based on the intent constraint labels to remove human actions that are inconsistent with the target device control direction or unrelated to the target operation type. The filtered human actions are associated and matched with each predicted intent in the predicted intent set. The matching degree between the human actions and the predicted intent is calculated, and the human actions with a matching degree greater than a preset threshold are identified as target human action features. Based on the correlation between target human motion features and predicted intent sets, the system identifies the user's corresponding device control needs, environmental adjustment needs, and device interaction needs, and generates implicit user demand information.

[0011] As a preferred embodiment of the vision-based smart home control method of the present invention, the step of fusing the implicit demand information of each user to generate a family demand set when multiple users are detected includes the following steps: Based on real-time visual data streams, the number of users in a scene is detected, and a correspondence between users and implicit demand information is established. If only a single user is detected, a set of family needs is generated directly based on that user's implicit needs information; When multiple users have conflicting needs for the same target device, the conflicting needs are analyzed by combining the users' historical interaction habits, current environmental status and the intensity of the needs, and information on the conflicting needs is generated. Based on the correspondence between users and implicit demand information, and combined with demand conflict information, the implicit demand information of multiple users is integrated to generate a family demand set.

[0012] As a preferred embodiment of the vision-based smart home control method of the present invention, the following steps are included: Based on a set of family needs, combined with information on conflicting needs and the visual attention migration trajectories of multiple users, the control rights of multi-user visual games are allocated to generate a dynamic control right sequence for devices, and a smart home control strategy is generated based on the dynamic control right sequence for devices: Based on the set of household needs, extract the target equipment needs, demand intensity, demand category, and demand conflict relationships for each user; When the demand conflict relationship information indicates that multiple users have control conflicts for the same target device, the visual attention migration trajectory of each user is combined to extract the visual attention intensity, attention duration and attention shift trend of each user for the target device, and the visual participation of each user for the target device is calculated accordingly. Based on the visual engagement with the target device, combined with the intensity of demand and users' historical interaction habits, the visual game evaluation value of each user for the target device is calculated. Based on the conflicting demand relationships, calculate the conflict probability of the target equipment, and calculate the cost of switching control based on the change in control ownership between adjacent time points. Based on visual game evaluation value, conflict probability and control switching cost, the control benefit of each user for the target device is continuously optimized to construct an optimizable control benefit function. Based on the control right benefit function, the control right benefits of each user within a continuous time window are predicted in a rolling manner to generate the initial control right allocation result, and a control right allocation neighborhood set is constructed with the initial control right allocation result as the center. A rolling search is performed within the control allocation neighborhood set, and the user to whom control of the target device belongs at each consecutive moment is determined by the joint constraint of maximizing benefits and minimizing switching costs. When the frequency of control switching of the target device within a preset time window exceeds a set threshold, a control retention factor is introduced to modify the control benefit function and suppress frequent control switching. The user who controls the target device at any given moment is arranged in chronological order to generate a dynamic sequence of device control rights. Based on the dynamic control sequence of devices, user needs in the set of household needs are matched to generate smart home control strategies.

[0013] Secondly, this invention provides a vision-based smart home control system, including: The multi-source data acquisition module collects images, spatial location information, and device status data in the smart home environment; The real-time visual data stream construction module continuously processes the acquired images and spatial information to generate a real-time data stream containing user dynamic behavior and environmental status. The head micro-pose detection module uses extended Haar-like features and a Boosted Cascade classifier to detect the user's head pose and construct a head micro-pose time series. The attention transfer analysis module generates cross-device visual attention relationships and user visual attention transfer trajectories based on head posture and eye orientation features, combined with a spatial correlation matrix. The user intent prediction module predicts the user's possible device control intents based on the visual attention migration trajectory and generates a set of predicted intents. The human micro-motion recognition module analyzes the user's body movements and local micro-motion features, associates and matches them with predicted intentions, and identifies the user's implicit needs information. The multi-user control allocation module integrates implicit demand information and visual attention trajectories from multiple users to perform game theory calculations to generate a dynamic sequence of device control rights, and generates smart home control strategies accordingly.

[0014] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the vision-based smart home control method described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the vision-based smart home control method described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By collecting multi-source data from the smart home environment and constructing a real-time visual data stream, it achieves collaborative perception of user status information, device spatial information, and environmental dynamic information, thereby improving the smart home's environmental adaptability and spatial perception capabilities in response to user behavior; by employing extended Haar-like features and Boosted... This paper presents a head micro-pose detection method combined with a Cascade classifier. By constructing a continuous head micro-pose time series based on the positional offset of the nose region relative to the face region, it achieves continuous tracking and pose perception of subtle head movements, more accurately reflecting changes in user visual attention and improving the stability of visual perception and the accuracy of user intent recognition. By combining head pose features, eye orientation features, and spatial correlation matrices to construct cross-device visual attention transfer relationships, it achieves dynamic mapping of the user's visual attention process across different devices, enabling analysis of the shifting trend of user visual attention and improving the system's ability to predict user control goals. By generating a predicted intent set based on the user's visual attention transfer trajectory and combining it with human micro-motion features to identify implicit user needs, it achieves joint analysis of explicit and implicit user visual behaviors, enabling early identification of potential user control needs and improving the initiative and naturalness of smart home control. Furthermore, by fusing implicit needs information from multiple users and combining it with information on conflicting needs and multi-user visual attention transfer trajectories for visual game control allocation, it achieves dynamic coordination of device control in multi-user scenarios, reducing multi-user device control conflicts and improving the synergy and intelligence level of smart home systems in complex home environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a vision-based smart home control method.

[0019] Figure 2 This is a schematic diagram of a vision-based smart home control system.

[0020] Figure 3 A flowchart for constructing a continuous head micro-pose time series.

[0021] Figure 4 To generate a flowchart for smart home control strategies. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a vision-based smart home control method, including the following steps: Collect multi-source data of the smart home environment, construct a real-time visual data stream, and generate a spatial association matrix between users and devices based on the real-time visual data stream.

[0026] Specifically, it collects environmental image data of the smart home environment, as well as spatial location data of users and devices; By continuously updating the location data of users and devices in the image through continuous frame target detection, a real-time visual data stream containing dynamic visual information of users and environmental spatial information is formed. Based on real-time visual data streams, correlation parameters are calculated according to the relative distance between the user and each device, the facial orientation angle, and the interactive range. The formula is as follows: ; in, Index for users, For device indexing, For the first The user and the first The associated parameters of each device For the first The user and the first Distance correlation value of each device For the first The user and the first The facial orientation correlation value of each device For the first The user and the first The interaction range associated with each device. , , These are the weighting coefficients for the distance association value, face orientation association value, and interaction range association value, respectively.

[0027] The distance correlation value is calculated based on the actual distance between the user and the device and the preset maximum effective interaction distance between the user and the device, using the following formula: ; in, For the first The user and the first The actual distance between the devices This is the preset maximum effective interaction distance.

[0028] The facial orientation correlation value is calculated based on the angle between the user's facial orientation direction and the direction the user is pointing at the device. The formula is as follows: ; in, For the first The user and the first The angle between the devices.

[0029] The interaction range correlation value is calculated based on the overlap ratio between the user's current location and the device's interactive area, using the following formula: ; in, This is a function for calculating the area of ​​a region. For the first The location area of ​​each user For the first Interactive areas of each device.

[0030] Users are represented as rows, devices as columns, and the association parameters between users and each device are used as matrix elements to construct a spatial association matrix that reflects the spatial relationship between users and devices.

[0031] By collecting image data of the smart home environment and spatial location data of users and devices, and using continuous frame target detection to dynamically update the positions of users and devices, a real-time visual data stream is constructed, which can continuously reflect user behavior and environmental status. Furthermore, based on this real-time visual data stream, correlation parameters are calculated according to the relative position of the user and each device, the angle of facial orientation, and the interactive range. The correlation parameters are then matrixed with the user as the row and the device as the column to construct a spatial correlation matrix. This achieves a quantitative representation of the spatial relationship between users and devices, provides an accurate user-device interaction basis for the smart home system, and improves the environmental adaptability of user behavior.

[0032] Based on real-time visual data streams, a head micro-pose detection method combining extended Haar-like features and a Boosted Cascade classifier is adopted. A continuous head micro-pose time series is constructed by the positional offset relationship between the nose region and the face region.

[0033] Specifically, image frames are acquired based on real-time visual data streams and grayscale processing is performed, and an integral image is constructed based on the grayscale image; Sliding detection windows are set at different scales to scan the image region, and extended Haar-like features are used to extract head contour features, facial edge features, and local features of the eyes and nose within each window to obtain the feature response values ​​of each window. Based on the matching degree between the feature response values ​​of each detection window and the preset head feature template, image regions that meet the head feature conditions are selected as candidate user head regions. The candidate regions of the user's head are sequentially input into the pre-trained Boosted Cascade classifier model, and feature matching and classification are performed step by step. Candidate regions that do not meet the conditions are eliminated, and finally the user's facial regions are determined. Locate the nose region within the facial region, calculate the horizontal and vertical offsets of the nose center point relative to the geometric center of the face, and obtain the nose offset vector at consecutive time points. Based on the changes in the nose offset vector at continuous time intervals, the micro-pose features of the user's head in the pitch, yaw and roll directions are extracted. The continuous head micro-pose features are arranged in chronological order to generate a continuous head micro-pose time series.

[0034] By converting image frames to grayscale based on real-time visual data streams and constructing integral maps, and then using sliding detection windows of different scales to extract extended Haar-like features, the head contour, facial edges, and local features of the eyes and nose are analyzed. This enables the screening of candidate regions for the user's head and accurate facial localization. Then, a Boosted Cascade classifier is used to progressively match and eliminate regions that do not meet the criteria. Furthermore, the center point of the nose is located and its offset relative to the geometric center of the face is calculated. The pitch, yaw, and tilt micro-pose features of the head at continuous time moments are extracted and a continuous head micro-pose time series is generated in chronological order. This enables continuous perception of the user's subtle head movements, providing a high-precision and quantifiable pose basis for visual attention transfer analysis and prediction of user control intentions. This improves the sensitivity of smart homes to changes in user behavior and the accuracy of control responses.

[0035] Head posture features are extracted based on continuous head micro-pose time series. Combined with user eye orientation features and spatial correlation matrix, cross-device visual attention transfer relationship is constructed to generate user visual attention transfer trajectory.

[0036] Specifically, based on real-time visual data streams, the user's eye region is identified, and features of the user's gaze direction and eye fixation direction are extracted to generate a sequence of the user's eye orientation. The continuous head micro-pose feature sequence is time-aligned with the user's eye orientation feature sequence, and the head pose and eye gaze direction at each time step are weighted and fused to obtain a comprehensive visual attention direction vector. During the fusion process, the head pose fusion weights and eye gaze fusion weights are constructed as a weight vector to be optimized; Multiple weight particles are randomly generated within a preset weight range, and each weight particle represents a set of candidate fusion weight combinations. Based on the deviation between the fused visual attention direction and the historical attention direction, a visual attention continuity evaluation index is constructed, and the visual attention continuity evaluation index is used as the fitness value of the weight particles. Based on the fitness value of each weighted particle, determine the weight combination corresponding to the highest fitness obtained by each weighted particle in the historical iteration process, and use it as the individual optimal weighted particle; The weight particle with the highest fitness value among all individual optimal weight particles is selected as the global optimal weight particle. During the iterative update of the weighted particles, the inertial weights are dynamically adjusted based on the nonlinear perturbation parameters generated by the Logistic chaotic mapping, so that the weighted particles maintain a strong global search capability in the initial stage and gradually enhance their local optimization capability in the subsequent stage. A mapping perturbation factor generated based on the Tent chaotic mapping is introduced to dynamically adjust the individual learning coefficient and the group learning coefficient, so that the weighted particles can update their own positions according to the guiding directions of the individual optimal weighted particles and the global optimal weighted particles. Based on the updated position, a new candidate fusion weight combination is generated, and the corresponding fitness value is recalculated. The update process of individual optimal weight particles and global optimal weight particles is repeated. When the maximum number of iterations is reached, the head pose fusion weight and eye gaze fusion weight corresponding to the globally optimal weight particle are determined as the optimal fusion weight; Based on the optimal fusion weight, the head pose direction vector and the eye gaze direction vector are weighted and fused to generate visual attention direction information; The visual attention direction information is mapped to the devices in the spatial association matrix to determine the target attention device, and the change events of the target attention device at adjacent time moments are recorded to form the visual attention transfer relationship between devices; The number of times the target device changes per unit of time is counted to obtain the device switching frequency; Calculate the device dwell time corresponding to the target device based on the length of time the same target device is continuously monitored within consecutive time periods. Based on the number of times the same target device is refocused within a preset time window, extract the device refocusing feature; The device switching frequency, device dwell time, and device repetition attention characteristics are normalized to construct a visual attention transfer feature vector; A visual attention transfer graph is established based on the transfer relationship between the target attention devices at continuous time, where the target attention devices are graph nodes and the visual attention transfer relationship between devices is graph edge. The weights of graph edges are determined based on the device switching frequency, the weights of node attention are determined based on the device dwell time, and the weights of return visits are determined based on the repeated attention characteristics of devices. Visual attention events are encoded into a visual attention migration map in chronological order to generate a user visual attention migration trajectory that reflects changes in user visual attention across devices.

[0037] By identifying the user's eye region and extracting gaze direction and eye fixation features based on real-time visual data streams, an eye orientation sequence is generated. This sequence is then time-aligned and weighted to fuse with a continuous head micro-pose feature sequence, achieving a high-precision representation of the user's comprehensive visual attention direction. During this process, particle swarm optimization combined with chaotic mapping dynamically adjusts the fusion weights, ensuring the continuity and stability of the visual attention direction vector and accurately reflecting changes in user attention across multiple devices. Furthermore, the fused visual attention direction is mapped to smart devices in a spatial correlation matrix, and device switching frequency, dwell time, and repeated attention features are statistically analyzed. A visual attention migration map is constructed, and the user's visual attention migration trajectory is encoded in chronological order. This enables dynamic analysis and quantitative representation of cross-device visual attention behavior, providing precise user behavior data for smart home control strategy formulation and improving the predictive ability and intelligent response level to users' implicit control needs.

[0038] Based on the user's visual attention migration trajectory, the user's control intention is predicted, a predicted intention set is generated, and the predicted intention set is used as a constraint condition to identify the user's implicit needs information in combination with human micro-motion features.

[0039] Specifically, based on the user's visual attention migration trajectory, the user's visual attention sequence, device dwell time, device switching frequency, and repeated device attention characteristics of smart home devices are extracted to generate user visual attention behavior characteristics; Based on user visual attention behavior characteristics and combined with historical device interaction records, the system predicts the user's possible device control direction and target operation category, and generates a set of predicted user intentions. Based on real-time visual data stream, the system identifies local areas of the user's limbs, extracts the user's hand movements, body posture changes, and local movement amplitude features, and generates a sequence of the user's human micro-movement features. The device control direction and target operation category in the predicted intent set are converted into intent constraint labels, and the human micro-motion feature sequence is filtered based on the intent constraint labels to remove human actions that are inconsistent with the target device control direction or unrelated to the target operation type. The filtered human actions are then matched with each predicted intent in the predicted intent set, and the matching degree between the human actions and the predicted intents is calculated using the following formula: ; in, For human movement index, For predicting intent index, For the first Individual physical movements and the first The degree of matching between predicted intentions The first one obtained through similarity calculation Individual physical movements and the first The predicted intention's action amplitude matching value. The first one obtained through similarity calculation Individual physical movements and the first A value matching the action direction of the predicted intent. The first one obtained through similarity calculation Individual physical movements and the first Each predicted intention operation type matches the value. , , These are the weighting coefficients for the set motion amplitude matching value, motion direction matching value, and operation type matching value, respectively.

[0040] Human movements with a matching degree greater than a preset threshold are identified as target human movement features; Based on the correlation between target human motion features and predicted intent sets, the system identifies the user's corresponding device control needs, environmental adjustment needs, and device interaction needs, and generates implicit user demand information.

[0041] By extracting visual attention behavior features based on user visual attention migration trajectories and generating a predicted intent set by combining historical device interaction records, the system enables early inference of users' potential control goals. This allows for the identification of users' possible operational tendencies before human actions are fully formed, improving the foresight and proactive response capabilities of user intent perception. Furthermore, by converting predicted intents into intent constraint labels and performing targeted filtering and correlation matching of human micro-motion features, the system achieves dual verification of visual and motor behavior information. This reduces the impact of irrelevant, habitual, and environmentally disruptive actions on the demand identification results, improving the consistency between implicit demand identification results and actual control intentions. Based on the correlation between target human motor features and predicted intents, the system identifies user device control needs, environmental adjustment needs, and device interaction needs. This realizes a progressive reasoning process from visual attention to behavioral expression to demand generation, enabling smart homes to accurately perceive latent needs not explicitly expressed by users, enhancing the proactive service capabilities of device control and the naturalness of human-computer interaction.

[0042] When multiple users are detected, the implicit demand information of each user is merged to generate a set of family needs.

[0043] Specifically, based on real-time visual data streams, the number of users in the scene is detected, and a correspondence between users and implicit demand information is established; If only a single user is detected, a set of family needs is generated directly based on that user's implicit needs information; When multiple users have conflicting needs for the same target device, the conflicting needs are analyzed by combining the users' historical interaction habits, current environmental status and the intensity of the needs, and information on the conflicting needs is generated. Based on the correspondence between users and implicit demand information, and combined with demand conflict information, the implicit demand information of multiple users is integrated to generate a family demand set.

[0044] By detecting the number of users in a scenario and establishing a correspondence between users and implicit demand information, the system distinguishes the sources of different user demands in a home setting, avoiding the conflation of demands from multiple users. In a single-user scenario, it directly generates a set of family demands, reducing unnecessary conflict analysis processes and improving control response efficiency. In a multi-user scenario, by combining users' historical interaction habits, current environmental status, and demand intensity to analyze the conflict relationships on the same device, it achieves reasonable identification and orderly integration of conflicting demands. This allows the set of family demands to simultaneously reflect individual demand differences and overall control priorities, improving the accuracy of demand integration and control coordination in a multi-user environment.

[0045] Based on the set of family needs, combined with information on conflicting needs and the visual attention migration trajectory of multiple users, the control rights of multi-user visual games are allocated, a dynamic control right sequence of devices is generated, and a smart home control strategy is generated based on the dynamic control right sequence of devices.

[0046] Specifically, based on the set of household needs, extract the target device needs, intensity of needs, categories of needs, and conflicting needs for each user; When the demand conflict relationship information indicates that multiple users have control conflicts for the same target device, the intensity, duration and trend of each user's visual attention to the target device are extracted by combining the visual attention migration trajectory of each user. The visual attention intensity, attention duration, and attention shift trend are normalized and linearly weighted according to preset weight coefficients to calculate the visual engagement of each user with respect to the target device. Based on the visual engagement with the target device, combined with the intensity of demand and users' historical interaction habits, the visual game evaluation value of each user for the target device is calculated. Based on the demand conflict relationship, calculate the conflict probability of the target equipment, and calculate the control handover cost based on the change of control ownership between adjacent time points. The formula is as follows: ; in, For time index, For the first Each device at time The cost of switching For the first Each device at time Control belongs to the user. For the first Each device at time The user had control at the previous moment. This is an indicator function; it returns 1 if the condition is true and 0 if the condition is false. To ensure that control belongs to the user The visual game evaluation value at any given moment. To ensure that control belongs to the user The visual game evaluation value of the previous moment. For the first The frequency of control switching for each device within the most recent time window. , , These are the weighting coefficients for control changes, visual game evaluation differences, and control switching frequency, respectively.

[0047] Based on the visual game evaluation value, conflict probability, and control switching cost, the control benefit for each user regarding the target device is continuously optimized to construct an optimizable control benefit function, the formula of which is: ; in, For the first Control allocation scheme for individual devices For the first The total benefit of control over a single device within a time window. For any moment within the time window, The length of the time window, The number of users participating in the competition for control. Assign variables to control rights, if the first The device is Control will be allocated to the first person at any time. If there are 1 users, then the value is 1; otherwise, the value is 0. For the first Individual users At any time for the first Visual game evaluation value of each device For the first The device is The probability of conflicting demands at any given moment. For the first The device is The cost of switching moments The weighting coefficients for the probability of conflicting demands are set. The weighting coefficient for the switching cost.

[0048] Based on the control right benefit function, the control right benefits of each user within a continuous time window are predicted in a rolling manner to generate the initial control right allocation result, and a control right allocation neighborhood set is constructed with the initial control right allocation result as the center. A rolling search is performed within the control allocation neighborhood set, and the user to whom control of the target device belongs at each consecutive moment is determined by the joint constraint of maximizing benefits and minimizing switching costs. When the frequency of control switching of the target device within a preset time window exceeds a set threshold, a control retention factor is introduced to modify the control benefit function and suppress frequent control switching. The user who controls the target device at any given moment is arranged in chronological order to generate a dynamic sequence of device control rights. Based on the dynamic control sequence of devices, user needs in the set of household needs are matched to generate smart home control strategies.

[0049] By extracting the target device needs, intensity, category, and conflict relationships of each user based on the family's needs set, and calculating each user's visual engagement with the target device by combining the user's visual attention migration trajectory, a quantitative representation of the differences in needs for the same device among multiple users is achieved. Furthermore, a visual game evaluation value is constructed based on visual engagement, demand intensity, and historical interaction habits, and rolling optimization and prediction are performed by combining conflict probability and control switching costs, realizing the reasonable allocation and dynamic adjustment of device control rights among multiple users. Finally, by generating a dynamic device control right sequence and matching it with the family's needs set, a smart home control strategy is formed, realizing the coordination and intelligence of device control in a multi-user environment, and improving the perception accuracy and execution efficiency of smart homes for potential needs of multiple users.

[0050] This embodiment also provides a vision-based smart home control system, including: The multi-source data acquisition module collects images, spatial location information, and device status data in the smart home environment; The real-time visual data stream construction module continuously processes the acquired images and spatial information to generate a real-time data stream containing user dynamic behavior and environmental status. The head micro-pose detection module uses extended Haar-like features and a Boosted Cascade classifier to detect the user's head pose and construct a head micro-pose time series. The attention transfer analysis module generates cross-device visual attention relationships and user visual attention transfer trajectories based on head posture and eye orientation features, combined with a spatial correlation matrix. The user intent prediction module predicts the user's possible device control intents based on the visual attention migration trajectory and generates a set of predicted intents. The human micro-motion recognition module analyzes the user's body movements and local micro-motion features, associates and matches them with predicted intentions, and identifies the user's implicit needs information. The multi-user control allocation module integrates implicit demand information and visual attention trajectories from multiple users to perform game theory calculations to generate a dynamic sequence of device control rights, and generates smart home control strategies accordingly.

[0051] This embodiment also provides a computer device applicable to the vision-based smart home control method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the vision-based smart home control method proposed in the above embodiment.

[0052] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0053] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the vision-based smart home control method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0054] In summary, this invention improves the smart home's environmental adaptability and spatial perception capabilities by: collecting multi-source data from the smart home environment and constructing a real-time visual data stream; and by employing extended Haar-like features and Boosted... This paper presents a head micro-pose detection method combined with a Cascade classifier. By constructing a continuous head micro-pose time series based on the positional offset of the nose region relative to the face region, it achieves continuous tracking and pose perception of subtle head movements, more accurately reflecting changes in user visual attention and improving the stability of visual perception and the accuracy of user intent recognition. By combining head pose features, eye orientation features, and spatial correlation matrices to construct cross-device visual attention transfer relationships, it achieves dynamic mapping of the user's visual attention process across different devices, enabling analysis of the shifting trend of user visual attention and improving the system's ability to predict user control goals. By generating a predicted intent set based on the user's visual attention transfer trajectory and combining it with human micro-motion features to identify implicit user needs, it achieves joint analysis of explicit and implicit user visual behaviors, enabling early identification of potential user control needs and improving the initiative and naturalness of smart home control. Furthermore, by fusing implicit needs information from multiple users and combining it with information on conflicting needs and multi-user visual attention transfer trajectories for visual game control allocation, it achieves dynamic coordination of device control in multi-user scenarios, reducing multi-user device control conflicts and improving the synergy and intelligence level of smart home systems in complex home environments.

[0055] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A vision-based smart home control method, characterized in that: include, Collect multi-source data of the smart home environment, construct a real-time visual data stream, and generate a spatial association matrix between users and devices based on the real-time visual data stream; Based on real-time visual data stream, a head micro-pose detection method combining extended Haar-like features and Boosted Cascade classifier is adopted. A continuous head micro-pose time series is constructed by the positional offset relationship between the nose region and the face region. Head posture features are extracted based on continuous head micro-pose time series, and user eye orientation features and spatial correlation matrix are combined to construct cross-device visual attention transfer relationship and generate user visual attention transfer trajectory. Based on the user's visual attention migration trajectory, the user's control intention is predicted, a predicted intention set is generated, and the predicted intention set is used as a constraint condition to identify the user's implicit demand information in combination with human micro-movement features. When multiple users are detected, the implicit demand information of each user is merged to generate a set of family needs; Based on the set of family needs, combined with information on conflicting needs and the visual attention migration trajectory of multiple users, the control rights of multi-user visual games are allocated, a dynamic control right sequence of devices is generated, and a smart home control strategy is generated based on the dynamic control right sequence of devices.

2. The vision-based smart home control method as described in claim 1, characterized in that: The process of collecting multi-source data from the smart home environment, constructing a real-time visual data stream, and generating a spatial association matrix between users and devices based on the real-time visual data stream includes the following steps: Collect environmental image data of the smart home environment, as well as spatial location data of users and devices; By continuously updating the location data of users and devices in the image through continuous frame target detection, a real-time visual data stream containing dynamic visual information of users and environmental spatial information is formed. Based on real-time visual data streams, correlation parameters are calculated according to the relative distance between the user and each device, the angle of facial orientation, and the interactive range. Users are represented as rows, devices as columns, and the association parameters between users and each device are used as matrix elements to construct a spatial association matrix that reflects the spatial relationship between users and devices.

3. The vision-based smart home control method as described in claim 2, characterized in that: The head micro-pose detection method, based on real-time visual data streams and employing a combination of extended Haar-like features and a Boosted Cascade classifier, constructs a continuous head micro-pose time series by analyzing the positional offset relationship between the nose region and the face region. The method includes the following steps: Image frames are acquired based on real-time visual data streams and grayscale processing is performed. An integral image is then constructed based on the grayscale image. Sliding detection windows are set at different scales to scan the image region, and extended Haar-like features are used to extract head contour features, facial edge features, and local features of the eyes and nose within each window to obtain the feature response values ​​of each window. Based on the matching degree between the feature response values ​​of each detection window and the preset head feature template, image regions that meet the head feature conditions are selected as candidate user head regions. The candidate regions of the user's head are sequentially input into the pre-trained Boosted Cascade classifier model, and feature matching and classification are performed step by step. Candidate regions that do not meet the conditions are eliminated, and finally the user's facial regions are determined. Locate the nose region within the facial region, calculate the horizontal and vertical offsets of the nose center point relative to the geometric center of the face, and obtain the nose offset vector at consecutive time points. Based on the changes in the nose offset vector at continuous time intervals, the micro-pose features of the user's head in the pitch, yaw and roll directions are extracted. The continuous head micro-pose features are arranged in chronological order to generate a continuous head micro-pose time series.

4. The vision-based smart home control method as described in claim 3, characterized in that: The process of establishing cross-device visual attention transfer relationships and generating user visual attention transfer trajectories includes the following steps: Based on real-time visual data stream, the user's eye area is identified, the user's gaze direction and eye fixation direction features are extracted, and a sequence of the user's eye orientation is generated. The continuous head micro-pose feature sequence is time-aligned with the user's eye orientation feature sequence, and the head pose and eye gaze direction at each time step are weighted and fused to obtain a comprehensive visual attention direction vector. During the fusion process, the head pose fusion weights and eye gaze fusion weights are constructed as a weight vector to be optimized; Multiple weight particles are randomly generated within a preset weight range, and each weight particle represents a set of candidate fusion weight combinations. Based on the deviation between the fused visual attention direction and the historical attention direction, a visual attention continuity evaluation index is constructed, and the visual attention continuity evaluation index is used as the fitness value of the weight particles. Based on the fitness value of each weighted particle, determine the weight combination corresponding to the highest fitness obtained by each weighted particle in the historical iteration process, and use it as the individual optimal weighted particle; The weight particle with the highest fitness value among all individual optimal weight particles is selected as the global optimal weight particle. During the iterative update of the weighted particles, the inertial weights are dynamically adjusted based on the nonlinear perturbation parameters generated by the Logistic chaotic mapping, so that the weighted particles maintain a strong global search capability in the initial stage and gradually enhance their local optimization capability in the subsequent stage. A mapping perturbation factor generated based on the Tent chaotic mapping is introduced to dynamically adjust the individual learning coefficient and the group learning coefficient, so that the weighted particles can update their own positions according to the guiding directions of the individual optimal weighted particles and the global optimal weighted particles. Based on the updated position, a new candidate fusion weight combination is generated, and the corresponding fitness value is recalculated. The update process of individual optimal weight particles and global optimal weight particles is repeated. When the maximum number of iterations is reached, the head pose fusion weight and eye gaze fusion weight corresponding to the globally optimal weight particle are determined as the optimal fusion weight; Based on the optimal fusion weight, the head pose direction vector and the eye gaze direction vector are weighted and fused to generate visual attention direction information; The visual attention direction information is mapped to the devices in the spatial association matrix to determine the target attention device, and the change events of the target attention device at adjacent time moments are recorded to form the visual attention transfer relationship between devices; The number of times the target device changes per unit of time is counted to obtain the device switching frequency; Calculate the device dwell time corresponding to the target device based on the length of time the same target device is continuously monitored within consecutive time periods. Based on the number of times the same target device is refocused within a preset time window, extract the device refocusing feature; The device switching frequency, device dwell time, and device repetition attention characteristics are normalized to construct a visual attention transfer feature vector; A visual attention transfer graph is established based on the transfer relationship between the target attention devices at continuous time, where the target attention devices are graph nodes and the visual attention transfer relationship between devices is graph edge. The weights of graph edges are determined based on the device switching frequency, the weights of node attention are determined based on the device dwell time, and the weights of return visits are determined based on the repeated attention characteristics of devices. Visual attention events are encoded into a visual attention migration map in chronological order to generate a user visual attention migration trajectory that reflects changes in user visual attention across devices.

5. The vision-based smart home control method as described in claim 4, characterized in that: The method of predicting user control intentions based on user visual attention migration trajectories, generating a predicted intention set, and using the predicted intention set as a constraint, combined with human micro-motion features to identify users' implicit needs, includes the following steps: Based on the user's visual attention migration trajectory, the user's visual attention sequence, device dwell time, device switching frequency and repeated device attention features are extracted to generate user visual attention behavior features. Based on user visual attention behavior characteristics and combined with historical device interaction records, the system predicts the user's possible device control direction and target operation category, and generates a set of predicted user intentions. Based on real-time visual data stream, the system identifies local areas of the user's limbs, extracts the user's hand movements, body posture changes, and local movement amplitude features, and generates a sequence of the user's human micro-movement features. The device control direction and target operation category in the predicted intent set are converted into intent constraint labels, and the human micro-motion feature sequence is filtered based on the intent constraint labels to remove human actions that are inconsistent with the target device control direction or unrelated to the target operation type. The filtered human actions are associated and matched with each predicted intent in the predicted intent set. The matching degree between the human actions and the predicted intent is calculated, and the human actions with a matching degree greater than a preset threshold are identified as target human action features. Based on the correlation between target human motion features and predicted intent sets, the system identifies the user's corresponding device control needs, environmental adjustment needs, and device interaction needs, and generates implicit user demand information.

6. The vision-based smart home control method as described in claim 5, characterized in that: When multiple users are detected, the implicit demand information of each user is fused to generate a family demand set, which includes the following steps: Based on real-time visual data streams, the number of users in a scene is detected, and a correspondence between users and implicit demand information is established. If only a single user is detected, a set of family needs is generated directly based on that user's implicit needs information; When multiple users have conflicting needs for the same target device, the conflicting needs are analyzed by combining the users' historical interaction habits, current environmental status and the intensity of the needs, and information on the conflicting needs is generated. Based on the correspondence between users and implicit demand information, and combined with demand conflict information, the implicit demand information of multiple users is integrated to generate a family demand set.

7. The vision-based smart home control method as described in claim 6, characterized in that: The process involves allocating control rights for multi-user visual games based on a set of family needs, combined with information on conflicting needs and multi-user visual attention migration trajectories, generating a dynamic control right sequence for devices, and then generating a smart home control strategy based on this sequence. This includes the following steps: Based on the set of household needs, extract the target equipment needs, demand intensity, demand category, and demand conflict relationships for each user; When the demand conflict relationship information indicates that multiple users have control conflicts for the same target device, the visual attention migration trajectory of each user is combined to extract the visual attention intensity, attention duration and attention shift trend of each user for the target device, and the visual participation of each user for the target device is calculated accordingly. Based on the visual engagement with the target device, combined with the intensity of demand and users' historical interaction habits, the visual game evaluation value of each user for the target device is calculated. Based on the conflicting demand relationships, calculate the conflict probability of the target equipment, and calculate the cost of switching control based on the change in control ownership between adjacent time points. Based on visual game evaluation value, conflict probability and control switching cost, the control benefit of each user for the target device is continuously optimized to construct an optimizable control benefit function. Based on the control right benefit function, the control right benefits of each user within a continuous time window are predicted in a rolling manner to generate the initial control right allocation result, and a control right allocation neighborhood set is constructed with the initial control right allocation result as the center. A rolling search is performed within the control allocation neighborhood set, and the user to which control rights belong for the target device at consecutive times is determined by the joint constraint of maximizing benefits and minimizing switching costs. When the frequency of control switching of the target device within a preset time window exceeds a set threshold, a control retention factor is introduced to modify the control benefit function and suppress frequent control switching. The user who controls the target device at any given moment is arranged in chronological order to generate a dynamic sequence of device control rights. Based on the dynamic control sequence of devices, user needs in the set of household needs are matched to generate smart home control strategies.

8. A vision-based smart home control system, based on the vision-based smart home control method according to any one of claims 1 to 7, characterized in that: include, The multi-source data acquisition module collects images, spatial location information, and device status data in the smart home environment; The real-time visual data stream construction module continuously processes the acquired images and spatial information to generate a real-time data stream containing user dynamic behavior and environmental status. The head micro-pose detection module uses extended Haar-like features and a Boosted Cascade classifier to detect the user's head pose and construct a head micro-pose time series. The attention transfer analysis module generates cross-device visual attention relationships and user visual attention transfer trajectories based on head posture and eye orientation features, combined with a spatial correlation matrix. The user intent prediction module predicts the user's possible device control intents based on the visual attention migration trajectory and generates a set of predicted intents. The human micro-motion recognition module analyzes the user's body movements and local micro-motion features, associates and matches them with predicted intentions, and identifies the user's implicit needs information. The multi-user control allocation module integrates implicit demand information and visual attention trajectories from multiple users to perform game theory calculations to generate a dynamic sequence of device control rights, and generates smart home control strategies accordingly.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the vision-based smart home control method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the vision-based smart home control method according to any one of claims 1 to 7.

Images