Virtual environment image home control method and system under digital twinning

Abstract

The invention relates to the technical field of smart home, and discloses a virtual environment image home control method and system under digital twinning, and the method comprises the steps: building a digital twinning body through the physical data of a home scene, and building a user 3D human body model based on the human body scattered wave data, so as to generate a virtual mirror image; creating a communication channel between the virtual mirror image and the home equipment so as to construct a control interface of the virtual mirror image to the home equipment; based on the current behavior intention, setting a personalized control instruction of the virtual mirror image to the home equipment; according to the time sequence image and the behavior tracking network, setting a real-time synchronization mechanism of the virtual mirror image and the home scene, and constructing data binding of the home device and the virtual mirror image; and in combination with the virtual mirror image, the control instruction, the scene synchronization mechanism and the data security interaction system, executing the control instruction of the home equipment. According to the invention, one-to-one correspondence between the pictures displayed by the digital twinning and the real scene can be realized, so that a user can accurately control the actual state of the supervised object.

Description

Technical Field

[0001] The present invention relates to a method and system for controlling a home using a virtual environment under digital twin conditions, and belongs to the technical field of smart homes. Background Art

[0002] A virtual mirror refers to a digital dynamic mapping model built based on a home environment. Its core is to perform three-dimensional modeling and real-time restoration of information such as people, environments, objects, and behaviors in the real world with centimeter-level or even millimeter-level accuracy, so as to build a digital twin that is completely synchronized with the real home environment. With the rapid development of artificial intelligence and Internet of Things technologies, smart homes have increasingly higher requirements in terms of device interconnection, user experience, and personalized services.

[0003] However, traditional smart home systems mainly rely on flat interactive technologies, such as basic smart app control, voice commands or physical buttons to achieve operations such as turning on and off and adjusting home appliances to meet users' daily needs. However, this control method lacks the ability to perceive the overall home environment and dynamically present spatial conditions, and cannot accurately grasp the user's situational needs, resulting in a poor user experience.

[0004] Therefore, there is an urgent need for a solution that can achieve a one-to-one correspondence between the images displayed by the digital twin and the real scene, so as to meet the user's precise control over the actual status of the supervised object. Summary of the Invention

[0005] The present invention provides a method and system for controlling home appliances using virtual images under digital twins. The main purpose of the method is to achieve a one-to-one correspondence between the virtual images displayed under digital twins and the real scenes, thereby satisfying the user's precise control over the actual status of the supervised object.

[0006] To achieve the above objectives, the present invention provides a method for controlling a home using a virtual environment under a digital twin, comprising:

[0007] Obtain the user's home scene and its corresponding home devices, collect physical data of the home scene, where the physical data includes object data, environmental state data, and spatial structure information, and use the physical data to construct a digital twin of the home scene;

[0008] collecting human body scattered wave data of the user according to the WiFi signal of the household device, and creating a 3D human body model of the user in the digital twin based on the human body scattered wave data;

[0009] Performing 3D rendering processing on the digital twin and the 3D human body model to generate a virtual image, establishing a communication channel between the virtual image and the home appliance, and constructing a control interface for the virtual image to control the home appliance based on the communication channel;

[0010] Based on the WiFi signal, a behavior tracking network of the user is constructed, and according to the behavior tracking network, the user's current behavior intention with respect to the home device is identified, and based on the current behavior intention and the control interface, personalized control instructions for the home device by the virtual mirror are set;

[0011] Collecting time-series images of the home scene in real time, and setting a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time-series images and the behavior tracking network, and building a data security interaction system between the home devices and the virtual mirror based on the real-time scene synchronization mechanism;

[0012] In combination with the virtual mirror, the personalized control instructions, the real-time scene synchronization mechanism and the data security interaction system, the intelligent control processing of the home appliance is performed to obtain a home control result.

[0013] Optionally, constructing the digital twin of the home scene using the physical data includes:

[0014] Determining the built-in home products of the home scene and their corresponding operating states and spatial layouts through the physical data, and identifying environmental state data of the home scene;

[0015] Analyzing the correlation between the operating status and the environmental status data;

[0016] identifying, based on the association relationship, operating modes of the built-in home products in different environments;

[0017] Building a behavior model of the built-in home product in the different environments through the operation mode;

[0018] Constructing a geometric model of the built-in home furnishing product using the appearance structure and the spatial layout of the built-in home furnishing product;

[0019] Based on the material properties of the object in the physical data, texture mapping is performed on the geometric model to generate a physical model;

[0020] The behavior model, the geometric model, and the physical model are combined to construct a digital twin of the home scene.

[0021] Optionally, collecting the user's human body scattered wave data based on the WiFi signal of the home device includes:

[0022] Determine the WiFi transceiver node corresponding to the home device according to the WiFi signal of the home device;

[0023] Collecting original WiFi signal data from the WiFi transceiver node and identifying interference signals in the original WiFi signal data;

[0024] Analyzing an interference type of the interference signal, and performing interference removal processing on the interference signal based on the interference type to obtain an optimized WiFi signal;

[0025] Extracting a subcarrier signal from the optimized WiFi signal and calculating a signal-to-noise ratio of the subcarrier signal;

[0026] Determining a sensitive subcarrier signal of the user according to the signal-to-noise ratio;

[0027] Based on the sensitive subcarrier signal, human body scattered wave data of the user is collected.

[0028] Optionally, creating a 3D human body model of the user in the digital twin based on the human body scattered wave data includes:

[0029] Identifying scattered wave intensity and phase difference in the human body scattered wave data;

[0030] determining a body contour of the user according to the scattered wave intensity;

[0031] Extracting internal structural features of the user's body based on the phase difference;

[0032] Creating a virtual human body of the user according to the human body contour and the internal structural features of the human body;

[0033] Extracting key feature points of the user from the human body contour and the human body internal structure features;

[0034] Extracting the coordinate parameters of the key feature points from the human body scattered wave data using a skeleton tracking algorithm;

[0035] Based on the coordinate parameters, identifying the body parts of the user and the corresponding motion states;

[0036] Constructing a mapping relationship between the virtual human body and the user according to the body part and the motion state;

[0037] Based on the mapping relationship and the virtual human body, a 3D human body model of the user in the digital twin is created.

[0038] Optionally, the creating a communication channel between the virtual mirror and the home appliance includes:

[0039] Identifying the spatial layout of the device display terminal of the virtual mirror and the home appliances;

[0040] Collecting the real-time three-dimensional coordinates of the device display terminal, and calculating the signal propagation path loss value between the virtual mirror and the home device in combination with the real-time three-dimensional coordinates and the spatial layout;

[0041] Setting adaptive communication parameters between the virtual mirror and the home device based on the signal propagation path loss value;

[0042] Identifying communication protocol characteristics of the device display terminal and a real-time computing load of the virtual image;

[0043] Setting a multimodal communication mode between the virtual mirror and the home appliance according to the communication protocol characteristics and the real-time computing load;

[0044] collecting historical channel quality data of the home device, and analyzing the network congestion probability of the home device based on the historical channel quality data;

[0045] Setting the error correction coding redundancy of the virtual image according to the network congestion probability;

[0046] generating an optimal communication strategy between the virtual mirror and the home device based on the network congestion probability and the error correction code redundancy;

[0047] A communication channel between the virtual mirror and the home appliance is created by combining the optimal communication strategy, the adaptive communication parameters and the multimodal communication method.

[0048] Optionally, constructing a control interface of the virtual mirror for the home appliance based on the communication channel includes:

[0049] Extracting a signal propagation path loss value corresponding to the communication channel and generating a dynamic protocol parsing module for the communication channel;

[0050] Identifying a spatial action vector in the virtual image and, based on the dynamic protocol parsing module, setting an instruction conversion engine for the spatial action vector;

[0051] Collecting status data of the home appliance and extracting rendering properties of the virtual image;

[0052] Creating a dynamic mapping table between the data type of the state data and the rendering attributes;

[0053] Based on the dynamic mapping table, construct a status feedback parser for the home appliance;

[0054] defining an adapter interface of the communication channel according to the state feedback parser and the dynamic protocol parsing module;

[0055] The dynamic protocol parsing module, the instruction conversion engine, the state feedback parser and the adapter interface are combined to construct a control interface of the virtual mirror for the home appliance.

[0056] Optionally, constructing the user's behavior tracking network based on the WiFi signal includes:

[0057] Performing filtering on the WiFi signal to generate a filtered signal;

[0058] extracting spatiotemporal joint features of the filtered signal, and identifying the user's behavior pattern based on the spatiotemporal joint features;

[0059] Analyzing the temporal evolution rules of the behavior pattern using a Bayesian causal network;

[0060] Generating a behavior state transition diagram of the user according to the time evolution rule;

[0061] Based on the behavior state transition graph, a behavior tracking network of the user is constructed.

[0062] Optionally, the setting of personalized control instructions of the virtual mirror to the home appliance based on the current behavioral intention and the control interface includes:

[0063] Collecting the user's historical interaction data and real-time behavior data, and collecting environmental perception data of the home device;

[0064] Calculating the intention confidence of the current behavior intention based on the historical interaction data and the real-time behavior data;

[0065] Identifying the user's fixed behavior pattern based on the historical interaction data;

[0066] Calculating the user's habit matching degree based on the current behavioral intention and the fixed behavioral pattern;

[0067] Identifying the current environmental status of the home appliance based on the environmental perception data;

[0068] Determining the scenario urgency of the home device based on the current environmental state and the current behavioral intention;

[0069] Creating a dynamic priority matrix for the home devices by combining the intention confidence, the habit matching, and the scenario urgency;

[0070] generating a personalized response mode of the control interface according to the dynamic priority matrix;

[0071] Based on the dynamic priority matrix and the personalized response mode, personalized control instructions of the virtual mirror to the home device are set.

[0072] Optionally, setting a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time series image and the behavior tracking network includes:

[0073] Identifying the tracked human joints and object movement trajectories in the home scene based on the time series images and the behavior tracking network;

[0074] Based on the tracked human joint points, identifying the movement speed of the human body in the home scene;

[0075] Determining the object displacement vector of the home scene according to the object movement trajectory;

[0076] generating a spatial synchronization element of the virtual mirror image according to the human body movement speed and the object displacement vector;

[0077] Setting an adaptive coordinate update system of the virtual mirror using the spatial synchronization elements;

[0078] Extracting the light intensity, color temperature, and light source direction of the time series image to identify the lighting difference between the virtual mirror image and the home scene;

[0079] According to the illumination difference, setting a real-time light matching mechanism of the virtual mirror;

[0080] In combination with the adaptive coordinate updating system and the real-time light matching mechanism, a real-time scene synchronization mechanism between the virtual mirror and the home scene is set.

[0081] In order to solve the above problems, the present invention also provides a virtual environment control home system under digital twin, which includes:

[0082] A scene modeling module is used to obtain the user's home scene and its corresponding home devices, collect physical data of the home scene, where the physical data includes object data, environmental state data and spatial structure information, and use the physical data to build a digital twin of the home scene;

[0083] a virtual character construction module, configured to collect human body scattered wave data of the user according to the WiFi signal of the household device, and create a 3D human body model of the user in the digital twin based on the human body scattered wave data;

[0084] A virtual image generation module is configured to perform 3D rendering processing on the digital twin and the 3D human body model to generate a virtual image, establish a communication channel between the virtual image and the home appliance, and construct a control interface for the virtual image to control the home appliance based on the communication channel;

[0085] a personalized control module, configured to construct a behavior tracking network for the user based on the WiFi signal, identify the user's current behavioral intention with respect to the home appliance based on the behavior tracking network, and set personalized control instructions for the virtual mirror with respect to the home appliance based on the current behavioral intention and the control interface;

[0086] a secure interaction module for collecting time-series images of the home scene in real time, and setting a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time-series images and the behavior tracking network, and building a data security interaction system between the home devices and the virtual mirror based on the real-time scene synchronization mechanism;

[0087] The home control module is used to combine the virtual mirror, the personalized control instructions, the real-time scene synchronization mechanism and the data security interaction system to perform intelligent control processing of the home appliances and obtain home control results.

[0088] Compared with the problems described in the background technology, the embodiment of the present invention uses the physical data to construct the digital twin of the home scene, which can accurately present the virtual scene of the home, helping to realize the remote control of the user on the virtual interface of the home appliance; further, the embodiment of the present invention collects the user's human body scattered wave data according to the WiFi signal of the home appliance, and can accurately calculate the user's position and movement trajectory in the room, thereby driving the virtual character to make corresponding postures and actions, enhancing the realism of the virtual mirror and the user's immersion; the embodiment of the present invention creates a 3D human body model of the user in the digital twin based on the human body scattered wave data, which can reflect the movement state and posture changes of the human body in real time, and realize real-time dynamic update of the human body in the virtual mirror; further, the embodiment of the present invention generates a virtual mirror by performing 3D rendering processing on the digital twin and the 3D human body model, so that the user can intuitively understand and perceive the scenes and characters in the virtual environment, enhance the user experience, improve interactivity and practicality; the embodiment of the present invention constructs the user's behavior tracking network based on the WiFi signal, which can realize the user's full-body movement recognition and promote the seamless interaction between the user and the home appliance; further, the embodiment of the present invention identifies the user's actions on the home appliance according to the behavior tracking network. The current behavioral intention of the device can provide personalized home services for the user; the embodiment of the present invention sets the personalized control instructions of the virtual mirror for the home device based on the current behavioral intention and the control interface, which can improve the collaborative efficiency between devices and also improve user satisfaction; further, the embodiment of the present invention sets the real-time scene synchronization mechanism between the virtual mirror and the home scene according to the time series image and the behavior tracking network, which can achieve accurate capture and response of user behavioral intention, enhance the environmental perception ability of the virtual mirror, and optimize the intelligent control of home devices; the embodiment of the present invention constructs a data security interaction system between the home device and the virtual mirror based on the real-time scene synchronization mechanism, which can ensure the confidentiality and integrity of data transmission and protect user privacy and behavioral data security; finally, the embodiment of the present invention combines the virtual mirror, the personalized control instructions, the real-time scene synchronization mechanism and the data security interaction system to perform intelligent control processing of the home device to obtain home control results, which can achieve three-dimensional perception and dynamic presentation of the home environment, accurately capture user behavioral intention and provide personalized control instructions, form a user-centered personalized and spatialized smart home control system, and significantly improve the user's interaction accuracy and the consistency of action in the real scene. Therefore, the virtual image control home method and system under the digital twin provided by the embodiment of the present invention can achieve a one-to-one correspondence between the image displayed by the digital twin and the real scene, thereby satisfying the user's accurate control of the actual status of the supervised object. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A flowchart of a method for controlling a home using a virtual environment in a digital twin according to an embodiment of the present invention is provided;

[0090] Figure 2 A home control diagram for a method of controlling a home using a virtual environment under a digital twin provided by one embodiment of the present invention;

[0091] Figure 3 A functional module diagram for implementing a virtual environment control home system under a digital twin provided in one embodiment of the present invention.

[0092] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0093] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0094] An embodiment of the present application provides a method for controlling a home using a virtual image under a digital twin. The execution subject of the method for controlling a home using a virtual image under a digital twin includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for controlling a home using a virtual image under a digital twin can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0095] Example 1:

[0096] Reference Figure 1 FIG2 is a flow chart of a method for controlling a home using a virtual environment under a digital twin according to an embodiment of the present invention. In this embodiment, the method for controlling a home using a virtual environment under a digital twin includes:

[0097] S1. Obtain the user's home scene and its corresponding home appliances, collect physical data of the home scene, wherein the physical data includes object data, environmental status data and spatial structure information, and use the physical data to construct a digital twin of the home scene.

[0098] The embodiments of the present invention can accurately restore the home layout in a virtual mirror by obtaining the user's home scene and its corresponding home appliances. The user refers to an individual or group who uses a virtual mirror to control the home system, such as elderly people at home. The home scene refers to a specific life scene in a home living environment that is composed of different spatial areas, functional requirements, and corresponding home appliances and decorations, including a living room, bedroom, kitchen, bathroom, etc. The home appliances refer to electrical appliances, devices, appliances, etc. installed or placed in a home living environment to meet various needs of people's daily life, such as televisions, refrigerators, air conditioners, smart toilets, etc.

[0099] Furthermore, the embodiments of the present invention can provide data support for constructing a virtual mirror and improve the authenticity of the virtual mirror by collecting physical data of the home scene. The physical data refers to various quantifiable information related to objects, environmental states and spatial structures in the home environment, including object data, environmental state data and spatial structure information. The object data includes basic information and dynamic information of the object, such as the type, shape, size of the object, the state of the object's movement, the state of equipment operation, etc. The environmental state data includes temperature, humidity, light intensity, air quality, noise level, etc. The spatial structure information includes the spatial layout and geometric characteristics of the home, such as the size of the room, the position of doors and windows, the wall structure, and the connection relationship between each room.

[0100] The embodiment of the present invention uses the physical data to construct a digital twin of the home scene, which can accurately present the virtual scene of the home, helping users to remotely control home appliances on a virtual interface. The digital twin refers to a virtual model that uses digital technology to comprehensively and accurately simulate and map physical entities in the home scene.

[0101] As an embodiment of the present invention, constructing the digital twin of the home scene using the physical data includes:

[0102] Determining the built-in home products of the home scene and their corresponding operating states and spatial layouts through the physical data, and identifying environmental state data of the home scene;

[0103] Analyzing the correlation between the operating status and the environmental status data;

[0104] identifying, based on the association relationship, operating modes of the built-in home products in different environments;

[0105] Building a behavior model of the built-in home product in the different environments through the operation mode;

[0106] Constructing a geometric model of the built-in home furnishing product using the appearance structure and the spatial layout of the built-in home furnishing product;

[0107] Based on the material properties of the object in the physical data, texture mapping is performed on the geometric model to generate a physical model;

[0108] The behavior model, the geometric model, and the physical model are combined to construct a digital twin of the home scene.

[0109] Among them, the built-in home products refer to various types of equipment, appliances and facilities that are fixedly installed or configured in the home scene and are used to achieve specific functions, including but not limited to smart home appliances, smart bathroom equipment, lighting systems, security equipment, etc. The operating status refers to the various parameter indicators and working conditions presented by the built-in home products during operation, including but not limited to the switch status, operating power, working mode, etc. of the equipment, such as the cooling and heating modes of the air conditioner. The spatial layout refers to the layout structure of all built-in home products and other spatial elements in the home scene in three-dimensional space, including the specific position, arrangement, mutual spacing and orientation relationship. The environmental status data refers to the physical environmental parameters in the home scene collected by various sensors, such as environmental quality parameters. The correlation relationship refers to the regular connection between the mutual influence and interaction between the operating status of the built-in home products and the environmental status data, for example, the positive correlation between ambient temperature and the operating power of the air conditioner, the triggering relationship between light intensity and the switch status of the lighting device. Development relationship, etc., the operating mode refers to the preset working mode adopted by the built-in home furnishing products in different environmental states to achieve specific functions or goals based on the association relationship, the behavioral model refers to a digital model constructed based on the operating mode of the built-in home furnishing products in different environments, which is used to simulate and predict the behavior of the product. It takes the environmental state data as input and outputs the expected operating state of the product according to the logical rules of the operating mode. The appearance structure refers to the external shape, size specifications, contour characteristics, material texture and assembly relationship and external manifestation of the built-in home furnishing products. The geometric model refers to a digital three-dimensional model with precise geometric dimensions and spatial position relationships constructed according to the appearance structure and spatial layout information of the built-in home furnishing products using three-dimensional modeling technology. The physical model refers to a digital model that is generated on the basis of the geometric model, combined with the physical material characteristics of the built-in home furnishing products, through technical means such as texture mapping and material parameter assignment, and can simulate the physical properties and real appearance of the product.

[0110] Optionally, the correlation between the operating status and the environmental status data can be analyzed through a nonlinear regression model. Based on the correlation, the operating mode of the built-in home product in different environments is identified using a hidden Markov model. Combined with the behavioral model, the geometric model and the physical model, the digital twin of the home scene can be constructed through the Unity3D engine.

[0111] S2. Collect human body scattered wave data of the user according to the WiFi signal of the home device, and create a 3D human body model of the user in the digital twin based on the human body scattered wave data.

[0112] By collecting the user's human scattered wave data based on the WiFi signals of the household devices, embodiments of the present invention can accurately calculate the user's position and movement trajectory in the room, thereby driving the virtual character to perform corresponding postures and movements, enhancing the realism of the virtual image and the user's immersion. The WiFi signal refers to a wireless radio frequency signal transmitted by a household device (such as a router) and complying with the IEEE802.11BF protocol. It achieves spatial diversity through a MIMO multi-antenna array and interacts with the user's body during propagation, generating a scattered wave signal containing characteristics such as the body's position, movement, and posture. The human scattered wave data refers to the data generated after the WiFi signal is scattered by the human body when it propagates in space and encounters the human body, including the signal's amplitude, phase, frequency, etc.

[0113] As an embodiment of the present invention, collecting the user's human body scattered wave data based on the WiFi signal of the home device includes:

[0114] Determine the WiFi transceiver node corresponding to the home device according to the WiFi signal of the home device;

[0115] Collecting original WiFi signal data from the WiFi transceiver node and identifying interference signals in the original WiFi signal data;

[0116] Analyzing an interference type of the interference signal, and performing interference removal processing on the interference signal based on the interference type to obtain an optimized WiFi signal;

[0117] Extracting a subcarrier signal from the optimized WiFi signal and calculating a signal-to-noise ratio of the subcarrier signal;

[0118] Determining a sensitive subcarrier signal of the user according to the signal-to-noise ratio;

[0119] Based on the sensitive subcarrier signal, human body scattered wave data of the user is collected.

[0120] The WiFi transceiver node refers to a collection of smart devices in a home environment that have WiFi signal transmission and reception capabilities, including but not limited to wireless routers, IoT terminals, and smart appliances. The raw WiFi signal data refers to the unprocessed channel state information (CSI) sequence collected by the WiFi transceiver node, containing amplitude and phase data of at least 100 subcarriers, with a sampling frequency of no less than 100 Hz. The interference signal refers to the signal component in the raw WiFi signal data that is not generated by human scattering, such as multipath reflection signals caused by walls. The interference type refers to the classified interference signal type obtained by classifying the interference signal based on time-frequency characteristics, such as periodic interference, pulsed interference, and environmental multipath interference. The interference removal processing refers to the signal processing operations performed on different interference types. For example, for periodic interference or device interference, at least three notches can be set within the range of ±1 MHz of its center frequency. The subcarrier signal refers to the independent signal component obtained by dividing the raw WiFi signal by frequency. The signal-to-noise ratio refers to the ratio of the signal power of the human scattering component to the noise power. The sensitive subcarrier signal refers to the subcarrier signal that is most sensitive to signal changes caused by human activity.

[0121] Optionally, the interference signal in the original WiFi signal data can be identified by spectrum analysis, and the interference type of the interference signal can be analyzed using a random forest algorithm. For example, an integrated model containing 50-100 decision trees can be constructed to determine the interference type through a voting mechanism.

[0122] Furthermore, the embodiment of the present invention creates a 3D human body model of the user in the digital twin based on the human body scattered wave data, which can reflect the human body's movement state and posture changes in real time and realize real-time dynamic update of the human body in the virtual mirror. The 3D human body model refers to the three-dimensional digital representation of the user's human body in the virtual mirror in the real world.

[0123] As an embodiment of the present invention, creating a 3D human body model of the user in the digital twin based on the human body scattered wave data includes:

[0124] Identifying scattered wave intensity and phase difference in the human body scattered wave data;

[0125] determining a body contour of the user according to the scattered wave intensity;

[0126] Extracting internal structural features of the user's body based on the phase difference;

[0127] Creating a virtual human body of the user according to the human body contour and the internal structural features of the human body;

[0128] Extracting key feature points of the user from the human body contour and the human body internal structure features;

[0129] Extracting the coordinate parameters of the key feature points from the human body scattered wave data using a skeleton tracking algorithm;

[0130] Based on the coordinate parameters, identifying the body parts of the user and the corresponding motion states;

[0131] Constructing a mapping relationship between the virtual human body and the user according to the body part and the motion state;

[0132] Based on the mapping relationship and the virtual human body, a 3D human body model of the user in the digital twin is created.

[0133] Among them, the scattered wave intensity refers to the energy of a specific subcarrier signal per unit time, quantified in decibels (dB); the phase difference refers to the phase change difference of the same subcarrier signal at different times or different paths, and the unit is radians (rad); the human body contour refers to a human body shape boundary model generated in three-dimensional space based on scattered wave intensity data through image processing algorithms such as threshold segmentation and edge detection. The specific generation method is: the area with a scattered wave intensity higher than a preset threshold (such as -30dB) is regarded as a human body area, and a closed three-dimensional contour line is generated through a contour extraction algorithm (such as the Canny operator); the internal structural features of the human body refer to the manifestation of structures such as tissues and organs inside the human body in electromagnetic wave scattering, which can be extracted through the phase difference of the scattered wave; the virtual human body refers to the human body contour and internal structural features After the fusion of the digital space. A three-dimensional human body model is constructed, wherein the key feature points refer to three-dimensional coordinate points selected from the human body contour and internal structure features, which are representative of the description of human body shape and movement, including joint points, contour key points, etc., such as facial feature point coordinates. The skeleton tracking algorithm refers to an algorithm that calculates the joint position and motion trajectory of the human skeleton structure in real time through a spatiotemporal convolutional network based on the key feature point coordinate parameters. The coordinate parameters refer to the position information of the key feature points extracted from the human body scattered wave data in the three-dimensional space, and are expressed in a Cartesian coordinate system (X, Y, Z). The body part refers to a component with a clear anatomical definition in the human body structure. The motion state refers to a set of motion parameters of the body part at a certain moment. The mapping relationship refers to a one-to-one correspondence between the joint points and body parts of the virtual human body and the corresponding parts of the real user in terms of spatial position and motion state.

[0134] Optionally, the extraction of the key feature points of the user from the human body contour and the internal structural features of the human body can be achieved using a Harris corner detection algorithm, and the mapping relationship between the virtual human body and the user based on the body parts and the motion state can be constructed through a motion mapping algorithm.

[0135] S3. Perform 3D rendering processing on the digital twin and the 3D human body model to generate a virtual image, create a communication channel between the virtual image and the home appliance, and construct a control interface for the virtual image to the home appliance based on the communication channel.

[0136] The embodiment of the present invention generates a virtual mirror image by performing 3D rendering processing on the digital twin and the 3D human body model, so that users can intuitively understand and perceive the scenes and characters in the virtual environment, enhance user experience, and improve interactivity and practicality. The 3D rendering processing refers to the process of using a 3D rendering engine to convert a digital twin or a 3D human body model into a realistic image or animation. The virtual mirror image refers to a virtual, highly realistic image generated by digitally modeling a home scene in the real world and performing 3D rendering processing.

[0137] Furthermore, the embodiment of the present invention can feed back the operating status information of the home appliance to the virtual mirror in real time by creating a communication channel between the virtual mirror and the home appliance, thereby facilitating the remote control of the home appliance by the virtual mirror. The communication channel refers to an information transmission path established between the virtual mirror and the home appliance.

[0138] As an embodiment of the present invention, the creating of a communication channel between the virtual mirror and the home appliance includes:

[0139] Identifying the spatial layout of the device display terminal of the virtual mirror and the home appliances;

[0140] Collecting the real-time three-dimensional coordinates of the device display terminal, and calculating the signal propagation path loss value between the virtual mirror and the home device in combination with the real-time three-dimensional coordinates and the spatial layout;

[0141] Setting adaptive communication parameters between the virtual mirror and the home device based on the signal propagation path loss value;

[0142] Identifying communication protocol characteristics of the device display terminal and a real-time computing load of the virtual image;

[0143] Setting a multimodal communication mode between the virtual mirror and the home appliance according to the communication protocol characteristics and the real-time computing load;

[0144] collecting historical channel quality data of the home device, and analyzing the network congestion probability of the home device based on the historical channel quality data;

[0145] Setting the error correction coding redundancy of the virtual image according to the network congestion probability;

[0146] generating an optimal communication strategy between the virtual mirror and the home device based on the network congestion probability and the error correction code redundancy;

[0147] A communication channel between the virtual mirror and the home appliance is created by combining the optimal communication strategy, the adaptive communication parameters and the multimodal communication method.

[0148] The device display terminal refers to a terminal device with three-dimensional spatial positioning and communication capabilities, used to present a virtual image and serve as an intermediary node for communication between the virtual image and home appliances, including but not limited to mobile smart devices such as smartphones, tablets, and AR / VR headsets. The spatial layout refers to a digital model of the geometric structure and device deployment information of the physical space in the home environment. The real-time three-dimensional coordinates refer to the real-time position coordinates of the device display terminal in the spatial layout coordinate system. The signal propagation path loss value refers to the power loss value caused by factors such as path attenuation, obstacle obstruction, and multipath effects during the spatial propagation of the wireless signal between the virtual image and the home appliance. The adaptive communication parameters refer to wireless communication parameters that are dynamically adjusted based on the signal propagation path loss value. The communication protocol characteristics refer to the types and protocol characteristics of wireless communication protocols supported by the device display terminal, wherein protocol types include short-range communication and wide-area network communication, and protocol characteristics include maximum transmission unit (MTU), quality of service (QoS) mechanism, and security encryption algorithm. The real-time computing load refers to the real-time computing power consumption during the operation of the virtual image, which can be monitored in real time through the terminal operating system performance monitoring interface (such as iOS's os_signpost and Linux's perf tool). Sample, sampling indicators include CPU utilization, GPU frame rate, memory occupancy, task queue delay, the multimodal communication mode refers to a composite communication strategy selected for virtual mirror and home appliance by combining communication protocol capabilities and real-time computing load, for example, low-latency control instructions can be transmitted via Bluetooth LE, and high-bandwidth status data can be transmitted via Wi-Fi, the historical channel quality data refers to the historical channel status information collected during the communication between the virtual mirror and the home appliance, the information content includes received signal strength indication, signal-to-noise ratio, bit error rate, channel occupancy, number of conflicts, etc., the network congestion probability refers to the historical channel quality data based on the historical channel quality data. , the probability value of data conflict or insufficient bandwidth occurring in the channel where the home appliance is located, predicted by a statistical model (value range: 0-1); the error correction code redundancy refers to the proportion of redundant check bits introduced in data transmission to resist channel noise and congestion, wherein the coding methods include low-density parity-check codes, Reed-Solomon codes, and convolutional codes; the redundancy parameters include the code rate (such as 1 / 2 code rate represents 100% redundancy) and the number of check blocks (such as adding 3 check blocks for every 10 data blocks); the optimal communication strategy refers to an end-to-end communication solution generated by an optimization algorithm based on the comprehensive network congestion probability, error correction code redundancy, and communication environment parameters.

[0149] Optionally, the real-time three-dimensional coordinates of the device display terminal can be collected through a GPS positioning system. Based on the historical channel quality data, the network congestion probability of the home device can be analyzed by calculating the proportion of the number of samples with channel quality indicators lower (or higher) than a threshold in unit time to the total number of samples, as a network congestion probability.

[0150] In an optional embodiment of the present invention, the signal propagation path loss value between the virtual mirror and the home appliance is calculated using the following formula in combination with the real-time three-dimensional coordinates and the spatial layout:

[0151]

[0152] Where PL(d) represents the signal propagation path loss value between the virtual mirror and the home device, PL0 represents the benchmark loss of the signal propagation between the virtual mirror and the home device at a distance d0, d0 represents the standardized benchmark distance of signal propagation (such as 1 meter), n represents the path loss exponent, d represents the real-time 3D Euclidean distance between the real-time 3D coordinates and the spatial layout, α and β represent weight coefficients, and f(θ, φ) represents the azimuth / elevation correction function of the signal propagation between the virtual mirror and the home device. represents the attenuation function of obstacles in the spatial layout, The position vector of the obstacle in three-dimensional space.

[0153] It should be noted that, in this application, the above formula can more accurately calculate the signal propagation path loss value between the virtual mirror and the home device. In particular, it should be noted that the formula not only uses The distance factor is taken into account, and the azimuth / elevation angle of signal propagation (corrected by f(θ,φ)) and the influence of obstacles in the spatial layout (corrected by Compared with the method of considering only a single factor or simple model, it is more adaptable to the calculation of signal propagation loss in complex home environments.

[0154] The embodiment of the present invention constructs a control interface for the home appliance through the virtual mirror based on the communication channel, thereby realizing centralized management of multiple home appliances and personalizing the home appliances in the control interface according to the user's own needs and usage habits. The control interface refers to an interface that allows the user to interact with the home appliance through the virtual mirror and issue control instructions.

[0155] As an embodiment of the present invention, constructing the control interface of the virtual mirror for the home appliance based on the communication channel includes:

[0156] Extracting a signal propagation path loss value corresponding to the communication channel and generating a dynamic protocol parsing module for the communication channel;

[0157] Identifying a spatial action vector in the virtual image and, based on the dynamic protocol parsing module, setting an instruction conversion engine for the spatial action vector;

[0158] Collecting status data of the home appliance and extracting rendering properties of the virtual image;

[0159] Creating a dynamic mapping table between the data type of the state data and the rendering attributes;

[0160] Based on the dynamic mapping table, construct a status feedback parser for the home appliance;

[0161] defining an adapter interface of the communication channel according to the state feedback parser and the dynamic protocol parsing module;

[0162] The dynamic protocol parsing module, the instruction conversion engine, the state feedback parser and the adapter interface are combined to construct a control interface of the virtual mirror for the home appliance.

[0163] Among them, the dynamic protocol parsing module refers to a functional module that dynamically selects the data encoding format and parsing rules according to the signal propagation path loss value, such as switching to compressed binary protocol parsing when the loss is high, and using JSON plain text parsing when the loss is low. The spatial action vector refers to the digital representation of the three-dimensional spatial action corresponding to the user operation in the virtual mirror. The vector dimension includes parameters such as position (X, Y, Z), posture (pitch angle θ, yaw angle φ, roll angle ψ), speed, etc., which can be calculated in real time by the gyroscope and accelerometer of the AR / VR headset. For example, the user's waving action is converted into a vector. The instruction conversion engine refers to an engine that converts the spatial action vector into a control instruction that can be recognized by the dynamic protocol parsing module. The status data refers to the home page data obtained through the device manufacturer's SDK. The semantic description of the home device status data includes the data type, numerical range and unit. The rendering attribute refers to the visual parameter type used to visualize the device status in the virtual mirror interface, including color gradient, icon change, particle animation, etc. The data type refers to the format classification of the home device status data, such as numeric type (such as temperature 25°C) and Boolean type (such as switch on / off). The dynamic mapping table refers to a configurable data structure that records the correspondence between the home device status data type and the virtual mirror rendering attribute. The state feedback parser refers to a functional module that uses the dynamic mapping table to convert the home device status data into visual attribute values ​​that can be rendered by the virtual mirror. The adapter interface refers to a standardized interface designed to be compatible with different communication protocols and rendering engines.

[0164] Optionally, based on the dynamic protocol parsing module, the instruction conversion engine of the spatial action vector can be set through a rule engine, such as the instruction conversion engine loads an editable rule template and generates instructions by matching action vectors in real time. The rendering properties of the virtual image can be extracted using a regular expression matching method, such as extracting key-value pairs through regular expression matching, and inferring the rendering type based on unit symbols (such as '℃', '%'). The dynamic mapping table between the data type of the status data and the rendering properties can be created by a low-code configuration tool, such as the Node-RED tool. Based on the dynamic mapping table, the status feedback parser of the home appliance is constructed using the Apache Camel parsing engine.

[0165] S4. Based on the WiFi signal, a behavior tracking network of the user is constructed. According to the behavior tracking network, the user's current behavior intention for the home device is identified. Based on the current behavior intention and the control interface, personalized control instructions for the home device by the virtual mirror are set.

[0166] The embodiment of the present invention builds a user behavior tracking network based on the WiFi signal, which can realize the user's full body movement recognition and promote seamless interaction between the user and home devices. The behavior tracking network refers to a system architecture that collects user behavior data through WiFi signals to realize real-time tracking and prediction of user behavior characteristics such as location, movement, trajectory, and habits.

[0167] As an embodiment of the present invention, the step of constructing the user's behavior tracking network based on the WiFi signal includes:

[0168] Performing filtering on the WiFi signal to generate a filtered signal;

[0169] extracting spatiotemporal joint features of the filtered signal, and identifying the user's behavior pattern based on the spatiotemporal joint features;

[0170] Analyzing the temporal evolution rules of the behavior pattern using a Bayesian causal network;

[0171] Generating a behavior state transition diagram of the user according to the time evolution rule;

[0172] Based on the behavior state transition graph, a behavior tracking network of the user is constructed.

[0173] The filtering process refers to the operation of noise suppression and signal enhancement on the original WiFi signal (including but not limited to CSI amplitude, phase, and RSSI), and includes operations such as dynamic wavelet packet decomposition, joint Kalman filtering, and phase linearization correction. The spatiotemporal joint features refer to a set of features extracted from the WiFi signal that simultaneously contain temporal evolution patterns and spatial distribution characteristics, including temporal and spatial dimension features. The behavior pattern refers to a semantic description of user actions obtained by classifying or clustering the spatiotemporal joint signal features using a machine learning model, such as "waving," "sitting down," "walking," and "opening the door." The Bayesian causal network refers to a directed acyclic graph model based on probabilistic reasoning, used to analyze the causal relationships and temporal evolution patterns between behavior patterns. The temporal evolution rules refer to the statistical characteristics and evolution patterns of user behavior patterns in the temporal dimension, including information such as the frequency, sequence, and transition probability of the behavior patterns. For example, "the probability of a user switching from the 'watching TV' behavior mode to the 'cooking' behavior mode at dinner time is 70%." The behavior state transition graph refers to a graph structure model with behavior patterns as nodes and the temporal evolution relationships between behavior patterns as directed edges.

[0174] Optionally, the spatiotemporal joint features of the filtered signal may be extracted using a tensor decomposition method, and the user behavior tracking network based on the behavior state transition graph may be constructed using a long short-term memory network model.

[0175] Furthermore, the embodiments of the present invention can provide personalized home services for users by identifying the user's current behavioral intention towards the home appliance based on the behavior tracking network. The current behavioral intention refers to the user's current purpose or need for performing a specific operation on the home appliance determined by the user through behavior, action or environmental interaction. For example, if the user stops in front of the sofa several times and raises his hand to adjust his sitting posture, the current intention is "wanting to sit down and watch TV", which can correspond to operations such as "turning on the TV" and "adjusting the lights".

[0176] Optionally, based on the behavior tracking network, the user's current behavioral intention towards the home device can be identified through action execution confidence, such as calculating the confidence of the user performing a certain type of operation based on the behavior state transition diagram of the behavior tracking network.

[0177] In an embodiment of the present invention, by setting personalized control instructions for the home devices by the virtual mirror based on the current behavioral intention and the control interface, the collaborative efficiency between devices can be improved, and user satisfaction can also be increased. The personalized control instructions refer to device control commands customized for home scenes generated by the virtual mirror system based on personalized characteristics such as individual user behavior habits, preferences, and scene requirements. For example, if a user sits in the study for 45 minutes or longer and has a heart rate greater than 100 beats per minute, a 'seat massage + play white noise' instruction will be generated.

[0178] As an embodiment of the present invention, setting personalized control instructions of the virtual mirror to the home appliance based on the current behavioral intention and the control interface includes:

[0179] Collecting the user's historical interaction data and real-time behavior data, and collecting environmental perception data of the home device;

[0180] Calculating the intention confidence of the current behavior intention based on the historical interaction data and the real-time behavior data;

[0181] Identifying the user's fixed behavior pattern based on the historical interaction data;

[0182] Calculating the user's habit matching degree based on the current behavioral intention and the fixed behavioral pattern;

[0183] Identifying the current environmental status of the home appliance based on the environmental perception data;

[0184] Determining the scenario urgency of the home device based on the current environmental state and the current behavioral intention;

[0185] Creating a dynamic priority matrix for the home devices by combining the intention confidence, the habit matching, and the scenario urgency;

[0186] generating a personalized response mode of the control interface according to the dynamic priority matrix;

[0187] Based on the dynamic priority matrix and the personalized response mode, personalized control instructions of the virtual mirror to the home device are set.

[0188] Among them, the historical interaction data refers to the set of interaction records generated by the user with home appliances through various control interfaces (such as APP, voice, touch panel) within a specified time window (such as the past 30 days); the real-time behavior data refers to the operation instructions and accompanying context data issued by the user through the control interface at the current moment (within ±5 seconds); the environmental perception data refers to the physical quantity data collected in real time by various sensors in the home environment, which is used to characterize the environmental state, including basic environmental data, safety data (such as gas concentration (0-1000ppm), smoke concentration (0-20%LEL)), device status data, etc.; the intention confidence refers to the probability value of the current behavior intention being a real and valid instruction. For example, if the user repeats the "open the curtains" instruction many times and the historical execution success rate is >90%, the confidence = 0.92; the fixed behavior pattern refers to the fixed behavior pattern obtained from the historical interaction data using the DBSCAN clustering algorithm. The data is extracted from the periodic and repetitive user behavior paradigm. The habit matching refers to the cosine similarity between the current behavioral intention and the fixed behavioral pattern, with a value range of [0,1]. The current environmental state refers to the data collected in real time by various sensors in the home environment. After analysis and integration, the actual current situation of the home environment is presented. For example, the bedroom temperature is 22°C and the humidity is 50%, reflecting the indoor temperature, dryness and wetness. The scene urgency refers to the urgency index calculated by the hierarchical analysis method based on the current environmental state and behavioral intention, with a value range of [1,10]. The dynamic priority matrix refers to a Cartesian coordinate system with intention confidence, habit matching, and scene urgency as three dimensions, and a decision model that generates priority values ​​through a weighted sum formula. The personalized response method refers to a control interface interaction strategy customized for different users based on the priority values ​​output by the dynamic priority matrix.

[0189] Optionally, the user's habit matching degree based on the current behavioral intention and the fixed behavioral pattern can be calculated by a Bayesian probability model.

[0190] S5. Collect time series images of the home scene in real time, set a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time series images and the behavior tracking network, and build a data security interaction system between the home devices and the virtual mirror based on the real-time scene synchronization mechanism.

[0191] The embodiments of the present invention can help identify the motion trajectory and spatial relationship changes of objects in the home scene and drive the real-time update of the virtual mirror by real-time acquisition of time series images of the home scene. The time series images refer to a sequence of home scene images continuously acquired at fixed time intervals (such as 10 frames per second) through visual sensors such as cameras. For example, a kitchen camera captures the image changes of water flow and stove flames in real time to form a continuous frame sequence.

[0192] Furthermore, the embodiment of the present invention sets a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time series image and the behavior tracking network, thereby accurately capturing and responding to user behavior intentions, enhancing the environmental perception ability of the virtual mirror, and optimizing the intelligent control of home appliances. The real-time scene synchronization mechanism refers to establishing a dynamic real-time mapping link between the virtual mirror and the real home scene through the coordinated operation of the time series image and the behavior tracking network.

[0193] As an embodiment of the present invention, setting a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time series image and the behavior tracking network includes:

[0194] Identifying the tracked human joints and object movement trajectories in the home scene based on the time series images and the behavior tracking network;

[0195] Based on the tracked human joint points, identifying the movement speed of the human body in the home scene;

[0196] Determining the object displacement vector of the home scene according to the object movement trajectory;

[0197] generating a spatial synchronization element of the virtual mirror image according to the human body movement speed and the object displacement vector;

[0198] Setting an adaptive coordinate update system of the virtual mirror using the spatial synchronization elements;

[0199] Extracting the light intensity, color temperature, and light source direction of the time series image to identify the lighting difference between the virtual mirror image and the home scene;

[0200] According to the illumination difference, setting a real-time light matching mechanism of the virtual mirror;

[0201] In combination with the adaptive coordinate updating system and the real-time light matching mechanism, a real-time scene synchronization mechanism between the virtual mirror and the home scene is set.

[0202] Among them, the tracked human joints refer to the joints of the target human body in the video image or real-time data stream, such as the head joints are manifested as the tip of the nose, ears, etc. The object movement trajectory refers to the position change sequence of movable objects (such as furniture, electrical appliances) in the scene in consecutive frames. The object displacement vector refers to the position change vector of the object in three-dimensional space, which contains the displacement components of the x / y / z axes. The spatial synchronization element refers to the core parameter set used to describe the dynamic changes of the scene, including human body speed, object displacement vector, etc. The human body movement speed refers to the displacement of the human body joints in unit time, reflecting the speed of human movement (such as walking, standing, waving, etc.). It is obtained by calculating the coordinate difference of the joints in adjacent frames and dividing it by the time interval. The adaptive coordinate update system is It refers to the dynamic adjustment mechanism of the virtual mirror coordinate system, which updates the position and posture of the virtual object in real time according to the spatial synchronization elements. The light intensity refers to the strength of the light in the scene. The color temperature refers to the temperature value of the light source color, which can be calculated by fitting the blackbody radiation curve. The light source direction refers to the three-dimensional direction vector of the incident light, which is used to determine the shadow position and the distribution of light and dark surfaces of the object. The lighting difference refers to the parameter difference between the virtual mirror and the real scene in light intensity, color temperature, and light source direction, which is determined by calculating the absolute value difference between the two. The real-time light matching mechanism refers to a closed-loop control system that dynamically adjusts the virtual light source parameters according to the lighting difference. The light source power, color, and position can be adjusted by a PID controller (proportional-integral-differential control) until the difference is lower than the threshold.

[0203] Optionally, based on the time series images and the behavior tracking network, the tracked human joints of the home scene can be identified by an OpenPose model, the object movement trajectory can be identified using ORB feature matching technology, and the object displacement vector of the home scene determined by the object movement trajectory can be determined by a world coordinate system.

[0204] The embodiment of the present invention constructs a data security interaction system between the home device and the virtual mirror based on the real-time scene synchronization mechanism, which can ensure the confidentiality and integrity of data transmission and protect user privacy and behavioral data security. The data security interaction system refers to a set of security mechanisms constructed in the smart home system to ensure the security, reliability and privacy compliance of data interaction between the virtual mirror and home devices.

[0205] As an embodiment of the present invention, the data security interaction system between the home device and the virtual mirror is constructed based on the real-time scene synchronization mechanism, including:

[0206] Based on the real-time scene synchronization mechanism, extracting the real-time scene data of the virtual mirror;

[0207] Identifying abnormal fluctuation factors of the real-time scene data, and locating data security vulnerabilities of the virtual image based on the abnormal fluctuation factors;

[0208] Analyze the causes of the data security vulnerabilities and, based on the causes, create a vulnerability self-repair mechanism for the virtual image;

[0209] Identifying the user identity of the home device and the corresponding device operation habits;

[0210] Generating an identity authentication system for the virtual image according to the user identity and the device operation habits;

[0211] Setting a data exposure boundary of the virtual image based on the user identity;

[0212] Generating a visual and secure interactive interface between the home appliance and the virtual image using a device display terminal of the virtual image;

[0213] By combining the vulnerability self-repair mechanism, the identity authentication system, the data exposure boundary and the visual security interaction interface, a data security interaction system between the home appliance and the virtual mirror is constructed.

[0214] Among them, the real-time scene data refers to a dynamic data set collected and transmitted to the virtual mirror by the real-time scene synchronization mechanism, the abnormal fluctuation factor refers to an abnormal pattern in the real-time scene data that deviates from the preset threshold or historical behavior pattern, such as the frequent switching of lamps without operation, which can be identified by the moving average method, the data security vulnerability refers to a potential security defect in the virtual mirror or home device data interaction process, such as weak identity authentication, the formation cause refers to the fundamental factor or triggering condition that leads to the occurrence of virtual mirror data security vulnerability, such as firmware vulnerability leading to data encryption module abnormality, the vulnerability self-repair mechanism refers to a closed-loop processing system based on automation technology (such as smart contracts, AI decision engines) to respond to identified data security vulnerabilities in real time, and the user identity refers to the user's identity through biometrics (fingerprints, face recognition), account credentials (passwords, tokens) or behavioral characteristics (operating habits, the device operation habit refers to the personalized interaction mode of the user with the home device, which is obtained by analyzing historical operation data (such as operation time distribution, instruction combination, and preference settings); the identity authentication system refers to a multi-level verification mechanism built based on user identity and operation habits. For example, when a user unlocks the virtual mirror access permission through fingerprint, the system also verifies whether the lighting parameters of the current environment match the user's operation habits (such as the user usually operates in a warm light environment); the data exposure boundary refers to the security boundary of the data range and operation permission open to the virtual mirror to the home device or user based on the user's identity and authority level; the visual security interaction interface refers to an interaction interface that displays the security interaction status of the home device and the virtual mirror to the user through visual elements such as charts, dashboards, and status indicators based on the device display terminal.

[0215] Optionally, based on the abnormal fluctuation factor, the data security vulnerability of the virtual image can be located by a protocol consistency verification method, and the cause of the data security vulnerability can be analyzed using an LSTM neural network.

[0216] S6. Combine the virtual mirror, the personalized control instructions, the real-time scene synchronization mechanism and the data security interaction system to perform intelligent control processing of the home appliance to obtain a home control result.

[0217] The embodiment of the present invention combines the virtual mirror, the personalized control instructions, the real-time scene synchronization mechanism and the data security interaction system to perform intelligent control processing of the home appliances and obtain home control results, which can realize three-dimensional perception and dynamic presentation of the home environment, accurately capture user behavior intentions and provide personalized control instructions, form a user-centered personalized, spatialized smart home control system, and significantly improve the user's interaction accuracy and scene adaptability. The intelligent control processing refers to the process of implementing closed-loop intelligent control of home appliances by integrating virtual mirrors, personalized control instructions, real-time scene synchronization mechanism, and data security interaction system. The home control result refers to the state change or actual response result made by the home appliance according to the instruction after the intelligent control processing process is executed. For example, the virtual mirror detects the elderly sitting up and automatically turns on the floor lamp within 30 cm of the bed.

[0218] See Figure 2 As shown in FIG, a home control diagram of a method for controlling a home using a virtual environment under a digital twin provided by an embodiment of the present invention. Figure 2 In the system, the controller serves as the core unit for data processing and decision-making. It receives data from temperature sensors, humidity sensors, and PM2.5 sensors, and combines the current scene information fed back by the virtual mirror to decide whether and how to control home appliances. The execution device then performs specific operations based on the decision made by the controller, ultimately realizing intelligent adjustment of the home environment and achieving home control results.

[0219] Compared with the problems described in the background technology, the embodiment of the present invention uses the physical data to construct the digital twin of the home scene, which can accurately present the virtual scene of the home, helping to realize the remote control of the user on the virtual interface of the home appliance; further, the embodiment of the present invention collects the user's human body scattered wave data according to the WiFi signal of the home appliance, and can accurately calculate the user's position and movement trajectory in the room, thereby driving the virtual character to make corresponding postures and actions, enhancing the realism of the virtual mirror and the user's immersion; the embodiment of the present invention creates a 3D human body model of the user in the digital twin based on the human body scattered wave data, which can reflect the movement state and posture changes of the human body in real time, and realize real-time dynamic update of the human body in the virtual mirror; further, the embodiment of the present invention generates a virtual mirror by performing 3D rendering processing on the digital twin and the 3D human body model, so that the user can intuitively understand and perceive the scenes and characters in the virtual environment, enhance the user experience, improve interactivity and practicality; the embodiment of the present invention constructs the user's behavior tracking network based on the WiFi signal, which can realize the user's full-body movement recognition and promote the seamless interaction between the user and the home appliance; further, the embodiment of the present invention identifies the user's actions on the home appliance according to the behavior tracking network. The current behavioral intention of the device can provide personalized home services for the user; the embodiment of the present invention sets the personalized control instructions of the virtual mirror for the home device based on the current behavioral intention and the control interface, which can improve the collaborative efficiency between devices and also improve user satisfaction; further, the embodiment of the present invention sets the real-time scene synchronization mechanism between the virtual mirror and the home scene according to the time series image and the behavior tracking network, which can achieve accurate capture and response of user behavioral intention, enhance the environmental perception ability of the virtual mirror, and optimize the intelligent control of home devices; the embodiment of the present invention constructs a data security interaction system between the home device and the virtual mirror based on the real-time scene synchronization mechanism, which can ensure the confidentiality and integrity of data transmission and protect user privacy and behavioral data security; finally, the embodiment of the present invention combines the virtual mirror, the personalized control instructions, the real-time scene synchronization mechanism and the data security interaction system to perform intelligent control processing of the home device to obtain home control results, which can achieve three-dimensional perception and dynamic presentation of the home environment, accurately capture user behavioral intention and provide personalized control instructions, form a user-centered personalized and spatialized smart home control system, and significantly improve the user's interaction accuracy and the consistency of action in the real scene. Therefore, the virtual image control home method and system under the digital twin provided by the embodiment of the present invention can achieve a one-to-one correspondence between the image displayed by the digital twin and the real scene, thereby satisfying the user's accurate control of the actual status of the supervised object.

[0220] Example 2:

[0221] like Figure 3 The figure shows a functional module diagram of a virtual environment control home system under a digital twin of the present invention.

[0222] The virtual environment control home system 200 based on digital twins described in the present invention can be installed in an electronic device. Depending on the functions implemented, the virtual environment control home system based on digital twins can include a scene modeling module 201, a virtual character construction module 202, a virtual image generation module 203, a personalized control module 204, a security interaction module 205, and a home control module 206. The modules described in the present invention can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can perform fixed functions, which are stored in the memory of the electronic device.

[0223] In the embodiment of the present invention, the functions of each module / unit are as follows:

[0224] The scene modeling module 201 is used to obtain the user's home scene and its corresponding home appliances, collect physical data of the home scene, wherein the physical data includes object data, environmental state data and spatial structure information, and use the physical data to build a digital twin of the home scene;

[0225] The virtual character construction module 202 is used to collect the user's human body scattered wave data according to the WiFi signal of the home device, and create a 3D human body model of the user in the digital twin based on the human body scattered wave data.

[0226] The virtual image generation module 203 is used to perform 3D rendering processing on the digital twin and the 3D human body model to generate a virtual image, establish a communication channel between the virtual image and the home appliance, and build a control interface for the virtual image to the home appliance based on the communication channel;

[0227] The personalized control module 204 is configured to construct a user behavior tracking network based on the WiFi signal, identify the user's current behavioral intention with respect to the home appliance based on the behavior tracking network, and set personalized control instructions for the virtual mirror to the home appliance based on the current behavioral intention and the control interface;

[0228] The security interaction module 205 is used to collect time-series images of the home scene in real time, and set a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time-series images and the behavior tracking network, and build a data security interaction system between the home devices and the virtual mirror based on the real-time scene synchronization mechanism;

[0229] The home control module 206 is used to combine the virtual mirror, the personalized control instructions, the real-time scene synchronization mechanism and the data security interaction system to perform intelligent control processing of the home appliances and obtain home control results.

[0230] In detail, the modules in the virtual environment control home system 200 under the digital twin embodiment of the present invention are used in the same manner as above. Figure 1 The technical means described in the article are similar to the method of controlling home appliances through virtual environments under digital twins and can produce the same technical effects, so I will not go into details here.

[0231] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for controlling home appliances using virtual images based on digital twins, characterized in that: The method comprises: Obtain the user's home scene and its corresponding home devices, collect physical data of the home scene, where the physical data includes object data, environmental state data, and spatial structure information, and use the physical data to construct a digital twin of the home scene; collecting human body scattered wave data of the user according to the WiFi signal of the household device, and creating a 3D human body model of the user in the digital twin based on the human body scattered wave data; Performing 3D rendering processing on the digital twin and the 3D human body model to generate a virtual image, establishing a communication channel between the virtual image and the home appliance, and constructing a control interface for the virtual image to control the home appliance based on the communication channel; Based on the WiFi signal, a behavior tracking network of the user is constructed, and according to the behavior tracking network, the user's current behavior intention with respect to the home device is identified, and based on the current behavior intention and the control interface, personalized control instructions for the home device by the virtual mirror are set; Collecting time-series images of the home scene in real time, and setting a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time-series images and the behavior tracking network, and building a data security interaction system between the home devices and the virtual mirror based on the real-time scene synchronization mechanism; In combination with the virtual mirror, the personalized control instructions, the real-time scene synchronization mechanism and the data security interaction system, the intelligent control processing of the home appliance is performed to obtain a home control result.

2. The method for controlling a home using a virtual environment under a digital twin as claimed in claim 1, wherein: The constructing of the digital twin of the home scene using the physical data includes: Determining the built-in home products of the home scene and their corresponding operating states and spatial layouts through the physical data, and identifying environmental state data of the home scene; Analyzing the correlation between the operating status and the environmental status data; identifying, based on the association relationship, operating modes of the built-in home products in different environments; Building a behavior model of the built-in home product in the different environments through the operation mode; Constructing a geometric model of the built-in home furnishing product using the appearance structure and the spatial layout of the built-in home furnishing product; Based on the material properties of the object in the physical data, texture mapping is performed on the geometric model to generate a physical model; The behavior model, the geometric model, and the physical model are combined to construct a digital twin of the home scene.

3. The method for controlling a home using a virtual environment based on a digital twin as claimed in claim 1, wherein: The collecting of the user's human body scattered wave data according to the WiFi signal of the home device includes: Determine the WiFi transceiver node corresponding to the home device according to the WiFi signal of the home device; Collecting original WiFi signal data from the WiFi transceiver node and identifying interference signals in the original WiFi signal data; Analyzing an interference type of the interference signal, and performing interference removal processing on the interference signal based on the interference type to obtain an optimized WiFi signal; Extracting a subcarrier signal from the optimized WiFi signal and calculating a signal-to-noise ratio of the subcarrier signal; Determining a sensitive subcarrier signal of the user according to the signal-to-noise ratio; Based on the sensitive subcarrier signal, human body scattered wave data of the user is collected.

4. The method for controlling a home using a virtual environment based on a digital twin as claimed in claim 1, wherein: The step of creating a 3D human body model of the user in the digital twin based on the human body scattered wave data includes: Identifying scattered wave intensity and phase difference in the human body scattered wave data; determining a body contour of the user according to the scattered wave intensity; Extracting internal structural features of the user's body based on the phase difference; Creating a virtual human body of the user according to the human body contour and the internal structural features of the human body; Extracting key feature points of the user from the human body contour and the human body internal structure features; Extracting the coordinate parameters of the key feature points from the human body scattered wave data using a skeleton tracking algorithm; Based on the coordinate parameters, identifying the body parts of the user and the corresponding motion states; Constructing a mapping relationship between the virtual human body and the user according to the body part and the motion state; Based on the mapping relationship and the virtual human body, a 3D human body model of the user in the digital twin is created.

5. The method for controlling a home using a virtual environment based on a digital twin as claimed in claim 1, wherein: The creating of a communication channel between the virtual mirror and the home appliance includes: Identifying the spatial layout of the device display terminal of the virtual mirror and the home appliances; Collecting the real-time three-dimensional coordinates of the device display terminal, and calculating the signal propagation path loss value between the virtual mirror and the home device in combination with the real-time three-dimensional coordinates and the spatial layout; Setting adaptive communication parameters between the virtual mirror and the home device based on the signal propagation path loss value; Identifying communication protocol characteristics of the device display terminal and a real-time computing load of the virtual image; Setting a multimodal communication mode between the virtual mirror and the home appliance according to the communication protocol characteristics and the real-time computing load; collecting historical channel quality data of the home device, and analyzing the network congestion probability of the home device based on the historical channel quality data; Setting the error correction coding redundancy of the virtual image according to the network congestion probability; generating an optimal communication strategy between the virtual mirror and the home device based on the network congestion probability and the error correction code redundancy; A communication channel between the virtual mirror and the home appliance is created by combining the optimal communication strategy, the adaptive communication parameters and the multimodal communication method.

6. The method for controlling a home using a virtual environment based on a digital twin as claimed in claim 1, wherein: The step of constructing a control interface for the home appliance by the virtual mirror based on the communication channel includes: Extracting a signal propagation path loss value corresponding to the communication channel and generating a dynamic protocol parsing module for the communication channel; Identifying a spatial action vector in the virtual image and, based on the dynamic protocol parsing module, setting an instruction conversion engine for the spatial action vector; Collecting status data of the home appliance and extracting rendering properties of the virtual image; Creating a dynamic mapping table between the data type of the state data and the rendering attributes; Based on the dynamic mapping table, construct a status feedback parser for the home appliance; defining an adapter interface of the communication channel according to the state feedback parser and the dynamic protocol parsing module; The dynamic protocol parsing module, the instruction conversion engine, the state feedback parser and the adapter interface are combined to construct a control interface of the virtual mirror for the home appliance.

7. The method for controlling a home using a virtual environment based on a digital twin as claimed in claim 1, wherein: The step of constructing a user behavior tracking network based on the WiFi signal includes: Performing filtering on the WiFi signal to generate a filtered signal; extracting spatiotemporal joint features of the filtered signal, and identifying the user's behavior pattern based on the spatiotemporal joint features; Analyzing the temporal evolution rules of the behavior pattern using a Bayesian causal network; Generating a behavior state transition diagram of the user according to the time evolution rule; Based on the behavior state transition graph, a behavior tracking network of the user is constructed.

8. The method for controlling a home using a virtual environment based on a digital twin as claimed in claim 1, wherein: The step of setting personalized control instructions of the virtual mirror to the home appliance based on the current behavioral intention and the control interface includes: Collecting the user's historical interaction data and real-time behavior data, and collecting environmental perception data of the home device; Calculating the intention confidence of the current behavior intention based on the historical interaction data and the real-time behavior data; Identifying the user's fixed behavior pattern based on the historical interaction data; Calculating the user's habit matching degree based on the current behavioral intention and the fixed behavioral pattern; Identifying the current environmental status of the home appliance based on the environmental perception data; Determining the scenario urgency of the home device based on the current environmental state and the current behavioral intention; Creating a dynamic priority matrix for the home devices by combining the intention confidence, the habit matching, and the scenario urgency; generating a personalized response mode of the control interface according to the dynamic priority matrix; Based on the dynamic priority matrix and the personalized response mode, personalized control instructions of the virtual mirror to the home device are set.

9. The method for controlling a home using a virtual environment based on a digital twin as claimed in claim 1, wherein: The step of setting a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time series image and the behavior tracking network includes: Identifying the tracked human joints and object movement trajectories in the home scene based on the time series images and the behavior tracking network; Based on the tracked human joint points, identifying the movement speed of the human body in the home scene; Determining the object displacement vector of the home scene according to the object movement trajectory; generating a spatial synchronization element of the virtual mirror image according to the human body movement speed and the object displacement vector; Setting an adaptive coordinate update system of the virtual mirror using the spatial synchronization elements; Extracting the light intensity, color temperature, and light source direction of the time series image to identify the lighting difference between the virtual mirror image and the home scene; According to the illumination difference, setting a real-time light matching mechanism of the virtual mirror; In combination with the adaptive coordinate updating system and the real-time light matching mechanism, a real-time scene synchronization mechanism between the virtual mirror and the home scene is set.

10. A virtual environment control home system based on digital twin, characterized by: The system comprises: A scene modeling module is used to obtain the user's home scene and its corresponding home devices, collect physical data of the home scene, where the physical data includes object data, environmental state data and spatial structure information, and use the physical data to build a digital twin of the home scene; a virtual character construction module, configured to collect human body scattered wave data of the user according to the WiFi signal of the household device, and create a 3D human body model of the user in the digital twin based on the human body scattered wave data; A virtual image generation module is configured to perform 3D rendering processing on the digital twin and the 3D human body model to generate a virtual image, establish a communication channel between the virtual image and the home appliance, and construct a control interface for the virtual image to control the home appliance based on the communication channel; a personalized control module, configured to construct a behavior tracking network for the user based on the WiFi signal, identify the user's current behavioral intention with respect to the home appliance based on the behavior tracking network, and set personalized control instructions for the virtual mirror with respect to the home appliance based on the current behavioral intention and the control interface; a secure interaction module for collecting time-series images of the home scene in real time, and setting a real-time scene synchronization mechanism between the virtual mirror and the home scene based on the time-series images and the behavior tracking network, and building a data security interaction system between the home devices and the virtual mirror based on the real-time scene synchronization mechanism; The home control module is used to combine the virtual mirror, the personalized control instructions, the real-time scene synchronization mechanism and the data security interaction system to perform intelligent control processing of the home appliances and obtain home control results.

Images