Control method and electronic equipment

Abstract

The invention discloses a control method and electronic equipment. Belongs to the technical field of communication. The method comprises the steps that under the condition that a three-dimensional model of a house where a user is located is displayed, first input of the user is received; in response to the first input, determining a target wall body in the three-dimensional model, and displaying at least one piece of Internet of Things equipment shielded by the target wall body in the three-dimensional model; and under the condition that a second input of the user is received, determining a target Internet of Things device in the at least one Internet of Things device, and sending a control instruction to the target Internet of Things device.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of communication technologies, and specifically to control methods and electronic devices. Background Art

[0002] With the popularization of smart homes, there are usually Internet of Things (IoT) devices distributed in multiple rooms in a user's home, such as smart lights, air conditioners, curtains, etc. Users can use Extended Reality (XR) devices to control the IoT devices.

[0003] In the prior art, XR devices only support visual control of IoT devices within the user's field of view, and cannot control IoT devices blocked by walls. If it is necessary to control an IoT device blocked by a wall, the user needs to move to the room where the device is located, which is a cumbersome and inefficient way of operation. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a control method and an electronic device that can improve the control efficiency of IoT devices blocked by walls.

[0005] In a first aspect, an embodiment of the present application provides a control method applied to an XR device. The method includes: when displaying a three-dimensional model of the user's house, receiving a first input from the user; in response to the first input, determining a target wall in the three-dimensional model and displaying at least one IoT device blocked by the target wall in the three-dimensional model; when receiving a second input from the user, determining a target IoT device among the at least one IoT device and sending a control instruction to the target IoT device.

[0006] In a second aspect, an embodiment of the present application provides a control device applied to an XR device. The device includes: a receiving unit, configured to receive a first input from the user when displaying a three-dimensional model of the user's house; a first determination unit, configured to determine a target wall in the three-dimensional model and display at least one IoT device blocked by the target wall in the three-dimensional model in response to the first input; a second determination unit, configured to determine a target IoT device among the at least one IoT device and send a control instruction to the target IoT device when receiving a second input from the user.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory. The memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] Fourthly, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] Fifthly, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement the method described in the first aspect.

[0010] Sixthly, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In the embodiment of the present application, when displaying a three-dimensional model of the house where the user is located, a first input of the user is received; in response to the first input, a target wall in the three-dimensional model is determined, and at least one Internet of Things device blocked by the target wall in the three-dimensional model is displayed; when a second input of the user is received, a target Internet of Things device among the at least one Internet of Things device is determined, and a control instruction is sent to the target Internet of Things device. By visualizing the Internet of Things devices blocked by the target wall in the three-dimensional model of the house where the user is located through an extended reality device, the user can directly complete the cross-room visual control of the target Internet of Things device through the extended reality device in the current room without moving to the room where the target Internet of Things device is located, reducing the complexity of user operations and improving the control efficiency of the Internet of Things devices blocked by the wall. Description of the Drawings

[0012] Figure 1 is a flowchart of the control method provided by an embodiment of the present application;

[0013] Figure 2A is a schematic diagram of an application scenario of the control method provided by an embodiment of the present application;

[0014] Figure 2B is a schematic diagram of an application scenario of the control method provided by an embodiment of the present application;

[0015] Figure 3A is a schematic diagram of an application scenario of the control method provided by an embodiment of the present application;

[0016] Figure 3B is a schematic diagram of an application scenario of the control method provided by an embodiment of the present application;

[0017] Figure 4A is a schematic diagram of the three-dimensional model generation process in the control method provided by an embodiment of the present application;

[0018] Figure 4B It is a schematic diagram of the three-dimensional model generation process in the control method provided by an embodiment of the present application;

[0019] Figure 5 It is a schematic structural diagram of the control device provided by an embodiment of the present application;

[0020] Figure 6 It is a schematic structural diagram of the electronic device provided by an embodiment of the present application;

[0021] Figure 7 It is a schematic hardware structure diagram of the electronic device suitable for implementing the embodiment of the present application. Detailed implementation manners

[0022] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, rather than all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0023] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0024] Next, the control method and device provided by the embodiments of the present application will be described in detail in conjunction with the accompanying drawings through specific embodiments and their application scenarios.

[0025] Please refer to Figure 1, which shows one of the flowcharts of the control method provided by the embodiments of the present application. The control method provided by the embodiments of the present application can be applied to Extended Reality (XR) devices. Extended Reality devices refer to devices that can integrate virtual content with the real environment, specifically including but not limited to Virtual Reality (VR) devices, Mixed Reality (MR) devices, etc. Among them, a Virtual Reality device is a device that can create a completely virtual three-dimensional environment and allows users to interact with the virtual environment. A Mixed Reality device encompasses technologies such as virtual reality, augmented reality, and mixed reality, and can create an environment that combines virtual and real elements. It can recognize and understand the real environment and superimpose or integrate virtual elements into the real world.

[0026] The flow of the control method provided by the embodiments of the present application includes the following steps:

[0027] Step 101, when displaying a three-dimensional model of the user's house, receive a first input from the user.

[0028] In this embodiment, the user's house can refer to a specific movable space for the user. For example, it can be a residence, a company, etc. The user's house can include at least one room. The rooms can be separated by walls. The user's house can include at least one floor, and the floors can also be separated by walls. In practice, the XR device can display the area corresponding to the user's current field of view in the three-dimensional model on the display screen in real time. As the user's field of view changes, the displayed area of the three-dimensional model changes accordingly.

[0029] In this embodiment, the three-dimensional model refers to a three-dimensional geometric model of the user's house generated through sensor scanning and modeling algorithms in extended reality, and can include the spatial positions and geometric information of walls, Internet of Things devices, and other indoor facilities. In practice, during the process of the user wearing the XR device and moving around the whole house, the XR device can pre-scan the whole house space through sensors such as lidar, cameras, and Inertial Measurement Unit (IMU), and generate a high-precision three-dimensional model in combination with semantic segmentation technology, etc.

[0030] In this embodiment, the first input can be used to select a target wall in the three-dimensional model. The first input can be an eye movement input, a gesture input, a voice input, or other feasible inputs, and can be specifically determined according to actual usage requirements, which is not limited in the embodiments of the present application.

[0031] Exemplarily, refer to Figure 2A, the user is located in the entrance hall of the residence. The extended reality device worn by the user can display the area within the user's field of view in the 3D model of the whole house, specifically including part of the living room area. When the user gazes at the wall between the entrance hall and the living room for more than 2 seconds, or points at the wall between the entrance hall and the living room with a finger for more than 2 seconds, the first input can be completed. At this time, the wall that the user gazes at or points at with a finger can be determined as the target wall, as shown in reference numeral 201.

[0032] Step 102, in response to the first input, determine the target wall in the 3D model and display at least one Internet of Things device blocked by the target wall in the 3D model.

[0033] In this embodiment, the target wall refers to the wall specified by the user through the first input. The Internet of Things device refers to a device that can be connected through the Internet and interact with other devices, the cloud, or the user. For example, it can include but is not limited to smart lights, air conditioners, curtains, floor cleaning robots, etc., and these devices can be controlled through the extended reality device.

[0034] In this embodiment, in response to the first input, the first line of sight direction can be determined first. The first line of sight direction refers to the line of sight direction of the user when triggering the first input, and can be determined by the tracking system of the extended reality device. The tracking system can use the built-in IMU sensor and camera to determine the position of the user's head and eyes, so as to calculate the line of sight direction.

[0035] In this embodiment, the display method and style of at least one Internet of Things device blocked by the target wall can be preset as needed, and are not limited here. It should be noted that while displaying the Internet of Things device blocked by the target wall, other objects blocked by the target wall, such as furniture, etc., can also be displayed, which is not specifically limited here.

[0036] Optionally, the target wall can be displayed in a weakened manner to perspectively display at least one Internet of Things device blocked by the target wall in the 3D model. Weakened display means that the display effect of the wall is weakened by adjusting the display effect to perspectively display the Internet of Things device behind the wall. The weakened display method can include but is not limited to increasing transparency, only displaying the outline, hiding the target wall, etc. Perspectively display can include but is not limited to at least one of the following: displaying the outline of at least one Internet of Things device blocked by the target wall in the area where the target wall is located, displaying the visual image of at least one Internet of Things device blocked by the target wall in the area where the target wall is located.

[0037] As an example, the target wall in the 3D model can be hidden so that the Internet of Things device blocked by the target wall can be displayed. As another example, the transparency of the target wall in the 3D model can be adjusted so that the Internet of Things device blocked by the target wall can be perspectively displayed. As yet another example, based on reference Figure 2AAfter the user gazes at the wall 201 between the entrance hall and the living room for more than 2 seconds, the transparency of the wall 201 can be increased, while the outline of the wall is retained, and the smart table lamp 202 blocked by the wall can be highlighted.

[0038] In practice, each object in the 3D model has a corresponding visual image, which can be obtained by sensors such as cameras when globally scanning the user's house. Thus, the appearance of the object can be seen from different positions and angles. The IoT device in the occluded area shown in perspective can include not only the outline of the IoT device in the 3D model, but also the visual image of the IoT device obtained when globally scanning the user's house.

[0039] By weakening the display of the target wall and showing the IoT device blocked by it in perspective, the user can directly see the IoT device blocked by the target wall, enhancing the spatial perception ability. For example, the user can view the air conditioner status in the bedroom in the living room without entering the bedroom.

[0040] Step 103: When receiving the second input from the user, determine the target IoT device among at least one IoT device, and send a control instruction to the target IoT device.

[0041] In this embodiment, the second input can be used to select the target IoT device and trigger the generation of the control instruction. The second input can include eye movement input, gesture input, voice input, or other feasible inputs, which can be specifically determined according to actual usage requirements and are not limited in this application embodiment. Among them, the above gesture input can include, but is not limited to, pinching, dragging, pointing, etc. The target IoT device can be any IoT device among the above at least one IoT device. The control instruction is an instruction for controlling the target IoT device.

[0042] In this embodiment, for different IoT devices, the corresponding relationship between different control instructions and input methods can be preset in advance. Based on the preset corresponding relationship for the target IoT device, the control instruction corresponding to the second input can be determined, and then the control instruction can be generated and sent.

[0043] Exemplarily, continue to refer to Figure 2A , the target IoT device in the 3D model is the smart table lamp 202 in the on state. After the user's eye movement targets the smart table lamp 202 and triggers the switch control area 203 of the smart table lamp 202 through gestures at the same time, the second input is completed. At this time, a turn-off instruction can be sent to the smart table lamp to turn it off.

[0044] Optionally, after sending a control instruction to the target Internet of Things device, the status information of the target Internet of Things device can also be received to dynamically update the displayed virtual interface based on this status information. For example, after adjusting the light, the brightness of the virtual interface can be synchronously updated. Thereby, the authenticity of the virtual interface can be improved.

[0045] Optionally, the control instruction can be sent to the target Internet of Things device through the following steps:

[0046] Step S11, receive the third input of the user for the target Internet of Things device in the 3D model.

[0047] Step S12, in response to the third input, display the virtual control panel of the target Internet of Things device.

[0048] Step S13, receive the fourth input of the user for the virtual control panel.

[0049] Step S14, in response to the fourth input, generate a control instruction and send the control instruction to the target Internet of Things device.

[0050] Among them, the third input can be used to trigger the display of the virtual control panel of the target Internet of Things device. The third input can include eye movement input, gesture input, voice input, or other feasible inputs, which can be specifically determined according to actual usage requirements, and the embodiments of the present application do not make limitations. The fourth input can be used to trigger the control instruction for the target Internet of Things device. The fourth input can include eye movement input, gesture input, voice input, or other feasible inputs, which can be specifically determined according to actual usage requirements, and the embodiments of the present application do not make limitations. The virtual control panel refers to an interface generated in the display interface of the extended reality device for controlling the target Internet of Things device. The virtual control panel can contain various interactive controls, such as buttons, sliders, switches, etc.; the virtual control panel can also include the information collected by the target Internet of Things device, such as temperature, humidity, etc. The virtual control panel can be generated by analyzing the functions of each Internet of Things device through a deep learning model such as PointNet++.

[0051] Exemplarily, referring to Figure 2B , the target Internet of Things device is the smart table lamp 202. When the user gazes at the smart table lamp 202 for more than 2 seconds, the third input can be completed. At this time, the virtual control panel 204 of the smart table lamp 202 can be displayed. Through the virtual control panel, not only can the smart table lamp 202 be turned on and off, but its brightness can also be adjusted. The user can interact with the virtual control panel 204 through gesture input or eye movement input. For example, if the user gazes at the on / off key in the virtual control panel 204, an on instruction can be sent to the smart table lamp 202 in the off state to control it to turn on.

[0052] It should be noted that the second input and the third input can also be the same input. In this case, when the second input of the user is received, while selecting the target Internet of Things device, the virtual control panel of the target Internet of Things device can be displayed.

[0053] By displaying the virtual control panel, the user can intuitively perform more refined operations on the Internet of Things device without having to remember complex operation procedures, improving the convenience and accuracy of controlling the Internet of Things device.

[0054] Optionally, the transmission of the control instruction can be achieved through the following steps: First, send the control instruction for the target Internet of Things device to the home gateway through the Message Queuing Telemetry Transport (MQTT) protocol. Then, through the home gateway, forward the control instruction to the target Internet of Things device.

[0055] Among them, the Message Queuing Telemetry Transport protocol is a lightweight message transmission protocol designed for low-bandwidth, high-latency or unreliable network environments and is commonly used for the communication of Internet of Things devices. The home gateway is the central device in the home network, responsible for connecting the home internal network and the external network, managing all Internet of Things devices in the home network, and forwarding control instructions and status data.

[0056] The lightweight feature of the Message Queuing Telemetry Transport protocol enables it to perform excellently in low-bandwidth and high-latency network environments, ensuring that the control instruction can be quickly transmitted to the home gateway. For example, after the user sends the "turn on the light" instruction, the light can respond within 100 milliseconds, providing instant feedback and ensuring the accuracy and timeliness of the control instruction sending. In addition, the Message Queuing Telemetry Transport protocol supports message confirmation and retransmission mechanisms to ensure the reliable transmission of control instructions. Even in the case of unstable network, the system can ensure the successful delivery of the instruction through the retransmission mechanism. Moreover, the Message Queuing Telemetry Transport protocol supports the connection and management of a large number of devices, and the home gateway can be easily expanded to support more Internet of Things devices. For example, the user can easily add new Internet of Things devices to the home network without modifying the existing communication architecture.

[0057] By transmitting the control instruction through the Message Queuing Telemetry Transport protocol and through the centralized management of the home gateway, the connection of Internet of Things devices and the sending of control instructions can be carried out conveniently and quickly, improving the stability and security of the network.

[0058] It should be noted that the objects that the user can operate on are not limited to the Internet of Things devices blocked by the target wall, but can also include other objects blocked by the target wall and Internet of Things devices not blocked by the target wall. As an example, see Figure 3A, the user can point at the floor of the living room in the 3D model through gestures and control the sweeping robot 301 to go for cleaning through voice. As another example, see Figure 3B , the user can fixate on the curtain shown by reference numeral 302 in the 3D model with eye movement for 2 seconds, making the curtain contour line highlighted and enter the interactive state, and perform pinching or dragging through gesture operations to trigger the generation of a closing instruction for the curtain. Further, the opening and closing ratio of the curtain can be mapped according to the amplitude of the user's dragging to precisely control it.

[0059] Further, the display style of irrelevant objects in the displayed 3D model can also be automatically adjusted according to the relative position of the user and the target Internet of Things device. For example, when the user controls the Internet of Things device in the living room, the wall of the bedroom can be weakened in display.

[0060] The traditional smart home control method requires the user to perform multiple steps of operations through an application or a voice assistant. For example, successively opening the application, selecting the room, selecting the device, adjusting the parameters, confirming the operation, etc. However, in the application embodiment, the Internet of Things device blocked by the target wall in the 3D model of the user's house is visualized through the extended reality device. Without moving to the room where the target Internet of Things device is located, the user can directly complete the cross-room visual control of the target Internet of Things device in the current room through the extended reality device, reducing the complexity of the user's operation and improving the control efficiency of the Internet of Things device blocked by the wall. At the same time, it can be applied to complex house structures with multiple rooms and multiple floors to meet the user's control requirements for the whole-house Internet of Things devices in complex scenarios.

[0061] In some alternative embodiments, the above 3D model is a 3D geometric model. Before displaying the 3D model of the user's house, the following steps can also be performed to generate the 3D model:

[0062] Step S21, globally scan the user's house through the sensors in the extended reality device to obtain sensor data.

[0063] Specifically, during the process of the user wearing the extended reality device and moving in the house, the extended reality device can globally scan the user's house through the sensors installed on it to obtain sensor data. Global scanning means comprehensive scanning. The sensor data is the data collected by the above sensors.

[0064] In practice, sensors in extended reality devices may include, but are not limited to, lidar, cameras, IMUs, etc. Correspondingly, sensor data may include, but are not limited to, point cloud data, image data, pose data, etc. Exemplarily, lidar can emit lasers and receive reflected signals to collect point cloud data of a house, which can reflect the depth information of objects in the house; cameras can collect image data of the house, which can reflect the visual information of objects in the house; IMUs can record the motion pose of the extended reality device to ensure the accuracy of scanning.

[0065] Step S22: Generate a three-dimensional point cloud model based on the sensor data.

[0066] Specifically, a three-dimensional point cloud model refers to a model containing the coordinate information of a large number of points generated from sensor data, which can be used to represent the position and shape of objects in space. Exemplarily, see Figure 4A as shown.

[0067] In practice, mapping algorithms such as ORB-SLAM3 (ORB-Simultaneous Localization and Mapping 3, the third version of ORB-based simultaneous localization and mapping) can be used to process the sensor data, fuse the point cloud data collected by lidar with the image data collected by cameras, and generate a three-dimensional point cloud model of the house. Among them, ORB (Oriented FAST and Rotated BRIEF) is an efficient feature detection and description algorithm. The error of the three-dimensional point cloud model can be controlled within 3 centimeters to ensure the accuracy of the model.

[0068] Step S23: Perform semantic segmentation on the three-dimensional point cloud model to obtain a semantic segmentation result, which includes walls, Internet of Things devices, and other indoor facilities.

[0069] Specifically, semantic segmentation refers to classifying each point in the three-dimensional point cloud model to identify different object categories. The semantic segmentation result may include the category labels of each point in the three-dimensional point cloud model. Exemplarily, the categories may include walls, Internet of Things devices, and other indoor facilities.

[0070] In practice, deep learning models such as PointNet++ can be used to achieve semantic segmentation of the three-dimensional point cloud model, identify objects such as sofas, dining tables, air conditioners, etc., and assign corresponding category labels to the point clouds of each object. Through semantic segmentation, walls, Internet of Things devices, and other indoor facilities in the house can be automatically identified without manual annotation, improving the efficiency and accuracy of modeling.

[0071] Step S24: Generate a three-dimensional geometric model based on the three-dimensional point cloud model and the above semantic segmentation result.

[0072] Specifically, a three-dimensional geometric model is a digital three-dimensional object representation constructed through mathematics and computer technology, used to describe the shape, structure, and properties of an object in three-dimensional space. First, based on the semantic segmentation results, the boundaries and features of each object in the house can be determined. Then, using the boundaries and features of the objects, a three-dimensional geometric model containing detailed geometric information can be generated. This model not only contains the position information of the objects but also their shapes and sizes, as shown in Figure 4B shown.

[0073] By combining mapping algorithms and semantic segmentation techniques, a high-precision three-dimensional model of the entire house can be constructed, providing an accurate data basis for subsequent user interactions.

[0074] In some alternative embodiments, before displaying the three-dimensional model of the house where the user is located, the following steps can also be performed to bind Internet of Things (IoT) devices in the three-dimensional model:

[0075] Step S31: Obtain the application-side data of each IoT device.

[0076] The application-side data may include, but is not limited to, the room information, appearance diagrams, control interfaces, device information, etc. of the IoT devices. Taking a smart table lamp as an example, the above-mentioned room information can be used to indicate the room where the IoT device is located, which can be manually edited and confirmed by the user. For example, it can be a bedroom, a living room, etc. The above-mentioned appearance diagrams may include the appearance diagrams of the IoT device in different states. For example, it may include the appearance diagram of the smart table lamp in the on state and the appearance diagram of the smart table lamp in the off state. The control interfaces may include, but are not limited to, switch interfaces, color temperature adjustment interfaces, brightness adjustment interfaces, mode setting interfaces, etc. The above-mentioned device information may include, but is not limited to, timing task information, environmental information, etc.

[0077] Step S32: Based on the signal strength and room information of each IoT device, determine the mapping positions of all IoT devices in the house in the three-dimensional model.

[0078] Step S33: Establish the correspondence between the mapping positions of each IoT device and the application-side data.

[0079] Specifically, for each IoT device, the unique signal feature of the IoT device can be obtained first. For example, the MAC address, signal strength curve, etc. Then, the unique signal feature is associated with the mapping position of the IoT device in the three-dimensional model. Since the unique signal feature can uniquely identify the IoT device, the association between the IoT device in the three-dimensional model and its application-side data can be realized.

[0080] By associating the mapped location of each Internet of Things device with its application - side data, it is ensured that the device location displayed in the 3D model can accurately reflect its actual location and status, thereby improving the accuracy of controlling Internet of Things devices. In addition, by establishing the above - mentioned correspondence, the status data of Internet of Things devices can be synchronized to the 3D model in real time, enabling the extended reality device to store or display the latest status information in real time.

[0081] In some alternative embodiments, the following steps may also be performed:

[0082] Step S41: Obtain the status information of each Internet of Things device in real time.

[0083] Among them, the status information is used to indicate the current operating status of the Internet of Things device. Exemplarily, it may include but is not limited to at least one of the following: temperature, humidity, energy consumption, switch status, sensor data, etc. Sensor data may include but is not limited to smoke concentration, gas leakage detection results, etc.

[0084] Step S42: Detect whether each Internet of Things device is abnormal based on the status information.

[0085] Specifically, a preset rule or machine - learning algorithm can be used to analyze the collected status information to determine whether each Internet of Things device is in an abnormal state. For example, temperature abnormality, energy - consumption abnormality, sensor - data abnormality, etc. In addition, the operating status of the device can also be dynamically evaluated based on historical data and real - time data to identify abnormal situations.

[0086] Step S43: When there are abnormal Internet of Things devices, display risk - warning information of the abnormal Internet of Things devices.

[0087] The risk - warning information can be used to prompt the user that there are abnormalities in the Internet of Things device. The risk - warning information may include visual information, which may include warning information related to the abnormal Internet of Things device. For example, a red flashing icon, detailed data of the abnormal Internet of Things device, etc., which are not specifically limited here. In addition, the risk - warning information may also include audio information, such as a prompt tone, etc.

[0088] By real - time monitoring the operating status of the whole - house Internet of Things devices, abnormal situations can be detected in time and the user can be reminded, so that the user can respond quickly and take measures. This precise risk management helps to prevent equipment failures and potential safety hazards, and improves the safety of the house where the user is located.

[0089] It should be noted that for the control method provided in the embodiments of the present application, the execution subject may be a control device. In the embodiments of the present application, taking the control device as the execution subject of the control method as an example, the control device provided in the embodiments of the present application is described.

[0090] Such as Figure 5As shown in the figure, the control device 500 of this embodiment includes: a receiving unit 501, configured to receive a first input from a user when a three-dimensional model of the house where the user is located is displayed; a first determining unit 502, configured to determine a target wall in the three-dimensional model in response to the first input, and display at least one Internet of Things device blocked by the target wall in the three-dimensional model; a second determining unit 503, configured to determine a target Internet of Things device from the at least one Internet of Things device in response to receiving a second input from the user, and send a control instruction to the target Internet of Things device.

[0091] In some optional implementation manners of this embodiment, the three-dimensional model is a three-dimensional geometric model; the device further includes a generating unit, configured to: perform a full-house scan on the house where the user is located through a sensor in the extended reality device to obtain sensor data; generate a three-dimensional point cloud model based on the sensor data; perform semantic segmentation on the three-dimensional point cloud model to obtain a semantic segmentation result, where the semantic segmentation result includes walls, Internet of Things devices, and other indoor facilities; generate the three-dimensional geometric model based on the three-dimensional point cloud model and the semantic segmentation result. By combining mapping algorithms and semantic segmentation technologies, a high-precision three-dimensional model of the whole house can be constructed, providing an accurate data basis for subsequent user interactions.

[0092] In some optional implementation manners of this embodiment, the device further includes an establishing unit, configured to: obtain application-side data of each Internet of Things device, where the application-side data includes room information indicating the room where the Internet of Things device is located; determine the mapping positions of each Internet of Things device in the three-dimensional model based on the signal strength and room information of each Internet of Things device; establish a correspondence between the mapping positions of each Internet of Things device and the application-side data. By associating the mapping position of each Internet of Things device with its application-side data, it is ensured that the device positions displayed in the three-dimensional model can accurately reflect their actual positions and states, thereby improving the accuracy of controlling Internet of Things devices. In addition, by establishing the correspondence, the status data of Internet of Things devices can be synchronized to the three-dimensional model in real time, enabling the extended reality device to store or display the latest status information in real time.

[0093] In some alternative implementation manners of this embodiment, the first determination unit 502 is further configured to: perform weakening display on the target wall to perspectively display at least one Internet of Things device blocked by the target wall in the three-dimensional model; wherein, the weakening display includes at least one of the following: increasing the transparency of the target wall, hiding the target wall, and only displaying the contour line of the target wall; the perspective display includes at least one of the following: displaying the contour line of at least one Internet of Things device blocked by the target wall in the area where the target wall is located, and displaying the visual image of at least one Internet of Things device blocked by the target wall in the area where the target wall is located. By weakening the display of the target wall and perspectively displaying the Internet of Things devices blocked by it, the user can intuitively see the Internet of Things devices blocked by the target wall, enhancing the spatial perception ability.

[0094] In some alternative implementation manners of this embodiment, the second determination unit 503 is further configured to: receive a third input from the user on the target Internet of Things device in the three-dimensional model; in response to the third input, display the virtual control panel of the target Internet of Things device; receive a fourth input from the user on the virtual control panel; in response to the fourth input, generate a control instruction and send the control instruction to the target Internet of Things device. By displaying the virtual control panel, the user can intuitively perform more refined operations on the Internet of Things device without memorizing complex operation processes, improving the convenience and accuracy of controlling the Internet of Things device.

[0095] In some alternative implementation manners of this embodiment, the second determination unit 503 is further configured to: send a control instruction for the target Internet of Things device to the home gateway through the Message Queuing Telemetry Transport (MQTT) protocol; through the home gateway, forward the control instruction to the target Internet of Things device. By transmitting the control instruction through the Message Queuing Telemetry Transport (MQTT) protocol and through the centralized management of the home gateway, the connection of the Internet of Things device and the sending of the control instruction can be facilitated and quick, improving the stability and security of the network.

[0096] In some alternative implementation manners of this embodiment, the device further includes a prompt unit, configured to: obtain the status information of each Internet of Things device in real time; based on the status information, detect whether each Internet of Things device is abnormal; in the case of an abnormal Internet of Things device, display the risk prompt information of the abnormal Internet of Things device. By monitoring the running status of the whole-house Internet of Things devices in real time, abnormal situations can be detected in time and the user can be reminded, so that the user can respond quickly and take measures. This accurate risk management helps prevent equipment failures and potential safety hazards, improving the safety of the house where the user is located.

[0097] The device provided in the above embodiments of the present application, when displaying the three-dimensional model of the house where the user is located, receives the first input of the user; in response to the first input, determines the target wall in the three-dimensional model, and displays at least one Internet of Things device blocked by the target wall in the three-dimensional model; when receiving the second input of the user, determines the target Internet of Things device among the at least one Internet of Things device, and sends a control instruction to the target Internet of Things device. By visualizing the Internet of Things devices blocked by the target wall in the three-dimensional model of the house where the user is located through the extended reality device, the user can directly complete the cross-room visual control of the target Internet of Things device in the current room through the extended reality device without moving to the room where the target Internet of Things device is located, reducing the complexity of the user operation and improving the control efficiency of the Internet of Things devices blocked by the wall.

[0098] The control device in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an Augmented Reality (AR) / Virtual Reality (VR) device, a robot, a wearable device, an Ultra-Mobile Personal Computer (UMPC), a netbook or a Personal Digital Assistant (PDA), etc., and can also be a server, a Network Attached Storage (NAS), a Personal Computer (PC), a Television (TV), a teller machine or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0099] The control device in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0100] The control device provided in the embodiments of the present application can implement Figure 1 each process implemented by the method embodiments, and for the sake of brevity, it will not be repeated here.

[0101] Optionally, as Figure 6As shown in the figure, an embodiment of the present application further provides an electronic device 600, which includes a processor 601 and a memory 602. A program or instruction that can run on the processor 601 is stored on the memory 602. When the program or instruction is executed by the processor 601, it implements each step of the above control method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0102] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0103] Figure 7 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application.

[0104] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710, etc.

[0105] Those skilled in the art can understand that the electronic device 700 may further include a power supply (such as a battery) for supplying power to each component. The power supply can be logically connected to the processor 710 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. Figure 7 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0106] Among them, the processor 710 is used to receive a first input from the user through the user input unit 707 when a three-dimensional model of the user's house is displayed through the display unit 706; in response to the first input, determine a target wall in the three-dimensional model, and display at least one Internet of Things device blocked by the target wall in the three-dimensional model; when a second input from the user is received through the user input unit 707, determine a target Internet of Things device among the at least one Internet of Things device, and send a control instruction to the target Internet of Things device.

[0107] By visualizing the Internet of Things devices blocked by the target wall in the three-dimensional model of the user's house through an extended reality device, the user can directly complete the cross-room visual control of the target Internet of Things device in the current room through the extended reality device without moving to the room where the target Internet of Things device is located, reducing the complexity of user operations and improving the control efficiency of the Internet of Things devices blocked by the wall.

[0108] In some alternative implementation manners of this embodiment, the three-dimensional model is a three-dimensional geometric model; the processor 710 is further configured to perform a full-house scan on the user's house through the sensors in the extended reality device to obtain sensor data; generate a three-dimensional point cloud model based on the sensor data; perform semantic segmentation on the three-dimensional point cloud model to obtain a semantic segmentation result, where the semantic segmentation result includes walls, Internet of Things devices, and other indoor facilities; generate the three-dimensional geometric model based on the three-dimensional point cloud model and the semantic segmentation result. By combining the mapping algorithm and semantic segmentation technology, a high-precision three-dimensional model of the whole house can be constructed, providing an accurate data basis for subsequent user interaction.

[0109] In some alternative implementation manners of this embodiment, the processor 710 is further configured to obtain the application-side data of each Internet of Things device, where the application-side data includes room information indicating the room where the Internet of Things device is located; determine the mapping positions of the Internet of Things devices in the three-dimensional model based on the signal strengths and room information of the Internet of Things devices; establish a correspondence between the mapping positions of the Internet of Things devices and the application-side data. By associating the mapping position of each Internet of Things device with its application-side data, it is ensured that the device position displayed in the three-dimensional model can accurately reflect its actual position and status, thereby improving the accuracy of controlling the Internet of Things devices. In addition, by establishing the correspondence, the status data of the Internet of Things devices can be synchronized to the three-dimensional model in real time, enabling the extended reality device to store or display the latest status information in real time.

[0110] In some alternative implementation manners of this embodiment, the processor 710 is further configured to perform a weakening display on the target wall through the display unit 706 to perspectively display at least one Internet of Things device blocked by the target wall in the three-dimensional model; where the weakening display includes at least one of the following: increasing the transparency of the target wall, hiding the target wall, and only displaying the contour line of the target wall; the perspective display includes at least one of the following: displaying the contour line of at least one Internet of Things device blocked by the target wall in the area where the target wall is located, and displaying the visual image of at least one Internet of Things device blocked by the target wall in the area where the target wall is located. By weakening the display of the target wall and perspectively displaying the Internet of Things devices blocked by it, the user can intuitively see the Internet of Things devices blocked by the target wall, enhancing the spatial perception ability.

[0111] In some alternative implementation manners of this embodiment, the processor 710 is further configured to receive, through the user input unit 707, a third input of the user for the target Internet of Things device in the 3D model; in response to the third input, display, through the display unit 706, a virtual control panel of the target Internet of Things device; receive, through the user input unit 707, a fourth input of the user for the virtual control panel; in response to the fourth input, generate a control instruction, and send the control instruction to the target Internet of Things device. By displaying the virtual control panel, the user can intuitively perform more refined operations on the Internet of Things device without memorizing complex operation processes, improving the convenience and accuracy of controlling the Internet of Things device.

[0112] In some alternative implementation manners of this embodiment, the processor 710 is further configured to send, through the Message Queuing Telemetry Transport protocol, a control instruction for the target Internet of Things device to the home gateway; and forward, through the home gateway, the control instruction to the target Internet of Things device. By transmitting the control instruction through the Message Queuing Telemetry Transport protocol and through the centralized management of the home gateway, the connection of the Internet of Things device and the sending of the control instruction can be facilitated and accelerated, improving the stability and security of the network.

[0113] In some alternative implementation manners of this embodiment, the processor 710 is further configured to obtain the status information of each Internet of Things device in real time; based on the status information, detect whether each Internet of Things device is abnormal; and in the case of an abnormal Internet of Things device, display, through the display unit 706, risk prompt information of the abnormal Internet of Things device. By monitoring the running status of the whole-house Internet of Things devices in real time, abnormal situations can be detected in a timely manner and the user can be reminded, so that the user can respond quickly and take measures. This precise risk management helps prevent equipment failures and potential safety hazards, improving the safety of the house where the user is located.

[0114] It should be understood that, in the embodiment of the present application, the input unit 704 may include a Graphics Processing Unit (GPU) 7041 and a microphone 7042. The graphics processor 7041 processes the image data of static pictures or videos obtained by an image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. The other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.

[0115] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 709 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0116] The processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 710 either.

[0117] The embodiments of the present application also provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above control method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0118] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disks, or optical discs, etc.

[0119] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above control method embodiment and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0120] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0121] The embodiments of the present application provide a computer program product. The program product is stored in a storage medium and is executed by at least one processor to implement each process of the above control method embodiment and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0122] It should be noted that in this article, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article, or device. Without more limitations, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0123] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0124] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection scope of the present application.

Claims

1. A control method, characterized in that, Applied to an extended reality device, the method includes: When displaying a three-dimensional model of the house where the user is located, receiving a first input from the user; In response to the first input, determining a target wall in the three-dimensional model and displaying at least one Internet of Things device blocked by the target wall in the three-dimensional model; When receiving a second input from the user, determining a target Internet of Things device among the at least one Internet of Things device and sending a control instruction to the target Internet of Things device.

2. The method according to claim 1, wherein The three-dimensional model is a three-dimensional geometric model; before displaying the three-dimensional model of the house where the user is located, the method further includes: Globally scanning the house where the user is located through a sensor in the extended reality device to obtain sensor data; Generating a three-dimensional point cloud model based on the sensor data; Performing semantic segmentation on the three-dimensional point cloud model to obtain a semantic segmentation result, where the semantic segmentation result includes walls, Internet of Things devices, and other indoor facilities; Generating the three-dimensional geometric model based on the three-dimensional point cloud model and the semantic segmentation result.

3. The method according to claim 1, wherein Before displaying the three-dimensional model of the house where the user is located, the method further includes: Obtaining application-side data of each Internet of Things device, where the application-side data includes room information indicating the room where the Internet of Things device is located; Determining the mapping positions of the Internet of Things devices in the three-dimensional model based on the signal strength of each Internet of Things device and the room information; Establishing a correspondence between the mapping positions of the Internet of Things devices and the application-side data.

4. The method according to claim 1, characterized in that, The displaying of at least one Internet of Things device blocked by the target wall in the three-dimensional model includes: Weakening the display of the target wall to transparently display at least one Internet of Things device blocked by the target wall in the three-dimensional model; Wherein, the weakening display includes at least one of the following: increasing the transparency of the target wall, hiding the target wall, and only displaying the contour line of the target wall; The transparent display includes at least one of the following: displaying the contour line of at least one Internet of Things device blocked by the target wall in the area where the target wall is located, and displaying the visual image of at least one Internet of Things device blocked by the target wall in the area where the target wall is located.

5. The method according to claim 1, characterized in that, The sending of the control instruction to the target Internet of Things device includes: Receiving a third input from the user on the target Internet of Things device in the three-dimensional model; In response to the third input, displaying a virtual control panel of the target Internet of Things device; Receiving a fourth input from the user on the virtual control panel; In response to the fourth input, generating a control instruction and sending the control instruction to the target Internet of Things device.

6. A control device, characterized in that, Applied to an extended reality device, the apparatus includes: A receiving unit, configured to receive a first input from the user when displaying a three-dimensional model of the house where the user is located; A first determination unit, configured to determine a target wall in the three-dimensional model in response to the first input and display at least one Internet of Things device blocked by the target wall in the three-dimensional model; A second determination unit, configured to determine a target Internet of Things device among the at least one Internet of Things device when receiving a second input from a user, and send a control instruction to the target Internet of Things device.

7. The device according to claim 6, characterized in that, The three-dimensional model is a three-dimensional geometric model; the device further includes a generation unit, configured to: Perform a whole-house scan of the house where the user is located through a sensor in the extended reality device to obtain sensor data; Generate a three-dimensional point cloud model based on the sensor data; Perform semantic segmentation on the three-dimensional point cloud model to obtain a semantic segmentation result, where the semantic segmentation result includes walls, Internet of Things devices, and other indoor facilities; Generate the three-dimensional geometric model based on the three-dimensional point cloud model and the semantic segmentation result.

8. The device according to claim 6, characterized in that, The device further includes an establishment unit, configured to: Obtain application-side data of each Internet of Things device, where the application-side data includes room information indicating the room where the Internet of Things device is located; Determine the mapping positions of the Internet of Things devices in the three-dimensional model based on the signal strength of each Internet of Things device and the room information; Establish a correspondence between the mapping positions of the Internet of Things devices and the application-side data.

9. The device according to claim 6, characterized in that, The first determination unit is further configured to: Weakly display the target wall to perspectively display at least one Internet of Things device blocked by the target wall in the three-dimensional model; Wherein, the weak display includes at least one of the following: increasing the transparency of the target wall, hiding the target wall, and only displaying the contour line of the target wall; The perspective display includes at least one of the following: displaying the contour line of at least one Internet of Things device blocked by the target wall in the area where the target wall is located, and displaying the visual image of at least one Internet of Things device blocked by the target wall in the area where the target wall is located.

10. The device according to claim 6, characterized in that, The second determination unit is further configured to: Receive a third input from the user to the target Internet of Things device in the three-dimensional model; In response to the third input, display a virtual control panel of the target Internet of Things device; Receive a fourth input from the user to the virtual control panel; In response to the fourth input, generate a control instruction and send the control instruction to the target Internet of Things device.

11. An electronic device, characterized in that, Includes a processor and a memory, the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the control method according to any one of claims 1-5 are implemented.

12. A readable storage medium, characterized in that, The program or instruction is stored on the readable storage medium, and when the program or instruction is executed by the processor, the steps of the control method according to any one of claims 1-5 are implemented.

Images