Equipment control method, control equipment, system, readable storage medium and chip

Abstract

The invention provides an equipment control method, control equipment, a system, a readable storage medium and a chip, and relates to the technical field of smart home. The method is applied to a control device and comprises the steps that a display interface comprising spatial information is displayed, and the spatial information comprises at least one area range; determining a first area range from the at least one area range according to the first control operation; according to the second control operation and the first area range, a target area range is determined from the at least one area range, and the first area range is the same as or different from the target area range; and controlling the corresponding electronic equipment in the target area range according to the second control operation. Through the technical scheme provided by the invention, a user can conveniently and quickly carry out intelligent control on the electronic equipment in the target area in a visual and natural manner, the operation process is simplified, and the use experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a device control method, a control device, a system, a readable storage medium and a chip. Background Art

[0002] In the field of smart home technology, a smart home system includes a control device and various smart home devices that are connected to the control device through the Internet of Things technology. Users can remotely control and intelligently manage various smart home devices through the device management application in the control device.

[0003] At present, in order to facilitate users to control multiple smart home devices at the same time, device management applications usually control and manage smart home devices based on scenario-based centralized control, that is, multiple smart home devices are set in the same scene mode, and users only need to trigger the scene to control multiple categories or cross-category devices without having to control individual devices one by one. However, in this way, although users can operate multiple devices in one step, users need to define the scenes in advance and add corresponding smart home devices to each scene, which leads to high initial operating costs for users. And when users need to control and adjust one or more smart home devices, they need to go through tedious search operations in the operation interface of the scenario-based centralized control to find the target device, resulting in insufficient intelligence in the management and control of the device and poor user experience. Summary of the invention

[0004] The present application provides a device control method, a control device, a system, a readable storage medium and a chip, which solve the problem in the prior art that the management and control of electronic devices by control devices are not intelligent enough, resulting in high operating costs and cumbersome operations when users control and search for target devices through control devices.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a device control method is provided, which is applied to control a device, the method comprising: displaying a display interface including spatial information, wherein the spatial information includes at least one area range; determining a first area range from at least one area range according to a first control operation; determining a target area range from at least one area range according to a second control operation and the first area range, wherein the first area range is the same as or different from the target area range; and controlling a corresponding electronic device within the target area range according to the second control operation.

[0007] The device control method provided in the first aspect of the present application is that the control device determines the first area range according to the user's first control operation, and then determines the target area range according to the determined first area range and the second control operation. In other words, the control device can further determine through the second control operation whether the target area range that the user wants to control is the first area range indicated by the first control operation, or other area ranges that are different from or associated with the first area range. In this way, the target area range that the user needs to control and the electronic equipment within the target area range can be more accurately determined, thereby achieving precise control.

[0008] In addition, the control operation in this method is based on the interaction of the spatial information displayed in the display interface. The user can quickly and easily perform intelligent control on one or more devices within the target area in an intuitive and natural way, which simplifies the operation process and improves the interactive efficiency of selecting and controlling the target device in the smart home scenario, thereby improving the user experience.

[0009] In some embodiments, according to the first control operation, a first area range is determined from at least one area range, including: according to the first control operation, determining the projection coordinates of the first control operation projected into the spatial information; when there is a corresponding area range for the projection coordinates, determining the area range corresponding to the projection coordinates as the first area range; or, when there is no corresponding area range for the projection coordinates, determining the area range adjacent to the projection coordinates as the first area range.

[0010] As mentioned above, the spatial information includes at least one area range, and each area range corresponds to its own projection coordinates, that is, there is a corresponding relationship between the area range and the projection coordinates, and the control device can determine the first area range according to the projection coordinates projected into the spatial information by the first control operation.

[0011] When the control device receives a first control operation, if the projection coordinates projected by the first control operation in the spatial information have a corresponding area range, the control device determines the area range corresponding to the projection coordinates as the first area range; if the projection coordinates projected by the first control operation in the spatial information do not have a corresponding area range, the area range adjacent to the projection coordinates is determined as the first area range.

[0012] In some embodiments, based on the second control operation and the first area range, a target area range is determined from at least one area range, including: performing semantic analysis on the control instruction in the second control operation; when the control instruction includes direction reference information, determining the target area range based on the direction reference information and the first area range.

[0013] Among them, the orientation reference information refers to the relevant content that describes the orientation information, including but not limited to: "here", "this room", "other places", "other rooms", "around", "next to" and "nearby". When the control instruction includes the orientation reference information and does not include the orientation reference information, it may express completely different control processes. The control instruction contained in the second control operation received by the control device may contain the orientation reference information or may not contain the orientation reference information. Based on this, after receiving the second control operation, the control device needs to analyze the control instruction in the second control operation to determine whether the control instruction contains the orientation reference information, and then determine the target area range according to the specific analysis results and in combination with the first area range.

[0014] In this embodiment, when the control instruction in the second control operation includes direction reference information, such as "other rooms", other area ranges outside the first area range are determined as target area ranges. At this time, the first area range is different from the target area range, and the target area range does not include the first area range.

[0015] In some embodiments, the method further includes: when the control instruction does not include the orientation reference information, determining the first area range as the target area range.

[0016] In this embodiment, when the control instruction does not include the orientation reference information, the control device assumes by default that the target area range to be controlled is the first area range initially determined. At this time, the first area range is the same as the target area range.

[0017] In some embodiments, the method further includes: when the control instruction does not include the orientation reference information, determining the upper level area range of the first area range as the target area range.

[0018] This embodiment is another implementation method when the control instruction does not include the azimuth reference information, which can be applied to the application scenario where the control instruction contains "implicit" azimuth reference information. In this case, the control device determines the upper level area range of the first area range as the target area range to meet the specific usage needs of the user. At this time, the first area range is different from the target area range, and the target area range is a larger range that includes the first area range.

[0019] In some embodiments, the spatial information includes: a floor plan of the apartment structure of the home environment; or a three-dimensional model diagram of the apartment structure of the home environment; or a three-dimensional model diagram of the soft furnishings layout information of the home environment.

[0020] In other embodiments, the spatial information may also be a topographic distribution map of a certain area of ​​a city.

[0021] In some embodiments, the area range includes: a point area range corresponding to the projection coordinates of the first control operation projected into the spatial information; or, a two-dimensional area range or a three-dimensional area range corresponding to the projection coordinates of the first control operation projected into the spatial information; or, an area range indicated by area identification information corresponding to the projection coordinates of the first control operation projected into the spatial information.

[0022] In this embodiment, the point area range can be any point in the two-dimensional spatial information or the three-dimensional spatial information. The two-dimensional area range can be a preset plane range around any point in the spatial information, wherein the point can be the center point of the preset plane range, for example, the center of a circular area range; or it can be any point within the preset plane range, for example, the vertex of a rectangular area range. The three-dimensional area range can be a preset spatial range around any point in the spatial information, wherein the point can be the center point of the preset spatial range, for example, the center of a sphere; or it can be any other point within the preset spatial range, such as the vertex of a cone. The area range indicated by the area identification information can be the preset area range corresponding to the name of a spatial area or the name of a direction reference object, for example, the area range indicated by the area ID "living room sofa" is the preset area range corresponding to "living room sofa"; the area range indicated by the area ID "bedroom" is the preset area range corresponding to "bedroom".

[0023] In some embodiments, according to a first control operation, a first area range is determined from at least one area range, including: receiving a user's selection operation on N area ranges in the spatial information; receiving a user's cancellation operation on M area ranges among the N area ranges; and determining the NM area ranges as the first area range, where N≥M≥1.

[0024] In this embodiment, the control device may determine one or more first area ranges according to user operations, so as to facilitate the subsequent control of multiple area ranges.

[0025] In some embodiments, after determining a first area range from at least one area range according to a first control operation and before receiving a second control operation, the method also includes: extracting candidate control instructions related to the first area range; displaying an element control of the candidate control instructions on a display interface, the element control being used to instruct a user to input a control instruction according to content corresponding to the element control.

[0026] In this embodiment, the user can trigger the corresponding control by controlling the operation options corresponding to the element control (for example, clicking on / off, or adjusting a slider), or speaking the corresponding text in these element controls.

[0027] In some embodiments, the second control operation includes a voice control operation or a touch control operation.

[0028] In some embodiments, displaying a display interface including spatial information includes: determining a position orientation of a control device in a physical space; and controlling a display direction of the spatial information in a first direction of the control device to be consistent with the position orientation of the control device.

[0029] In some embodiments, the method further includes: when the position orientation of the control device in the physical space changes, adjusting the display direction of the spatial information in the display interface of the control device according to the changed position orientation.

[0030] In this way, users can more intuitively correspond the spatial information in the display interface, such as a three-dimensional model diagram of the apartment structure of a home environment or a floor plan of the apartment structure of a home environment, to the real physical space, and the view display direction of the spatial information can be adaptively adjusted according to the position and orientation of the control device in the physical space, thereby improving the user experience.

[0031] In a second aspect, a device control device is provided, which is applied to control a device, and the device includes: a display module, which is used to display a display interface including spatial information, wherein the spatial information includes at least one area range; a first determination module, which is used to determine a first area range from at least one area range according to a first control operation; a second determination module, which is used to determine a target area range from at least one area range according to a second control operation and the first area range, wherein the first area range is the same as or different from the target area range; and a control module, which is used to control a corresponding electronic device within the target area range according to the second control operation.

[0032] In a third aspect, a control device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method as shown in the first aspect above when executing the computer program.

[0033] In a fourth aspect, a device control system is provided, including a control device and a plurality of electronic devices connected to the control device, wherein the control device is configured to execute the method as shown in the first aspect above.

[0034] According to a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method shown in the first aspect is implemented.

[0035] In a sixth aspect, a chip is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method shown in the first aspect is implemented.

[0036] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of a smart home system provided for one embodiment of the present application;

[0038] Figure 2A A schematic diagram of the structure of a smart home system provided by another embodiment of the present application;

[0039] Figure 2B A schematic diagram of the structure of a smart home system provided in yet another embodiment of the present application;

[0040] Figure 3 A schematic diagram of the structure of a control device provided in an embodiment of the present application;

[0041] Figure 4A A schematic diagram of a scenario in which a control device provided in an embodiment of the present application controls a single device in a smart home device;

[0042] Figure 4B A schematic diagram of a scenario for centralized control of smart home devices by a control device according to an embodiment of the present application;

[0043] Figure 5 A schematic diagram of an interactive process of a device control method provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of the structure of spatial information provided by an embodiment of the present application;

[0045] Fig. 7A A schematic diagram of the area scope provided for an embodiment of the present application;

[0046] Figure 7B A schematic diagram of a region range provided for another embodiment of the present application;

[0047] Figure 7C A schematic diagram of a region range provided for yet another embodiment of the present application;

[0048] Fig. 8A A schematic diagram of a scene for displaying spatial information according to a first orientation is provided for an embodiment of the present application;

[0049] Figure 8B A schematic diagram of a scene for displaying spatial information according to a second orientation is provided for an embodiment of the present application;

[0050] Fig.9A A schematic diagram of the corresponding relationship between the projection coordinates and the point-shaped area range provided in an embodiment of the present application;

[0051] Fig. 9B A schematic diagram of the corresponding relationship between the projection coordinates and the two-dimensional / three-dimensional area range provided in an embodiment of the present application;

[0052] Fig. 9C A schematic diagram of the corresponding relationship between the projection coordinates and the area range corresponding to the area ID provided in an embodiment of the present application;

[0053] Fig. 10A A schematic diagram of a scenario in which a control device according to an embodiment of the present application determines a first area range according to a first control operation;

[0054] Fig. 10B A schematic diagram of a scenario in which a control device according to another embodiment of the present application determines a first area range according to a first control operation;

[0055] Fig. 10C A schematic diagram of a scenario in which a control device according to another embodiment of the present application determines a first area range according to a first control operation;

[0056] Fig. 10D A schematic diagram of a scenario in which a control device according to another embodiment of the present application determines a first area range according to a first control operation;

[0057] Fig.11A A schematic diagram of a control device provided in an embodiment of the present application displaying a control interface in response to a first control operation;

[0058] Fig. 11B A schematic diagram of a control device provided by another embodiment of the present application displaying a control interface in response to a first control operation;

[0059] Fig. 12A A schematic diagram of the correspondence between electronic device types and control instruction sets provided in an embodiment of the present application;

[0060] Fig. 12B A schematic diagram of the correspondence between electronic devices and spatial information provided in an embodiment of the present application;

[0061] Fig.13A A schematic diagram of a scenario in which a control device determines a target area range provided in an embodiment of the present application;

[0062] Fig. 13B A schematic diagram of a scenario in which a control device determines a target area range provided by another embodiment of the present application;

[0063] Fig. 13C A schematic diagram of a scenario in which a control device determines a target area range according to another embodiment of the present application;

[0064] Fig.13DA schematic diagram of the hierarchical relationship of regional scopes provided in an embodiment of the present application;

[0065] Fig.14A A schematic diagram of a scenario in which a control device controls electronic devices within a target area provided by an embodiment of the present application;

[0066] Fig. 14B A schematic diagram of a scenario in which a control device controls electronic devices within a target area provided by another embodiment of the present application;

[0067] Fig.15 A schematic diagram of the structure of the device control device provided in the embodiment of the present application;

[0068] Fig.16 A schematic diagram of the structure of the chip provided in the embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solution provided in the embodiments of the present application is described below with reference to the accompanying drawings.

[0070] It should be understood that in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0071] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0072] The smart home system is a residential remote control system that uses advanced computer technology, network communication technology, integrated wiring technology, etc. to organically combine various subsystems related to home life. Through remote control and overall management, it makes home life more comfortable, safe and convenient, while saving energy and resources and improving the quality of life.

[0073] Figure 1 This is a schematic diagram of the structure of the smart home system provided in the embodiment of the present application. Figure 1As shown, the system includes a control device and multiple smart home devices (also referred to as electronic devices). The control device is wirelessly connected to each smart home device, and each smart home device may be connected or not connected to each other. In addition, the wireless connection method between different devices may be wireless fidelity (Wi-Fi), Bluetooth (BT), etc., which is not specifically limited in this embodiment.

[0074] Smart home devices are objects managed and controlled by control devices, and they can perform corresponding control operations according to the control instructions of the control devices. For example, smart home devices can be TVs, speakers, headphones, air purifiers, refrigerators, air conditioners, sweeping robots, electric lights, cameras, routers, power strips, desk lamps, routers, humidifiers, sockets, smart door locks, water purifiers, treadmills, etc. Depending on the type of device, the control operations performed by smart home devices according to control instructions are different. For example, the control operations performed by electric lights can be turning on / off lights, adjusting the brightness of lights, etc., the control operations performed by speakers can be playing / pausing music, and the control operations performed by air conditioners can be turning on / off, adjusting the temperature, etc. The embodiments of the present application do not limit their specific types.

[0075] The control device plays a management and control role in the smart home system, and can intelligently manage and control each smart home device through device management applications (such as smart life applications). In this embodiment, the control device can be a portable terminal device with a display screen such as a mobile phone or a tablet computer, or a home management device with a display screen such as a central control screen, a smart screen, or a central control panel. It can be understood that when the type of control device is different, the connection method between it and each smart home device may be different.

[0076] In some embodiments, see Figure 2A As shown, when the control device is a terminal device or a home management device, the control device can directly connect to and control various smart home devices.

[0077] In other embodiments, see Figure 2B As shown, when the control device is a terminal device such as a mobile phone or a tablet computer, the control device can be indirectly connected to each smart home device through a central device. Based on this, when the control device controls each smart home device, it needs to transfer signaling or data through the central device. For example, when the control device controls the air conditioner to turn on, it needs to first send the power-on command to the central device, and then the central device forwards it to the air conditioner. Among them, the central device can be the home management device mentioned in the above embodiment, or it can be a router, home gateway and other devices, which is not limited in this embodiment.

[0078] Figure 3 3 is a schematic diagram of the structure of the control device provided in the embodiment of the present application. The control device includes a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a sensor module 380, a button 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a subscriber identification module (SIM) card interface 395, etc.

[0079] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the control device. In other embodiments of the present application, the control device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0080] The processor 310 may include one or more processing units, for example: the processor 310 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. Among them, the controller can be the nerve center and command center of the control device. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0081] The processor 310 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. The memory may store instructions or data that the processor 310 has just used or cyclically used. If the processor 310 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0082] The charging management module 340 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 can receive charging input from a wired charger through the USB interface 330. In some wireless charging embodiments, the charging management module 340 can receive wireless charging input through a wireless charging coil of a control device. While the charging management module 340 is charging the battery 342, it can also power the control device through the power management module 341.

[0083] The power management module 341 is used to connect the battery 342, the charging management module 340 and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, and supplies power to the processor 310, the internal memory 321, the external memory, the display screen 394, the camera 393, and the wireless communication module 360. The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc.

[0084] The wireless communication function of the control device can be implemented through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, modem processor and baseband processor.

[0085] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the control device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0086] The mobile communication module 350 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the control device. The mobile communication module 350 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0087] In some embodiments, at least some functional modules of the mobile communication module 350 may be disposed in the processor 310. In some embodiments, at least some functional modules of the mobile communication module 350 and at least some modules of the processor 310 may be disposed in the same device.

[0088] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio playback device (not limited to a speaker 370A, a receiver 370B, etc.), or displays an image or video through a display screen 394. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 310 and be set in the same device as the mobile communication module 350 or other functional modules.

[0089] The wireless communication module 360 ​​can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the control device. The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 310. The wireless communication module 360 ​​can also receive the signal to be sent from the processor 310, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0090] The camera 393 is used to capture static images or videos. In some embodiments, the control device may include 1 or N cameras 393, where N is a positive integer greater than 1.

[0091] The display screen 394 is used to display images, videos, etc., such as various types of device management interfaces in the embodiments of the present application. The display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the control device may include 1 or N display screens 394, where N is a positive integer greater than 1.

[0092] The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the control device. The external memory card communicates with the processor 310 through the external memory interface 320 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0093] The internal memory 321 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 310 executes various functional applications and data processing of the control device by running the instructions stored in the internal memory 321. The internal memory 321 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the control device (such as audio data, a phone book, etc.).

[0094] In addition, the internal memory 321 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0095] The control device can implement audio functions through the audio module 370, the speaker 370A, the receiver 370B, the microphone 370C, the headphone interface 370D, and the application processor.

[0096] The audio module 370 is used to convert digital audio signals into analog audio signals for output, and is also used to convert analog audio inputs into digital audio signals. The audio module 370 can also be used to encode and decode audio signals. In some embodiments, the audio module 370 can be arranged in the processor 310, or some functional modules of the audio module 370 can be arranged in the processor 310.

[0097] The speaker 370A, also called a "speaker", is used to convert audio electrical signals into sound signals. The control device can listen to music or listen to hands-free calls through the speaker 370A. For example, the speaker can play the comparison analysis results provided in the embodiment of the present application.

[0098] The receiver 370B, also called a "handset", is used to convert audio electrical signals into sound signals. When the control device answers a call or voice message, the voice can be heard by placing the receiver 370B close to the human ear.

[0099] Microphone 370C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting their mouth close to the microphone 370C to input the sound signal into the microphone 370C. The control device can be provided with at least one microphone 370C. In other embodiments, the control device can be provided with two microphones 370C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the control device can also be provided with three, four or more microphones 370C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0100] In some embodiments, the control device may receive ultrasonic signals sent by other electronic devices through the microphone 370C, and identify the frequency and reception strength of the ultrasonic signals through the processor 310 .

[0101] The earphone interface 370D is used to connect a wired earphone and can be a USB interface 330 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0102] The sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, an air pressure sensor 380C, a magnetic sensor 380D, an acceleration sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, an ambient light sensor 380L, a bone conduction sensor 380M, etc.

[0103] The key 390 includes a power key, a volume key, etc. The key 390 may be a mechanical key or a touch key. The control device may receive key input and generate key signal input related to user settings and function control of the control device.

[0104] Motor 391 can generate vibration prompts. Motor 391 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. Motor 391 can also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 394.

[0105] Indicator 392 may be an indicator light, which may be used to indicate the charging status and power changes of the control device, and may also be used to indicate messages, missed calls, notifications, etc.

[0106] The SIM card interface 395 is used to connect a SIM card. The SIM card can be connected to or disconnected from the control device by inserting the SIM card interface 395 or pulling the SIM card out of the SIM card interface 395. The control device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 395 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 395 at the same time. The types of the multiple cards can be the same or different.

[0107] At present, because smart home systems have more devices and functions than traditional home appliance systems, and with the development of smart home technology, smart home systems will include more devices and realize more functions of more devices in the future. Therefore, a simpler, more intuitive and natural way is needed to control and manage.

[0108] Typically, in order to facilitate users to control and manage smart home devices through device management applications, users and control devices can transmit control commands to and from smart home devices through voice interaction and / or touch interaction.

[0109] Among them, voice interaction means that the control device, based on voice recognition and speech synthesis technology, allows users to communicate with the smart home system by speaking to implement various instructions and queries. For example, users can use voice commands to turn on or off smart home devices such as lights, air conditioners, curtains, or ask smart home devices for information such as weather, news, and schedules.

[0110] Touch interaction means that the control device is based on touch screen or gesture recognition technology, allowing users to interact with the smart home system by touching or showing gestures to achieve various settings and adjustments. For example, users can use icons or menus on the touch screen to select related parameters of smart home devices in different modes or scenarios, or adjust parameters such as volume and brightness of a single device.

[0111] Based on user operations, the control device can individually control a single device (also referred to as a single device) in the smart home device, or can centrally control multiple electronic devices in the smart home device. The following is an explanation with reference to specific examples.

[0112] Example 1: Individually control a single device based on user operations

[0113] The specific process of the control device individually controlling a single device based on user operations may be that the control device responds to the user's click operation on the control card button of the target device or the drag operation of the slide bar in the touch interface, and performs operations such as turning on and off, adjusting the brightness, etc. on the target device; or when the user is far away from the control device, the control device performs operations such as turning on and off, adjusting the brightness, etc. on the target device based on voice commands input by the user.

[0114] In some implementations, the control device can complete the control of the target device in a touch interface based on the user operation. Figure 4A The figure is a schematic diagram of the process of the control device controlling a single device provided in this embodiment, which involves the process of the control device controlling "turning on the downlight on the background wall of the master bedroom". Figure 4A Figure (a) shows the main interface of the control management application, which displays the devices, spaces and "my home" operation options that can be controlled by the device management application. It should be understood that the main interface can also include more or fewer other operation options, which is not limited in the embodiments of the present application.

[0115] like Figure 4A As shown, after the user starts the control management application in the control device, the following is displayed Figure 4A The interface 401 shown in (a) in FIG. 4 shows operation options for different spaces, such as the master bedroom, bathroom, and children's room. The user performs the following operations: Figure 4AThe operation shown in (a) in FIG. 1 shows that the user clicks the operation option of the master bedroom, and the following is displayed: Figure 4A The interface 402 shown in (b) of FIG. 4 shows operation options of different types of equipment in the master bedroom, such as the master bedroom TV, the master bedroom lighting equipment, the master bedroom audio, the master bedroom air conditioner, etc. The user clicks the operation option of the master bedroom lighting equipment on the interface 402, and the control device responds to the user's click operation and displays the following Figure 4A The interface 403 shown in FIG. (c) in FIG. 4 shows operation options of different types of lamps, such as background wall downlights, table lamps, floor lamps, and bedside lamps. The user clicks the operation option of the background wall downlight on the interface 403, and the control device responds to the user's click operation and displays the following Figure 4A The interface 404 shown in Figure (d) displays the switch options corresponding to each downlight on the background wall. The user clicks on the switch option, and the control device responds to the click operation to complete the process of turning on the downlight on the background wall.

[0116] It can be seen that through Figure 4A In the operation process shown, the control device is based on the user's operation. When the touch interface is used to control the master bedroom background wall downlight to turn on, the user needs to select the space, master bedroom, and master bedroom lighting equipment in turn on each interface, and find the option corresponding to "background wall downlight" among the multiple lighting equipment in the master bedroom, and then click to turn it on. During this step, finding the option corresponding to "background wall downlight" from the multiple lighting equipment in the master bedroom is the most tedious step, because the user needs to read the names of different lighting equipment one by one to determine whether they correspond to the lighting equipment they want to operate. After confirmation, click and complete the final control operation.

[0117] In other implementations, the control device can complete the control of the target device through voice interaction based on user operations. For example, for the scenario where the control device controls "turn on the downlight on the background wall of the master bedroom", the user needs to say to the control device in voice interaction mode: "Turn on the downlight on the background wall of the master bedroom", and the control device controls the turning on of the downlight on the background wall of the master bedroom according to the voice command. In this way, the user needs to name each lamp in the space in the device management application in advance, that is, each lamp corresponds to a unique voice command, otherwise it cannot be implemented.

[0118] Since smart home systems have more devices and more functions than traditional home systems, and the number has been increasing. In addition, in the application scenarios of smart home systems, the most important lighting system with the highest user interaction frequency is currently trending towards "no main light design", that is, in a single space, there is no longer only one main light as before, but a more uniform light environment is created through multiple downlights, light strips and other lighting equipment, and the existence of lamps is almost invisible in the physical space, thus achieving "seeing light but not seeing lamps". In other words, there may be multiple devices of the same type in the same space. In this application scenario, whether the control device controls a single device based on user touch or voice, the user's operating cost or pre-preparation cost is very high, which affects the user experience.

[0119] Example 2: Scenario-based centralized control of devices based on user operations

[0120] Scenario-based centralized control means that the control device packages multiple smart home devices into a "scene" according to user needs, such as movie mode, family mode, relaxation mode or focus mode. The user only needs to trigger the scene, and the control device responds to the user's control operation on the triggered scene mode to achieve the control of multiple smart home devices of the same category or across categories, without having to control each single device one by one. In different scene modes, the control device controls the smart home devices to be in a preset switch state or parameter state. For example, the user can create a "movie mode" in which the curtains are closed, the main lights are turned off, and the atmosphere lights are turned on to a darker mode.

[0121] In some embodiments, the control device can complete the scene-based centralized control of smart home devices in the touch interface based on user operations. Figure 4B As shown, it is a schematic diagram of the process of the control device provided in this embodiment to centrally control multiple devices based on the scenario, involving the process of the control device controlling the "starting movie viewing mode" according to the user operation.

[0122] like Figure 4B As shown, the user clicks Figure 4B After clicking the "Scene" operation option shown in (a) of FIG. 1 , the control device displays the following in response to the click operation: Figure 4B The interface 405 shown in (b) in FIG. 4 shows the operation options corresponding to different scene modes. The user clicks the operation option of the movie viewing mode on the interface 405, and the control device responds to the user's click operation and displays the following Figure 4BInterface 406 shown in (c) in FIG. The interface 406 includes the switch status or parameter status of each smart home device in the movie viewing mode, as well as the operation options of "Edit more" and "Execute". For example, only 3 lights are turned on in the lighting equipment, the parameters of the cooling and heating equipment are adjusted to a temperature of 24°C and a humidity of 40%, the curtains are closed, and the audio is turned on. After the user clicks the "Execute" option in the interface 406, the control device responds to the click operation and controls each device in the movie viewing mode.

[0123] Through the control method in Example 2 above, the control device can control multiple devices with one click based on user operation, but the user needs to define each scene in the device management application in advance, which also has the problem of high user preparation cost. In addition, in the process of defining scenes, it is also necessary to set the device parameters and startup status of each smart device in different scenes, which will also cause the problem of "difficulty in selecting single-product devices" and high user operation costs, affecting the user experience.

[0124] In addition, "scenario-based centralized control" is an integrated control process based on fixed execution rules. When using smart home devices, users will not be in the state corresponding to each scenario all the time, and will often want to fine-tune certain parameters of certain devices in life. In this scenario, the problem of "difficulty in selecting single-product devices" and high user operation costs will also be faced.

[0125] To sum up, the above-mentioned various implementation methods all have the problem of high operation cost or preliminary preparation cost for users, and it is impossible to intuitively and naturally select one or more devices that users want to control. In addition, due to the limitation of more devices and more functions in the smart home system, users have a poor user experience when controlling smart home devices based on traditional methods.

[0126] To this end, the embodiment of the present application provides a device control method that can solve the problems of difficult device selection and high initial preparation costs for users in smart home scenarios. Users can conveniently and quickly perform intelligent control of one or more devices within a target area in an intuitive and natural way, which simplifies the operation process, improves the interactive efficiency of selecting target devices and controlling target devices in smart home scenarios, and improves the user experience.

[0127] The following will be combined Figures 5 to 14B , an exemplary explanation is given of the process of device control provided in the embodiments of the present application.

[0128] Figure 5 is a schematic diagram of the interactive flow of the device control method provided by an embodiment of the present application, see Figure 5 As shown, the method includes the following steps S501 to S506.

[0129] S501: Control a device to display a display interface including spatial information, wherein the spatial information includes at least one area range.

[0130] In this embodiment, the display interface can be the main operation interface of the device management application (such as a smart life application), or it can be other interfaces, such as the operation interface displayed in response to the user's left or right swipe operation on the main interface of the device management application, or the operation interface displayed in a certain operation mode (such as a regional operation mode). Exemplarily, after the device management application is started and run, its main interface displays different operation modes, such as a single device operation mode, a scenario-based centralized control operation mode, and a regional operation mode, etc. Different operation modes correspond to different operation methods for smart home devices. Specifically, the single device operation mode corresponds to the operation process of individually controlling a single device based on user operation as given in Example 1 above, the scenario-based centralized control operation mode corresponds to the operation process of scenario-based centralized control of a device based on user operation as given in Example 2 above, and the regional operation mode corresponds to the operation process of controlling smart home devices based on user operation and combining the spatial information displayed in the display interface. After starting the device management application, the control device can display the display interface according to the user operation.

[0131] Among them, spatial information refers to information reflecting the spatial distribution characteristics of geographic entities, which include the location, shape, spatial relationship between entities, regional spatial structure, etc. For example, the spatial information can be a floor plan of a home environment, a three-dimensional model of a home environment, or a three-dimensional model of soft furnishings layout information of a home environment. For example, see Figure 6 The floor plan of the home environment shown in Figure (a) is shown in Figure 6 The three-dimensional model diagram of the household structure shown in Figure (b) of the home environment is shown in Figure 1. Figure 6 The three-dimensional model diagram of the soft furnishings layout information of the home environment shown in Figure (c) of FIG. The spatial information can also be a topographic distribution map of a certain area in a city.

[0132] In some embodiments, the spatial information may come from a floor plan, a three-dimensional model diagram, etc. stored in a host / server. The floor plan may be in a picture format, such as an exchangeable image file (EXIF), portable network graphics (PNG), joint photographic experts group (JPEG) or web page image (weppy, webp) format, etc.; the three-dimensional model diagram may be in a format including a motion capture tool (filmbox, FBX), a standard three-dimensional model (STEP) or an object file (OBJ) format, etc., wherein the three-dimensional model diagram may be displayed in a display interface by a 3D rendering engine.

[0133] In this embodiment, the spatial information may be obtained and stored in the host / server by, but not limited to, the following methods.

[0134] (1) Some home decoration design software has preset floor plans of different types of houses in different areas and communities.

[0135] (2) Computer aided design (CAD) drawings or floor plans of houses obtained through on-site surveying. The CAD drawings or floor plans of houses can be automatically generated into three-dimensional models through tools, or the three-dimensional model drawings can be manually generated through three-dimensional design software.

[0136] (3) A three-dimensional model of a house including soft furnishings obtained by manually adding the layout information of soft furnishings to the three-dimensional model of the room.

[0137] (4) Using sensors with environmental depth perception, such as cameras and / or lidar, to generate a three-dimensional model of the house through on-site scanning.

[0138] Usually, since the preset floor plan does not include the layout information of soft furnishings, all the spatial information including "floor plan" and "soft furnishings" can be obtained at one time through the above two methods (3) or (4), and can be updated at any time.

[0139] In this embodiment, when the spatial information is a three-dimensional model or a plan view of a home environment, the spatial information includes the layout of each spatial area in the user's home environment and the location references of each room. The location reference refers to a landmark entity used to refer to a certain area in the spatial area. For example, see Figure 6In the floor plan of the apartment structure shown in Figure (a), the space information includes several space areas such as the master bedroom, the second bedroom, the living room, the dining room, the kitchen and the bathroom; each space area includes various orientation indicators, such as the sofa, coffee table, and living room door in the living room, the master bed, dressing table and desk in the master bedroom or the second bedroom, and the vanity and toilet in the bathroom. Among them, the sofa is used to refer to the area in the living room including the location of the sofa, and the master bed is used to refer to the area in the master bedroom including the location of the master bed.

[0140] It should be noted that the area range involved in the present embodiment may be an area range including the entire spatial information, for example, the area range of all spatial areas included in the floor plan of the apartment structure in a home environment; it may also be the area range corresponding to different spatial areas in the spatial information, for example, the area range included in the master bedroom in the floor plan of the apartment structure, or the area range included in the living room, etc.; it may also be the area range corresponding to the location reference objects in each spatial area, such as the area range corresponding to the position of the sofa in the living room, the area range corresponding to the position of the master bed in the master bedroom, etc.

[0141] In this embodiment, the area range may include but is not limited to the following forms.

[0142] (1) Point-like area range

[0143] For example, the region range may be any point in the spatial information, see Fig. 7A As shown in (a) and (b) in the figure, the area range includes the dotted area range 1 corresponding to the restaurant in (a), the dotted area range 2 corresponding to the living room, and the dotted area range 3 corresponding to the master bedroom, and the dotted area range 1 corresponding to the living room in (b), the dotted area range 2 corresponding to the bathroom, and the dotted area range 3 corresponding to the restaurant.

[0144] (2) Two-dimensional area range

[0145] For example, when the spatial information is a floor plan of a house structure, the area range can be a preset plane range around any point in the spatial information, wherein the point can be the center point of the preset plane range, for example, the center of a circular area range; or it can be any point within the preset plane range, for example, the vertex of a rectangular area range. Figure 7B As shown in the figure, the spatial information shown in the figure includes a plane area range 1 corresponding to the restaurant, a plane area range 2 corresponding to the living room, and a plane area range 3 corresponding to the master bedroom. The plane area ranges can be as follows: Fig. 7A The circular area shown in the figure may also be other shapes, such as triangle, rectangle, ellipse, polygon or other irregular shapes.

[0146] (3) Three-dimensional area range

[0147] For example, when the spatial information is a three-dimensional diagram of a house structure, the region range may be a preset spatial range around any point in the spatial information, wherein the point may be the center point of the preset spatial range, such as the center of a sphere; or any other point within the preset spatial range, such as the vertex of a cone. Figure 7C As shown in FIG. 1 , the spatial information shown in the figure includes a spatial area range 1 corresponding to the living room, a spatial area range 2 corresponding to the bathroom, and a spatial area range 3 corresponding to the dining room. The area ranges may be as follows: Figure 7B The spherical space region shown in may also be a cubic or rectangular space region, etc.

[0148] It should be understood that Fig. 7A , Figure 7B and Figure 7C The spatial information shown may also include more or fewer, larger or smaller area ranges, and each area range is provided with smart home devices corresponding to the area range. The location and number of the area ranges can be specifically set up in the background by the operation and maintenance personnel according to actual needs, and are not listed one by one in this embodiment.

[0149] In some embodiments, when the control device displays a display interface including spatial information, it first determines the position orientation of the control device in the physical space, and then controls the display direction of the spatial information in the first direction of the control device to be consistent with the position orientation of the control device. The display direction of the control device in the first direction may be the upward display direction of the control device. When the position orientation of the control device in the physical space changes, the display direction of the spatial information in the display interface of the control device is adjusted according to the changed position orientation. For example, see Fig. 8A As shown in , when the user holds the control device in the physical space in the direction Fig. 8A When the first orientation (i.e., position orientation) is shown in FIG. (a), the control device displays as follows Fig. 8A In the display interface 801a of the spatial information shown in FIG. (b), the upward display direction (also referred to as the first direction) of the control device is consistent with the first direction of the user in the physical space. Figure 8B As shown in , when the user holds the control device in the physical space in the direction Figure 8B In the second orientation shown in Figure (a), the control device displays Figure 8BIn the display interface 801b of the spatial information shown in Figure (b), in the display interface 801b, the upward display direction of the control device is consistent with the second direction of the user in the physical space. When the direction of the user holding the control device in the physical space keeps changing, the display direction of the spatial information in the display interface of the control device is adaptively adjusted according to the direction of the user in the physical space. In other words, the "left-up-right-down" in the spatial information corresponds to the "left-front-right-back" relative to the user in the real physical space, which is consistent with the user's habit of looking at maps.

[0150] It should be noted that, in this embodiment, the upward display direction of the control device can be defined in the following manner:

[0151] (1) If the control device is placed vertically (or at an angle greater than 0° to the horizontal plane), then "upward" means the direction from the bottom of the control device to the top.

[0152] (2) If the control device is placed horizontally, the "upward" direction is inherited from the upward direction before the horizontal placement (when placed vertically); it can also be based on sensors such as a front camera to obtain the person's orientation (based on the orientation of the face) and adjust the upward direction of the interface according to the person's orientation.

[0153] In this way, users can more intuitively correspond the spatial information in the display interface, such as a three-dimensional model diagram of the apartment structure of a home environment or a floor plan of the apartment structure of a home environment, to the real physical space, and the view display direction of the spatial information can be adaptively adjusted according to the position and orientation of the control device in the physical space, thereby improving the user experience.

[0154] In some embodiments, the orientation of the control device relative to the indoor environment is obtained by:

[0155] (1) The control device is fixed at a certain position in the indoor environment, and the orientation information can be preset in a storage module in the control device.

[0156] (2) The control device is provided with an accessory of a magnetic base, which is fixed to the wall. The magnetic base encodes a direction information through near field communication (NFC) technology. When the control device contacts the magnetic base, the direction information can be obtained through NFC.

[0157] (3) The north-south orientation of the indoor environment is configured in advance in the control device. The orientation of the control terminal in the earth coordinate system is obtained through sensors such as a magnetometer built into the control device, so that the orientation of the control device relative to the indoor environment can be calculated.

[0158] S502: The control device determines a first area range from at least one area range according to a first control operation.

[0159] Among them, the first control operation includes touch operations such as clicking, long pressing, double clicking, and sliding.

[0160] In this embodiment, the control device determines the first area range from at least one area range according to the first control operation, including: the control device responds to the user's first control operation on the spatial information, and determines the first area range from at least one area range included in the spatial information according to the projection coordinates of the first control operation falling within a certain area range in the spatial information. Exemplarily, when the user clicks on any position on the display interface of the control device, the control device projects the click position onto a plane or three-dimensional map of the spatial information, and obtains the projection coordinates corresponding to the click position in the plane or three-dimensional map, and then determines the first area range according to the area range indicated by the projection coordinates. Optionally, when there is no corresponding area range for the projection coordinates, the area range adjacent to the projection coordinates is determined as the first area range.

[0161] In some embodiments, the region range includes a point region range corresponding to the projection coordinates of the first control operation projected into the spatial information. The region range may be a plane coordinate (x n ,y n ), or it can be a spatial coordinate (x n ,y n , z n ). For example, see Fig.9A As shown, the projection coordinates of each point in the spatial information correspond to a point-like area range. For example, the projection coordinates (x1, y1) or (x1, y1, z1) of point 1 fall at the location of the coffee table in the living room in the spatial information, and its corresponding area range is point-like area range 1, which may include electronic devices such as coffee table ceiling lamps; the projection coordinates (x2, y2) or (x2, y2, z2) of point 2 fall at the location of the sofa in the living room in the spatial information, and its corresponding area range is point-like area range 2, which may include electronic devices such as sofa wall lamps, and so on.

[0162] In other embodiments, the area range includes a two-dimensional area range or a three-dimensional area range corresponding to the projection coordinates of the first control operation projected into the spatial information. For example, when the projection coordinates of the first control operation projected into the spatial information are (x1, y1), the area range referred to is a circular plane area range with a radius of r and a projection coordinate (x1, y1) as the geometric center, or a rectangular space area range (i.e., a three-dimensional area range) with a length, width, and height of l, w, and h, respectively, and with a projection coordinate (x1, y1, z1) as the geometric center. For example, see Fig. 9BAs shown, the projection coordinates of each point in the spatial information correspond to a two-dimensional or three-dimensional area range. For example, the projection coordinates (x4, y4) or (x4, y4, z4) of point 4 fall at the location of the master bed in the spatial information, and the corresponding area range is the two-dimensional / three-dimensional area range 4 with the projection coordinates (x4, y4) or (x4, y4, z4) as the geometric center, and the two-dimensional / three-dimensional area range 4 may include electronic devices such as the master bed lamp; the projection coordinates (x5, y5) or (x5, y5, z5) of point 5 fall at the location of the master bedroom wardrobe in the spatial information, and the corresponding area range is the two-dimensional / three-dimensional area range 5 with the projection coordinates (x5, y5) or (x5, y5, z5) as the geometric center, and the two-dimensional / three-dimensional area range 5 may include electronic devices such as smart wardrobes, and so on.

[0163] In some further embodiments, the area range includes area identification information (IDentity, ID) corresponding to the projection coordinates projected into the spatial information by the first control operation, referred to as the area range indicated by the area ID. Each area range corresponds to an area ID of the area range, and the area ID can be the name of a certain spatial area, such as "bedroom", "living room", "cinema viewing area", "leisure area", etc.; it can also be the name of a direction reference, such as "living room sofa", "coffee table", "master bedroom bed", etc. Exemplarily, the area range indicated by the area ID "living room sofa" is the preset area range corresponding to the "living room sofa"; the area range indicated by the area ID "bedroom" is the preset area range corresponding to the "bedroom". Exemplarily, see Fig. 9C As shown in the figure, the projection coordinates of each point in the spatial information correspond to an area range indicated by an area ID. For example, the projection coordinates (x9, y9) or (x9, y9, z9) of point 9 fall on the location of the desk in the second bedroom in the spatial information, and its corresponding area range is the area range corresponding to the area ID "desk", which may include electronic devices such as desk lamps; the projection coordinates (x9, y9) of point 14 fall on the location of the desk in the second bedroom in the spatial information, and its corresponding area range is the area range corresponding to the area ID "desk", which may include electronic devices such as desk lamps; 14 ,y 14 ) or (x 14 ,y 14 , z 14 ) falls on the location of the restaurant tea bar area in the spatial information, and its corresponding area range is the area range corresponding to the area ID "tea bar area". The area range corresponding to the area ID "tea bar area" can include electronic equipment such as tea bar machines, and so on.

[0164] It should be noted that each area may include one or more electronic devices. For example, the area corresponding to the area ID "tea bar area" may include a tea bar machine, and may also include other electronic devices such as a tea bar machine ceiling light and a water dispenser. Specific settings may be made during configuration, and this embodiment will not list them one by one.

[0165] Exemplarily, the process of controlling the device to determine the first area range is described by taking the corresponding area range indicated by the projection coordinates as the area range indicated by the area identification information where the projection coordinates fall as an example.

[0166] See also Fig. 10A As shown in , the user clicks Fig. 10A When the user clicks the position shown in FIG. (a), the control device projects the user's click position onto the apartment structure plan, and obtains that the projection coordinates corresponding to the user's click position in the apartment structure plan fall within the plane area range corresponding to the "living room sofa", and then, based on the area range indicated by the projection coordinates including the plane area range indicated by the area identification information, the plane area range corresponding to the "living room sofa" is determined as the first area range; the user clicks Fig. 10A When the user clicks the position shown in Figure (b), the control device projects the user's click position onto the apartment structure plan, and obtains that the projection coordinates corresponding to the user's click position in the apartment structure plan fall within the plane area corresponding to the "master bedroom", and then based on the area indicated by the projection coordinates including the plane area indicated by the area identification information, the area corresponding to the "master bedroom" is determined as the first area. It can be understood that when the spatial information is a three-dimensional diagram of the apartment structure, see Fig. 10B As shown in Figure (a) in FIG. 1 , when the user clicks on the location of the sofa in the area identification information in the three-dimensional diagram of the apartment structure, the control device determines the spatial area range corresponding to the sofa as the first area range according to the above determination process; see Fig. 10B As shown in Figure (b), when the user clicks on the location of the master bedroom in the area identification information in the three-dimensional diagram of the apartment structure, the control device determines the spatial area range corresponding to the master bedroom as the first area range according to the above determination process.

[0167] It should be noted that the first area range may be one or more, which may be specifically determined according to the user's operation.

[0168] In some embodiments, the control device receives a user's selection operation on N area ranges in the spatial information, for example, the user can select N area ranges in the spatial information by "multiple clicks"; receives a user's cancellation operation on M area ranges in the N area ranges, for example, the user can cancel the selection of M area ranges in the N area ranges by "sliding", and at this time, the control device determines the NM area ranges as the first area range, where N≥M≥1. For example, see Fig. 10C As shown in , the user selects area range ①, area range ② and area range ③ by “multiple clicks”, and the control device determines the area range ①, area range ② and area range ③ as the first area range; see Fig. 10DAs shown in , the user draws out area range ③ by “sliding”, and the control device determines the remaining area range except the drawn area range as the first area range, that is, the control device determines area range ① and area range ② as the first area range.

[0169] In some embodiments, the control device responds to the user's first control operation on the spatial information, determines the first area range, extracts candidate control instructions related to the first area range, and displays element controls of the candidate control instructions on the display interface. The element control is used to instruct the user to input a control instruction according to the content corresponding to the element control.

[0170] In one implementation, see Fig.11A As shown in the figure, the user clicks on the "living room sofa", and the control interface displays element controls of the devices near the "living room sofa", such as "lighten this place, darken other places", "turn off this room light", etc. The user can trigger the corresponding control by clicking the operation option corresponding to the element control, or speaking the corresponding text in these element controls.

[0171] In another implementation, see Fig. 11B As shown in the figure, the user clicked on the "living room sofa", and the control interface displays the element controls of the devices near the "living room sofa", such as the switch and brightness adjustment control of the coffee table ceiling light, the switch and brightness adjustment control of the sofa wall light, and the switch and temperature adjustment control of the air conditioner, etc. The user can trigger the corresponding control by clicking the element control corresponding to each device, or adjusting the element control of the slider.

[0172] S503: The control device receives a second control operation.

[0173] The second control operation includes a voice control operation (such as voice input) or a touch control operation (such as touch input), etc. The electronic device receiving the second control operation includes the electronic device receiving a control instruction inputted by a user through voice input or touch input.

[0174] It should be noted that when the user expresses the type of electronic device he wants to control to the control device through voice input or touch input, the control instruction contained in the second control operation may contain directional reference information or may not contain directional reference information. At this time, after receiving the second control operation, the control device needs to analyze the control instruction in the second control operation to determine whether the control instruction contains directional reference information. Optionally, the second control operation can be a user's Fig. 10A or Fig. 10B The operation options in the control interface shown or the voice or touch operation of the control card.

[0175] The directional reference information refers to the relevant content describing the directional information, including but not limited to: "here", "this room", "other places", "other rooms", "around", "next to" and "nearby".

[0176] In this embodiment, the control commands input by the user through voice include but are not limited to the following: "Make it brighter here", "Make it darker elsewhere", "Turn off the lights in this room", "Turn off the lights in other rooms", "Turn on the lights", "Clean up here", and "Don't let the wind blow here", etc.

[0177] The control commands input by the user through touch include but are not limited to: "turn on / off the light", "brightness adjustment", "area cleaning", "wind direction adjustment", etc.

[0178] In this embodiment, different control instructions correspond to different types of electronic devices, or different types of electronic devices correspond to their own control instruction sets, see Fig. 12A For example, "light switch" corresponds to "lighting equipment", "area cleaning" corresponds to "sweeping robot", "wind direction adjustment" corresponds to "air conditioner / fan", etc. At the same time, when each electronic device is adaptively configured with the spatial information, different types or different electronic devices of the same type are also associated with the area range in the spatial information. For example, see Fig. 12B As shown, the spatial information includes not only fields for describing the area range of the electronic device, but also the spatial area (such as the living room, master bedroom or dining room), area ID (for example, area ID "sofa", area ID "coffee table" and corresponding projection coordinates (for example, (x1, y1) or (x1, y1, z1), (x2, y2) or (x2, y2, z2)), etc.). Different electronic devices may have one or more area IDs due to their different locations. For example, a spotlight located on the background wall of the sofa may have an area ID "sofa" and an additional area ID "against the wall" to specifically describe its location. The number of area IDs of different electronic devices may be different.

[0179] As can be seen from the above, the control instructions input by the user through voice input or touch input may include the type of electronic device to be controlled and the control instructions for the action to be performed on the electronic device, such as turning on the light, turning off the light, shutting down the computer, etc., where turning on and off are the actions to be performed, and the light and the computer are the electronic devices to be controlled; it can also be a control instruction that includes both the type of electronic device to be controlled and the action to be performed on the electronic device, and the direction reference information, such as making it brighter here, darker elsewhere, and cleaning here, etc., where bright, dark, and cleaning correspond to the electronic devices being lights, lamps, and sweeping robots, respectively, and the corresponding execution actions are brightening, dimming, and cleaning; here, other places, and here correspond to the direction reference information, respectively. Therefore, after receiving different types of control instructions, the control device needs to analyze the control instructions, and then determine the operation to be performed based on the analysis results.

[0180] In some embodiments, the control device extracts semantic information from the control instruction input by the user's voice or touch input through natural language processing (NLP) technology or based on a large language model (LLM). When the extracted voice information includes any of the above-mentioned location reference information, it is determined that the control instruction contains the location reference information. For example, the control instruction of the second control operation input by the user is "turn off the lights in other rooms", and the control device determines that the control instruction includes the voice information of "other rooms" through the NLP technology, and then determines that the control instruction of the second control operation contains the location reference information.

[0181] S504: The control device determines a target area range from at least one area range according to the second control operation and the first area range, wherein the first area range is the same as or different from the target area range.

[0182] In the above step S502, the control device has determined the first area range from at least one area range in response to the user's first control operation on the spatial information. In this step, the control device determines the target area range to be controlled based on the determined first area range and the second control operation; wherein the first area range and the target area range may be the same or different. In other words, the control device needs to further determine through the second control operation whether the target area range that the user wants to control is the first area range indicated by the first control operation, or other area ranges that are different from or associated with the first area range. The control instruction based on the second control operation includes the type of electronic device that needs to be controlled and the action that needs to be performed on the electronic device. Therefore, after determining the target area range, the control device also immediately determines the electronic device that needs to be controlled within the target area range.

[0183] In one implementation, see Fig.13A As shown, in response to the user's first control operation, the control device has determined the area range corresponding to the area ID "living room sofa" as the first area range; then after receiving the user's second control operation "lighten here" input by voice, the control device extracts the semantic information in the second control operation. After analyzing the semantic information, the control device determines that the semantic information includes the directional pronoun "here"; then, based on the directional pronoun "here", the target area range is determined as the first area range, that is, the area range corresponding to the area ID "living room sofa". At this time, the first area range is the same as the target area range.

[0184] In this example, if the control device receives the second control operation "turn on the light in this room" input by the user through voice, the semantic information in the second control operation is extracted, and after analysis, it is determined that the voice information includes the directional pronoun "this room". Based on the directional pronoun "this room", the control device determines the target area range as the entire living room range including the area range corresponding to the area ID "living room sofa", rather than just the area range corresponding to the area ID "living room sofa". At this time, the first area range is different from the target area range, and the target area range is a larger range that includes the first area range.

[0185] In another implementation, see Fig. 13B As shown, after receiving the second control operation of "turn off the lights in other rooms" input by the user through voice, the control device extracts the semantic information in the second control operation and analyzes it, and determines that the semantic information includes the directional pronoun "other rooms", and then determines the target area range as other area ranges other than the first area range based on the semantic information "other rooms". At this time, the first area range is different from the target area range, and the target area range does not include the first area range.

[0186] In yet another implementation, see Fig. 13C As shown, after receiving the second control operation of "turn off the lights" input by the user through voice, the control device extracts the semantic information in the second control operation and analyzes it, and determines that the semantic information does not include direction reference information. The control device then determines the target area range as the first area range. At this time, the first area range is also the same as the target area range.

[0187] From the above, it can be seen that when the control device determines the target area range, the content related to the direction reference information in its control instruction can also play a decisive role. When the control instruction includes the direction reference information, the control device determines the target area range based on the direction reference information and the first area range; when the control instruction does not include the direction reference information, the first area range is determined as the target area range.

[0188] In addition, in some other implementations, the control instructions input by the user through voice or touch may not contain direction reference information, but only contain the device to be controlled and the action to be performed, but the area range that the user actually wants to control is not limited to the first area range. For example, the control device responds to the first control operation of the user clicking on the location of the "living room sofa", and receives the second control operation of the user inputting the control instruction of "turn on the lights" through voice or touch. At this time, although the user is pointing to a specific "living room sofa" location, what he expresses is that he wants to control the lights of the entire viewing area to turn on, or adjust the viewing area to the default lighting mode. In this embodiment, this situation is regarded as a situation where the control instruction of the second control operation contains "implicit" direction reference information.

[0189] For the above situation, the control device in this embodiment can determine the range of the target area based on different regional attributes. Among them, the regional attributes include specific areas and broad areas. There is a certain hierarchical correspondence between the broad area and the specific area. The specific area is characterized by an area without a lower-level regional scope, such as "sofa", "dining table", etc. The broad area is characterized by containing other sub-areas, such as "movie viewing area", "dining area", etc., wherein the "movie viewing area" may include other sub-areas such as "sofa", "TV cabinet", "coffee table", etc., and the "dining area" may include other sub-areas such as "tea bar area" and "dining table area".

[0190] For example, see Fig.13DAs shown in, it is a schematic diagram of the hierarchical relationship of the regional scope provided by the embodiment of the present application. As shown in the figure, the broad area "living room" corresponds to the next-level sub-area "movie viewing area", "dining area", etc., and the sub-area "movie viewing area" corresponds to its next-level sub-area "tea table", "TV cabinet", etc.; the broad area "dining area" corresponds to its next-level sub-area "dining table", "tea bar machine", etc. Based on the fact that the sub-areas "tea table", "TV cabinet", "dining table", and "tea bar machine" have no next-level regional scope, therefore, "tea table", "TV cabinet", "dining table", and "tea bar machine" can also be referred to as specific areas, and the sub-areas "movie viewing area" and "dining area" also contain other sub-areas, therefore, "movie viewing area" and "dining area" can also be referred to as broad areas. In the process of determining the target area range, when the control device determines the first area range as a "specific area" through the first control operation, according to the second control operation of the control instruction containing the "implicit" orientation reference information and the first area range, the upper-level area range of the first area range is determined as the target area range. For example, the position clicked by the user is the position of the "sofa", and the control device determines the area range of the upper level as the "movie viewing area" according to the area ID "sofa". At this time, the control device determines the "movie viewing area" as the target area range. When the first area range determined by the control device through the first control operation is the "broad area", according to the second control operation of the control instruction containing the "implicit" orientation reference information and the first area range, the first area range, namely the "broad area", is determined as the target area range. For example, the user clicks on a certain position next to the sofa. Since the projection coordinates corresponding to the position correspond to the area range of the "movie viewing area", but do not belong to the area range corresponding to the area ID "sofa", the control device determines the first area range, namely the "broad area", as the target area range. In this implementation process, if the user only wants to turn on the light of the "sofa", the user needs to explicitly input the control instruction containing the orientation reference information when executing the second control operation, such as "turn on the light here" or "lighten it a little here".

[0191] In this way, the control device can determine whether it only needs to control the electronic device corresponding to the area clicked by the user, so as to achieve more accurate determination of the target area range.

[0192] S505: The control device determines electronic devices within the target area.

[0193] After determining the range of the target area, the control device can determine the electronic device that ultimately needs to be controlled and the action that the electronic device needs to perform according to the control instruction in the second control operation.

[0194] For example, see Fig.14AAs shown in , if the control instruction input by the user through voice is "brighten here a little", the control device determines that the electronic device is "lamp" according to the control instruction, and the action required to be performed by the electronic device is "increase the brightness". At this time, the display interface of the control device displays a prompt message "the nearby lights have been brightened for you" and an element control of the nearby lighting device. The element control is used for secondary adjustment when the user is not satisfied with the brightness adjusted by the control device; see Fig. 14B As shown in , if the control instruction input by the user through voice is "turn off the lights in other rooms", the control device determines that the electronic device is "light" according to the control instruction, and the action that the electronic device needs to perform is "turn off". At this time, the display interface of the control device displays the prompt message "The lights in other rooms have been turned off for you"; if the control instruction input by the user through voice is "make it darker in other places", the control device determines that the electronic device is "light" according to the control instruction, and the action that the electronic device needs to perform is "lower the brightness". Similarly, the display interface of the control device may display the prompt message "The lights in other places have been dimmed for you".

[0195] In the method provided in this embodiment, the control device responds to the user's first control operation on the spatial information, determines the first area range from at least one area range, and then according to the control instruction of the second control operation in this step, can more accurately determine the target area range that the user needs to control and the electronic devices within the target area range. When there are multiple electronic devices of the same type or different types in the user's home, this method can determine the specific device that the user needs to control.

[0196] S506: The control device controls the corresponding electronic device within the target area according to the second control operation.

[0197] In some embodiments, step S506 specifically includes the following steps S506a-S506b.

[0198] S506a, the control device sends an operation instruction to the electronic device.

[0199] The operation instruction is used for the electronic device to perform the corresponding operation according to the operation instruction after receiving the operation instruction. The electronic device is all the electronic devices in the target area, which may be one or more, depending on the type and number of the electronic devices configured in the target area.

[0200] S506b, the electronic device performs a corresponding operation according to the received operation instruction.

[0201] After receiving the operation instruction, the electronic device performs the corresponding operation according to the operation, for example, turning on / off the bedside lamp; turning on / off the air conditioner, etc.

[0202] It should be noted that in some other application scenarios, the second control operation input by the user may include multiple control instructions. For example, after the user inputs the first control instruction of "turn off the lights in other rooms" through voice, he or she immediately inputs the second control instruction of "turn on the air conditioner" to the control device.

[0203] In this case, the control device may first determine the first target area range and the electronic devices within the first target area range according to the first area range and the first control instruction in the second control operation, and control the electronic devices within the first target area to perform corresponding operations according to the first control instruction. Then, the control device may determine the second target area range and the electronic devices within the second target area range according to the first area range and the second control instruction in the second control operation, and control the electronic devices within the second target area to perform corresponding operations according to the second control instruction.

[0204] Through the device control method provided in the embodiment of the present application, the control device can conveniently and quickly control various types of electronic devices in the smart home scene according to the user's operation, and can realize the single control operation of a single electronic device, and can also realize the collective control operation of multiple electronic devices, thereby improving the interaction efficiency of users selecting target devices and controlling target devices in the smart home scene. The method does not require the user to name each electronic device or define the scene in advance, which reduces the user's operating cost and pre-production cost, and improves the user's experience.

[0205] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0206] The present application also provides a device control apparatus for controlling a device, see Fig.15 As shown, the device includes a display module 1501, a first determination module 1502, a second determination module 1503 and a control module 1504. The functions of each module are specifically as follows.

[0207] The display module 1501 is used to display a display interface including spatial information, wherein the spatial information includes at least one area range.

[0208] The first determining module 1502 is configured to determine a first area range from at least one area range according to a first control operation.

[0209] The second determining module 1503 is used to determine a target area range from at least one area range according to the second control operation and the first area range, wherein the first area range is the same as or different from the target area range.

[0210] The control module 1504 is used to control the corresponding electronic device within the target area according to the second control operation.

[0211] An embodiment of the present application also provides a control device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the device control method shown in the above embodiments.

[0212] The present application also provides a chip, see Fig.16 As shown, the chip includes a processor and a memory, in which a computer program is stored. When the computer program is executed by the processor, the device control method in the above embodiments is implemented.

[0213] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the device control method provided in the above embodiments is implemented.

[0214] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a controlled device, the controlled device implements the device control method provided in the above embodiments.

[0215] It should be understood that the processor mentioned in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0216] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus random access memory (DR RAM).

[0217] In the embodiments provided in the present application, the division of each framework or module is only a logical function division. There may be other division methods in actual implementation. For example, multiple frameworks or modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0218] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0219] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0220] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0221] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A device control method, characterized in that: Applied to a control device, the method comprises: Displaying a display interface including spatial information, wherein the spatial information includes at least one area range; According to a first control operation, determining a first area range from the at least one area range; Determine a target area range from the at least one area range according to a second control operation and the first area range, wherein the first area range is the same as or different from the target area range; According to the second control operation, the corresponding electronic device within the target area is controlled.

2. The method according to claim 1, characterized in that The step of determining a first area range from the at least one area range according to the first control operation includes: Determining, according to the first control operation, a projection coordinate of the first control operation projected onto the spatial information; When the projection coordinates have a corresponding area range, the area range corresponding to the projection coordinates is determined as the first area range; or, When there is no corresponding area range for the projection coordinates, the area range adjacent to the projection coordinates is determined as the first area range.

3. The method according to claim 1 or 2, characterized in that: The step of determining a target area range from the at least one area range according to the second control operation and the first area range includes: performing semantic analysis on the control instruction in the second control operation; When the control instruction includes position reference information, the target area range is determined according to the position reference information and the first area range.

4. The method according to claim 3, characterized in that The method further comprises: When the control instruction does not include the orientation reference information, the first area range is determined as the target area range.

5. The method according to claim 3, characterized in that: The method further comprises: When the control instruction does not include the orientation reference information, the upper level area range of the first area range is determined as the target area range.

6. The method according to any one of claims 1 to 5, characterized in that: The spatial information includes: A floor plan of the home environment; or, A three-dimensional model of the home environment; or A three-dimensional model diagram of the soft furnishings layout information for the home environment.

7. The method according to any one of claims 1 to 6, characterized in that: The area covered includes: The first control operation is projected to a point-shaped area corresponding to the projection coordinates in the spatial information; or The first control operation is projected into a two-dimensional area range or a three-dimensional area range corresponding to the projection coordinates in the spatial information; or The first control operation is projected onto an area range indicated by area identification information corresponding to projection coordinates in the spatial information.

8. The method according to any one of claims 1 to 7, characterized in that: The step of determining a first area range from the at least one area range according to the first control operation includes: Receiving a user's selection operation on N area ranges in the spatial information; receiving a user's cancel operation on M area ranges among the N area ranges; NM area ranges are determined as the first area range, where N≥M≥1.

9. The method according to any one of claims 1 to 8, characterized in that: After determining the first area range from the at least one area range according to the first control operation, and before determining the target area range from the at least one area range according to the second control operation and the first area range, the method further includes: Extracting candidate control instructions related to the first area range; An element control of the candidate control instruction is displayed on a display interface, and the element control is used to instruct a user to input the control instruction according to content corresponding to the element control.

10. The method according to any one of claims 1 to 9, characterized in that: The second control operation includes a voice control operation or a touch control operation.

11. The method according to any one of claims 1 to 10, characterized in that: The display interface for displaying spatial information includes: Determining the position orientation of the control device in physical space; The display direction of the spatial information in the first direction of the control device is controlled to be consistent with the position orientation of the control device.

12. The method according to claim 10, characterized in that The method further comprises: When the position orientation of the control device in the physical space changes, the display direction of the space information in the display interface of the control device is adjusted according to the changed position orientation.

13. A control device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 12 when executing the computer program.

14. A device control system, characterized in that: The method comprises a control device and a plurality of electronic devices connected to the control device, wherein the control device is configured to execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A chip, characterized in that: The chip includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 12 is implemented.

Images