Device control method, control device, system, readable storage medium and chip

Abstract

The application provides a device control method, a control device, a system, a readable storage medium and a chip, and relates to the technical field of smart home. The method is applied to a control device, and the method comprises the following steps: displaying a display interface comprising space information, wherein the space information comprises at least one region range; determining a first region range from the at least one region range according to a first control operation; determining a target region range from the at least one region range according to a second control operation and the first region range, wherein the first region range is the same as or different from the target region range; and controlling corresponding electronic devices in the target region range according to the second control operation. Through the technical scheme provided in the application, a user can conveniently and quickly control electronic devices in a target region in an intuitive and natural manner, so that the operation process is simplified, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and in particular to a device control method, control device, system, readable storage medium, and chip. Background Technology

[0002] In the field of smart home technology, a smart home system includes a control device and various smart home devices connected to the control device via Internet of Things (IoT) technology. Users can remotely control and intelligently manage various smart home devices through device management applications in the control device.

[0003] Currently, to facilitate simultaneous control of multiple smart home devices, device management applications typically employ a scenario-based centralized control approach. This involves setting multiple smart home devices within the same scenario mode, allowing users to control multiple or cross-category devices simply by triggering that scenario, eliminating the need to control each individual device. However, while this method allows users to operate multiple devices at once, it requires pre-defining scenarios and adding corresponding smart home devices to each scenario, resulting in high initial operational costs. Furthermore, when users need to control or adjust one or more smart home devices, they must navigate through a cumbersome search within the scenario-based centralized control interface to locate the target device, leading to a lack of intelligence in device management and control, and a poor user experience. Summary of the Invention

[0004] This application provides a device control method, control device, system, readable storage medium, and chip, which solves the problem that the existing control devices are not intelligent enough in managing and controlling electronic devices, resulting in high operating costs and cumbersome operation when users control and search for target devices through the control devices.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, a device control method is provided, applied to a control device. The method includes: displaying a display interface including spatial information, wherein the spatial information includes at least one area range; determining a first area range from the at least one area range according to a first control operation; determining a target area range from the at least one area range according to a second control operation and the first area range, wherein the first area range is the same as or different from the target area range; and controlling an electronic device corresponding to the target area range according to the second control operation.

[0007] The device control method provided in the first aspect of this application involves a control device determining a first area range based on a user's first control operation, and then determining a target area range based on the already determined first area range and a second control operation. In other words, the control device can further determine, through the second control operation, whether the target area range that the user wants to control is the first area range indicated by the first control operation, or a different area range that is related to or distinct from the first area range. This method enables a more accurate determination of the target area range that the user needs to control, as well as the electronic equipment within that target area range, thereby achieving precise control.

[0008] In addition, the control operations in this method are based on the spatial information displayed on the display interface. Users can conveniently and quickly control one or more devices within the target area in an intuitive and natural way, which simplifies the operation process, improves the interaction efficiency of selecting and controlling target devices in smart home scenarios, and enhances the user experience.

[0009] In some embodiments, determining a first region range from at least one region range according to a first control operation includes: determining projection coordinates of the first control operation projected into spatial information according to the first control operation; when a region range corresponding to the projection coordinates exists, determining the region range corresponding to the projection coordinates as the first region range; or, when a region range adjacent to the projection coordinates does not exist, determining the region range as the first region range.

[0010] As mentioned above, the spatial information includes at least one area, and each area corresponds to its own projection coordinates. That is, there is a correspondence between the area and the projection coordinates. The control device can determine the first area by projecting the projection coordinates onto the spatial information according to the first control operation.

[0011] When the control device receives the first control operation, if the projection coordinates of the first control operation projected into the spatial information have a corresponding area range, the control device determines the area range corresponding to the projection coordinates as the first area range; if the projection coordinates of the first control operation projected into the spatial information do not have a corresponding area range, the area range adjacent to the projection coordinates is determined as the first area range.

[0012] In some embodiments, determining a target region range from at least one region range based on a second control operation and a first region range includes: performing semantic analysis on a control instruction in the second control operation; and, when the control instruction includes directional reference information, determining the target region range based on the directional reference information and the first region range.

[0013] The directional reference information refers to the relevant content describing directional information, including but not limited to: "here," "this room," "other places," "other rooms," "around," "next to," and "nearby." Control commands that include directional reference information and those that do not may express completely different control processes. The control command contained in the second control operation received by the control device may or may not contain directional reference information. Therefore, after receiving the second control operation, the control device needs to analyze the control command in the second control operation to determine whether the control command contains directional reference information, and then determine the target area range based on the specific analysis results and in conjunction with the first area range.

[0014] In this embodiment, when the control command in the second control operation includes directional information, such as "other rooms," the area outside the first area is determined as the target area. In this case, the first area and the target area are different; the target area does not include the first area.

[0015] In some embodiments, the method further includes: when the control command does not include directional reference information, determining the first region range as the target region range.

[0016] In this embodiment, when the control command does not include directional information, the control device defaults to the target area range that needs to be controlled being the initially determined first area range. In this case, the first area range is the same as the target area range.

[0017] In some embodiments, the method further includes: when the control command does not include directional reference information, determining the next higher level region of the first region as the target region.

[0018] This embodiment presents an alternative implementation method when the control command does not include directional reference information, and is applicable to application scenarios where the control command contains "implicit" directional reference information. In this case, the control device determines the next higher-level region of the first region as the target region to meet the user's specific needs. Here, the first region is different from the target region; the target region is a larger region that includes the first region.

[0019] In some embodiments, the spatial information includes: a floor plan of the home environment; or a three-dimensional model of the home environment; or a three-dimensional model of the layout of the home's furnishings.

[0020] In other embodiments, the spatial information may also be a topographical distribution map of a certain area of ​​a city, etc.

[0021] In some embodiments, the area range includes: a point-like area range corresponding to the projection coordinates of the first control operation projected into the spatial information; or a two-dimensional or three-dimensional area range corresponding to the projection coordinates of the first control operation projected into the spatial information; or an area range indicated by the area identification information corresponding to the projection coordinates of the first control operation projected into the spatial information.

[0022] In this embodiment, the point-like region can be any point in two-dimensional or three-dimensional spatial information. A two-dimensional region can be a preset planar region surrounding any point in the spatial information, where the point can be the center point of the preset planar region, for example, the center of a circular region; or any point within the preset planar region, such as the vertex of a rectangular region. A three-dimensional region can be a preset spatial region surrounding any point in the spatial information, where the point can be the center point of the preset spatial region, for example, the center of a sphere; or any other point within the preset spatial region, such as the vertex of a cone. The region indicated by the region identifier information can be a preset region corresponding to the name of a spatial region or the name of a directional reference object. For example, the region ID "living room sofa" indicates the preset region corresponding to "living room sofa"; the region ID "bedroom" indicates the preset region corresponding to "bedroom".

[0023] In some embodiments, determining a first region range from at least one region range according to a first control operation includes: receiving a user's selection operation on N region ranges in spatial information; receiving a user's cancellation operation on M region ranges among the N region ranges; and determining NM region ranges as the first region range, where N≥M≥1.

[0024] In this embodiment, the control device can determine one or more first area ranges based on user operations, so as to facilitate subsequent control of multiple area ranges.

[0025] In some embodiments, after determining a first region range from at least one region range according to a first control operation and before receiving a second control operation, the method further includes: extracting candidate control instructions related to the first region range; and displaying element controls of the candidate control instructions on a display interface, the element controls being used to instruct the user to input control instructions according to the content corresponding to the element controls.

[0026] In this embodiment, the user can trigger the corresponding control by controlling the operation options corresponding to the element control (e.g., clicking on / off, or adjusting the slider), or by speaking the corresponding text in these element controls.

[0027] In some embodiments, the second control operation includes a voice control operation or a touch control operation.

[0028] In some embodiments, displaying a display interface including spatial information includes: determining the position orientation of the control device in physical space; and ensuring that the display direction of the spatial information in a first direction of the control device is consistent with the position orientation of the control device.

[0029] In some embodiments, the method further includes: when the position orientation of the control device in the physical space changes, adjusting the display orientation of the spatial information in the display interface of the control device according to the changed position orientation.

[0030] In this way, users can more intuitively view the spatial information in the display interface, such as a 3D model of the home's floor plan or a floor plan of the home's floor plan, and map it to the actual physical space. Furthermore, the view orientation of the spatial information can be adaptively adjusted according to the location and orientation of the control device in the physical space, thereby improving the user experience.

[0031] In a second aspect, a device control apparatus is provided for controlling a device. The apparatus includes: a display module for displaying a display interface including spatial information, wherein the spatial information includes at least one area range; a first determining module for determining a first area range from the at least one area range according to a first control operation; a second determining module for determining a target area range from the at least one area range according to a second control operation and the first area range, wherein the first area range may be the same as or different from the target area range; and a control module for controlling an electronic device corresponding to the target area range according to the second control operation.

[0032] Thirdly, a control device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method shown in the first aspect above.

[0033] Fourthly, a device control system is provided, including a control device and a plurality of electronic devices connected to the control device, the control device being configured to perform the method as shown in the first aspect above.

[0034] Fifthly, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the method shown in the first aspect above.

[0035] In a sixth aspect, a chip is provided, the chip including a processor and a memory storing a computer program that, when executed by the processor, implements the method shown in the first aspect above.

[0036] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0037] Figure 1 A schematic diagram of the structure of a smart home system provided in one embodiment of this application;

[0038] Figure 2A This is a schematic diagram of the structure of a smart home system provided in another embodiment of this application;

[0039] Figure 2B This is a schematic diagram of the structure of a smart home system provided in another embodiment of this application;

[0040] Figure 3 This is a schematic diagram of the structure of the control device provided in the embodiments of this application;

[0041] Figure 4A This is a schematic diagram illustrating a scenario where a control device controls a single smart home device according to an embodiment of this application.

[0042] Figure 4B This is a schematic diagram illustrating a scenario of a control device for a smart home device with contextualized control, provided in one embodiment of this application.

[0043] Figure 5 A schematic diagram of the interaction flow of a device control method provided in an embodiment of this application;

[0044] Figure 6 This is a schematic diagram of the structure of spatial information provided in one embodiment of this application;

[0045] Figure 7A A schematic diagram of the area provided in one embodiment of this application;

[0046] Figure 7B A schematic diagram of the area provided for another embodiment of this application;

[0047] Figure 7C A schematic diagram of the area provided for yet another embodiment of this application;

[0048] Figure 8A This application provides a schematic diagram of a scene where spatial information is displayed according to a first orientation, as an embodiment of the present application.

[0049] Figure 8B This application provides a schematic diagram of a scene where spatial information is displayed according to a second orientation, as an embodiment of the present application.

[0050] Figure 9A A schematic diagram showing the correspondence between projected coordinates and point-like region ranges provided in an embodiment of this application;

[0051] Figure 9B A schematic diagram illustrating the correspondence between projected coordinates and two-dimensional / three-dimensional region ranges provided in an embodiment of this application;

[0052] Figure 9C A schematic diagram illustrating the correspondence between projected coordinates and the area range corresponding to the area ID provided in an embodiment of this application;

[0053] Figure 10A A schematic diagram of a scenario in which a control device provided in an embodiment of this application determines the range of a first region according to a first control operation;

[0054] Figure 10B A schematic diagram of a scenario in which a control device, according to a first control operation, determines the range of a first region, as provided in another embodiment of this application.

[0055] Figure 10C A schematic diagram of a scenario in which a control device, according to a first control operation, determines the range of a first region, as provided in another embodiment of this application;

[0056] Figure 10D A schematic diagram of a scenario in which a control device, according to a first control operation, determines the range of a first region, as provided in another embodiment of this application.

[0057] Figure 11A A schematic diagram of a control device displaying a control interface in response to a first control operation, according to an embodiment of this application;

[0058] Figure 11B A schematic diagram of a control device displaying a control interface in response to a first control operation, provided in another embodiment of this application;

[0059] Figure 12A A schematic diagram showing the correspondence between electronic device types and control instruction sets provided in an embodiment of this application;

[0060] Figure 12B This is a schematic diagram illustrating the correspondence between electronic devices and spatial information provided in an embodiment of this application;

[0061] Figure 13A A schematic diagram illustrating a scenario where a control device according to an embodiment of this application determines the scope of a target area;

[0062] Figure 13B A schematic diagram illustrating a scenario where a control device, according to another embodiment of this application, determines the scope of a target area;

[0063] Figure 13C A schematic diagram illustrating a scenario where a control device, according to another embodiment of this application, determines the range of a target area;

[0064] Figure 13DThis is a schematic diagram of the hierarchical relationship of the region range provided in an embodiment of this application;

[0065] Figure 14A A schematic diagram illustrating a scenario where a control device controls electronic devices within a target area, as provided in an embodiment of this application.

[0066] Figure 14B A schematic diagram illustrating a scenario where a control device controls electronic devices within a target area, according to another embodiment of this application.

[0067] Figure 15 This is a schematic diagram of the structure of the device control apparatus provided in the embodiments of this application;

[0068] Figure 16 This is a schematic diagram of the chip structure provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0070] It should be understood that in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0071] In this embodiment, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0072] A smart home system is a residential remote control system that uses advanced computer technology, network communication technology, and integrated wiring technology to organically combine various subsystems related to home life. Through remote control and overall management, it makes home life more comfortable, safe, and convenient, while saving energy and resources and improving the quality of life.

[0073] Figure 1 A schematic diagram of the structure of a smart home system provided in an embodiment of this application. See also... Figure 1As shown, the system includes a control device and multiple smart home devices (also referred to as electronic devices). The control device is wirelessly connected to each smart home device, and the smart home devices may or may not be connected to each other. Furthermore, the wireless connection method between different devices can be Wi-Fi, Bluetooth, etc., and this embodiment does not impose specific limitations on this.

[0074] Smart home devices are objects managed and controlled by control devices, capable of executing corresponding control operations based on control commands from the control devices. Examples of smart home devices include televisions, speakers, headphones, air purifiers, refrigerators, air conditioners, robot vacuums, lights, cameras, routers, power strips, table lamps, humidifiers, sockets, smart door locks, water purifiers, treadmills, etc. Depending on the device type, the control operations performed by smart home devices according to control commands vary. For example, the control operations performed by a light might be turning the light on / off or adjusting its brightness; the control operations performed by a speaker might be playing / pausing music; and the control operations performed by an air conditioner might be turning it on / off or adjusting the temperature. This application does not limit the specific types of smart home devices.

[0075] Control devices play a management and control role in smart home systems, enabling intelligent management and control of various smart home devices through device management applications (such as smart living applications). In this embodiment, the control device can be a portable terminal device with a display screen, such as a mobile phone or tablet, or a home management device with a display screen, such as a central control screen, smart screen, or central control panel. It is understood that the connection methods between the control device and various smart home devices may differ depending on the type of control device.

[0076] In some embodiments, see Figure 2A As shown, when the control device is a terminal device or a home management device, the control device can directly connect to and control various smart home devices.

[0077] In other embodiments, see Figure 2B As shown, when the control device is a terminal device such as a mobile phone or tablet, it can indirectly connect to various smart home devices through a central device. Therefore, when controlling various smart home devices, the control device needs to relay signaling or data through the central device. For example, when controlling an air conditioner to turn on, the control device needs to first send the power-on command to the central device, which then forwards it to the air conditioner. The central device can be the home management device mentioned in the above embodiments, or it can be a router, home gateway, or other devices; this embodiment does not impose any restrictions on this.

[0078] Figure 3 This is a schematic diagram of the control device provided in an embodiment of this application. The control device includes a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, antenna 1, antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, a headphone jack 370D, a sensor module 380, buttons 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a subscriber identification module (SIM) card interface 395, etc.

[0079] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the control device. In other embodiments of this application, the control device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0080] Processor 310 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center for controlling the device. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0081] The processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. This memory can store instructions or data that the processor 310 has just used or that are used repeatedly. If the processor 310 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0082] The charging management module 340 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 receives charging input from the wired charger via a USB interface 330. In some wireless charging embodiments, the charging management module 340 receives wireless charging input via the wireless charging coil of the control device. While charging the battery 342, the charging management module 340 can also supply power to the control device via the power management module 341.

[0083] The power management module 341 connects the battery 342, the charging management module 340, and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, providing power to the processor 310, internal memory 321, external memory, display screen 394, camera 393, and wireless communication module 360. The power management module 341 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance).

[0084] The wireless communication function of the control device can be implemented through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, modem processor, and baseband processor.

[0085] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the control device can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0086] The mobile communication module 350 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G in control equipment. The mobile communication module 350 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0087] In some embodiments, at least some functional modules of the mobile communication module 350 may be disposed in the processor 310. In some embodiments, at least some functional modules of the mobile communication module 350 and at least some modules of the processor 310 may be disposed in the same device.

[0088] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio playback device (not limited to speaker 370A, receiver 370B, etc.) or displays images or videos through a display screen 394. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 310 and may be housed in the same device as the mobile communication module 350 or other functional modules.

[0089] The wireless communication module 360 ​​can provide solutions for wireless communication applications in control devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 310. The wireless communication module 360 ​​can also receive signals to be transmitted from processor 310, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0090] Camera 393 is used to capture still images or videos. In some embodiments, the control device may include one or N cameras 393, where N is a positive integer greater than 1.

[0091] The display screen 394 is used to display images, videos, etc., such as the various device management interfaces in the embodiments of this application. The display screen 394 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the control device may include one or N display screens 394, where N is a positive integer greater than 1.

[0092] The external storage interface 320 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the control device. The external storage card communicates with the processor 310 through the external storage interface 320 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0093] Internal memory 321 can be used to store executable program code, including instructions. Processor 310 executes various functional applications and data processing of the control device by running the instructions stored in internal memory 321. Internal memory 321 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.). The data storage area may store data created during the use of the control device (such as audio data, phonebook, etc.).

[0094] In addition, the internal memory 321 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0095] The control device can implement audio functions through audio module 370, speaker 370A, receiver 370B, microphone 370C, headphone jack 370D, and application processor.

[0096] The audio module 370 is used to convert digital audio signals into analog audio signals for output, and also to convert analog audio inputs into digital audio signals. The audio module 370 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 370 may be located in the processor 310, or some functional modules of the audio module 370 may be located in the processor 310.

[0097] The speaker 370A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The control device can listen to music or hands-free calls through the speaker 370A. For example, the speaker can play the comparison analysis results provided in the embodiments of this application.

[0098] The receiver 370B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the control device is answering a phone call or voice message, the receiver 370B can be brought close to the listener's ear to hear the voice.

[0099] Microphone 370C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 370C, inputting the sound signal into microphone 370C. The control device can have at least one microphone 370C. In some embodiments, the control device can have two microphones 370C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, the control device can also have three, four, or more microphones 370C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0100] In some embodiments, the control device may receive ultrasonic signals sent by other electronic devices via microphone 370C and identify the frequency and received intensity of the ultrasonic signals via processor 310.

[0101] The 370D headphone jack is used to connect wired headphones. The 370D headphone jack can be a USB 330 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0102] The sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, a barometric pressure sensor 380C, a magnetic sensor 380D, an accelerometer sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, an ambient light sensor 380L, a bone conduction sensor 380M, etc.

[0103] Buttons 390 include a power button, volume buttons, etc. Buttons 390 can be mechanical buttons or touch-sensitive buttons. The control device can receive button inputs and generate key signal inputs related to user settings and function control of the control device.

[0104] Motor 391 can generate vibration alerts. Motor 391 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 391 can also provide different vibration feedback effects for touch operations applied to different areas of the display screen 394.

[0105] Indicator 392 can be an indicator light, which can be used to indicate the charging status of the control device, changes in power level, or to indicate messages, missed calls, notifications, etc.

[0106] The SIM card interface 395 is used to connect SIM cards. The SIM card can be inserted into or removed from the SIM card interface 395 to achieve contact and separation with the control device. The control device can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 395 supports Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 395 simultaneously. The multiple cards can be of the same or different types.

[0107] Currently, smart home systems have more devices and functions than traditional home appliance systems. Furthermore, with the development of smart home technology, future smart home systems will encompass even more devices and implement even more functions. Therefore, a simpler, more intuitive, and natural way to control and manage them is needed.

[0108] Typically, to facilitate users' control and management of smart home devices through device management applications, users can transmit control commands to smart home devices through voice interaction and / or touch interaction.

[0109] Voice interaction refers to the control devices using speech recognition and speech synthesis technologies, allowing users to interact with the smart home system by speaking, and to issue various commands and queries. For example, users can use voice commands to turn on or off smart home devices such as lights, air conditioners, and curtains, or ask smart home devices for information such as the weather, news, and schedules.

[0110] Touch interaction refers to the use of touchscreens or gesture recognition technology to allow users to interact with smart home systems through touch or gestures, enabling various settings and adjustments. For example, users can use icons or menus on the touchscreen to select parameters for smart home devices in different modes or scenarios, or adjust parameters such as volume and brightness of individual devices.

[0111] The control device can control individual devices (also known as single devices) within a smart home system based on user input, or it can centrally control multiple electronic devices within a smart home system. The following examples illustrate this.

[0112] Example 1: Individual control of single devices based on user operations

[0113] The specific process of a control device individually controlling a single device based on user operation can be as follows: the control device responds to the user's click operation on the control card button of the target device or the drag operation of the slider on the touch interface to perform operations such as turning the target device on or off, and adjusting the brightness; or when the user is far away from the control device, the control device performs operations such as turning the target device on or off, and adjusting the brightness based on the user's voice command input.

[0114] In some implementations, the control device can control the target device via a touch interface based on user actions. For example, see [link to example]. Figure 4A The diagram shown illustrates the process of the control device controlling a single device provided in this embodiment, specifically the process of the control device controlling the "turn on the master bedroom background wall downlight". Wherein, Figure 4A Figure (a) shows the main interface of the control management application, which displays the devices, spaces, and "My Home" operation options that the device management application can control. It should be understood that the main interface may also include more or fewer operation options, and this embodiment of the application does not limit this.

[0115] like Figure 4A As shown, after the user starts the control management application in the control device, the following is displayed: Figure 4A Interface 401, shown in (a) above, displays operation options for various spaces, such as the master bedroom, bathroom, and children's room. The user performs actions such as... Figure 4AThe operation shown in (a) refers to the user clicking the operation option for the master bedroom, which displays as shown in [image / description]. Figure 4A The interface 402 shown in (b) displays operation options for different types of equipment in the master bedroom, such as the master bedroom television, lighting fixtures, audio system, and air conditioner. When the user clicks on the lighting fixture operation option on interface 402, the control device responds to the user's click and displays, as shown in (b). Figure 4A The interface 403 shown in Figure (c) displays operation options for different types of lighting devices, such as background wall downlights, table lamps, floor lamps, and bedside lamps. When the user clicks the background wall downlight operation option on interface 403, the control device responds to the user's click and displays, as shown in Figure (c). Figure 4A The interface 404 shown in Figure (d) displays the switch options corresponding to each downlight on the background wall. By clicking on the switch option, the user controls the device to complete the process of turning on the downlights on the background wall.

[0116] It can be seen that, through Figure 4A The illustrated operation process involves user-controlled devices. When turning on the master bedroom's accent wall downlight via a touchscreen interface, the user needs to sequentially select "Space," "Master Bedroom," and then "Master Bedroom Lighting Fixtures" on various screens. They then locate the "Actress Wall Downlight" option among the multiple lighting fixtures in the master bedroom and select it to turn on. The most tedious step is finding the "Actress Wall Downlight" option from the multiple lighting fixtures in the master bedroom, as the user needs to read the names of each different lighting fixture, determine if it corresponds to the desired fixture, and then click to complete the final control operation.

[0117] In other implementations, the control device can be controlled via voice interaction based on user input. For example, in a scenario where the control device is used to "turn on the master bedroom background wall downlight," the user needs to say to the control device in voice interaction mode, "Turn on the master bedroom background wall downlight," and the control device will then turn on the downlight based on this voice command. In this method, the user needs to pre-name each light in the space in the device management application, meaning each light corresponds to a unique voice command; otherwise, the operation will not be possible.

[0118] Smart home systems have more devices and functions than traditional home systems, and this number continues to grow. Furthermore, in smart home applications, the most important and frequently interacted lighting system is currently trending towards a "no-main-light design." This means that instead of a single main light in a space, multiple spotlights, LED strips, and other lighting devices create a more uniform lighting environment, making the light fixtures virtually invisible – achieving "seeing the light but not the lamp." In other words, multiple similar devices may exist in the same space. In this application scenario, whether users control individual devices via touch or voice, the operational and setup costs are high, impacting the user experience.

[0119] Example 2: Contextualized centralized control of devices based on user operations

[0120] Scenario-based centralized control refers to a control device packaging multiple smart home devices into a "scene" based on user needs, such as a movie-watching mode, a home mode, a relaxation mode, or a focus mode. Users only need to trigger this scene, and the control device responds to the user's control operation on the triggered scene mode, enabling control of multiple smart home devices of the same or different categories, without needing to control each individual device. In different scene modes, the control device keeps the smart home devices in preset on / off states or parameter states. For example, a user can create a "movie-watching mode," in which the curtains will be closed, the main light will be off, and the ambient lighting will be turned on to a dimmer setting.

[0121] In some embodiments, the control device can perform scenario-based centralized control of smart home devices via a touch interface based on user operations. For example, see [link to example]. Figure 4B The diagram shown is a schematic of the process of the control device in this embodiment controlling multiple devices in a scenario-based manner, involving the process of the control device controlling the "movie viewing mode" according to the user's operation.

[0122] like Figure 4B As shown, the user clicks Figure 4B After the "Scene" operation option shown in (a) is selected, the control device responds to the click operation by displaying the following: Figure 4B Interface 405, shown in (b), displays operation options corresponding to different scene modes. When a user clicks on the viewing mode operation option on interface 405, the control device responds to the user's click and displays, as shown in (b). Figure 4BInterface 406 is shown in (c). Interface 406 includes the on / off status or parameter status of each smart home device in movie-watching mode, as well as "Edit More" and "Execute" operation options. For example, only three lights in the lighting system can be turned on, the parameters of the air conditioning and ventilation system can be adjusted to a temperature of 24°C and a humidity of 40%, the curtains can be closed, and the audio system can be turned on. After the user clicks the "Execute" option in interface 406, the control device responds to the click operation and performs control operations on each device in movie-watching mode.

[0123] Using the control method described in Example 2 above, the control device can control multiple devices with a single click based on user operation. However, users need to pre-define various scenarios in the device management application, which also presents the problem of high upfront preparation costs for users. Furthermore, in the process of defining scenarios, it is also necessary to set the device parameters and startup status of each smart device under different scenarios, which also leads to problems such as "difficulty in selecting individual devices" and high user operation costs, affecting the user experience.

[0124] Furthermore, "scenario-based centralized control" is an integrated control process based on fixed execution rules. During the use of smart home devices, users are not always in the state corresponding to each scenario; they often want to fine-tune certain parameters of some devices. In this scenario, the same problems arise regarding the difficulty of selecting individual devices and the high operating costs for users.

[0125] In summary, all of the above-mentioned different implementation methods suffer from high user operating costs or upfront preparation costs. They also fail to allow users to intuitively and naturally select the one or more devices they want to control. Furthermore, due to the greater number of devices and functions in a smart home system, users experience a poor user experience when controlling smart home devices using traditional methods.

[0126] Therefore, this application provides a device control method that can solve the problems of difficult device selection and high user preparation costs in smart home scenarios. Users can conveniently and quickly control one or more devices within a target area in an intuitive and natural way, simplifying the operation process, improving the interactive efficiency of selecting and controlling target devices in smart home scenarios, and enhancing the user experience.

[0127] The following will combine Figures 5 to 14B The process of device control provided in the embodiments of this application will be explained and illustrated by way of example.

[0128] Figure 5 This is a schematic diagram of the interaction flow of a device control method provided in an embodiment of this application. See also: Figure 5 As shown, the method includes the following steps S501 to S506.

[0129] S501, the control device displays a display interface including spatial information, wherein the spatial information includes at least one area range.

[0130] In this embodiment, the display interface can be the main interface of a device management application (such as a smart living application), or other interfaces, such as the interface displayed in response to a user's left or right swipe on the main interface of the device management application, or the interface displayed under a certain operation mode (such as a regional operation mode). For example, after the device management application starts running, its main interface displays different operation modes, such as a single-device operation mode, a scenario-based centralized control operation mode, and a regional operation mode. Different operation modes correspond to different ways of operating smart home devices. Specifically, the single-device operation mode corresponds to the operation process of individually controlling a single device based on user operation, as given in Example 1 above; the scenario-based centralized control operation mode corresponds to the operation process of scenario-based centralized control of devices based on user operation, as given in Example 2 above; and the regional operation mode corresponds to the operation process of controlling smart home devices based on user operation and the spatial information displayed on the display interface. After the control device starts the device management application, it can display this interface according to the user operation.

[0131] Spatial information refers to information reflecting the spatial distribution characteristics of geographical entities. These characteristics include the entity's location, shape, spatial relationships between entities, and regional spatial structure. For example, this spatial information could be a floor plan of a home, a 3D model of the home's floor plan, or a 3D model showing the layout of interior decoration and furniture in a home. See also... Figure 6 The floor plan of the family environment shown in Figure (a) is available in [reference]. Figure 6 Figure (b) shows a 3D model of the family environment's floor plan. (See Figure b for details.) Figure 6 Figure (c) shows a 3D model of the interior furnishings layout in a home environment. This spatial information could also be a topographical map of a certain area of ​​a city, etc.

[0132] In some embodiments, the spatial information may originate from floor plans, 3D models, etc., stored on a host / server. The floor plan may be in an image format, such as an exchangeable image file (EXIF), portable network graphics (PNG), joint photographic experts group (JPEG), or web page image (Weppy, WebP) format. The 3D model may be in a format such as a motion capture tool (Filmbox, FBX), a standard 3D model (STEP), or an object file (OBJ) format. The 3D model can be displayed on the display interface using a 3D rendering engine.

[0133] In this embodiment, spatial information may be obtained and stored in the host / server through, but is not limited to, the following methods.

[0134] (1) Pre-set floor plans of different types of houses in different areas of various communities in some home decoration design software.

[0135] (2) Computer-aided design (CAD) drawings or floor plans of the house obtained from on-site surveying. Among them, the CAD drawings or floor plans of the house can be automatically generated into a 3D model by tools, or the 3D model drawings can be manually generated by 3D design software.

[0136] (3) A three-dimensional model of a house including soft furnishings, obtained by manually adding the layout information of soft furnishings to the three-dimensional model of the room.

[0137] (4) Using sensors with environmental depth perception, such as cameras and / or lidar, to scan and generate a 3D model of the house on site.

[0138] Normally, since the pre-set floor plan does not include the layout information of soft furnishings, the above two methods (3) or (4) can be used to obtain all the spatial information including "floor plan" and "soft furnishings" at once, and can be updated at any time.

[0139] In this embodiment, when the spatial information is a 3D model or floor plan of the home environment, the spatial information includes the layout of various spatial areas in the user's home environment, as well as directional landmarks for each room. Here, directional landmarks refer to iconic entities used to indicate a specific area within the spatial region. For example, see... Figure 6The floor plan shown in Figure (a) includes several spatial areas such as the master bedroom, secondary bedroom, living room, dining room, kitchen, and bathroom. Each spatial area includes various directional objects, such as the sofa, coffee table, and living room door in the living room; the master bed, dressing table, and desk in the master or secondary bedroom; and the vanity and toilet in the bathroom. The term "sofa" refers to the area in the living room including the sofa's location, and "master bed" refers to the area in the master bedroom including the bed's location.

[0140] It should be noted that the area involved in this embodiment can be an area encompassing the entire spatial information, such as the area encompassing all spatial areas included in the floor plan of a family home; it can also be the area corresponding to different spatial areas in the spatial information, such as the area encompassing the master bedroom in the floor plan, or the area encompassing the living room, etc.; it can also be the area corresponding to the directional reference objects in each spatial area, such as the area corresponding to the location of the sofa in the living room, or the area corresponding to the location of the master bedroom bed in the master bedroom, etc.

[0141] In this embodiment, the area range can be, but is not limited to, the following forms.

[0142] (1) Scope of point-like areas

[0143] For example, the area can be any point in the spatial information, see [link to relevant documentation]. Figure 7A As shown in Figures (a) and (b), the area includes point area 1 corresponding to the dining room, point area 2 corresponding to the living room, and point area 3 corresponding to the master bedroom in Figure (a), and point area 1 corresponding to the living room, point area 2 corresponding to the bathroom, and point area 3 corresponding to the dining room in Figure (b).

[0144] (2) Two-dimensional region range

[0145] For example, when the spatial information is a floor plan, the area can be a preset planar range surrounding any point in the spatial information. This point can be the center point of the preset planar range, such as the center of a circular area; or it can be any point within the preset planar range, such as the vertex of a rectangular area. See also Figure 7B As shown in the diagram, the spatial information includes planar area 1 corresponding to the dining room, planar area 2 corresponding to the living room, and planar area 3 corresponding to the master bedroom. These planar areas can be as follows: Figure 7A The circular area shown can also be other shapes, such as triangles, rectangles, ellipses, polygons, or other irregular shapes.

[0146] (3) Three-dimensional region range

[0147] For example, when the spatial information is a 3D floor plan, the area can be a preset spatial range surrounding any point in the spatial information. This point can be the center of the preset spatial range, such as the center of a sphere; or any other point within the preset spatial range, such as the vertex of a cone. See also... Figure 7C As shown in the diagram, the spatial information includes spatial area 1 corresponding to the living room, spatial area 2 corresponding to the bathroom, and spatial area 3 corresponding to the dining room. These areas can be as follows: Figure 7B The spherical space region shown can also be a cube or cuboid space region, etc.

[0148] It should be understood that Figure 7A , Figure 7B and Figure 7C The spatial information shown may include more or fewer, larger or smaller areas. Each area is equipped with smart home devices corresponding to that area. The location and number of areas can be specifically set by maintenance personnel in the backend according to actual needs. This embodiment does not list them all.

[0149] In some embodiments, when displaying a display interface including spatial information, the control device first determines its orientation in physical space, and then ensures that the display direction of the spatial information in a first direction of the control device is consistent with the orientation of the control device. The first direction of the control device can be an upward-facing orientation. When the orientation of the control device in physical space changes, the display direction of the spatial information in the control device's display interface is adjusted according to the changed orientation. For example, see [link to example]. Figure 8A As shown, when the user holds the control device, its orientation in physical space is as follows: Figure 8A When the first orientation (i.e., position orientation) is shown in Figure (a), the control device displays as follows: Figure 8A The spatial information display interface 801a shown in Figure (b) has the control device facing upwards (also referred to as the first direction) in line with the user's first orientation in the physical space. See also... Figure 8B As shown, when the user holds the control device, its orientation in physical space is as follows: Figure 8B When the second orientation is shown in Figure (a), the control device displays as follows: Figure 8BThe spatial information display interface 801b shown in Figure (b) has the control device facing upwards in a direction consistent with the user's second orientation in the physical space. As the user's orientation while holding the control device changes in the physical space, the display direction of the spatial information on the control device's interface adaptively adjusts accordingly. In other words, the "left-up-right-down" in the spatial information corresponds to "left-front-right-back" relative to the user in the actual physical space, consistent with the user's map viewing habits.

[0150] It should be noted that, in this embodiment, the upward display direction of the control device can be defined in the following way:

[0151] (1) If the control device is placed vertically (or at an angle greater than 0° with the horizontal plane), then “upward” means the direction from the bottom of the control device to the top.

[0152] (2) If the control device is placed horizontally, the "upward" direction is inherited from the upward direction before it was placed horizontally (when it was placed vertically); or it can be based on sensors such as the front camera to obtain the direction of the person (judged according to the direction of the face), and adjust the upward direction of the interface according to the direction of the person.

[0153] In this way, users can more intuitively view the spatial information in the display interface, such as a 3D model of the home's floor plan or a floor plan of the home's floor plan, and map it to the actual physical space. Furthermore, the view orientation of the spatial information can be adaptively adjusted according to the location and orientation of the control device in the physical space, thereby improving the user experience.

[0154] In some embodiments, the orientation of the control device relative to the indoor environment is obtained by:

[0155] (1) The control device is fixed at a certain position in the indoor environment, and the orientation information can be pre-stored in the storage module of the control device.

[0156] (2) The control device is equipped with a magnetic base accessory that is fixed to the wall. The magnetic base encodes orientation information via near field communication (NFC) technology. When the control device comes into contact with the magnetic base, it can obtain the orientation information via NFC.

[0157] (3) The north-south orientation of the indoor environment is pre-configured in the control equipment. The orientation of the control terminal in the Earth coordinate system is obtained by the magnetometer and other sensors built into the control equipment, so that the orientation of the control equipment relative to the indoor environment can be calculated.

[0158] S502, the control device determines a first area range from at least one area range according to a first control operation.

[0159] The first control operation includes touch operations such as clicking, long pressing, double-clicking, and swiping.

[0160] In this embodiment, determining the first region range from at least one region range based on a first control operation includes: responding to a user's first control operation on spatial information, the control device determines the first region range from at least one region range included in the spatial information based on the projection coordinates of a certain region range within the spatial information where the first control operation falls. For example, when a user clicks any location on the control device's display interface, the control device projects the clicked location onto a planar or three-dimensional map of the spatial information, thus obtaining the projection coordinates corresponding to the clicked location in the planar or three-dimensional map, and then determines the first region range based on the region range indicated by the projection coordinates. Optionally, if no corresponding region range exists for the projection coordinates, the region range adjacent to the projection coordinates is determined as the first region range.

[0161] In some embodiments, the region range includes a point-like region range corresponding to the projection coordinates of the first control operation onto the spatial information. This region range can be a planar coordinate (x...) n y n It can also be a spatial coordinate (x) n y n , z n For example, see Figure 9A As shown, the projected coordinates of each point in the spatial information correspond to a point-like region. For example, the projected coordinates (x1, y1) or (x1, y1, z1) of point 1 fall on the location of the coffee table in the living room in the spatial information, and its corresponding region is point-like region 1. Point-like region 1 can include electronic devices such as coffee table ceiling lights; the projected coordinates (x2, y2) or (x2, y2, z2) of point 2 fall on the location of the sofa in the living room in the spatial information, and its corresponding region is point-like region 2. Point-like region 2 can include electronic devices such as sofa wall lights, and so on.

[0162] In other embodiments, the region range includes a two-dimensional or three-dimensional region range corresponding to the projection coordinates of the first control operation onto the spatial information. For example, when the projection coordinates of the first control operation onto the spatial information are (x1, y1), the region range it refers to is a circular planar region range with a radius of r and the geometric center of the projection coordinates (x1, y1), or a rectangular spatial region range (i.e., a three-dimensional region range) with the geometric center of the projection coordinates (x1, y1, z1) and the length, width, and height being l, w, and h, respectively. See also, for an example... Figure 9BAs shown, the projected coordinates of each point in the spatial information correspond to a two-dimensional or three-dimensional region. For example, the projected coordinates (x4, y4) or (x4, y4, z4) of point 4 fall on the location of the master bedroom bed in the spatial information, and its corresponding region is a two-dimensional / three-dimensional region 4 with the projected coordinates (x4, y4) or (x4, y4, z4) as the geometric center. This two-dimensional / three-dimensional region 4 may include electronic devices such as the master bedroom bed lamp; the projected coordinates (x5, y5) or (x5, y5, z5) of point 5 fall on the location of the master bedroom wardrobe in the spatial information, and its corresponding region is a two-dimensional / three-dimensional region 5 with the projected coordinates (x5, y5) or (x5, y5, z5) as the geometric center. This two-dimensional / three-dimensional region 5 may include electronic devices such as the smart wardrobe, and so on.

[0163] In some embodiments, the area range includes the area identification information (IDentity, ID) corresponding to the projection coordinates of the first control operation projected into the spatial information, referred to as the area range indicated by the area ID. Each area range corresponds to an area ID, which can be the name of a spatial area, such as "bedroom," "living room," "movie viewing area," "relaxation area," etc.; or it can be the name of a directional object, such as "living room sofa," "coffee table," "master bedroom bed," etc. For example, the area range indicated by the area ID "living room sofa" is a preset area range corresponding to "living room sofa"; the area range indicated by the area ID "bedroom" is a preset area range corresponding to "bedroom." See also, for examples... Figure 9C As shown, the projected coordinates of each point in the spatial information correspond to a region range indicated by a region ID. For example, the projected coordinates of point 9 (x9, y9) or (x9, y9, z9) fall on the location of the desk in the secondary bedroom in the spatial information, and its corresponding region range is the region range corresponding to the region ID "desk". The region range corresponding to the region ID "desk" can include electronic devices such as desk lamps; the projected coordinates of point 14 (x9, y9) are also shown. 14 y 14 ) or (x 14 y 14 , z 14 The location of the restaurant tea bar area in the spatial information is the area range corresponding to the area ID "tea bar area". The area range corresponding to the area ID "tea bar area" can include electronic devices such as tea bar machines, and so on.

[0164] It should be noted that each area may include one or more electronic devices. For example, the area corresponding to the area ID "Tea Bar Area" may include a tea bar machine, as well as other electronic devices such as tea bar machine lights, water dispensers, etc. Specific settings can be configured during setup; this embodiment does not list them all.

[0165] For example, the process of the control device determining the first area range is described using the area range indicated by the area identification information where the projected coordinates fall as an example.

[0166] See Figure 10A As shown, the user clicks Figure 10A When the user clicks at the location shown in Figure (a), the control device projects the user's click location onto the floor plan, obtaining that the projected coordinates of the user's click location fall within the planar area corresponding to the "living room sofa". Then, based on the area indicated by the projected coordinates being the planar area indicated by the area identification information, the planar area corresponding to the "living room sofa" is determined as the first area. Figure 10A When the user clicks at the location shown in Figure (b), the control device projects the user's click location onto the floor plan. This results in the projected coordinates of the clicked location falling within the area corresponding to the "master bedroom bed." Then, based on the area indicated by the projected coordinates, which includes the area indicated by the area identifier information, the area corresponding to the "master bedroom bed" is determined as the first area. It can be understood that when the spatial information is a 3D floor plan, see [reference needed]. Figure 10B As shown in Figure (a), when the user clicks on the sofa location in the 3D floor plan map, the control device determines the spatial area corresponding to the sofa as the first area according to the above determination process; see also Figure 10B As shown in Figure (b), when the user clicks on the location of the master bedroom bed in the 3D floor plan, the control device determines the spatial area corresponding to the master bedroom bed as the first area according to the above determination process.

[0167] It should be noted that the first area can be one or multiple, depending on the user's actions.

[0168] In some embodiments, the control device receives a user's selection operation on N regions of spatial information, for example, the user can select N regions of spatial information by "multiple clicks"; it also receives a user's cancellation operation on M regions out of the N regions, for example, the user can cancel the selection on the M regions out of the N regions by "sliding". In this case, the control device determines the NM regions as the first region range, where N ≥ M ≥ 1. For example, see [link to documentation]. Figure 10C As shown, the user selects area range ①, area range ②, and area range ③ through "multiple clicks," and the control device identifies area range ①, area range ②, and area range ③ as the first area range; see also Figure 10DAs shown, the user draws out area ③ by "sliding", and the control device determines the remaining area outside the drawn area as the first area. That is, the control device determines area ① and area ② as the first area.

[0169] In some embodiments, in response to a user's first control operation on spatial information, the control device determines a first area range, extracts candidate control instructions related to the first area range, and displays element controls of the candidate control instructions on a display interface. These element controls are used to instruct the user to input control instructions based on the content corresponding to the element control.

[0170] In one implementation, see Figure 11A As shown in the image, the user clicked on the "living room sofa." The control interface displays element controls for devices near the "living room sofa," such as options to brighten this area, darken other areas, or turn off the lights in this room. The user can trigger the corresponding control by clicking on the operation option corresponding to the element control or by speaking the corresponding text within these element controls.

[0171] In another implementation, see Figure 11B As shown, the user clicked on the "living room sofa," and the control interface displays element controls for devices near the "living room sofa," such as the on / off switch and brightness adjustment controls for the coffee table ceiling light, the sofa wall light, and the air conditioner's on / off switch and temperature adjustment controls. The user can trigger corresponding controls by clicking on the element controls for each device or adjusting the sliders on the element controls.

[0172] S503, the control device receives the second control operation.

[0173] The second control operation includes voice control operations (such as voice input) or touch control operations (such as touch input). Receiving the second control operation by the electronic device includes the electronic device receiving control commands from the user via voice input or touch input.

[0174] It should be noted that when a user expresses the type of electronic device they want to control via voice input or touch input, the control command included in the second control operation may or may not contain directional information. In this case, after receiving the second control operation, the control device needs to analyze the control command to determine whether it contains directional information. Optionally, the second control operation can be performed by the user through voice input or touch input. Figure 10A or Figure 10B The operation options or voice or touch operations of the control cards in the control interface shown.

[0175] Among them, directional information refers to the relevant content that describes directional information, including but not limited to: "here", "this house", "other places", "other rooms", "around", "next to", and "nearby".

[0176] In this embodiment, the control commands input by the user via voice include, but are not limited to, the following: "Make this brighter," "Make other places darker," "Turn off the lights in this room," "Turn off the lights in other rooms," "Turn on the lights," "Clean this place," and "Don't let the wind blow here," etc.

[0177] User-input control commands via touch include, but are not limited to: "turn lights on / off", "brightness adjustment", "area cleaning", and "airflow adjustment".

[0178] In this embodiment, different control commands correspond to different types of electronic devices, or different types of electronic devices have their own sets of control commands. See [link to relevant documentation]. Figure 12A As shown, for example, "turning lights on / off" corresponds to "lighting equipment," "area cleaning" corresponds to "robot vacuum cleaner," and "airflow adjustment" corresponds to "air conditioner / fan," etc. Simultaneously, when each electronic device is adaptively configured with spatial information, it also associates different types of electronic devices or different types of the same device with the area range in the spatial information. For example, see [link to example]. Figure 12B As shown, the spatial information includes fields describing the area where the electronic device is located, as well as the spatial area it occupies (e.g., living room, master bedroom, or dining room), area IDs (e.g., area ID "sofa", area ID "coffee table", and corresponding projected coordinates (e.g., (x1, y1) or (x1, y1, z1), (x2, y2) or (x2, y2, z2) etc.). Different electronic devices may have one or more area IDs due to their different locations. For example, a spotlight located on the sofa's background wall may have an area ID "sofa" and an additional area ID "against the wall" to specifically describe its location. The number of area IDs may vary between different electronic devices.

[0179] As shown above, user control commands via voice or touch input can include the type of electronic device to be controlled and the action to be performed on that device, such as "turn on the light," "turn off the light," or "shut down the computer." Here, "turn on" and "turn off" are the actions to be performed, and the light and computer are the electronic devices to be controlled. Alternatively, the commands can include both the type of electronic device to be controlled and the action to be performed, as well as directional information, such as "make this brighter," "make other places darker," or "clean this area." Here, "brighter," "darker," and "clean" correspond to the electronic devices being a light, a lamp, and a robot vacuum cleaner, respectively, and the corresponding actions are adjusting brightness, dimming, and cleaning. "This area," "other places," and "here" correspond to directional information. Therefore, after receiving different types of control commands, the control device needs to analyze the commands and then determine the operation to be performed based on the analysis results.

[0180] In some embodiments, the control device extracts semantic information from user voice or touch input control commands using natural language processing (NLP) technology or based on a large language model (LLM). When the extracted voice information includes any of the aforementioned directional reference information, the control command is determined to contain directional reference information. For example, if the user inputs the control command for the second control operation as "turn off the lights in other rooms," and the control device determines through NLP technology that the control command includes the voice information "other rooms," then the control command for the second control operation is determined to contain directional reference information.

[0181] S504, the control device determines a target area range from at least one area range based on a second control operation and a first area range, wherein the first area range is the same as or different from the target area range.

[0182] In step S502 above, the control device, responding to the user's first control operation on spatial information, has already determined a first area range from at least one area range. In this step, the control device determines the target area range to be controlled based on the already determined first area range and the second control operation. The first area range and the target area range may be the same or different. That is, the control device needs to further determine, through the second control operation, whether the target area range the user wants to control is the first area range indicated by the first control operation, or another area range that is different from or related to the first area range. The control command based on the second control operation includes the type of electronic device to be controlled and the action to be performed on that electronic device. Therefore, after determining the target area range, the control device also immediately determines the electronic devices to be controlled within that target area range.

[0183] In one implementation, see Figure 13A As shown, in response to the user's first control operation, the control device has already determined the area corresponding to the area ID "living room sofa" as the first area range. Then, after receiving the user's second control operation, "Light up here," via voice input, the control device extracts the semantic information from the second control operation. After analyzing this semantic information, the control device determines that it includes the directional pronoun "here." Based on the directional pronoun "here," the target area range is then determined as the first area range, i.e., the area corresponding to the area ID "living room sofa." At this point, the first area range is the same as the target area range.

[0184] In this example, if the control device receives a second control operation from the user via voice input, "Turn on the lights in this room," the semantic information in this second control operation is extracted and analyzed. It is determined that the voice information includes the locative pronoun "this room." Based on the locative pronoun "this room," the control device defines the target area as the entire living room area, including the area corresponding to the area ID "living room sofa," and not just the area corresponding to the area ID "living room sofa." In this case, the first area range differs from the target area range; the target area range is a larger area that includes the first area range.

[0185] In another implementation, see Figure 13B As shown, after receiving the second control operation "turn off the lights in other rooms" input by the user via voice, the control device extracts and analyzes the semantic information in the second control operation, and determines that the semantic information includes the directional pronoun "other rooms". Based on the semantic information "other rooms", the target area is determined to be an area other than the first area. At this time, the first area is different from the target area, and the target area does not include the first area.

[0186] In another implementation, see Figure 13C As shown, after receiving the second control operation of "turn off the lights" input by the user via voice, the control device extracts and analyzes the semantic information in the second control operation. After determining that the semantic information does not include directional reference information, the control device determines the target area range as the first area range. At this time, the first area range is the same as the target area range.

[0187] As can be seen from the above, when the control device determines the target area range, the content related to azimuth information in its control command can also play a decisive role. When the control command includes azimuth information, the control device determines the target area range based on the azimuth information and the first area range; when the control command does not include azimuth information, the first area range is determined as the target area range.

[0188] Furthermore, in some other implementations, the control commands input by the user via voice or touch may not contain directional information, only the device to be controlled and the action to be performed. However, the area the user actually wants to control is not limited to the first area. For example, the control device responds to the user's first control operation of clicking on the location of the "living room sofa" and receives the user's second control operation of "turn on the lights" via voice or touch. In this case, although the user is pointing to a specific "living room sofa" location, what they are expressing is wanting to control the lights in the entire viewing area to turn on, or to adjust the viewing area to the default lighting mode. In this embodiment, this situation is considered as the case where the control command of the second control operation contains "implicit" directional information.

[0189] In response to the above situation, the control device in this embodiment can determine the target area range based on different area attributes. These area attributes include specific areas and broad areas. There is a hierarchical correspondence between specific and broad areas. Specific areas are characterized by having no lower-level area range, such as "sofa" or "dining table." Broad areas are characterized by including other sub-areas, such as "viewing area" or "dining area." A "viewing area" can include other sub-areas such as "sofa," "TV cabinet," and "coffee table," while a "dining area" can include other sub-areas such as "tea bar area" and "dining table area."

[0190] For example, see Figure 13DThe diagram shown illustrates the hierarchical relationship of the area range provided in this application embodiment. As shown, the broad area "living room" corresponds to sub-areas such as "viewing area" and "dining area," and the sub-area "viewing area" corresponds to sub-areas such as "coffee table" and "TV cabinet." The broad area "dining area" corresponds to sub-areas such as "dining table" and "tea bar." Since the sub-areas "coffee table," "TV cabinet," "dining table," and "tea bar" do not have a lower-level area range, they can also be called specific areas. However, the sub-areas "viewing area" and "dining area" also contain other sub-areas; therefore, they can also be called broad areas. During the process of determining the target area range, when the control device determines the first area range as a "specific area" through the first control operation, it determines the upper-level area range of the first area range as the target area range based on the second control operation containing "implicit" directional reference information and the first area range. For example, if the user clicks on the location of the "sofa," the control device determines the parent area of ​​the "sofa" area as the "viewing area" based on the area ID "sofa." In this case, the control device identifies the "viewing area" as the target area. When the first area determined by the control device through the first control operation is a "broad area," the second control operation, which includes "implicit" directional information, and the first area, determine this first area, i.e., the "broad area," as the target area. For example, if the user clicks on a location next to the sofa, since the projected coordinates of this location correspond to the "viewing area" area but not to the area corresponding to the area ID "sofa," the control device identifies the first area, i.e., the "broad area," as the target area. In this implementation, if the user only wants to turn on the light on the "sofa," the user needs to explicitly input a control command containing directional information when performing the second control operation, such as "turn on the light here" or "make it brighter here."

[0191] In this way, the control device can determine whether it only needs to control the electronic devices corresponding to the area selected by the user, thus enabling a more accurate determination of the target area.

[0192] S505, The control device determines the electronic equipment within the target area.

[0193] After determining the target area, the control device can determine the electronic device to be controlled and the actions to be performed by that electronic device based on the control instructions in the second control operation.

[0194] For example, see Figure 14AAs shown, if the user's voice input command is "Make this brighter," the control device determines the electronic device as a "lamp" based on this command. The action required from this electronic device is "Increase brightness." At this time, the control device's display shows the message "Nearby lights have been brightened for you," along with element controls for nearby lighting devices. These element controls allow the user to make secondary adjustments if they are not satisfied with the brightness adjusted by the control device. See also... Figure 14B As shown, if the user's voice input command is "turn off the lights in other rooms," the control device determines the electronic device as "light" based on this command, and the action required by the electronic device is "turn off." In this case, the control device's display screen shows the message "The lights in other rooms have been turned off for you." If the user's voice input command is "make other places darker," the control device determines the electronic device as "light" based on this command, and the action required by the electronic device is "reduce brightness." Similarly, the control device's display screen can show the message "The lights in other places have been dimmed for you."

[0195] In the method provided in this embodiment, the control device, responding to the user's first control operation on spatial information, determines a first area range from at least one area range, and then, based on the control command of the second control operation in this step, can more accurately determine the target area range that the user needs to control and the electronic devices within that target area range. When there are multiple electronic devices of the same or different types in the user's home, this method can determine the specific device that the user needs to control.

[0196] S506, the control device controls the corresponding electronic equipment within the target area according to the second control operation.

[0197] In some embodiments, step S506 specifically includes the following steps S506a to S506b.

[0198] S506a, the control device sends operation commands to the electronic device.

[0199] The operation instruction is used by the electronic device to perform the corresponding operation upon receiving the instruction. The electronic device refers to all electronic devices within the target area; it may be one or more, depending on the type and number of electronic devices configured within the target area.

[0200] S506b: The electronic device executes the corresponding operation according to the received operation instruction.

[0201] After receiving an operation command, the electronic device performs the corresponding action. For example, turning the bedside lamp on / off, or turning the air conditioner on / off.

[0202] It should be noted that in some other application scenarios, the second control operation input by the user may include multiple control commands. For example, after the user inputs the first control command "turn off the lights in other rooms" via voice, they immediately input the second control command "turn on the air conditioner" into the control device.

[0203] In this situation, the control device can first determine the first target area range and the electronic devices within that range based on the first control instruction in the first control operation and the first area range, and then control the electronic devices within the first target area to perform corresponding operations according to the first control instruction. Then, based on the second control instruction in the first control operation and the first area range, the control device can determine the second target area range and the electronic devices within that range, and then control the electronic devices within the second target area to perform corresponding operations according to the second control instruction.

[0204] The device control method provided in this application allows the control device to conveniently and quickly control various types of electronic devices in a smart home scenario, enabling both single-device control and centralized control of multiple electronic devices. This improves the efficiency of user interaction in selecting and controlling target devices within a smart home environment. Furthermore, this method eliminates the need for users to pre-name or define scenes for each electronic device, reducing operational and upfront costs and enhancing the user experience.

[0205] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0206] This application also provides a device control apparatus for controlling equipment; see [link to relevant documentation]. Figure 15 As shown, the device includes a display module 1501, a first determination module 1502, a second determination module 1503, and a control module 1504. The functions of each module are as follows.

[0207] Display module 1501 is used to display a display interface including spatial information, wherein the spatial information includes at least one area range.

[0208] The first determining module 1502 is used to determine a first region range from at least one region range according to a first control operation.

[0209] The second determining module 1503 is used to determine a target area range from at least one area range based on the second control operation and the first area range, wherein the first area range is the same as or different from the target area range.

[0210] The control module 1504 is used to control the corresponding electronic devices within the target area according to the second control operation.

[0211] This application also provides a control device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the device control method shown in the above embodiments.

[0212] This application also provides a chip, see [link to example]. Figure 16 As shown, the chip includes a processor and a memory, in which a computer program is stored. When the computer program is executed by the processor, it implements the device control methods described in the above embodiments.

[0213] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the device control methods provided in the above embodiments.

[0214] This application also provides a computer program product, which includes a computer program that, when run by a controlled device, enables the controlled device to implement the device control methods provided in the above embodiments.

[0215] It should be understood that the processor mentioned in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0216] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0217] In the embodiments provided in this application, the division of each framework or module is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple frameworks or modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0218] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0219] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0220] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0221] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A device control method, characterized in that, Applied to control equipment, the method includes: The display interface includes spatial information, wherein the spatial information includes at least one area range; According to the first control operation, a first region range is determined from the at least one region range; Based on the second control operation and the first area range, a target area range is determined from the at least one area range, wherein the first area range is the same as or different from the target area range, and the second control operation is used to determine the relationship between the target area range and the first area range; According to the second control operation, the corresponding electronic devices within the target area are controlled.

2. The method according to claim 1, characterized in that, Determining the first region range from the at least one region range according to the first control operation includes: Based on the first control operation, determine the projection coordinates of the first control operation onto the spatial information; When the projected coordinates have a corresponding region, the region corresponding to the projected coordinates is determined as the first region; or, When there is no corresponding region range for the projected coordinates, the region range adjacent to the projected coordinates is determined as the first region range.

3. The method according to claim 1 or 2, characterized in that, Determining the target region range from the at least one region range based on the second control operation and the first region range includes: Perform semantic analysis on the control instructions in the second control operation; When the control command includes directional reference information, the target area range is determined based on the directional reference information and the first area range.

4. The method according to claim 3, characterized in that, The method further includes: When the control command does not include directional reference information, the first area range is determined as the target area range.

5. The method according to claim 3, characterized in that, The method further includes: When the control command does not include directional reference information, the next higher level region of the first region is determined as the target region.

6. The method according to any one of claims 1-2 and 4-5, characterized in that, The spatial information includes: A floor plan showing the layout of the home; or, A 3D model of the floor plan of the home environment; or, A 3D model of the layout of soft furnishings and furniture in a home environment.

7. The method according to any one of claims 1-2 and 4-5, characterized in that, The area includes: The first control operation is projected onto the point-like region corresponding to the projection coordinates in the spatial information; or... The first control operation is projected onto a two-dimensional or three-dimensional region corresponding to the projection coordinates in the spatial information; or... The first control operation is projected onto the area indicated by the region identifier information corresponding to the projection coordinates in the spatial information.

8. The method according to any one of claims 1-2 and 4-5, characterized in that, Determining the first region range from the at least one region range according to the first control operation includes: Receive user selection operations for N regions in the spatial information; Receive cancellation requests from users for M of the N regions; The first region is defined as NM regions, where N ≥ M ≥ 1.

9. The method according to any one of claims 1-2 and 4-5, characterized in that, After determining a first region range from the at least one region range according to a first control operation, and before determining a target region range from the at least one region range according to a second control operation and the first region range, the method further includes: Extract candidate control instructions related to the first region range; The candidate control command element controls are displayed on the display interface. The element controls are used to instruct the user to input the control command according to the content corresponding to the element controls.

10. The method according to any one of claims 1-2 and 4-5, characterized in that, The second control operation includes voice control or touch control.

11. The method according to any one of claims 1-2 and 4-5, characterized in that, The display includes a spatial information display interface, comprising: Determine the orientation of the control device in physical space; The display direction of the spatial information in the first direction of the control device is kept consistent with the position orientation of the control device.

12. The method according to claim 11, characterized in that, The method further includes: When the orientation of the control device in physical space changes, the display direction of the spatial information on the control device display interface is adjusted according to the changed orientation.

13. A control device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1 to 12.

14. A device control system, characterized in that, The device includes a control device and a plurality of electronic devices connected to the control device, the control device being configured to perform the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 12.

16. A chip, characterized in that, The chip includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 12.

Images