Device control method, electronic device, storage medium and computer program product

Abstract

The invention provides an equipment control method and device, electronic equipment, a storage medium and a computer program product, and relates to the technical field of Internet of Things. The method comprises the following steps: acquiring a voice message acquired by a first device; generating a first control instruction based on the voice message to instruct a second device to execute a corresponding target operation in response to the first control instruction; the second equipment is intelligent equipment different from the first equipment; and generating a second control instruction based on the detected motion data and / or the voice message from the first device to instruct the second device to adjust a corresponding target parameter in response to the second control instruction. According to the invention, the problem of poor convenience and flexibility of equipment control in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of Internet of Things, and specifically, to a device control method, electronic device, storage medium and computer program product. Background Art

[0002] With the development of Internet of Things technology, the control of smart devices has been gradually and widely used in different scenarios. For example, the application scenarios include but are not limited to smart home scenarios, office scenarios, entertainment scenarios, traffic safety scenarios, etc.

[0003] At present, the control of smart devices mainly relies on control devices (such as remote controls), voice, action, and automated control solutions configured in the client. However, the above control methods can either only control a single smart device, or users need to repeat multiple steps when controlling the same smart device to perform different actions, resulting in poor control convenience and flexibility of smart devices and failure to meet user needs.

[0004] Therefore, how to improve the convenience and flexibility of equipment control remains to be solved. Summary of the invention

[0005] The present application provides a device control method, apparatus, electronic device, storage medium and computer program product, which can solve the problem of poor convenience and flexibility of device control in related technologies. The technical solution is as follows:

[0006] According to one aspect of the present application, a device control method includes: acquiring a voice message collected by a first device; generating a first control instruction based on the voice message to instruct a second device to perform a corresponding target operation in response to the first control instruction; the second device is a smart device different from the first device; generating a second control instruction based on the detected motion data from the first device and / or the voice message to instruct the second device to adjust the corresponding target parameters in response to the second control instruction.

[0007] According to one aspect of the present application, a device control apparatus comprises: a message acquisition module for acquiring a voice message collected by a first device; a first instruction generation module for generating a first control instruction based on the voice message to instruct a second device to perform a corresponding target operation in response to the first control instruction; the second device is a smart device different from the first device; and a second instruction generation module for generating a second control instruction based on the detected motion data from the first device and / or the voice message to instruct the second device to adjust the corresponding target parameters in response to the second control instruction.

[0008] In an exemplary embodiment, the first instruction generation module is used to perform voice recognition processing on the voice message to obtain the candidate device pointed to by the voice message and the target operation; if the clarity of the candidate device pointed to by the voice message is fuzzy, then according to the set association conditions, the second device is obtained by associating the candidate device pointed to by the voice message; the second device is a smart device with the same device attributes as the candidate device; based on the second device and the target operation, the first control instruction is generated to instruct the second device to perform the target operation.

[0009] In an exemplary embodiment, the first instruction generation module is also used to determine the space where the device is located based on the set association condition; associate the candidate device pointed to by the voice message with the space where the device is located to obtain the second device; the second device refers to the candidate device located in the space where the device is located.

[0010] In an exemplary embodiment, the first instruction generating module is further configured to use the candidate device as the second device if the clarity of the candidate device to which the voice message is directed is clear.

[0011] In an exemplary embodiment, the apparatus is further configured to detect whether the first device is in a first state during the process of the second device performing the target operation; if yes, obtain motion data of the first device corresponding to the first state.

[0012] In an exemplary embodiment, the apparatus is further configured to control the first device to enter a dormant state if it is detected that the first device is not in the first state, until the voice message is acquired again.

[0013] In an exemplary embodiment, the apparatus is further used to detect whether a motion operation is triggered on a component configured on the first device when the first device maintains a second state; if so, obtain motion data of the first device corresponding to the second state.

[0014] In an exemplary embodiment, the second instruction generation module is also used to determine the target parameter to be adjusted of the second device based on the voice message and / or the motion data, and determine the adjustment ratio corresponding to the target parameter according to the motion data; and generate the second control instruction instructing the second device to adjust the target parameter according to the adjustment ratio.

[0015] In an exemplary embodiment, the second instruction generating module is further used to determine the motion type of the first device in the first state based on the motion data if the voice message does not include the target parameter; if it is monitored that the motion type of the first device meets the trigger condition in the configured first automation scheme, the target parameter associated with the satisfied trigger condition is extracted from the first automation scheme; the first automation scheme controls the second device to adjust the target parameter by monitoring whether the motion type of the first device meets the trigger condition.

[0016] In an exemplary embodiment, the second instruction generation module is also used to determine at least one candidate parameter that can be adjusted by the second device if the voice message does not contain the target parameter, and obtain the configured priority of each candidate parameter; and use the candidate parameter with the highest priority as the target parameter.

[0017] In an exemplary embodiment, the second instruction generating module is also used to determine the motion parameters of the first device indicated by the motion data, wherein the motion parameters include one or more of motion direction, motion amplitude, motion angle, motion duration, motion speed, and acceleration; and determine the adjustment ratio corresponding to the target parameter based on the motion parameters of the first device.

[0018] In an exemplary embodiment, the apparatus is further used to detect whether there is a wake-up operation triggered for the first device; if so, the first device is controlled to enter a wake-up state from a sleep state, so that the first device in the wake-up state collects the voice message.

[0019] In an exemplary embodiment, the device is also used to determine the time interval between the acquisition time of the voice message and the detection time of the motion data; if the time interval does not exceed the set threshold, it is determined that the control object of the first device is the second device, so as to generate the second control instruction instructing the second device to adjust the corresponding target parameters based on the motion data and / or the voice message.

[0020] In an exemplary embodiment, the apparatus is further configured to monitor the motion type of the first device in the first state if the time interval exceeds the set threshold; extract the target operation associated with the satisfied trigger condition from the second automation scheme if the motion type of the first device in the first state is monitored to satisfy the trigger condition in the second automation scheme; the second automation scheme controls the target device to perform the target operation by monitoring whether the motion type of the first device satisfies the trigger condition; and generate a third control instruction based on the target device and the target operation to instruct the target device to perform the target operation in response to the third control instruction.

[0021] According to one aspect of the present application, a device control system includes: a first device, a second device, and a control end; wherein the first device is used to collect voice messages and send the voice messages to the control end; the control end is used to generate a first control instruction based on the voice message; the second device is used to receive the first control instruction sent by the control end, and perform a corresponding target operation in response to the first control instruction; the first device is also used to detect motion data and send the motion data to the control end; the control end is also used to generate a second control instruction based on the motion data and / or the voice message; the second device is also used to receive the second control instruction sent by the control end, and adjust the corresponding target parameter in response to the second control instruction.

[0022] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the device control method as described above is implemented.

[0023] According to one aspect of the present application, a storage medium stores a computer program thereon, and when the computer program is executed by one or more processors, the device control method as described above is implemented.

[0024] According to one aspect of the present application, a computer program product includes a computer program, and when the computer program is executed by one or more processors, the device control method as described above is implemented.

[0025] The beneficial effects of the technical solution provided by this application are:

[0026] In the above technical solution, a first control instruction is generated based on a voice message collected by the first device, and a second control instruction is generated based on the motion data and / or voice message of the first device, so that the second device can not only respond to the first control instruction to perform the corresponding target operation, but also respond to the second control instruction to adjust the corresponding target parameters, wherein the second device can be any smart device different from the first device. Therefore, with the help of the voice and motion capture functions of the first device, the user is no longer limited to controlling a single smart device, but can control any smart device different from the first device, which greatly improves the flexibility of device control and can perform more dimensional operations on these smart devices without repeating multiple steps. It is simple and convenient, thereby effectively solving the problems of poor convenience and flexibility of device control in related technologies and greatly meeting user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0028] Figure 1 It is a schematic diagram of the implementation environment involved in this application;

[0029] Figure 2 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0030] Figure 3 is a flow chart of a device control method according to an exemplary embodiment;

[0031] Figure 4 is a schematic diagram showing a first device in a second state according to an exemplary embodiment;

[0032] Figure 5 yes Figure 3 A flowchart of step 330 in one embodiment corresponding to the embodiment;

[0033] Figure 6 yes Figure 3 A flowchart of step 350 in one embodiment corresponding to the embodiment;

[0034] Figure 7 is a flow chart of another device control method according to an exemplary embodiment;

[0035] Figure 8 It is a schematic diagram of a specific implementation of a device control method in an application scenario;

[0036] Fig. 9 is a structural block diagram of a device control apparatus according to an exemplary embodiment;

[0037] Fig.10 is a structural block diagram of a device control system according to an exemplary embodiment;

[0038] Fig.11 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0040] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0041] As mentioned above, existing smart devices are mainly controlled by voice, action, client or single-function control devices (such as remote controls), which are too simple and lack flexible and multi-dimensional device control capabilities.

[0042] Specifically, on the one hand, only a single smart device can be controlled: for example, existing control devices such as knobs, joysticks, switches, etc. are usually for a single smart device, or users control a single smart device through a single action. Or, if the client is configured to automatically control the living room light, it can only control the living room light later. If other lamps, curtains or more smart devices need to be controlled, they need to be configured again in the client for different smart devices, and it is impossible to control different smart devices in real time according to the user's intentions.

[0043] On the other hand, users need to repeat multiple steps when controlling the same smart device to perform different actions: for example, if they need to adjust the brightness and color temperature of the living room lights, they often need to perform multiple steps, or operate multiple different control devices separately, such as one knob or button to control the brightness, and another knob or button to control the brightness. This is bound to affect the user experience, which is not only not smooth enough, but also relatively costly.

[0044] To this end, the device control method provided in the present application can effectively improve the flexibility of device control. Accordingly, the device control method is applicable to a device control device, which can be deployed in an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a laptop computer, a server, etc.; the electronic device can also be an electronic device with a central control function, for example, the electronic device includes a gateway, etc.; the electronic device can also refer to an electronic device with voice and motion capture functions, for example, the electronic device can be a voice remote control configured with components such as a microphone and a six-axis sensor.

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0046] Figure 1 Schematic diagram of an implementation environment involved in a device control method. The implementation environment at least includes a user terminal 110, a smart device 130, a server 170, and a network device. Figure 1 In the embodiment, the network devices include a gateway 150 and a router 190, which is not specifically limited herein.

[0047] Among them, the user terminal 110, which can also be considered as a user end or terminal, can deploy (also understood as install) the client associated with the smart device 130. This user terminal 110 can be an electronic device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart control panel, other devices with display and control functions, etc., which are not limited here.

[0048] Among them, the client is associated with the smart device 130, which essentially means that the user registers an account in the client and configures the smart device 130 in the client. For example, the configuration includes adding a device identifier to the smart device 130, so that when the client is run in the user terminal 110, it can provide the user with functions such as device display and device control of the smart device 130. This client can be in the form of an application or a web page. Accordingly, the interface for device display on the client can be in the form of a program window or a web page, which is not limited here.

[0049] The smart device 130 is deployed in the gateway 150, and communicates with the gateway 150 through its own configured communication module, and is then controlled by the gateway 150. It should be understood that the smart device 130 generally refers to one of the multiple smart devices 130, and the embodiment of the present application only takes the smart device 130 as an example, that is, the embodiment of the present application does not limit the number and device type of smart devices deployed in the gateway 150. In an application scenario, the smart device 130 accesses the gateway 150 through a local area network, and is thus deployed in the gateway 150. The process of the smart device 130 accessing the gateway 150 through the local area network includes: the gateway 150 first establishes a local area network, and the smart device 130 joins the local area network established by the gateway 150 by connecting to the gateway 150. This local area network includes, but is not limited to: ZIGBEE or Bluetooth. Among them, the smart device 130 can be a smart printer, a smart fax machine, a smart camera, a smart air conditioner, a smart door lock, a smart light, or a human body sensor equipped with a communication module, a door and window sensor, a temperature and humidity sensor, a water immersion sensor, a natural gas alarm, a smoke alarm, a wall switch, a wall socket, a wireless switch, a wireless wall sticker switch, a magic cube controller, a curtain motor, a millimeter wave radar, a voice remote control, etc.

[0050] The interaction between the user terminal 110 and the smart device 130 can be achieved through a local area network or a wide area network. In one application scenario, the user terminal 110 establishes a communication connection between the router 190 and the gateway 150 in a wired or wireless manner. For example, the wired or wireless manner includes but is not limited to WIFI, so that the user terminal 110 and the gateway 150 are deployed in the same local area network, so that the user terminal 110 can interact with the smart device 130 through the local area network path. In another application scenario, the user terminal 110 establishes a communication connection between the server 170 and the gateway 150 in a wired or wireless manner. For example, the wired or wireless manner includes but is not limited to 2G, 3G, 4G, 5G, WIFI, etc., so that the user terminal 110 and the gateway 150 are deployed in the same wide area network, so that the user terminal 110 can interact with the smart device 130 through the wide area network path.

[0051] The server end 170 may also be considered as a cloud, a cloud platform, a platform end, a service end, etc. The server end 170 may be a server, a server cluster consisting of multiple servers, or a cloud computing center consisting of multiple servers, so as to better provide background services to a large number of user terminals 110. For example, the background service includes a device control service.

[0052] In one application scenario, the device control method can be implemented through the interaction between different smart devices 130 (i.e., the first device and the second device), the gateway 150, and the server 170. Specifically, the first device collects voice messages and detects motion data, and sends the voice messages and motion data to the gateway 150, which is forwarded to the server 170 by the gateway 150. The server 170 generates a first control instruction and a second control instruction based on the voice message and / or motion data, and returns the first control instruction and the second control instruction to the gateway 150, which is forwarded to the second device by the gateway 150 to control the second device to perform the target operation and adjust the target parameters.

[0053] In one application scenario, the device control method can be implemented through the interaction between different smart devices 130 (i.e., the first device and the second device) and the gateway 150. Specifically, the first device collects voice messages and detects motion data, and sends the voice messages and motion data to the gateway 150, and the gateway 150 generates the first control instruction and the second control instruction based on the voice message and / or the motion data, and sends the first control instruction and the second control instruction to the second device to control the second device to perform the target operation and adjust the target parameters.

[0054] In another application scenario, the device control method can also be implemented through the interaction between different smart devices 130 (i.e., the first device and the second device) and the gateway 150. Specifically, the first device collects voice messages and detects motion data, and generates first control instructions and second control instructions based on the voice messages and / or motion data, and sends the first control instructions and the second control instructions to the gateway 150, which is forwarded by the gateway 150 to the second device to control the second device to perform the target operation and adjust the target parameters.

[0055] See also Figure 2 , Figure 2 is a hardware structure diagram of an electronic device according to an exemplary embodiment. The electronic device is suitable for Figure 1 The server side 170 in an implementation environment is shown.

[0056] It should be noted that the electronic device is only an example adapted to the present application and cannot be considered to provide any limitation on the scope of use of the present application. The electronic device cannot be interpreted as requiring dependence on or having Figure 2 One or more components of exemplary electronic device 200 are shown.

[0057] The hardware structure of the electronic device 200 may vary greatly due to different configurations or performances, such as Figure 2As shown, the electronic device 200 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0058] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 200 .

[0059] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Figure 1 In the illustrated implementation environment, the smart device 130 configured with the interface 231 interacts with the gateway 150 .

[0060] Of course, in other examples adapted by the present application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 2 As shown, this is not intended to be a specific limitation.

[0061] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0062] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 200 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0063] The application program 253 is a computer program formed by computer-readable instructions based on the operating system 251 to complete at least one specific task, and may include at least one module ( Figure 2 Each module may include corresponding computer-readable instructions. For example, the device control device may be regarded as an application 253 deployed on the electronic device 200.

[0064] Data 255 may be photos, pictures, etc. stored in a disk, or may be voice messages, motion data, automation control schemes, etc., stored in memory 250 .

[0065] The central processor 270 may include one or more processors and is configured to communicate with the memory 250 through at least one communication bus to read the computer program stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the device control method is completed in the form of the central processor 270 reading the application program 253 stored in the memory 250.

[0066] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.

[0067] See also Figure 3 The present application embodiment provides a device control method, which is applicable to an electronic device, for example, the electronic device may be Figure 1 The gateway 150 or server 170 in the implementation environment is shown to facilitate data transmission between the first device and the second device; the electronic device can also be the first device itself. In this case, the first device not only has voice and motion capture functions, but also has a communication function and can interact with the second device. The hardware structure of the electronic device can be as follows Figure 2 shown.

[0068] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0069] like Figure 3 As shown, the method may include the following steps:

[0070] Step 310: Acquire the voice message collected by the first device.

[0071] Among them, the voice message is data used to indicate the user's device control intention. It can be understood that the user's device control intention should not only be able to reflect which smart device the user needs to control, but also what the user needs to control the smart device for. In other words, the voice message is used to indicate the smart device to be controlled and the operation to be performed by the smart device. It is explained here that the second device can be any smart device different from the first device. For example, in a smart home scenario, the first device can be a voice remote control, and the second device can be a smart light, smart curtain, smart air conditioner, smart TV, etc.

[0072] The voice message is collected by the first device. The first device may be a smart device with a voice capture function, such as a voice remote control, an audio companion, etc. The environment in which the first device is deployed may also vary depending on the application scenario. For example, in a smart home scenario, the first device may be deployed in any room that the user enters and exits (such as a living room), or in a traffic safety scenario, the first device may be deployed in a vehicle driven by the user.

[0073] The acquisition of voice messages can come from local storage. For example, if the electronic device is a first device, the first device can directly acquire the voice message after collecting it and storing it. It can also come from data transmission. For example, if the electronic device is a gateway, the first device can acquire the voice message and send it to the gateway. Accordingly, the gateway can receive the voice message.

[0074] Step 330: Generate a first control instruction based on the voice message to instruct the second device to perform a corresponding target operation in response to the first control instruction.

[0075] The first control instruction is used to instruct the second device to perform a corresponding target operation. The target operation refers to an operation to be performed in at least one operation that can be performed by the second device. For example, for a smart light, in some embodiments, the executable operations include but are not limited to turning on and off; in other embodiments, the executable operations may also refer to adjusting parameters of the smart light, such as brightness, temperature, color, amplitude, etc.

[0076] As mentioned above, the voice message indicates the smart device to be controlled and the operation to be performed by the smart device. In a possible implementation, the smart device to be controlled is the second device, and accordingly, the operation to be performed by the smart device to be controlled is the target operation. In other words, the smart device to which the voice message points can be clear. That is, the voice message clearly indicates the user's device control intention. For example, the voice message is "Turn on living room lamp A". In this case, the second device is living room lamp A, and the target operation is to turn on.

[0077] Furthermore, the inventors also realized that if the voice message clearly indicates the user's device control intention, this device control method can often only control a single smart device. For example, in the above example, the user can only control the living room lamp A. Assuming that the living room lamp B is also deployed, the living room lamp B cannot be controlled at the same time. Therefore, in order to achieve dynamic selection of the controlled device, in a possible implementation method, the smart device pointed to by the voice message can be vague. In other words, the user's device control intention indicated by the voice message can be vague. For example, the voice message is "turn on the living room lamp", if at least lamp A and lamp B are deployed in the living room, it is considered that the voice message does not clearly indicate which lamp in the living room the user wants to control.

[0078] Based on this, if the smart device pointed to by the voice message is ambiguous, the second device can be further determined by the ambiguous smart device pointed to by the voice message through the dynamic selection of the controlled device. Still using the above example to illustrate, assuming that the ambiguous smart device pointed to by the voice message is the "living room lamp", and there are at least lamps A and B in the living room, then the second device can be lamps A and B, and accordingly, the corresponding target operations of lamps A and B are both turned on.

[0079] After determining the second device and the target operation, the electronic device can generate a first control instruction for instructing the second device to perform the target operation, and send the first control instruction to the second device. The second device can receive the first control instruction and perform the corresponding target operation in response to the first control instruction.

[0080] Step 350: Generate a second control instruction based on the detected motion data and / or voice message from the first device to instruct the second device to adjust the corresponding target parameter in response to the second control instruction.

[0081] The second control instruction is used to instruct the second device to adjust the corresponding target parameter. The target parameter refers to the parameter to be adjusted among at least one parameter adjustable by the second device. For example, for a smart light, the adjustable parameters include but are not limited to brightness, temperature, color, adjustment range, etc.

[0082] In this embodiment, the second control instruction is generated based on the motion data detected by the first device and / or the voice message collected by the first device. It can be seen that for the first device, it not only has a voice capture function, that is, it can collect voice messages, but also has a motion capture function, that is, it can detect motion data. Taking the voice remote control as an example, the voice remote control is equipped with a microphone and a six-axis sensor. The microphone can be used to collect voice messages, and the six-axis sensor can accurately capture motions, and provides more sophisticated motion recognition capabilities to obtain motion data. Among them, motion data refers to data generated by the first device when it is in overall motion or relatively still, including but not limited to motion direction, motion angle, motion duration, speed, acceleration, etc.

[0083] In one possible implementation, the motion data of the first device includes motion data corresponding to the first state. The first state can also be understood as an overall motion state, which means that the first device is in motion as a whole. For example, for a voice remote control, a user can rotate, flip, push, vibrate, shake the voice remote control, and tap other objects such as a table with the voice remote control. Then, when the voice remote control is performing the above motion, it can be considered that the voice remote control is in the first state.

[0084] In one possible implementation, the motion data of the first device includes motion data corresponding to the second state. The second state can also be understood as a relatively static state, which means that the first device as a whole remains static while the components configured by the first device are moving relative to the whole. In other words, the second motion state can refer to the state in which the first device as a whole remains static relative to the moving components. For example, the user can perform operations such as tapping or rotating a knob on a relatively static voice remote control. Figure 4 As shown, for a voice remote controller equipped with a knob, if the user rotates 401 the knob without rotating the voice remote controller, it can be considered that the voice remote controller is in the second state at this time, that is, the voice remote controller remains stationary as a whole relative to the rotated knob.

[0085] After determining the voice message and / or motion data, the electronic device will further determine the target parameters based on this. In one possible implementation, the target parameters can be extracted from the voice message containing the target parameters. In one possible implementation, the target parameters can be determined according to the priority of several parameters adjustable by the second device when the voice message does not contain the target parameters. In one possible implementation, the target parameters can be determined based on at least one of the motion direction, motion amplitude, motion angle, motion duration, motion speed, and acceleration contained in the motion data. In one possible implementation, the target parameters can also be determined based on whether the motion type of the first device in the first state determined by the motion data meets the trigger conditions in the automation solution.

[0086] Therefore, after determining the target parameters, the electronic device can generate a second control instruction for instructing the second device to adjust the target parameters based on the second device and the target parameters, and send the second control instruction to the second device. The second device can receive the second control instruction and perform the corresponding target operation in response to the second control instruction.

[0087] It is worth mentioning that the device control method is not limited to executing step 330 first and then step 350. It can also be considered that the user performs voice control first and then action control. The above two steps can also be performed simultaneously, that is, it can be considered that the user can combine voice control and action control at the same time. Specifically, it means that the first device collects a voice message and detects motion data at the same time. Accordingly, the first control instruction and the second control instruction can be generated based on the voice message and the motion data to control the second device to perform the target operation and adjust the target parameters. This embodiment does not constitute a specific limitation on this.

[0088] Through the above process, the user can control any smart device different from the first device with the help of the voice and motion capture functions of the first device, instead of just a single smart device. Moreover, the user can perform multi-dimensional operations on these smart devices without having to repeat multiple steps. This effectively solves the problem of poor control flexibility of devices in related technologies and greatly meets user needs.

[0089] In order to reduce the power consumption of the first device, the first device first enters a sleep state after being turned on. In the sleep state, the first device does not work until it is awakened, and then enters an awake state from the sleep state, and then can start working in the awake state, for example, collecting voice messages.

[0090] Specifically, before step 310, the method may further include the following steps: detecting whether there is a wake-up operation triggered for the first device; if yes, controlling the first device to enter the wake-up state from the sleep state, so that the first device in the wake-up state collects voice messages.

[0091] Among them, the wake-up operation refers to the user operation that triggers the first device to start working. The wake-up operation can be flexibly set according to the actual needs of the user. For example, in some embodiments, the wake-up operation can refer to "shaking" the first device. Then, when the user shakes the first device in a dormant state, the first device can detect the wake-up operation and then enter the awakened state from the dormant state. Then, for the first device in the awakened state, the voice message can be collected; when the wake-up operation is action wake-up, the other functions of the first device such as voice collection and recognition do not work or are in a turned-off state during the dormant period. In other embodiments, the wake-up operation can also refer to rotating, flipping, pushing, tapping, and vibrating the first device, which is not specifically limited here.

[0092] Of course, in other embodiments, the first device enters the awake state from the sleep state, which is not limited to the detection of the wake-up operation triggered for the first device, but can also be the collection of a set voice message for the first device, and the set voice message can also be flexibly set according to the actual needs of the user. For example, the set voice message can be a preset wake-up word, such as "Xiao Qiao", "Xiao Ai", etc., which does not constitute a specific limitation here. It should be noted that in this case, the first device can still collect voice messages in the sleep state, but the remaining functions of the first device, such as motion recognition, do not work or are in a closed state, so that the first device maintains low power consumption until the first device is awakened by the set voice message and enters the awake state from the sleep state, and the remaining functions of the first device start to work, for example, detecting the motion data of the first device.

[0093] In this way, the first device will collect voice messages only after being awakened, otherwise it always maintains low power consumption, which is not only beneficial to reducing power consumption and effectively extending the service life of the first device, but also beneficial to reducing the cost of device control.

[0094] See also Figure 5 In an exemplary embodiment, step 330 may include the following steps:

[0095] Step 331: Perform voice recognition processing on the voice message to obtain the candidate devices and target operations pointed to by the voice message.

[0096] In this embodiment, the candidate device refers to the smart device to which the voice message is directed.

[0097] As mentioned above, in order to achieve dynamic selection of the controlled device, the smart device pointed to by the voice message can be clear or blurred. Therefore, in order to determine whether the smart device pointed to by the voice message is clear, voice recognition processing technology is used.

[0098] In a possible implementation, the speech recognition process includes but is not limited to: speech conversion, semantic analysis, etc. Specifically, the speech recognition process may include the following steps: first, converting the speech message into text; second, performing semantic analysis on the text to obtain the candidate device and target operation pointed to by the speech message.

[0099] For example, if the voice message is "turn on the light", through voice recognition processing, it can be obtained that the candidate device pointed to by the voice message is "light" and the target operation is to turn it on.

[0100] After determining the candidate device to which the voice message is directed, the second device can be further determined based on whether the candidate device to which the voice message is directed is clear or blurred. The second device refers to a smart device that has the same device attributes as the candidate device. The device attributes can also be understood as the functions of the smart device. For example, the device attributes of an air conditioner refer to its temperature adjustment function, while the device attributes of a temperature sensor refer to its temperature detection function.

[0101] If the candidate device pointed to by the voice message is ambiguous, the second device can be obtained by associating the ambiguous candidate device pointed to by the voice message, that is, executing step 333 .

[0102] On the contrary, if the candidate device to which the voice message is directed is clear, the second device can be uniquely determined by the clear candidate device to which the voice message is directed, ie, step 335 is executed.

[0103] Step 333: If the clarity of the candidate device pointed to by the voice message is fuzzy, then according to the set association condition, the second device is associated with the candidate device pointed to by the voice message.

[0104] The clarity level is used to indicate whether the candidate device to which the voice message is directed is clear.

[0105] If the clarity of the candidate device pointed to by the voice message is fuzzy, it means that the voice message does not clearly indicate the user's device control intention. In this case, it can be obtained by associating the fuzzy candidate device pointed to by the voice message. For example, if the voice message is "turn on the living room light", the candidate device pointed to by the voice message is "living room light". If the lights deployed in the living room include at least lights A and lights B, the clarity of the candidate device "living room light" is fuzzy. In this case, by associating the candidate device "living room light", it can be determined that the second device can be at least lights A and lights B.

[0106] In this embodiment, association is achieved based on setting association conditions. The setting of association conditions refers to conditions used to associate with candidate devices, which can be flexibly adjusted according to the actual needs of the application scenario and are not limited here. For example, setting association conditions can be based on device type, device space / user space, user history, etc.

[0107] The following is an example of setting association conditions to associate based on the space where the device is located:

[0108] Specifically, the space where the device is located is determined based on the set association condition. After determining that the set association condition is based on the room where the device is located, the electronic device will locate the first device to determine the position of the first device, that is, the space where the device is located.

[0109] The candidate device pointed to by the voice message is associated with the space where the device is located to obtain the second device. Therefore, based on the association made in the space where the device is located, the second device refers to the candidate device located in the space where the device is located. For example, if the first device is in the living room and the candidate device is a "lamp", all the lamps in the living room can be used as the second device.

[0110] Similarly, based on the association of device types, the second device refers to a smart device of the same device type as the candidate device. For example, if the candidate device is "air conditioner", then the second device can be an air conditioner, but not a fan.

[0111] Based on the association of the user's historical operations, the second device refers to the historical smart device that was associated with the candidate device when the user last controlled it. For example, the last voice command issued by the user was "turn off the light", that is, the candidate device is "light", and assuming that the historical smart device associated with the candidate device "light" is living room light A, then the second device is living room light A.

[0112] Of course, in other application scenarios, the association conditions can also be set based on the space where the gateway to which the device is connected is located, based on the target space (such as the entire house in a smart home scenario), etc., which will not be described one by one here. The association conditions can be pre-configured by the user in the client. For example, in the client, the user is presented with selectable association conditions. When the user selects any one of the association conditions, the configuration of the association conditions is deemed to be completed.

[0113] Step 335: If the clarity of the candidate device to which the voice message is directed is clear, the candidate device is used as the second device.

[0114] If the clarity of the candidate device pointed to by the voice message is clear, it means that the voice message has clearly indicated the user's device control intention, so the second device can be uniquely determined by the candidate device pointed to by the voice message. For example, if the voice message is "Turn on living room lamp A", the candidate device pointed to by the voice message is "living room lamp A". It can be seen that the clarity of the candidate device "living room lamp A" is clear, so the candidate device "living room lamp A" is the second device.

[0115] Step 337: Generate a first control instruction for instructing the second device to execute the target operation according to the second device and the target operation.

[0116] After determining the second device and the target operation, the electronic device can generate a first control instruction for instructing the second device to perform the target operation, and send the first control instruction to the second device. The second device can receive the first control instruction and perform the corresponding target operation in response to the first control instruction.

[0117] With the help of the above-mentioned embodiments, dynamic selection of controlled devices is realized, so that device control is no longer limited to a single device, and multiple devices can be obtained by associating candidate devices, which greatly expands the flexibility of device control.

[0118] In an exemplary embodiment, before step 330, the method may further include the following steps:

[0119] The motion data of the first device is obtained.

[0120] As mentioned above, the motion data of the first device may include motion data corresponding to a first state and motion data corresponding to a second state. The first state refers to a state in which the first device as a whole is in motion, i.e., an overall motion state; the second state refers to a state in which the first device as a whole remains stationary relative to the moving component, i.e., a relative stationary state.

[0121] For the first state, obtaining the motion data may include the following steps:

[0122] During the process of the second device executing the target operation, it is detected whether the first device is in the first state; if it is detected that the first device is in the first state, the motion data of the first device corresponding to the first state is obtained.

[0123] In a possible implementation, the user first performs voice control and then motion control. That is, after the first device collects the voice message, the electronic device first generates a first control instruction based on the voice message to control the second device to perform the target operation, and then detects whether the first device is in the first state, thereby determining whether there is motion data of the first device corresponding to the first state. If there is motion data corresponding to the first state on the first device, a second control instruction is generated based on the motion data and / or the voice message to control the second device to adjust the target parameter.

[0124] The motion data corresponding to the first state refers to the data generated by the first device when it moves as a whole, including but not limited to the motion direction, motion angle, motion duration, speed, acceleration, etc. Taking the voice remote control as an example, the six-axis sensor configured therein can accurately capture the motion and provide more sophisticated motion recognition capabilities to obtain motion data.

[0125] In this way, the user can control the second device to perform the target operation through the voice capture function provided by the first device, and control the second device to adjust the target parameters through the motion capture function provided by the first device, including but not limited to rotating, flipping, pushing, knocking, vibrating, shaking the first device and other actions, thereby achieving dual control of voice and action and realizing precise control of smart home devices.

[0126] Furthermore, in order to reduce the power consumption of the first device, the method may further include the following steps:

[0127] If it is detected that the first device is not in the first state, the first device is controlled to enter a sleep state until a voice message is obtained again.

[0128] As mentioned above, after the first device enters the sleep state, it can detect a wake-up operation triggered for the first device, for example, the wake-up operation is to "shake" the first device, enter the wake-up state from the sleep state, and then start collecting voice messages in the wake-up state.

[0129] That is, in one possible implementation, the user only performs voice control. That is, after the electronic device controls the second device to perform the target operation, if it does not detect that the first device is moving as a whole, it is considered that the user has no intention to continue to control the second device, and the first device is controlled to enter a dormant state until the user has the intention to control the second device next time, and the first device is reawakened to collect voice messages again.

[0130] In this way, the first device always maintains low power consumption when the user has no intention to control the second device, which is not only beneficial to reducing power consumption and effectively extending the service life of the first device, but also beneficial to reducing the cost of device control.

[0131] For the second state, obtaining the motion data may include the following steps:

[0132] When the first device is in the second state, it is detected whether a relative movement operation of a component configured in the first device is triggered.

[0133] In one possible implementation, the user performs voice control first and then motion control. On the one hand, after controlling the second device to perform the target operation, the electronic device can detect whether the first device is in the second state, and when the first device is in the second state, continue to detect whether the component configured by the first device is in motion. On the other hand, for the electronic device, when it is detected that the first device is not in the first state, it will continue to detect whether the first device is in the second state, and when the first device is in the second state, it will determine whether the component configured by it is in motion. Among them, the relative motion operation refers to the operation of triggering the component configured by the first device to move. Taking the voice remote control equipped with a knob as an example, if the user only rotates the knob while the voice remote control remains stationary as a whole, the rotation operation can be considered as a relative motion operation triggered by the component configured by the first device.

[0134] In this way, the electronic device can determine whether there is motion data corresponding to the second state of the first device. If there is motion data corresponding to the second state of the first device, it can generate a second control instruction based on the motion data and / or voice message to control the second device to adjust the target parameters.

[0135] If yes, the motion data of the first device corresponding to the second state is obtained.

[0136] The motion data corresponding to the second state refers to the data generated by the first device when it is relatively still, including but not limited to the motion direction, motion angle, motion duration, speed, acceleration, etc. Taking the voice remote control as an example, the six-axis sensor configured therein can accurately capture the motion and provide more sophisticated motion recognition capabilities to obtain motion data.

[0137] In this way, the user can control the second device to perform the target operation through the voice capture function provided by the first device, and control the second device to adjust the target parameters through the motion capture function provided by the first device, including but not limited to rotating, pressing components configured by the first device (such as knobs, buttons), etc., thereby achieving dual control of voice and action and realizing precise control of smart home devices.

[0138] Through the coordination of the above-mentioned embodiments, a more natural, intuitive and multi-dimensional control method is provided, and a smooth multi-dimensional device control experience is achieved only through the same control device (ie, the first device).

[0139] See also Figure 6 In an exemplary embodiment, step 350 may include the following steps:

[0140] Step 351: Determine a target parameter to be adjusted for the second device based on the voice message and / or the motion data.

[0141] The target parameter refers to a parameter to be adjusted among at least one adjustable parameter of the second device.

[0142] In a possible implementation, the target parameter to be adjusted by the second device can be directly determined based on the voice message. Specifically, by performing voice analysis on the voice message, if the voice message contains the target parameter, the target parameter is extracted from the voice message. For example, if the voice message is "adjust the brightness of the living room light", the target parameter is brightness.

[0143] In a possible implementation, when the voice message does not contain the target parameter, the target parameter to be adjusted by the second device can be determined based on the motion data. The determination process can specifically include the following steps: the first step is to determine the motion type of the first device in the first state based on the motion data. Among them, the motion type can also be considered as the action type of the first device as a whole when in motion, and the action type includes but is not limited to rotation, flipping, pushing, knocking, vibration, shaking and other actions. The second step is, if the motion type of the first device is monitored to meet the trigger condition in the configured first automation scheme, the target parameter associated with the satisfied trigger condition is extracted from the first automation scheme. Among them, the first automation scheme controls the second device to adjust the target parameter by monitoring whether the motion type of the first device meets the trigger condition. For example, assuming that the first automation scheme = {trigger condition = the motion type of the first device is detected to be rotation, the controlled action = the light adjusts the color}, that is, if the motion type of the first device is detected to be rotation, the second device is a light, and the target parameter is color.

[0144] In a possible implementation, when the voice message does not contain the target parameter, it can also be determined based on the priority of the candidate parameters that can be adjusted by the second device. The determination process can specifically include the following steps: Step 1, determine at least one candidate parameter that can be adjusted by the second device. The candidate parameter refers to the adjustable parameter of the second device. Taking the second device as a lamp as an example, the candidate parameters of the lamp can be brightness, temperature, and color. Step 2, obtain the configured priority of each candidate parameter. The priority can be pre-configured by the user on the client, or it can be configured by the administrator on the control end (such as the default priority) based on the user's daily control frequency. Figure 1 The server side 170 in the implementation environment is shown as pre-configured. For example, for a lamp, the priority of brightness may be greater than the priority of color, and the priority of color may be greater than the priority of temperature. In the third step, the candidate parameter with the highest priority is used as the target parameter. Using the above example, the target parameter is the brightness with the highest priority.

[0145] Step 353: Determine the adjustment ratio corresponding to the target parameter according to the motion data.

[0146] It should be understood that for some target parameters that can undergo gradual changes, adjustment is actually a gradual change process. Taking the second device as a curtain, the opening and closing of the curtain is 0-100%, then the curtain can be adjusted to open / close 10% each time. As mentioned above, the motion data includes but is not limited to the motion direction, motion amplitude, motion angle, motion duration, motion speed, acceleration, etc. Based on this, in this embodiment, the adjustment ratio corresponding to the target parameter can be determined according to at least one of the motion direction, motion amplitude, motion angle, motion duration, motion speed, and acceleration contained in the motion data. It is explained here that the motion amplitude can refer to the amplitude when the first device is shaken or pushed, and the motion angle can refer to the angle when the first device is rotated or flipped. Correspondingly, the motion direction can refer to the up and down, left and right, front and back directions when the first device is shaken or pushed, and can refer to the clockwise, counterclockwise, etc. directions when the first device is rotated or flipped.

[0147] In a possible implementation, the adjustment ratio is related to the motion direction and / or motion amplitude of the first device. Specifically, step 353 may include the following steps: determining the motion parameters of the first device indicated by the motion data, the motion parameters including one or more of the motion direction, motion amplitude, motion angle, motion duration, motion speed, and acceleration; determining the adjustment ratio corresponding to the target parameter according to the motion parameters of the first device.

[0148] For example, the adjustment ratio can be determined according to the movement direction and the movement angle. Specifically, counterclockwise rotation can be used for decrease, while clockwise rotation can be used for increase. The decrease / increase ratio is further determined by the movement angle.

[0149] Alternatively, the adjustment ratio can be determined based on the movement angle. Specifically, the movement angle refers to the movement angle (i.e., the rotation angle). At this time, the adjustment ratio is evenly distributed according to the rotation angle by default. For example, if the brightness of the light is 0-100%, it corresponds to a rotation angle of 0-360 degrees.

[0150] It can also be understood that a mapping relationship is preset between the motion parameters of the first device and the adjustment ratio corresponding to the target parameter. Then, based on this mapping relationship, the adjustment ratio with the mapping relationship can be obtained from the motion parameters of the first device.

[0151] Of course, in other embodiments, the adjustment ratio may also be determined based on the movement amplitude, movement duration, movement speed, acceleration, etc., which does not constitute a specific limitation here.

[0152] Step 355: Generate a second control instruction to instruct the second device to adjust the target parameter according to the adjustment ratio.

[0153] Taking the above example as an example, assuming that the second device is a lamp, the target parameter is brightness, and the adjustment ratio is 30% determined based on a rotation angle of 30 degrees, the second control instruction can be used to adjust the brightness of the indicator light by 30%.

[0154] In the above process, the control of the adjustable parameters of the second device is realized, which can not only control different parameters of the second device, but also realize the control of different adjustment ratios of the same parameter, further improving the flexibility of device control and helping to improve the user's device control experience.

[0155] See also Figure 7 In an exemplary embodiment, before step 350, the method may further include the following steps:

[0156] Step 510, determining the time interval between the acquisition time of the voice message and the detection time of the motion data.

[0157] As mentioned above, for the second device, the user's device control intention has various forms, which may be voice control first and then action control, voice control only, or action control only. In order to determine the form of the user's device control intention, in this embodiment, the time interval = detection time of motion data - acquisition time of voice message.

[0158] For an electronic device, during data transmission with a first device, each received data may be timestamped. For example, the timestamp of a received voice message may be marked as t1, and the timestamp of received motion data may be marked as t2. At this time, the time interval = t2-t1.

[0159] Step 530: If the time interval does not exceed the set threshold, it is determined that the control object of the first device is the second device.

[0160] After determining that the second device is the controlled object, it indicates that the user's device control intention is to combine voice control and motion control. At this time, the electronic device can generate a second control instruction based on motion data and / or voice message, and send the second control instruction to the second device, so that the second device adjusts the corresponding target parameters in response to the second control instruction.

[0161] In this case, the user hopes to control the second device by combining dual control of voice and action, that is, the user can continue to rotate the first device after saying "turn on the living room light" to adjust the target parameters of the "living room light" (such as brightness).

[0162] Step 550: If the time interval exceeds the set threshold, determine that the control object of the first device is the target device.

[0163] The target device refers to the controlled device in the second automation solution, which can be the second device or other intelligent device different from the second device. It should be noted that, unlike the first automation solution which is used to control the second device to adjust the target parameters, the second automation solution controls the target device to perform the corresponding target operation by monitoring whether the motion type of the first device meets the trigger condition.

[0164] After determining that the target device is a controlled object, it indicates that the user's device control intention at this time is action control only. At this time, the electronic device can generate a third control instruction based on the second automation scheme, and send the third control instruction to the target device, so that the target device responds to the third control instruction and performs the corresponding target operation.

[0165] The specific process may include the following steps:

[0166] In the first step, if the time interval exceeds a set threshold, the movement type of the first device in the first state is monitored.

[0167] The motion type may also be considered as the action type of the first device as a whole when in motion, and the action type includes but is not limited to rotation, flipping, pushing, knocking, vibration, shaking and the like.

[0168] In the second step, if it is monitored that the motion type of the first device in the first state meets the trigger condition in the second automation solution, a target operation associated with the satisfied trigger condition is extracted from the second automation solution.

[0169] The second automation scheme controls the target device to perform the target operation by monitoring whether the motion type of the first device meets the trigger condition. For example, assuming that the second automation scheme = {trigger condition = the motion type of the first device is detected to be flipping, controlled action = turning on the air conditioner}, that is, if the motion type of the first device is detected to be flipping, the target device is the air conditioner (different from the first automation scheme in the previous example where the second device is a light), and the target operation is turning on.

[0170] The third step is to generate a third control instruction according to the target device and the target operation, so as to instruct the target device to perform the target operation in response to the third control instruction.

[0171] The third control instruction is used to instruct the target device to perform a corresponding target operation. The target device can be distinguished from the second device, and the target operation refers to an operation to be performed in at least one operation that the target device can perform. For example, for a smart light, the executable operations include but are not limited to turning on and off.

[0172] After determining the target device and the target operation, the electronic device can generate a third control instruction for instructing the target device to perform the target operation, and send the third control instruction to the target device. The target device can receive the third control instruction and perform the corresponding target operation in response to the third control instruction.

[0173] In this case, the user wishes to control only the second device through voice control, and to control a target device different from the second device through motion control.

[0174] Of course, when the control object of the first device cannot be determined in the above situations, the electronic device can also generate a prompt message and output the prompt message according to the set output method (such as voice broadcast, display screen display, alarm buzzer, etc.) to prompt the user to further clarify his device control intention.

[0175] Through the coordination of the above embodiments, users can arbitrarily combine voice control and motion control, which can be used to associate and control the same device or to control different devices, further improving the flexibility of device control and helping to enhance the user's device control experience.

[0176] Figure 8 This is a schematic diagram of a specific implementation of a device control method in an application scenario. Figure 8 As shown, this application scenario is a smart home scenario, which includes a cloud (i.e. Figure 1 The server side 170 in the implementation environment is shown), the gateway (ie Figure 1 The gateway 150 in the implementation environment is shown, and the first device and the second device (ie, Figure 1 A smart device 130 is shown in the implementation environment.

[0177] In this smart home scenario, the first device may be a voice remote controller with voice and motion capture functions, which is equipped with a microphone for capturing voice and a six-axis sensor for capturing motion, thereby realizing voice and motion dual control of the second device by the voice remote controller. The second device may be a smart device different from the first device, for example, a smart lamp or a smart curtain.

[0178] Combine the following Figure 8 The device control method in the smart home scenario is described in the embodiments of the present application:

[0179] Scenario 1: Smart Light Control

[0180] 1. The user issues a command through the voice remote control: "Turn on the living room light."

[0181] 2. The voice remote's microphone captures the voice message and converts it into text through the built-in speech recognition software.

[0182] 3. The six-axis sensor simultaneously detects the motion data corresponding to the user's rotation of the voice remote control, and uses the built-in motion recognition algorithm to analyze the rotation of the voice remote control, and converts the rotation angle into a brightness adjustment ratio.

[0183] 4. The gateway receives the voice message and motion data and uploads it to the cloud.

[0184] 5. The cloud performs semantic analysis, combines voice messages and action data, and determines that the user's intention is to turn on the living room light and adjust the brightness. It is worth mentioning that if there is only one smart light A deployed in the living room, the user's intention is determined to be to turn on the living room light A. If there are more than one smart lights deployed in the living room, for example, including smart lights A and smart lights B, the user's intention is determined to be to turn on the living room lights A and B, thereby realizing the dynamic selection of the controlled devices and effectively avoiding the singleness of device control.

[0185] 6. The cloud generates a first control instruction and a second control instruction, the first control instruction is used to turn on the living room light, and the second control instruction is used to adjust the brightness of the living room light to the adjustment ratio expected by the user, and sends the above control instructions back to the gateway.

[0186] 7. The gateway receives these instructions and sends them to the living room light, first turning on the living room light, then adjusting the brightness of the living room light according to the adjustment ratio, and finally achieving a smooth transition of the brightness of the living room light.

[0187] Scenario 2: Smart Curtain Control

[0188] 1. The user issues a command through the voice remote control: "Open the curtains."

[0189] 2. The above voice commands are captured by the voice remote control and converted into text.

[0190] 3. Users can also rotate the knob on the voice remote control to adjust the opening and closing ratio of the curtains. Correspondingly, the six-axis sensor can capture the rotation of the knob and convert the rotation angle into the adjustment ratio of the curtain opening and closing.

[0191] 4. The gateway uploads the voice message and motion data to the cloud.

[0192] 5. The cloud performs semantic analysis to generate a first control instruction and a second control instruction for controlling the opening and closing of the curtains, wherein the first control instruction is used to open the curtains, and the second control instruction is used to indicate the specific proportion of the curtains being opened. It should be noted that in a smart home scenario, curtains can often be deployed in more than one room. At this time, the room where the user is located (for example, the user is in the living room) can be used as a setting association condition, and the "curtains" in the voice message can be associated with the room, and the user's intention is determined to be opening the living room curtains, thereby realizing dynamic selection of the controlled device and effectively avoiding the singleness of device control.

[0193] 6. The cloud sends the above control instructions to the gateway.

[0194] 7. The gateway receives these instructions and sends them to the smart curtains, which eventually open the curtains according to the adjustment ratio.

[0195] In this application scenario, the combination of voice and action brought by the voice remote control enables users to control smart devices in multiple dimensions, which not only improves the flexibility and accuracy of smart home control and greatly improves the convenience of user control of devices, but also allows users to control any smart device in the smart home scene more naturally and intuitively, improving the user experience.

[0196] Of course, in other application scenarios, the voice and actions brought by the voice remote control are not limited to voice used to control the second device to perform operations, and actions used to control the second device to adjust parameters. Voice can also be used to control the second device to adjust one parameter, for example, the color temperature of the living room light, and the action is used to control the second device to adjust another parameter, for example, the brightness of the living room light. The device control principles are basically the same, and this application scenario does not constitute a specific limitation.

[0197] Compared with related technologies, 1) Dual control of voice and action: a more precise dual control method for smart home is achieved through voice and action, providing a more natural, intuitive and multi-dimensional control method; 2) Dynamic selection of controlled devices: the user's voice can be analyzed through voice recognition technology to analyze the user's intention and determine the smart device that the user needs to control, overcoming the shortcoming of the existing technology that can only specify a single device; 3) Multi-dimensional control of devices: a smooth multi-dimensional device control experience is provided through the same voice remote control, that is, not only can the operations to be performed by the controlled device be specified by voice, but also the parameters to be adjusted by the controlled device can be specified by capturing actions through the six-axis sensor; 4) Intelligent semantic analysis: semantic analysis is performed locally (i.e., voice remote control) or in the cloud, and the user's intention is determined through voice and / or action, and control instructions are generated. The device that the user needs to control can be judged in real time, which increases the accuracy of control.

[0198] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0199] The following is an embodiment of the device of the present application, which can be used to execute the device control method involved in the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the device control method involved in the present application.

[0200] See also Fig. 9 In an embodiment of the present application, a device control apparatus 900 is provided, including but not limited to: a message acquisition module 910, a first instruction generation module 930, and a second instruction generation module 950.

[0201] The message acquisition module 910 is used to acquire the voice message collected by the first device.

[0202] The first instruction generating module 930 is used to generate a first control instruction based on the voice message to instruct the second device to perform a corresponding target operation in response to the first control instruction; the second device is a smart device different from the first device.

[0203] The second instruction generating module 950 is used to generate a second control instruction based on the motion data and / or voice message detected from the first device, so as to instruct the second device to adjust the corresponding target parameter in response to the second control instruction.

[0204] In an exemplary embodiment, the first instruction generation module 930 is used to perform voice recognition processing on the voice message to obtain the candidate device and target operation pointed to by the voice message; if the clarity of the candidate device pointed to by the voice message is fuzzy, then according to the set association conditions, the second device is associated with the candidate device pointed to by the voice message; the second device is a smart device with the same device attributes as the candidate device; based on the second device and the target operation, a first control instruction is generated to instruct the second device to perform the target operation.

[0205] In an exemplary embodiment, the first instruction generation module 930 is also used to determine the space where the device is located based on the set association conditions; associate the candidate device pointed to by the voice message with the space where the device is located to obtain the second device; the second device refers to the candidate device located in the space where the device is located.

[0206] In an exemplary embodiment, the first instruction generating module 930 is further configured to use the candidate device as the second device if the clarity of the candidate device to which the voice message is directed is clear.

[0207] In an exemplary embodiment, the apparatus 900 is further configured to detect whether the first device is in the first state during the process of the second device performing the target operation; if yes, obtain motion data of the first device corresponding to the first state.

[0208] In an exemplary embodiment, the apparatus 900 is further configured to control the first device to enter a dormant state if it is detected that the first device is not in the first state, until a voice message is acquired again.

[0209] In an exemplary embodiment, the apparatus 900 is further configured to detect whether a motion operation is triggered on a component configured on the first device when the first device maintains the second state; if so, obtain motion data of the first device corresponding to the second state.

[0210] In an exemplary embodiment, the second instruction generation module 950 is also used to determine the target parameter to be adjusted of the second device based on the voice message and / or motion data, and determine the adjustment ratio corresponding to the target parameter according to the motion data; and generate a second control instruction instructing the second device to adjust the target parameter according to the adjustment ratio.

[0211] In an exemplary embodiment, the second instruction generating module 950 is further used to determine the motion type of the first device in the first state based on the motion data if the voice message does not contain the target parameter; if it is monitored that the motion type of the first device meets the trigger condition in the configured first automation scheme, the target parameter associated with the satisfied trigger condition is extracted from the first automation scheme; the first automation scheme controls the second device to adjust the target parameter by monitoring whether the motion type of the first device meets the trigger condition.

[0212] In an exemplary embodiment, the second instruction generation module 950 is also used to determine at least one candidate parameter that can be adjusted by the second device if the voice message does not contain the target parameter, and obtain the configured priority of each candidate parameter; and use the candidate parameter with the highest priority as the target parameter.

[0213] In an exemplary embodiment, the second instruction generating module 950 is also used to determine the motion parameters of the first device indicated by the motion data, wherein the motion parameters include one or more of the motion direction, motion amplitude, motion angle, motion duration, motion speed, and acceleration; and determine the adjustment ratio corresponding to the target parameter based on the motion parameters of the first device.

[0214] In an exemplary embodiment, the apparatus 900 is further configured to detect whether a wake-up operation is triggered for the first device; if yes, the first device is controlled to enter a wake-up state from a sleep state, so that the first device in the wake-up state collects voice messages.

[0215] In an exemplary embodiment, the device 900 is also used to determine the time interval between the acquisition time of the voice message and the detection time of the motion data; if the time interval does not exceed the set threshold, it is determined that the control object of the first device is the second device, so as to generate a second control instruction instructing the second device to adjust the corresponding target parameters based on the motion data and / or the voice message.

[0216] In an exemplary embodiment, the device 900 is also used to monitor the movement type of the first device in the first state if the time interval exceeds a set threshold; if it is monitored that the movement type of the first device in the first state satisfies the trigger condition in the second automation scheme, extract the target operation associated with the satisfied trigger condition from the second automation scheme; the second automation scheme controls the target device to perform the target operation by monitoring whether the movement type of the first device satisfies the trigger condition; and generate a third control instruction based on the target device and the target operation to instruct the target device to perform the target operation in response to the third control instruction.

[0217] See also Fig.10 In an embodiment of the present application, a device control system 1000 is provided, including but not limited to: a first device 1010, a second device 1030, and a control terminal 1050.

[0218] by Figure 1 Taking the implementation environment shown as an example, in some embodiments of the present application, the control terminal 1050 may be Figure 1 The gateway 150 in the embodiment, the first control instruction and the second control instruction can be generated by the gateway 150 and transmitted to the second device through the local area network path. In other embodiments, the control terminal 1050 can also include Figure 1 In the gateway 150 and the server 170, at this time, the first control instruction and the second control instruction can be generated by the server 170 and forwarded to the second device via the gateway 150 through the wide area network path.

[0219] Among them, the first device 1010 is used to collect voice messages and send voice messages to the control end.

[0220] The control terminal 1050 is used to generate a first control instruction based on the voice message.

[0221] The second device 1030 is used to receive the first control instruction sent by the control end, and execute the corresponding target operation in response to the first control instruction.

[0222] The first device 1010 is also used to detect motion data and send the motion data to the control end.

[0223] The control terminal 1050 is further configured to generate a second control instruction based on the motion data and / or the voice message.

[0224] The second device 1030 is further configured to receive a second control instruction sent by the control terminal, and adjust a corresponding target parameter in response to the second control instruction.

[0225] It should be noted that the device control device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing device control. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device control device will be divided into different functional modules to complete all or part of the functions described above.

[0226] In addition, the device control apparatus and the device control method provided in the above embodiments belong to the same concept, wherein the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0227] See also Fig.11 In an embodiment of the present application, an electronic device 4000 is provided, and the electronic device 4000 may include: a voice remote control, a gateway, a desktop computer, a laptop computer, a server, etc.

[0228] exist Fig.11 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0229] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0230] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0231] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0232] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store a computer program in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to this.

[0233] The memory 4003 stores a computer program, and the processor 4001 can read the computer program stored in the memory 4003 through the communication bus 4002 .

[0234] The computer program is executed by one or more processors 4001 to implement the device control method in the above-mentioned embodiments.

[0235] In addition, an embodiment of the present application provides a storage medium, on which a computer program is stored. The computer program is executed by one or more processors to implement the device control method as described above.

[0236] A computer program product is provided in an embodiment of the present application, including a computer program, which is executed by one or more processors to implement the device control method as described above.

[0237] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A device control method, characterized in that: The method comprises: Acquire a voice message collected by the first device; generating a first control instruction based on the voice message to instruct a second device to perform a corresponding target operation in response to the first control instruction; the second device is a smart device different from the first device; A second control instruction is generated based on the detected motion data from the first device and / or the voice message to instruct the second device to adjust the corresponding target parameter in response to the second control instruction.

2. The method according to claim 1, characterized in that The generating a first control instruction based on the voice message comprises: Performing voice recognition processing on the voice message to obtain a candidate device pointed to by the voice message and the target operation; If the clarity of the candidate device pointed to by the voice message is fuzzy, then according to the set association condition, the second device is obtained by associating the candidate device pointed to by the voice message; the second device is a smart device having the same device attributes as the candidate device; According to the second device and the target operation, the first control instruction is generated to instruct the second device to perform the target operation.

3. The method according to claim 2, characterized in that The step of associating the candidate device pointed to by the voice message to obtain the second device according to the set association condition includes: Determine the space where the device is located based on the set association condition; The candidate device pointed to by the voice message is associated with the space where the device is located to obtain the second device; the second device refers to the candidate device located in the space where the device is located.

4. The method according to claim 1, characterized in that Before generating the second control instruction based on the detected motion data from the first device and / or the voice message, the method further includes: During the process of the second device executing the target operation, detecting whether the first device is in a first state; If yes, then obtain motion data of the first device corresponding to the first state.

5. The method according to claim 4, characterized in that After detecting whether the first device is in the first state, the method further includes: If it is detected that the first device is not in the first state, the first device is controlled to enter a dormant state until the voice message is acquired again.

6. The method according to claim 1, characterized in that Before generating the second control instruction based on the detected motion data from the first device and / or the voice message, the method further includes: When the first device is in the second state, detecting whether a relative motion operation is triggered on a component configured in the first device; If yes, then obtain the motion data of the first device corresponding to the second state.

7. The method according to claim 1, characterized in that The generating a second control instruction based on the detected motion data from the first device and / or the voice message comprises: determining a target parameter to be adjusted for the second device based on the voice message and / or the motion data, and determining an adjustment ratio corresponding to the target parameter according to the motion data; Generate the second control instruction to instruct the second device to adjust the target parameter according to the adjustment ratio.

8. The method according to claim 7, characterized in that The determining the target parameter to be adjusted by the second device based on the voice message and / or the motion data includes: If the voice message does not include the target parameter, determining the type of motion of the first device in the first state based on the motion data; If it is monitored that the motion type of the first device meets the trigger condition in the configured first automation scheme, the target parameter associated with the satisfied trigger condition is extracted from the first automation scheme; the first automation scheme controls the second device to adjust the target parameter by monitoring whether the motion type of the first device meets the trigger condition.

9. The method according to claim 7, characterized in that The determining the target parameter to be adjusted by the second device based on the voice message and / or the motion data includes: If the voice message does not include the target parameter, determining at least one candidate parameter adjustable by the second device, and obtaining a configured priority of each of the candidate parameters; The candidate parameter with the highest priority is used as the target parameter.

10. The method according to claim 7, characterized in that The determining, according to the motion data, the adjustment ratio corresponding to the target parameter includes: Determine a motion parameter of the first device indicated by the motion data, wherein the motion parameter includes one or more of a motion direction, a motion amplitude, a motion angle, a motion duration, a motion speed, and an acceleration; An adjustment ratio corresponding to the target parameter is determined according to the motion parameter of the first device.

11. The method according to any one of claims 1 to 10, characterized in that Before acquiring the voice message collected by the first device, the method further includes: Detecting whether there is a wake-up operation triggered for the first device; If yes, the first device is controlled to enter a wake-up state from a sleep state, so that the first device in the wake-up state collects the voice message.

12. The method according to any one of claims 1 to 10, characterized in that: Before generating the second control instruction based on the detected motion data from the first device and / or the voice message, the method further includes: Determining a time interval between a time of acquiring the voice message and a time of detecting the motion data; If the time interval does not exceed the set threshold, it is determined that the control object of the first device is the second device, so as to generate the second control instruction instructing the second device to adjust the corresponding target parameter based on the motion data and / or the voice message.

13. The method according to claim 12, characterized in that After determining the time interval between the acquisition time of the voice message and the detection time of the motion data, the method further includes: If the time interval exceeds the set threshold, monitoring the movement type of the first device in the first state; If it is monitored that the motion type of the first device in the first state meets the trigger condition in the second automation solution, a target operation associated with the satisfied trigger condition is extracted from the second automation solution; the second automation solution controls the target device to perform the target operation by monitoring whether the motion type of the first device meets the trigger condition; A third control instruction is generated according to the target device and the target operation to instruct the target device to perform the target operation in response to the third control instruction.

14. A device control device, characterized in that: The device comprises: A message acquisition module, used to acquire a voice message collected by the first device; a first instruction generating module, configured to generate a first control instruction based on the voice message, so as to instruct a second device to perform a corresponding target operation in response to the first control instruction; the second device is a smart device different from the first device; The second instruction generating module is used to generate a second control instruction based on the detected motion data from the first device and / or the voice message to instruct the second device to adjust the corresponding target parameter in response to the second control instruction.

15. A device control system, characterized in that: The system comprises: a first device, a second device, and a control terminal; wherein, The first device is used to collect voice messages and send the voice messages to the control terminal; The control terminal is used to generate a first control instruction based on the voice message; The second device is used to receive the first control instruction sent by the control terminal, and execute a corresponding target operation in response to the first control instruction; The first device is further used to detect motion data and send the motion data to the control end; The control terminal is further used to generate a second control instruction based on the motion data and / or the voice message; The second device is further configured to receive the second control instruction sent by the control terminal, and adjust the corresponding target parameter in response to the second control instruction.

16. An electronic device comprising at least one processor and at least one memory, wherein: A computer program is stored in the memory, wherein the computer program, when executed by the processor, implements the device control method according to any one of claims 1 to 13.

17. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by one or more processors, the device control method according to any one of claims 1 to 13 is implemented.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by one or more processors, the device control method according to any one of claims 1 to 13 is implemented.

Images