Equipment control method, storage medium and electronic equipment

Abstract

The invention relates to the technical field of smart home, and discloses a device control method, a storage medium and an electronic device.The method comprises the steps that in the process that a user sends out a voice instruction to expect to control a certain device, the user can send out gesture information used for determining a target device expected to be controlled through gesture operation within a preset period of time; therefore, the voice instruction is executed by the target device expected to be controlled. Specifically, the equipment control method comprises the following steps: after a voice instruction sent by a user is recognized, performing image recognition on image data in a preset time period in an environment based on voice generation time when the user sends the voice instruction, and obtaining gesture information used for determining target equipment expected to be controlled by the user through an image recognition result, and finally, outputting a control instruction through the gesture information and the voice instruction to control target equipment expected to be controlled by the user to execute the voice instruction sent by the user.

Description

Technical Field

[0001] The present invention relates to the field of smart home technology, and in particular to a device control method, a storage medium and an electronic device. Background Art

[0002] When users use voice control to control smart home devices such as smart air conditioners and smart TVs, the control device can perform voice recognition on the user's voice and determine the voice commands that the user wants the smart home device to execute based on the voice recognition results, and then control the corresponding smart home device to execute the voice commands, thereby enabling the user to control the smart home device.

[0003] However, when there are multiple smart home devices in the user's environment, similar or identical voice commands may be executed by multiple identical or same type of smart home devices, causing the voice commands issued by the user to be executed by the wrong smart home device, affecting the user experience. Summary of the invention

[0004] The embodiments of the present invention provide a device control method, a storage medium and an electronic device.

[0005] In a first aspect, the present invention provides a device control method, applied to a first device, comprising: receiving a voice command issued by a user; determining a voice generation time corresponding to the voice command, and obtaining a target image of the user within a preset time period, wherein the preset time period is determined based on the voice generation time, and the target image includes body language information made by the user for controlling a second device; based on the voice command and the target image, outputting a control command, the control command being used to control the second device to execute the voice command.

[0006] In an embodiment of the present application, after the first device recognizes the voice command issued by the user, it performs image recognition on image data within a preset time period in the environment based on the voice generation time when the user issues the voice command, obtains the user's limb information used to determine the second device that the user wants to control through the image recognition result, and finally outputs a control instruction through a target image including the limb information and the voice instruction to control the second device that the user wants to control to execute the voice command issued by the user.

[0007] In a possible implementation of the first aspect above, obtaining a target image of a user within a preset time period further includes: determining a sound source position corresponding to a voice command, and obtaining a target image of the user within the preset time period corresponding to the sound source position.

[0008] In a possible implementation of the first aspect above, obtaining a target image of a user within a preset time period includes: determining an image generation time of the target image; and determining that the target image is an image generated within the preset time period when a time difference between the image generation time and the voice generation time satisfies a preset time threshold.

[0009] In a possible implementation of the first aspect above, obtaining a target image of a user within a preset time period also includes: based on the limb information in the target image, determining a target area that the limb corresponding to the limb information is facing and the regional coordinates of the target area; corresponding to the device coordinates preset for the second device and the regional coordinates satisfying a preset coordinate matching relationship, determining that the limb information is facing the second device.

[0010] In a possible implementation of the first aspect above, outputting a control instruction includes: determining the execution device type corresponding to the voice instruction; and outputting the control instruction when the device type corresponding to the second device and the execution device type satisfy a preset type association relationship.

[0011] In a possible implementation of the first aspect above, outputting a control instruction includes: determining multiple executable instructions corresponding to the second device; and outputting a control instruction in response to the voice instruction and any instruction among the multiple executable instructions satisfying a preset instruction similarity relationship.

[0012] In a possible implementation of the first aspect above, the limb information includes any one or more combinations of the following: hand movements, head movements, leg movements, and overall body movements.

[0013] In a second aspect, the present invention provides a device control system, comprising: a voice collection device, for collecting sounds made by a user and converting the sounds into voice data and sending them to a first device; an image collection device, for collecting user images and converting the user images into image data and sending them to the first device; a first device, for receiving the voice data sent by the voice collection device to determine voice instructions, and for receiving the image data sent by the image collection device to determine a target image including body information made by the user for controlling a second device, and for controlling the second device to execute voice instructions; a second device, for executing voice instructions; a third device, which is the same or the same type of device as the second device, and the third device can be used to execute voice instructions.

[0014] In the third aspect, the present invention provides a device control apparatus, comprising: a voice recognition module, for receiving voice data issued by a user and determining the voice instruction corresponding to the voice data, as well as the voice generation time and sound source position corresponding to the voice instruction; an image recognition module, for acquiring image data of the user corresponding to the sound source position within a preset time period based on the sound source position and the voice generation time, and for identifying image information in the image data, so as to use the image in the image data that includes body language information made by the user for controlling a second device as a target image; a control module, for outputting a control instruction based on the voice instruction and the target image, the control instruction being used to control the second device to execute the voice instruction.

[0015] In a fourth aspect, an embodiment of the present invention provides a readable storage medium having instructions stored thereon, which, when executed on an electronic device, enables the electronic device to implement any one of the device control methods provided by the first aspect and various possible implementations of the first aspect.

[0016] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device; and a processor, which is one of the processors of the electronic device, for executing the instructions stored in the memory to implement any one of the device control methods provided by the first aspect and various possible implementations of the first aspect.

[0017] In a sixth aspect, an embodiment of the present invention provides a program product, which includes instructions, and when the instructions are executed by an electronic device, the electronic device can implement any device control method provided by the first aspect and various possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 According to some embodiments of the present invention, a scene graph of voice control is shown;

[0019] Figure 2 According to some embodiments of the present invention, a scene graph of device control is shown;

[0020] Figure 3A and 3B According to some embodiments of the present invention, an example schematic diagram of a gesture is shown;

[0021] Figure 4 According to some embodiments of the present invention, a module diagram of a smart home hub is shown;

[0022] Figure 5 According to some embodiments of the present invention, a flow chart is shown;

[0023] Figure 6According to some embodiments of the present invention, a flow chart of a device control method is shown;

[0024] Figure 7 According to some embodiments of the present invention, a structural diagram of a smart home hub is shown. DETAILED DESCRIPTION

[0025] Illustrative embodiments of the present invention include, but are not limited to, a device control method, a storage medium, and an electronic device.

[0026] The following is combined with Figures 1 to 7 The technical solution of the present invention is introduced.

[0027] Figure 1 According to some embodiments of the present invention, a scene diagram of voice control is shown. Figure 1 As shown, Figure 1 The scene shown includes smart home devices such as a voice collection device 11 (such as a speaker), a smart home hub 12 (such as a router), a TV 13, an air conditioner 14, and an air conditioner 15, and the voice collection device 11, the TV 13, the air conditioner 14, and the air conditioner 15 are all connected to the smart home hub 12 through a network for communication. It can be understood that Figure 1 The environment of the scene shown can be any environment where smart home devices are deployed, such as an office, a residential living room, a shopping mall, etc., without specific limitation.

[0028] In some embodiments, the voice collection device 11 is used to collect the voices emitted by the user in the environment, and send the collected voice data to the smart home hub 12. After receiving the voice data, the smart home hub 12 performs voice recognition on the voice data and determines the voice instructions emitted by the user in the voice data for controlling the device according to the voice recognition result. Subsequently, the smart home hub 12 generates a control instruction based on the voice instruction and the execution device corresponding to the voice instruction to control the corresponding execution device to execute the voice instruction, thereby realizing the user's voice control over the execution device.

[0029] In some embodiments, Figure 1 As shown, when the user issues a voice command "turn on the air conditioner" in the hope of controlling the air conditioner 15 to turn on, the voice collection device 11 can collect the voice data of "turn on the air conditioner" and send the voice data to the smart home hub 12. After performing voice recognition on the received voice data, the smart home hub 12 can determine that the voice command issued by the user is: turn on the air conditioner, and the execution device corresponding to the voice command is: air conditioner. Subsequently, the smart home hub 12 can search for air conditioners in the environment that have established a communication connection with the smart home hub 12 in advance through the determined execution device.

[0030] In some embodiments, Figure 1 As shown, Figure 1 The scene shown includes two air-conditioning devices, air-conditioning 14 and air-conditioning 15, and both air-conditioning devices establish communication connections with the smart home hub 12. When the smart home hub 12 searches for air-conditioning devices in the environment that have pre-established communication connections with the smart home hub 12 through a determined execution device, it may search for both air-conditioning devices, air-conditioning 14 and air-conditioning 15. That is, the smart home hub 12 may control air-conditioning 14 and air-conditioning 15 to turn on at the same time according to the voice command to turn on the air conditioner.

[0031] However, when the user issues a voice command "turn on the air conditioner", he only wants to control the air conditioner 15 to turn on, and does not want the air conditioner 14 to turn on. However, since the voice commands corresponding to the same or same type of smart home devices may be the same or similar, when the user issues a voice command "turn on the air conditioner" and expects the air conditioner 15 to turn on, the correct air conditioner 15 and the wrong air conditioner 14 may be turned on at the same time.

[0032] In this way, when there are multiple smart home devices of the same or same type in the user's environment, similar or identical voice commands may be executed by multiple smart home devices, causing the voice commands issued by the user to be executed by the wrong smart home device, affecting the user experience.

[0033] To this end, the present invention proposes a device control method, in which when a user issues a voice command to control a certain device, the user can, within a preset time period, issue gesture information for determining the target device to be controlled through body movements with directional meanings, such as hand gestures, to ensure that the voice command is executed by the target device to be controlled. Specifically, the device control method includes: after the smart home hub recognizes the voice command issued by the user, based on the voice generation time when the user issues the voice command, the smart home hub performs image recognition on the image data within a preset time period in the environment, obtains the gesture information of the user for determining the target device to be controlled through the image recognition result, and finally outputs the control instruction through the gesture information and the voice instruction to control the target device to be controlled by the user to execute the voice instruction issued by the user.

[0034] Thus, when there are multiple smart home devices of the same or same type in the environment (such as Figure 1 When the user is trying to control the two air-conditioning devices shown in the figure), the smart home hub can combine voice recognition and image recognition to determine the voice command issued by the user and the target device that the user wants to control, so as to prevent the voice command issued by the user from being executed by the wrong device that is the same or of the same type as the target device. It can be understood that the smart home hub determines the image data corresponding to the preset time period through the voice generation time of the voice command, which can ensure that the gesture information in the image recognition and the voice command are the commands issued by the user for the same target device, thereby improving the accuracy of device control.

[0035] In some embodiments, the user can perform a gesture operation for determining the target device to be controlled before or after issuing a voice command, or the user can also perform a corresponding gesture operation while issuing a voice command. The smart home hub can obtain image data of the environment within a preset time period, for example, from 3 seconds before to 3 seconds after the voice generation time, based on the voice command and the voice generation time when the user issues the voice command. It can be understood that the image data can be a single-frame image, or a combination of multiple frames of continuous or non-continuous images, etc., without specific limitation.

[0036] In other embodiments, the smart home hub can also obtain video data of the environment within a preset time period, for example, from 5 seconds before to 5 seconds after the voice generation time, based on the voice command and the voice generation time when the user issues the voice command. It can be understood that the preset time period based on the voice generation time can be customized according to the different types of data to be obtained, such as when the data type is image, the image data from 3 seconds before to 3 seconds after the voice generation time is obtained, and when the data type is video, the image data from 5 seconds before to 5 seconds after the voice generation time is obtained; or the user can also set the preset time period before or after the voice generation time according to the user's own operating habits, etc., without specific limitation.

[0037] In some embodiments, the smart home hub can also obtain image data in a preset period of time corresponding to the sound source position in the environment based on the voice command and the sound source position when the user issues the voice command. It can be understood that the user position can be located by the sound source position, that is, the image data obtained by the smart home hub based on the sound source position is the user image corresponding to the user who issued the voice command, thereby ensuring that the gesture information and voice command in the image recognition are the same user's instructions for the same target device, thereby improving the accuracy of device control.

[0038] In some embodiments, the position space coordinate system of the deployment environment of the smart home device can be constructed in advance by planar measurement methods (such as rangefinders), indoor positioning methods (such as wireless, Bluetooth signals), photogrammetry methods (such as photographs), etc., and the device coordinates of each smart home device in the environment can be determined by the position space coordinate system; and the sound space coordinate system of the deployment environment of the smart home device can be constructed by sound source positioning, multi-microphone positioning, ultrasonic positioning, sound intensity positioning, etc. Subsequently, the position space coordinate system in the deployment environment corresponding to the smart home device, the device coordinates of each smart home device, and the sound space coordinate system are deployed to the smart home hub so that the smart home hub can realize the corresponding function. For example, the smart home hub can determine the sound source position of the user's voice command through the sound space coordinate system, and determine the user position corresponding to the sound source position through the sound source position and the position space coordinate system, so as to obtain the user image of the user at the corresponding sound source position when the user issues the voice command.

[0039] In some embodiments, the smart home hub uses images with directional body movements as target images during image recognition of image data. It is understood that the directional body movements referred to herein include: hand movements, such as hand raising, finger extension, elbow bending and other gestures; head movements, such as nodding in a certain direction; leg movements, such as leg raising, foot raising and other operations; overall body movements, such as trunk tilting, trunk rotation and other operations; in other embodiments, it can also be the operation of other parts of the user's body, without specific limitation.

[0040] In some embodiments, taking gesture operation as an example, after obtaining the target image, the smart home hub obtains the gesture direction of the gesture information in the target image and the target area of ​​the gesture direction, and based on the area coordinates of the target area in the position space coordinate system and the device coordinates of each smart home device, the smart home device corresponding to the device coordinates overlapping with the area coordinates is taken as the target device.

[0041] In some embodiments, after the smart home hub determines the voice command issued by the user and the target device that the user wants to control, it can further match the determined target device with the voice command to ensure that the target device is the device that the user wants to control when issuing the voice command. For example, matching is performed by matching the execution device type corresponding to the voice command with the device type of the target device, or matching the voice command with the executable instruction corresponding to the target device.

[0042] In some embodiments, when the target device does not match the voice command, for example, when the execution device type corresponding to the voice command and the device type of the target device do not satisfy the preset type association relationship, or the voice command and multiple executable instructions corresponding to the target device do not satisfy the preset instruction similarity relationship, the intelligent hub device can re-perform image recognition on the acquired image data to re-determine the target device, and match it with the voice command based on the re-determined target device, ensuring that the target device is the device that the user expects to control when issuing the voice command, thereby improving the accuracy of device control.

[0043] Figure 2 According to some embodiments of the present invention, a scene graph of device control is shown. Figure 2 As shown, Figure 2 The scene shown includes smart home devices such as voice collection device 21, smart home hub 22, TV 23, air conditioner 24, air conditioner 25 and image collection device 26 (such as point cloud extraction devices such as depth camera and stereo camera installed on TV 23), and voice collection device 21, image collection device 23, air conditioner 24, air conditioner 25 and image collection device 26 are all connected to the smart home hub 22 through the network to form a device control system. It can be understood that Figure 2 The environment of the scene shown can be any environment where smart home devices are deployed, such as an office, a residential living room, a shopping mall, etc., and the smart home hub 22 pre-stores Figure 2 The location space coordinate system of the environment, the device coordinates of each smart home device, and the sound space coordinate system are shown.

[0044] In some embodiments, Figure 2As shown, when the user issues a voice of "27 degrees" and points to the air conditioner 25 in the hope of controlling the air conditioner 25 to adjust to 27 degrees, the voice collection device 21 can collect the voice data of "27 degrees" and send the voice data to the smart home hub 22. The smart home hub 22 can determine that the voice command issued by the user is: adjust the temperature to 27 degrees after performing voice recognition based on the received voice data. At the same time, the smart home hub 22 determines the voice generation time corresponding to the voice command based on the collection time of the voice data, that is, the time when the user issues the voice of "27 degrees", and determines the sound source position corresponding to the voice command based on the sound space coordinate system, that is, the position when the user issues the voice of "27 degrees". Subsequently, the smart home hub 22 obtains the image data corresponding to the position of the user when the user issues the voice of "27 degrees" in the period from 3 seconds before to 3 seconds after the user issues the voice of "27 degrees" through the image collection device 26, and performs image recognition on the acquired image data to use the image including the gesture information of the user's gesture operation as the target image. Finally, the smart home hub 22 determines the gesture direction and the target area of ​​the gesture direction according to the arm direction of the gesture information in the target image, and obtains the smart home device corresponding to the device coordinates overlapping with the area coordinates of the target area based on the position space coordinate system, thereby determining that the target device that the user wants to control is: air conditioner 25.

[0045] In some embodiments, after determining the target device, the smart home hub 22 can further match the target device with the voice command, for example, by matching the execution device type corresponding to the voice command and the device type of the target device, or by matching the voice command and multiple executable instructions corresponding to the target device, to ensure that the target device is the device that the user expects to control when issuing the voice command, thereby improving the accuracy of device control.

[0046] In some embodiments, when the smart home hub 22 determines that the target device matches the voice command, such as the voice command to adjust the temperature to 27 degrees and the temperature adjustment command corresponding to the air conditioner 25 are of the same type of command, the air conditioner 25 is controlled to adjust the temperature to 27 degrees to enable the user to control the air conditioner 25.

[0047] Understandably, Figure 2In the scenario shown, both air conditioners 24 and 25 can be used to execute the voice command of adjusting the air conditioner temperature corresponding to the voice of "27 degrees" issued by the user, but when the user issues the voice of "27 degrees", he only wants to control air conditioner 25 to adjust the temperature. In this way, the smart home hub 22 recognizes the gesture operation of the user for determining the target device to be controlled based on voice recognition, and determines the voice command issued by the user and the target device to be controlled by the user by combining voice recognition and image recognition, ensuring that the voice command issued by the user is executed by the target device to be controlled by the user.

[0048] It can be understood that the target device that the user wants to control when issuing a voice command can be determined through image recognition, that is, the user can omit the voice command such as the wake-up word and the specified device when issuing a voice command, and can directly control the target device to perform the corresponding operation based on the voice command of the operation that the target device wants to perform. Figure 2 As shown, when the user utters "27 degrees" and points to the corresponding air conditioner 25, the temperature of the pointed air conditioner 25 can be adjusted to 27 degrees, omitting the voice commands for specifying the device such as "control the air conditioner" and "turn on the air conditioner".

[0049] In other embodiments, the target image acquired by the smart home hub 22 may be an overall image of the user's body, for example Figure 2 As shown, the user's arm direction is identified through the overall image of the user's body to determine that the target device is the air conditioner 25. Alternatively, the acquired target image may also be a partial image of the user's body, for example Figure 3A When the user points his finger at the air conditioner and utters the voice command "27 degrees", the user's finger direction can be identified through a partial image of the user's finger to determine that the user wants the air conditioner to execute a voice command to adjust the temperature to 27 degrees; or for example Figure 3B When the user points his finger at the ceiling light and utters the voice command "turn off", the direction of the user's finger can be identified through a partial image of the user's finger to determine that the user wants the ceiling light to execute the voice command to turn off the light.

[0050] Figure 4 According to some embodiments of the present application, a module diagram of a smart home hub 22 is shown. Figure 4 As shown, the smart home hub 22 includes a voice recognition module 41, an image recognition module 42 and a control module 43. It can be understood that Figure 4 The modules of the smart home hub 22 shown are only examples. In other embodiments, the smart home hub 22 may include more or fewer modules than shown in the figure, without limitation.

[0051] In some embodiments, the speech recognition module 41 is used to receive speech data, such as Figure 2The voice data sent by the voice collection device 21 is collected, and the voice instruction, the voice generation time of the voice instruction, and the sound source position of the voice instruction are determined according to the voice data.

[0052] In some embodiments, the image recognition module 42 is used to receive image data corresponding to the sound source position within a preset time period determined by the speech generation time, such as Figure 2 The control module 43 is configured to collect image data sent by the image collection device 26, and determine a target image including gesture information of the user's gesture operation according to the image data. In some embodiments, the control module 43 is configured to determine the target device corresponding to the target image and the matching degree between the target device and the voice command, and to generate a control command to control the corresponding target device to execute the voice command when the target device matches the voice command.

[0053] In some embodiments, Figure 2 The voice acquisition module and image acquisition module shown are respectively deployed in the voice collection device 21 and the image collection device 26, that is, the voice acquisition module and the image acquisition module are respectively deployed in different devices; in other embodiments, the voice acquisition module and the image acquisition module can also be deployed in the same device, or deployed together with the voice recognition module 41, the image recognition module 42 and the control module 43 in the smart home hub 23, etc., without specific limitation.

[0054] The following Figure 2 The scenes shown and Figure 4 Taking the modules shown as an example, some embodiments of the present application are illustrated.

[0055] Figure 5 According to some embodiments of the present application, a flow chart is shown.

[0056] In some embodiments, in combination Figure 5 and Figure 2 As shown, after receiving the voice data sent by the voice collection device 21, the voice recognition module 41 performs voice recognition analysis on the voice data to obtain the voice command corresponding to the voice data, the voice generation time of the voice command, and the sound source position of the voice command. Subsequently, the voice recognition module 41 sends the voice command to the control module 43, and sends the voice generation time of the voice command and the sound source position of the voice command to the image recognition module 42.

[0057] In some embodiments, after the image recognition module 42 receives the speech generation time and sound source location sent by the speech recognition module 41, it obtains image data within a preset time period determined by the corresponding speech generation time and corresponding to the sound source location through the image collection device 26, performs image recognition analysis on the image data, and sends the target image including gesture information of the user's gesture operation to the control module 43.

[0058] In some embodiments, the control module 43 determines the target device corresponding to the gesture information in the target image based on the received target image, and determines whether there is a target device that can execute the voice instruction based on the matching degree between the target device and the voice instruction. For example, when the execution device type corresponding to the voice instruction and the device type of the target device meet the preset type association relationship, or the voice instruction and any executable instruction corresponding to the target device meet the preset instruction similarity relationship, it can be determined that there is a target device that can execute the voice instruction, and the voice instruction is sent to the target device to control the target device to execute the voice instruction. For another example, when the execution device type corresponding to the voice instruction and the device type of the target device do not meet the preset type association relationship, or the voice instruction and multiple executable instructions corresponding to the target device do not meet the preset instruction similarity relationship, it can be determined that there is no target device that can execute the voice instruction, and the operation ends.

[0059] Figure 6 According to some embodiments of the present application, a flow chart of a device control method is shown. It can be understood that Figure 6 The execution subject of each step of the process shown is the smart home hub. Figure 6 The execution entities of each step will not be described repeatedly in the steps of the process shown. Figure 6 As shown, the process includes but is not limited to the following steps:

[0060] S601: Determine the voice command issued by the user, as well as the time information and location information when the user issues the voice command.

[0061] In some embodiments, Figure 2 As shown, when the user issues a voice of "27 degrees" and points to the air conditioner 25 in the hope of controlling the air conditioner 25 to adjust to 27 degrees, the smart home hub 22 can obtain the user's voice data through the voice collection device 21, and perform voice recognition on the voice data to determine that the voice command issued by the user is: adjust the temperature to 27 degrees. Then the smart home hub 22 determines the time information when the user issues the voice command through the collection time of the voice data, and determines the location information when the user issues the voice command through the sound source location of the voice data.

[0062] S602: Determine a target time period based on the time information, and obtain a target image corresponding to the position information within the target time period and including gesture information.

[0063] In some embodiments, the smart home hub 22 obtains the time period from 3 seconds before to 3 seconds after the time information as the target time period based on the determined time information, and obtains the image data of the corresponding position information in the target time period through the image collection device 26. Then the smart home hub 22 performs image recognition on the image data to identify the image including the gesture information of the gesture operation such as raising the hand and bending the elbow as the target image.

[0064] S603: Determine a target device based on a target area of ​​the gesture direction corresponding to the gesture information.

[0065] In some embodiments, the smart home hub 22 obtains the gesture direction of the gesture information in the target image and the target area of ​​the gesture direction, and based on the area coordinates of the target area in the position space coordinate system and the device coordinates of each smart home device, takes the smart home device corresponding to the device coordinates overlapping with the area coordinates as the target device.

[0066] S604: When the target device and the voice instruction meet a matching condition, output a control instruction to control the target device to execute the voice instruction.

[0067] In some embodiments, after the smart home hub determines the voice command issued by the user and the target device that the user expects to control, the matching degree between the target device and the voice command can be determined by the execution device type corresponding to the voice command and the device type of the target device. For example, when the execution device type corresponding to the voice command is an air-conditioning device, and the device type of the target device is also an air-conditioning device, the execution device type and the device type satisfy the preset type association relationship, that is, the target device and the voice command satisfy the matching condition, and the target device is the device that the user expects to control when issuing the voice command. Or for example, when the execution device type corresponding to the voice command is an air-conditioning device, but the device type of the target device is a television device, the execution device type and the device type do not satisfy the preset type association relationship, that is, the target device and the voice command do not satisfy the matching condition, and the target device may not be the device that the user expects to control when issuing the voice command.

[0068] In other embodiments, the smart home hub can also determine the matching degree between the target device and the voice command through the voice command and the multiple executable instructions corresponding to the target device. For example, corresponding to the air-conditioning device, the executable voice command includes: temperature adjustment command (such as 26 degrees, lower the temperature, etc.), open command (such as open, etc.), close command (such as close); corresponding to the TV device, the executable voice command includes: play command (such as play video, pause video, etc.), open command, close command. It can be understood that when the voice command is a temperature adjustment command and the target device is an air-conditioning device, the voice command and the temperature adjustment command in the multiple executable instructions corresponding to the air-conditioning device meet the preset command similarity relationship, that is, the target device and the voice command meet the matching conditions, and the target device is the device that the user expects to control when issuing the voice command; and when the voice command is a temperature adjustment command, but the target device is a TV device, the voice command and the multiple executable instructions corresponding to the TV device do not meet the preset command similarity relationship, that is, the target device and the voice command do not meet the matching conditions, and the target device may not be the device that the user expects to control when issuing the voice command.

[0069] Furthermore, when the smart home hub 22 determines that the execution device type corresponding to the voice command and the device type of the target device are the same or similar types, or determines that the voice command and any executable instruction corresponding to the target device are the same or similar instructions, it can be determined that the target device and the voice command meet the matching conditions, and output a control instruction to control the target device that the user expects to control to execute the voice command issued by the user.

[0070] It can be understood that when a user issues a voice command and determines the target device for executing the voice command through gestures, the gesture operation corresponding to the gesture is usually performed from a few seconds before to a few seconds after the voice command is issued. In this way, the gesture operation is determined by the image data of the target time period determined by the time information when the user issues the voice command. This can ensure that the acquired voice command and gesture operation are made by the user for the same device, avoiding the situation where the user controls multiple devices by voice and gesture respectively, resulting in an incorrect match between the voice command and the gesture operation, which causes the voice command issued by the user to be converted into an erroneous device command corresponding to the erroneous gesture operation.

[0071] It can be understood that, on the basis of ensuring through the target time period that the acquired voice commands and gesture operations are made by the user for the same device, the image data within the target time period is further located through the position information of the sound source position when the user issues the voice command, so as to ensure that the acquired voice commands and gesture operations are made by the same user for the same device, and avoid the situation in which there are multiple users in the environment who respectively control multiple devices with corresponding voices and gestures, resulting in the voice commands issued by the user being replaced by incorrect gesture operations or incorrect device commands corresponding to the gesture operations of another user.

[0072] It can be understood that after determining the target device and the voice command, the target device and the voice command are further matched, such as by matching the execution device type corresponding to the voice command and the device type of the target device, or by matching the voice command with the executable instruction corresponding to the target device, etc. This can avoid the situation where the user controls multiple devices by voice and gesture respectively, and an incorrect match between the voice command and the gesture operation, resulting in the voice command issued by the user being executed by the wrong device instruction corresponding to the wrong gesture operation, thereby ensuring that each voice command issued by the user is executed by the correct target device expected by the corresponding user.

[0073] Figure 7 According to some embodiments of the present application, a hardware diagram of a smart home hub 22 is shown. Figure 7 As shown, the smart home hub 22 includes: a processor 101, a communication interface 103 and a memory 102. Among them, the processor 101, the communication interface 103 and the memory 102 can be connected to each other through an internal bus 104, and can also communicate through other means such as wireless transmission. In the embodiment of the present application, the connection through the bus 104 is taken as an example. The bus 104 can be a peripheral component interconnect express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a computer express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 104 can be divided into an address bus, a data bus, a control bus, etc. In addition to the data bus, the bus 104 can also include a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus 104 in the figure.

[0074] The processor 101 may be composed of at least one general-purpose processor, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 101 executes various types of digital storage instructions, such as software or firmware programs stored in the memory 102, which enables the smart home hub 22 to provide a variety of services.

[0075] The memory 102 is used to store program codes, and is controlled and executed by the processor 101 to execute the processing steps of the device control method in the above embodiment.

[0076] It should be noted that this embodiment can be implemented by a general physical server, for example, an ARM server or an X86 server, or it can be implemented by a virtual machine based on a general physical server combined with NFV technology. A virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. This application does not make any specific limitations.

[0077] The memory 102 may include a volatile memory, such as a random access memory (RAM); the memory 102 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 102 may also include a combination of the above types. The memory 102 may store program codes to specifically execute the steps of the device control method in the figure above, which will not be described in detail here.

[0078] The communication interface 103 can be a wired interface (such as an Ethernet interface), an internal interface (such as a PCIe bus interface), a wired interface (such as an Ethernet interface) or a wireless interface (such as a cellular network interface or a wireless LAN interface) for communicating with other devices or modules.

[0079] Understandably, Figure 7 The structure of the smart home hub 22 shown is only an example. In other embodiments, the smart home hub 22 may include more or fewer components than shown, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware.

[0080] In some embodiments, the embodiments of the present application further provide a computer-readable medium, which stores a program code. When the computer program code runs on a computer, the computer executes the methods in the above aspects.

[0081] In some embodiments, the embodiments of the present application further provide a computer program product, which includes: a computer program code, which enables the computer to execute the methods in the above aspects when the computer program code runs on a computer.

[0082] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0083] It should be noted that the units / modules mentioned in the various device embodiments of the present invention are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present invention. In addition, in order to highlight the innovative part of the present invention, the above-mentioned device embodiments of the present invention do not introduce units / modules that are not closely related to solving the technical problems proposed by the present invention, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0084] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0085] While the present invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention.

Claims

1. A device control method, applied to a first device, characterized in that: include: Receiving a voice command from a user; Determine the voice generation time corresponding to the voice instruction, and obtain the target image of the user within a preset time period, wherein the preset time period is determined based on the voice generation time, and the target image includes body movement information made by the user for controlling the second device; Based on the voice instruction and the target image, a control instruction is output, where the control instruction is used to control the second device to execute the voice instruction.

2. The method according to claim 1, characterized in that The acquiring the target image of the user within a preset period of time includes: The sound source position corresponding to the voice command is determined, and the target image of the user at a position corresponding to the sound source position within the preset time period is acquired.

3. The method according to claim 1, characterized in that The acquiring the target image of the user within a preset period of time includes: Determining an image generation time of the target image; Corresponding to the time difference between the image generation time and the speech generation time satisfying a preset time threshold, it is determined that the target image is an image generated within the preset time period.

4. The method according to claim 1, characterized in that: The acquiring of the target image of the user within a preset period of time further includes: Based on the limb information in the target image, determining a target area toward which the limb corresponding to the limb information is directed and area coordinates of the target area; The device coordinates preset for the second device and the area coordinates satisfy a preset coordinate matching relationship, and it is determined that the limb information is directed toward the second device.

5. The method according to claim 1, characterized in that The output control instruction includes: Determining the execution device type corresponding to the voice command; The control instruction is outputted corresponding to the device type of the second device and the execution device type satisfying a preset type association relationship.

6. The method according to claim 1, characterized in that The output control instruction includes: Determining a plurality of executable instructions corresponding to the second device; Corresponding to the voice instruction and any instruction among the multiple executable instructions satisfying a preset instruction similarity relationship, the control instruction is output.

7. The method according to claim 1, characterized in that The body information includes any one or more combinations of the following: Hand movements, head movements, leg movements, overall body movements.

8. A device control system, characterized in that: include: A voice collection device, used to collect the voice of the user and convert the voice into voice data and send it to the first device; An image collection device, used for collecting user images and converting the user images into image data and sending them to the first device; The first device is used to receive the voice data sent by the voice collection device to determine the voice command, and to receive the image data sent by the image collection device to determine the target image including the body information made by the user for controlling the second device, and to control the second device to execute the voice command; The second device is used to execute the voice command; The third device is the same as or of the same type as the second device, and the third device can be used to execute the voice command.

9. A device control device, characterized in that: include: A speech recognition module, used to receive speech data sent by a user and determine the speech instruction corresponding to the speech data, as well as the speech generation time and sound source position corresponding to the speech instruction; an image recognition module, which acquires image data of a user at a position corresponding to the sound source position within a preset time period based on the sound source position and the speech generation time, and recognizes image information in the image data, so as to use an image in the image data including body information made by the user for controlling the second device as a target image; A control module is used to output a control instruction based on the voice instruction and the target image, wherein the control instruction is used to control the second device to execute the voice instruction.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores instructions, which, when executed on an electronic device, enable the electronic device to implement the method according to any one of claims 1 to 7.

11. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device; And a processor, which is one of the processors of the electronic device, configured to execute instructions stored in the memory to implement the method according to any one of claims 1 to 7.

Images