Smart home equipment control method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a smart home equipment control method and device, equipment and a storage medium. The method comprises the following steps: acquiring target voice information of a target user collected by a microphone; if the target voice information does not support the determination of the target smart home device, acquiring sound source positioning information of the target user according to the target voice information; detecting a person in an image shot by the camera; determining the target user from people contained in the image according to the sound source positioning information; performing gesture detection on the target user, and determining the target smart home device corresponding to the gesture of the target user; and controlling the target smart home device according to the target voice information. According to the embodiment of the invention, the target smart home equipment pointed by the default target voice information can be quickly determined, and the operation failure caused by entering multi-round dialogue selection or incapability of determining the target equipment is avoided, so that the user experience is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a smart home device control method and apparatus, an electronic device, and a storage medium. Background Art

[0002] With the development of the smart home industry, the functions and coverage of IoT smart control are becoming increasingly rich, and people's requirements for smart home intelligence are also constantly increasing. In terms of voice control, from the initial single-device single-intention control through basic models to the current support for complex intents, sentence generation and execution scenarios, etc., the richness of functions and user experience have undergone significant upgrades. However, there are still many details in the voice control of smart home devices that need to be solved.

[0003] When users use voice control of smart home devices, they often face the problem of unclear device reference. For example, when there are fans A, B and C, if the user uses the general phrase "turn on the fan", the system will enter multiple rounds of dialogue and ask the user to make a selection, such as "You have multiple fans, which one do you want to control, the first fan A..." In addition, when multiple devices have the same function (such as fans, air conditioners, and electric heaters all support wind sweeping), the user directly uses the voice control command of the default device type "turn on wind sweeping", and the system cannot process this default command and cannot determine which type of specific device it will be sent to.

[0004] Therefore, how to effectively handle the default voice control statements of the target smart home devices is an important issue to improve the smart home voice experience. Summary of the invention

[0005] The embodiment of the present application provides a smart home device control method to solve the problem of how to effectively handle the default voice control statements of the target smart home device.

[0006] Correspondingly, the embodiments of the present application also provide a smart home device control device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the embodiment of the present application discloses a smart home device control method, which is applied to a smart home system. The smart home system includes smart home devices, a microphone and a camera. The method includes:

[0008] Acquire target voice information of a target user collected by the microphone;

[0009] If the target voice information does not support determining the target smart home device, acquiring the sound source localization information of the target user according to the target voice information;

[0010] Detecting a person in an image captured by the camera;

[0011] Determine the target user from the persons included in the image according to the sound source localization information;

[0012] Performing gesture detection on the target user to determine the target smart home device corresponding to the gesture of the target user;

[0013] The target smart home device is controlled according to the target voice information.

[0014] In an optional embodiment of the present application, after acquiring the target voice information of the target user collected by the microphone, the method further includes:

[0015] Extracting target voiceprint information corresponding to the target voice information;

[0016] Matching the target voiceprint information with pre-stored voiceprint feature information to determine the target voiceprint feature information;

[0017] The target user identity of the target user is determined according to the target voiceprint feature information.

[0018] In an optional embodiment of the present application, determining the target user from people included in the image according to the sound source localization information includes:

[0019] According to the target user identity of the target user, obtaining pre-stored target facial feature information of the target user;

[0020] Performing facial recognition on people at the position corresponding to the sound source localization information in the image, to obtain facial recognition information corresponding to each person at the position corresponding to the sound source localization information in the image;

[0021] The target facial feature information is matched with the facial recognition information corresponding to each person to determine the target user in the image.

[0022] In an optional embodiment of the present application, determining the target user from people included in the image according to the sound source localization information includes:

[0023] If the target user has not pre-stored the target voiceprint information and / or the target facial feature information, performing gesture detection on the person at the position corresponding to the sound source localization information in the image, and determining the smart home device corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image;

[0024] If the number of the smart home devices corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image is one, taking the smart home device corresponding to the gesture of the person in the image as the target smart home device;

[0025] If the number of smart home devices corresponding to the gestures of the person at the position corresponding to the sound source localization information in the image is greater than one, randomly selecting a smart home device corresponding to the gesture of the person in the image;

[0026] The randomly selected smart home device is used as the target smart home device.

[0027] In an optional embodiment of the present application, the smart home device has a unique device name, and the method further includes:

[0028] Acquire the device recognition image captured by the camera;

[0029] Identifying the smart home device in the device recognition image;

[0030] If the smart home device is identified from the device identification image, receiving a first touch operation of the user on the device name, wherein the first touch operation is used to match the device name with the smart home device in the device identification image;

[0031] If the smart home device is not recognized from the device recognition image, receiving a second touch operation of the user on the device name, wherein the second touch operation is used to match the device name with a corresponding position of the smart home device in the device recognition image;

[0032] A target device recognition image including the matched device name is obtained.

[0033] In an optional embodiment of the present application, the performing gesture detection on the target user to determine the target smart home device corresponding to the gesture of the target user includes:

[0034] Performing gesture detection on the target user to obtain a target position corresponding to the gesture of the target user;

[0035] Detecting the smart home device in the image captured by the camera;

[0036] If the smart home device exists at the target position of the image captured by the camera, determining the device name of the smart home device according to the target device recognition image;

[0037] Determining the target smart home device from the image according to the device name of the smart home device;

[0038] If there is no smart home device at the target location of the image taken by the camera, the target user is reminded to add the target smart home device.

[0039] In an optional embodiment of the present application, the smart home system stores a device list, the smart home device has a unique device name, and the smart home device has a corresponding device type. After acquiring the target voice information of the target user collected by the microphone, the method further includes:

[0040] If the target device type of the target smart home device is supported to be determined according to the target voice information, and the device name in the target voice information is missing, then filtering the smart home device of the target device type from the device list;

[0041] If the smart home device of the target device type is not obtained through screening, prompting the user to add the target smart home device;

[0042] If a smart home device of the target device type is obtained through screening, the smart home device of the target device type is used as the target smart home device;

[0043] If more than one smart home device of the target device type is obtained through screening, it is determined that the target voice information does not support the determination of the target smart home device.

[0044] In an optional embodiment of the present application, the performing gesture detection on the target user to determine the target smart home device corresponding to the gesture of the target user includes:

[0045] If more than one smart home device of the target device type is obtained through screening, and there is more than one smart home device of the target device type in the target location corresponding to the gesture of the target user, the target user is prompted to voice input the device name of the target smart home device.

[0046] In an optional embodiment of the present application, the smart home system stores a device function map, the smart home device has a unique device name, the smart home device has a corresponding device type, and the gesture detection of the target user to determine the target smart home device corresponding to the gesture of the target user includes:

[0047] If the target device function of the target smart home device is supported to be determined according to the target voice information, and the device name and the device type in the target voice information are missing, then, according to the target voice information, the smart home device having the target device function is selected in the device function map;

[0048] If the smart home device having the target device function exists at the target location corresponding to the gesture of the target user, the smart home device at the target location corresponding to the gesture of the target user is used as the target smart home device;

[0049] If the smart home device having the target device function does not exist in the target location corresponding to the gesture of the target user, the target user is reminded to add the target smart home device.

[0050] The embodiment of the present application further discloses a smart home device control device, which is applied to a smart home system. The smart home system includes smart home devices, a microphone and a camera. The device includes:

[0051] A voice information acquisition module, used to acquire the target voice information of the target user collected by the microphone;

[0052] A sound source localization module, configured to obtain sound source localization information of the target user according to the target voice information if the target voice information does not support determining the target smart home device;

[0053] An image detection module, used to detect people in the image captured by the camera;

[0054] A target user determination module, configured to determine the target user from the people in the image according to the sound source localization information;

[0055] A gesture detection module, used to detect the gesture of the target user and determine the target smart home device corresponding to the gesture of the target user;

[0056] The device control module is used to control the target smart home device according to the target voice information.

[0057] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the smart home device control method as described in one or more of the embodiments of the present application.

[0058] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the smart home device control method as described in one or more of the embodiments of the present application.

[0059] Compared with the prior art, the embodiments of the present application have the following advantages:

[0060] In an embodiment of the present application, the target voice information of the target user collected by the microphone is obtained; if the target voice information does not support the determination of the target smart home device, the sound source positioning information of the target user is obtained according to the target voice information; the person in the image taken by the camera is detected; according to the sound source positioning information, the target user is determined from the people contained in the image; gesture detection is performed on the target user to determine the target smart home device corresponding to the gesture of the target user; according to the target voice information, the target smart home device is controlled. In view of the situation where there is a default target voice information in the control of the target smart home device, the embodiment of the present application determines the specific target user by combining the sound source positioning information and image recognition technology, and identifies the target smart home device in combination with the gesture of the target user. Through this multimodal approach, the target smart home device pointed to by the default target voice information can be quickly determined, avoiding entering multiple rounds of dialogue selection or operation failure due to the inability to determine the target device, thereby significantly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flowchart of a step of an embodiment of a smart home device control method of the present application;

[0062] Figure 2 is a flow chart of an embodiment of a smart home device control method of the present application;

[0063] Figure 3 It is a structural block diagram of an embodiment of a smart home device control device of the present application;

[0064] Figure 4 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0066] Reference Figure 1 , is a flowchart of a smart home device control method embodiment of the present application, comprising the following steps:

[0067] Step 101: Acquire target voice information of a target user collected by the microphone.

[0068] A smart home device control method shown in an embodiment of the present application is applied to a smart home system, which includes smart home devices, microphones and cameras. In one embodiment, in order to realize sound source localization combined with image detection to determine the target user, the microphone needs to be placed near the camera, and the microphones can be arranged in a 4-microphone square matrix or a 4-microphone cross matrix.

[0069] In step 101, the smart home system may collect target voice information of a target user through a microphone. The target voice information is an instruction for the target user to control smart home devices.

[0070] Step 102: If the target voice information does not support determining the target smart home device, then obtain the sound source localization information of the target user according to the target voice information.

[0071] In this embodiment, if the target voice information does not support determining the target smart home device, it is necessary to obtain the sound source localization information of the target user based on the target voice information.

[0072] The target voice information does not support the determination of the target smart home device, that is, the target user is unclear about the device when using voice control of the smart home device. For example, when there are fans A, B, and C, if the target voice information is "turn on the fan", "turn on the air sweep", etc., the system cannot process such default instructions and thus cannot determine the target smart home device.

[0073] In one embodiment, the sound source localization information of the target user can be obtained by a sound source localization algorithm. The following shows examples of using several sound source localization algorithms to obtain the sound source localization information of the target user.

[0074] 1. Estimate the sound source location based on the time difference of arrival (TDOA):

[0075] The target speech information collected by the microphone is preprocessed, including filtering, denoising and other operations, to improve the quality of the signal.

[0076] Calculate the cross-correlation function between the signals collected by different microphones. The cross-correlation function can reflect the similarity between two signals under different time delays. By finding the peak of the cross-correlation function, the time delay between the signals can be determined. Commonly used algorithms include the generalized cross-correlation (GCC) algorithm, which transforms two sound signals from the time domain to the frequency domain, obtains the cross-power spectrum, and then weights them. Then, the cross-correlation function is obtained by inverse Fourier transform back to the time domain. The position corresponding to the peak is the time delay.

[0077] Based on the obtained time difference and the known microphone position information, the position of the sound source is calculated using geometric relationships. For example, in three-dimensional space, with four microphones as a group, it can be determined that the sound source is located on a hyperbolic surface with these four microphones as the focus, and the position of the sound source is determined by the intersection of multiple hyperbolic surfaces.

[0078] 2. Locate the direction of the sound source based on controllable beamforming technology:

[0079] The signals of the microphone array are weighted and summed to enhance the response of the array in a specific direction, forming beams pointing in different directions. The calculation of the weighting coefficients is usually based on the position of the microphone and the desired beam direction. Within a certain angle range, the direction of the beam is gradually changed while monitoring the strength of the received signal. The direction with the strongest received signal is the direction of the sound source. For a matrix of four microphones or a cross of four microphones, it is necessary to determine the appropriate beamforming method and direction search strategy based on the layout of the microphones and the signal processing algorithm.

[0080] 3. Determine the sound source angle based on high-resolution spectrum estimation technology:

[0081] A mathematical model of the microphone array receiving signal is established, taking into account factors such as the signal propagation characteristics, the microphone response characteristics, and noise.

[0082] According to the signals received by the microphone array, the correlation matrix is ​​calculated. The correlation matrix reflects the correlation between different microphone signals. The correlation matrix is ​​subjected to eigendecomposition or other mathematical operations to obtain the spatial spectrum. The direction corresponding to the peak of the spatial spectrum is the direction of the sound source. In practical applications, it is necessary to select a suitable spectrum estimation method, such as the Multiple Signal Classification (MUSIC) algorithm, the Estimation of Signal Parameters via Rotational Invariance Techniques (ESPRIT) algorithm, etc.

[0083] Step 103: Detecting people in the image captured by the camera.

[0084] In step 103, an image is captured in real time by a camera, and a person in the image is detected. In one embodiment, a target detection algorithm such as YOLO (You Only Look Once, a real-time target detection algorithm) or Mask-RCNN (Mask Region-based Convolutional Neural Network, a deep learning model for instance segmentation) can be used to detect a person in the image captured by the camera.

[0085] Step 104: Determine the target user from the persons included in the image according to the sound source localization information.

[0086] After detecting a person in the image captured by the camera, the person at the position corresponding to the sound source positioning information in the image is determined according to the sound source positioning information, that is, the person is determined as the target user.

[0087] Step 105: Perform gesture detection on the target user to determine the target smart home device corresponding to the gesture of the target user.

[0088] In this embodiment, in order to clearly define the target smart home device to be controlled, the target user needs to make a preset gesture for the target smart home device, so that the gesture of the target user in the image can be detected to determine the target smart home device corresponding to the gesture of the target user.

[0089] In one embodiment, the preset gesture may be a finger pointing at a target smart home device. When the target user in the image points to a target smart home device, the target smart home device is used as the target smart home device.

[0090] Step 106: Control the target smart home device according to the target voice information.

[0091] After determining the target smart home device, the target smart home device is controlled to perform corresponding actions according to the content of the target voice information.

[0092] In an embodiment of the present application, the target voice information of the target user collected by the microphone is obtained; if the target voice information does not support the determination of the target smart home device, the sound source positioning information of the target user is obtained according to the target voice information; the person in the image taken by the camera is detected; according to the sound source positioning information, the target user is determined from the people contained in the image; gesture detection is performed on the target user to determine the target smart home device corresponding to the gesture of the target user; according to the target voice information, the target smart home device is controlled. In view of the situation where there is a default target voice information in the control of the target smart home device, the embodiment of the present application determines the specific target user by combining the sound source positioning information and image recognition technology, and identifies the target smart home device in combination with the gesture of the target user. Through this multimodal approach, the target smart home device pointed to by the default target voice information can be quickly determined, avoiding entering multiple rounds of dialogue selection or operation failure due to the inability to determine the target device, thereby significantly improving the user experience.

[0093] In an optional embodiment of the present application, after step 101, the method further includes:

[0094] Extracting target voiceprint information corresponding to the target voice information;

[0095] Matching the target voiceprint information with pre-stored voiceprint feature information to determine the target voiceprint feature information;

[0096] The target user identity of the target user is determined according to the target voiceprint feature information.

[0097] In this embodiment, if there are multiple people in the area corresponding to the sound source localization information, after obtaining the target voice information, the identity of the target user can be determined according to the target voiceprint information.

[0098] In one embodiment, the target user may be asked to read a few paragraphs of designated text in advance, the target user's voiceprint feature information may be collected, and the target user's target facial feature information may be input simultaneously, and the target user's voiceprint feature information may be bound to the target facial feature information.

[0099] In this embodiment, the target voiceprint information corresponding to the target voice information is extracted, and the target voiceprint information is matched with the pre-stored voiceprint feature information to determine the target voiceprint feature information. As an example, the system stores several voiceprint feature information, and the target voiceprint feature information corresponding to the target voice information is obtained through matching. Then, the target user identity of the bound target user is determined based on the target voiceprint feature information.

[0100] The embodiment of the present application can determine the identity of the target user based on the target voiceprint information, so that when there are multiple people in the area corresponding to the sound source positioning information, the identity of the specific target user can be determined immediately, and the target user in the image can be further determined based on the sound source positioning information and the face recognition results.

[0101] In an optional embodiment of the present application, after determining the identity of the target user according to the target voiceprint information, step 104 includes:

[0102] According to the target user identity of the target user, obtaining pre-stored target facial feature information of the target user;

[0103] Performing facial recognition on people at the position corresponding to the sound source localization information in the image, to obtain facial recognition information corresponding to each person at the position corresponding to the sound source localization information in the image;

[0104] The target facial feature information is matched with the facial recognition information corresponding to each person to determine the target user in the image.

[0105] In this embodiment, after the identity of the target user is confirmed, the target user can be determined from the image based on face recognition.

[0106] Specifically, the target facial feature information of the target user bound to the pre-stored target user identity can be obtained according to the target user identity of the target user. The face recognition is performed on the person at the position corresponding to the sound source localization information in the image to obtain the face recognition information corresponding to each person at the position corresponding to the sound source localization information in the image, and the target facial feature information is matched with the face recognition information corresponding to each person, so as to determine the target user in the image and obtain the gesture of the target user.

[0107] In one embodiment, if the target user cannot be determined from the image based on face recognition, the multi-round dialogue logic for selecting the target smart home device is directly entered. For example, for the target voice message of "turn on the air conditioner", the system can reply: "You have multiple air conditioners, which one do you want to turn on? The first one is A, the second one is B, please tell me which one or all of them"; for the target voice message of "turn on the ventilation mode", the system can reply: "You have multiple devices that can turn on the ventilation mode. Which one do you want to choose? The first one is purifier C, the second one is fresh air fan D, and the third one is air conditioner E". Among them, A, B, C, D, and E are the device names of smart home devices.

[0108] After confirming the identity of the target user, the embodiment of the present application can determine the target user from the image based on face recognition, thereby obtaining the target user's gesture, and determining the target smart home device based on the target user's gesture.

[0109] In an optional embodiment of the present application, if the target user does not pre-store the target voiceprint information and / or the target facial feature information, the step 104 includes:

[0110] If the target user has not pre-stored the target voiceprint information and / or the target facial feature information, performing gesture detection on the person at the position corresponding to the sound source localization information in the image, and determining the smart home device corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image;

[0111] If the number of the smart home devices corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image is one, taking the smart home device corresponding to the gesture of the person in the image as the target smart home device;

[0112] If the number of smart home devices corresponding to the gestures of the person at the position corresponding to the sound source localization information in the image is greater than one, randomly selecting a smart home device corresponding to the gesture of the person in the image;

[0113] The randomly selected smart home device is used as the target smart home device.

[0114] In this embodiment, if the target user does not store the target voiceprint information and / or the target facial feature information in the system in advance, gesture detection is directly performed on the person at the position corresponding to the sound source localization information in the image to determine the smart home device corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image. In one embodiment, the smart home device pointed to by the finger of the person at the position corresponding to the sound source localization information in the image can be determined.

[0115] If the number of smart home devices corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image is one, the smart home device corresponding to the gesture of the person in the image is directly used as the target smart home device; if the number of smart home devices corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image is greater than one, a smart home device corresponding to the gesture of the person in the image is randomly selected, and the randomly selected smart home device is used as the target smart home device.

[0116] In the embodiment of the present application, even if the target user has not stored the target voiceprint information and / or the target facial feature information in the system in advance, the target smart home device can be determined through sound source localization information and image gesture detection, thereby significantly improving the user experience.

[0117] In an optional embodiment of the present application, the smart home device has a unique device name. Before step 101, the method further includes:

[0118] Acquire the device recognition image captured by the camera;

[0119] Identifying the smart home device in the device recognition image;

[0120] If the smart home device is identified from the device identification image, receiving a first touch operation of the user on the device name, wherein the first touch operation is used to match the device name with the smart home device in the device identification image;

[0121] If the smart home device is not recognized from the device recognition image, receiving a second touch operation of the user on the device name, wherein the second touch operation is used to match the device name with a corresponding position of the smart home device in the device recognition image;

[0122] A target device recognition image including the matched device name is obtained.

[0123] In this embodiment, after the smart home device is equipped with the system, a specific device name can be assigned to it. For example, "air conditioner" can be named "air conditioner 1" or "Xiaobai" (duplicate names are generally not allowed). In this way, during general voice control, the target device can be quickly identified and operated by "turn on air conditioner 1 / turn on Xiaobai".

[0124] Furthermore, in the system, the device name of the smart home device can be matched with the smart home device in the image captured by the camera, thereby establishing a connection between the smart home device and the smart home device in the image.

[0125] Specifically, the device recognition image captured by the camera is obtained, and the smart home devices in the device recognition image are identified. At this time, only the type of smart home devices can be identified, such as air conditioners, fans, purifiers, etc., but the corresponding device name cannot be bound to the smart home device on the device recognition image, and the device name and approximate location of the smart home device outside the camera area cannot be marked.

[0126] If a smart home device is identified from the device recognition image, a first touch operation of the user on the device name is received, and the first touch operation is used to match the device name with the smart home device in the device recognition image. In one embodiment, the user can match the device name with the smart home device in the device recognition image by long pressing the device name and dragging it to the smart home device in the device recognition image on the terminal device.

[0127] If no smart home device is identified from the device identification image, a second touch operation of the user on the device name is received, and the second touch operation is used to match the device name with the corresponding position of the smart home device in the device identification image. In one embodiment, the user can match the device name with the corresponding position of the smart home device in the device identification image by long pressing the device name on the terminal device and dragging it to the corresponding position of the smart home device in the device identification image.

[0128] According to the above operation, a target device recognition image including the matched device name can be obtained.

[0129] The embodiment of the present application can match the device name of the smart home device with the smart home device in the image taken by the camera, thereby establishing a connection between the smart home device and the smart home device in the image. In the case of general voice control, the target smart home device can be quickly determined by the device name.

[0130] In an optional embodiment of the present application, in the case of acquiring a target device recognition image containing a matched device name, step 105 includes:

[0131] Performing gesture detection on the target user to obtain a target position corresponding to the gesture of the target user;

[0132] Detecting the smart home device in the image captured by the camera;

[0133] If the smart home device exists at the target position of the image captured by the camera, determining the device name of the smart home device according to the target device recognition image;

[0134] Determining the target smart home device from the image according to the device name of the smart home device;

[0135] If there is no smart home device at the target location of the image taken by the camera, the target user is reminded to add the target smart home device.

[0136] In this embodiment, after obtaining the matched target device recognition image, the target smart home device can be determined based on the target device recognition image combined with gesture detection.

[0137] Specifically, gesture detection is performed on the target user in the image to obtain the target position corresponding to the gesture of the target user. In one embodiment, the gesture of the target user may be a gesture of pointing a finger to a certain place, and the target position corresponding to the finger of the target user pointing to may be detected.

[0138] Detect smart home devices in the image captured by the camera, and determine whether there are smart home devices in the target direction of the image. If there are smart home devices in the target direction of the image captured by the camera, determine the device name of the smart home device according to the target device recognition image, and then determine the target smart home device in the image according to the device name of the smart home device; if there are no smart home devices in the target direction of the image captured by the camera, remind the target user to add the target smart home device.

[0139] The embodiment of the present application can determine the target smart home device based on the target device recognition image and combined with gesture detection, avoiding entering multiple rounds of dialogue selection or operation failure due to the inability to determine the target device, thereby significantly improving the user experience.

[0140] In an optional embodiment of the present application, the smart home system stores a device list, the smart home device has a unique device name, and the smart home device has a corresponding device type. After step 101, the method further includes:

[0141] If the target device type of the target smart home device is supported to be determined according to the target voice information, and the device name in the target voice information is missing, then filtering the smart home device of the target device type from the device list;

[0142] If the smart home device of the target device type is not obtained through screening, prompting the user to add the target smart home device;

[0143] If a smart home device of the target device type is obtained through screening, the smart home device of the target device type is used as the target smart home device;

[0144] If more than one smart home device of the target device type is obtained through screening, it is determined that the target voice information does not support the determination of the target smart home device.

[0145] In this embodiment, after the target voice information is acquired, it can be determined whether the target smart home device can be determined according to the target voice information.

[0146] If the target device type of the target smart home device is determined based on the target voice information, and the device name is missing in the target voice information, the smart home devices of the target device type are filtered from the device list. For example, if the target voice information is "adjust the air conditioner to cooling mode", the target voice information supports determining the target device type of the target smart home device, but the device name is missing in the target voice information, and the air conditioner is filtered from the device list.

[0147] If no smart home device of the target device type is screened, the user is reminded to add the target smart home device; if one smart home device of the target device type is screened, the smart home device of the target device type is used as the target smart home device; if more than one smart home device of the target device type is screened, it is determined that the target voice information does not support the determination of the target smart home device.

[0148] In the above example, if no air conditioner is found through screening, the user is prompted to add the device first; if one air conditioner is found through screening, the instruction corresponding to the target voice information is directly sent to the air conditioner; if there are multiple air conditioners, it is determined that the target voice information does not support the determination of the target air conditioner device, and the step of determining the target air conditioner device based on the sound source localization information and gesture detection is entered.

[0149] In one embodiment, if, when detecting the target user gesture, the smart home device corresponding to the target user gesture is not a device of the target device type, or there is no smart home device at the location corresponding to the target user gesture, a multi-round dialogue is entered for the user to select, and the system replies, "You don't mean the air conditioner. You have X air conditioners at home. Which one do you want to control? The first air conditioner is..."

[0150] After acquiring the target voice information, the embodiment of the present application first determines whether the target smart home device can be determined based on the target voice information, and then directly determines the target smart home device or determines the target smart home device based on the sound source positioning information and gesture detection, which can improve the user experience.

[0151] In an optional embodiment of the present application, after screening smart home devices of the target device type from the device list, in step 105, it also includes:

[0152] If more than one smart home device of the target device type is obtained through screening, and there is more than one smart home device of the target device type in the target position corresponding to the gesture of the target user, the target user is prompted to voice input the device name of the target smart home device.

[0153] In this embodiment, if more than one smart home device of the target device type is screened and it is determined that the target voice information does not support the determination of the target smart home device, when gesture detection is performed, if it is found that there are more than one smart home devices of the target device type in the target direction corresponding to the target user's gesture, the target user is prompted to voice input the device name of the target smart home device.

[0154] In an embodiment of the present application, when more than one smart home device of the target device type is screened and more than one smart home device of the target device type exists in the target direction corresponding to the target user's gesture, the user can be reminded to voice input the device name, thereby helping the user to determine the target smart home device.

[0155] In an optional embodiment of the present application, the smart home system stores a device function map, the smart home device has a unique device name, and the smart home device has a corresponding device type. Step 105 includes:

[0156] If the target device function of the target smart home device is supported to be determined according to the target voice information, and the device name and the device type in the target voice information are missing, then, according to the target voice information, the smart home device having the target device function is selected in the device function map;

[0157] If the smart home device having the target device function exists at the target location corresponding to the gesture of the target user, the smart home device at the target location corresponding to the gesture of the target user is used as the target smart home device;

[0158] If the smart home device having the target device function does not exist in the target location corresponding to the gesture of the target user, the target user is reminded to add the target smart home device.

[0159] In this embodiment, a device function map may be pre-constructed. Constructing a device function map may include the following steps:

[0160] Through the function book and requirement documents, the corresponding device functions of each smart home device are manually sorted (extracted) and connected hierarchically. For example, the air conditioner has the functions of adjusting temperature, adjusting mode, adjusting windshield, etc., the temperature adjustment has the functions of adjusting up, adjusting down, and adjusting temperature directly, and the direct temperature adjustment has the functions of adjusting 16 degrees, 17 degrees, etc. These device functions are called "entities".

[0161] Create tables for entity categories, such as "home appliance category table", "major category function table", "secondary function table", "third-level function table", etc., connect entities through foreign keys between tables, and use relational databases, such as MySQL, to store data. When using the device function map, you can index the specific device functions level by level to obtain supported smart home devices.

[0162] In one embodiment, the data in the device function map can be updated in a timely manner as new smart home devices are developed, old smart home devices are modified, etc.

[0163] In this embodiment, if the target device function of the target smart home device is determined based on the target voice information, and the device name and device type in the target voice information are missing, then the smart home device with the target device function is screened in the device function map based on the target voice information. For example, if the target voice information is "turn on the cooling mode", the target voice information supports determining that the target device function of the target smart home device is "dehumidification mode", but the device name and device type in the target voice information are missing. By entering "dehumidification mode" in the device function map, air conditioners, electric heaters, purifiers, dehumidifiers, etc. can be matched.

[0164] If there is a smart home device with the target device function at the target location corresponding to the target user's gesture, the smart home device at the target location corresponding to the target user's gesture is used as the target smart home device; if there is no smart home device with the target device function at the target location corresponding to the target user's gesture, the target user is reminded to add the target smart home device.

[0165] For example, if there is an air conditioner with a dehumidification function at the target location corresponding to the target user's gesture, the air conditioner will be used as the target smart home device; if there is no smart home device with the target device function at the target location corresponding to the target user's gesture, or the smart home device at the target location corresponding to the target user's gesture does not support the dehumidification function, the user can be prompted: "The device you pointed to does not support the cooling mode / why don't you add the category name and try again."

[0166] By constructing a device function map, the embodiment of the present application can process target voice information that supports determining the target device function of the target smart home device but has a missing device name and device type, thereby avoiding entering multiple rounds of dialogue selection or operation failure due to the inability to determine the target device, thereby significantly improving the user experience.

[0167] In an embodiment of the present application, the target voice information of the target user collected by the microphone is obtained; if the target voice information does not support the determination of the target smart home device, the sound source positioning information of the target user is obtained according to the target voice information; the person in the image taken by the camera is detected; according to the sound source positioning information, the target user is determined from the people contained in the image; gesture detection is performed on the target user to determine the target smart home device corresponding to the gesture of the target user; according to the target voice information, the target smart home device is controlled. In view of the situation where there is a default target voice information in the control of the target smart home device, the embodiment of the present application determines the specific target user by combining the sound source positioning information and image recognition technology, and identifies the target smart home device in combination with the gesture of the target user. Through this multimodal approach, the target smart home device pointed to by the default target voice information can be quickly determined, avoiding entering multiple rounds of dialogue selection or operation failure due to the inability to determine the target device, thereby significantly improving the user experience.

[0168] Reference Figure 2 , is a flow chart of an embodiment of a smart home device control method of the present application.

[0169] Figure 2 A flow chart of an embodiment of a smart home device control method of the present application is shown in FIG. 1 , which specifically includes the following steps:

[0170] Step 201: Annotate the device name and location based on the image.

[0171] Step 202: Determine the user's position by sound source localization.

[0172] Step 203: Determine the specific user through voiceprint and facial information.

[0173] Step 204: Determine the pointing device through a gesture.

[0174] Step 205: Determine through the knowledge graph whether the target device has the corresponding function.

[0175] Step 206: Complete the semantics of the control statement and issue it.

[0176] In an embodiment of the present application, the target voice information of the target user collected by the microphone is obtained; if the target voice information does not support the determination of the target smart home device, the sound source positioning information of the target user is obtained according to the target voice information; the person in the image taken by the camera is detected; according to the sound source positioning information, the target user is determined from the people contained in the image; gesture detection is performed on the target user to determine the target smart home device corresponding to the gesture of the target user; according to the target voice information, the target smart home device is controlled. In view of the situation where there is a default target voice information in the control of the target smart home device, the embodiment of the present application determines the specific target user by combining the sound source positioning information and image recognition technology, and identifies the target smart home device in combination with the gesture of the target user. Through this multimodal approach, the target smart home device pointed to by the default target voice information can be quickly determined, avoiding entering multiple rounds of dialogue selection or operation failure due to the inability to determine the target device, thereby significantly improving the user experience.

[0177] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0178] On the basis of the above embodiments, this embodiment further provides a smart home device control device, which is applied to electronic devices such as terminal devices and servers.

[0179] Reference Figure 3 , shows a structural block diagram of an embodiment of a smart home device control device of the present application, which may specifically include the following modules:

[0180] A voice information acquisition module 301 is used to acquire the target voice information of the target user collected by the microphone;

[0181] A sound source localization module 302 is configured to obtain sound source localization information of the target user according to the target voice information if the target voice information does not support determining the target smart home device;

[0182] An image detection module 303, used to detect people in the image captured by the camera;

[0183] A target user determination module 304 is used to determine the target user from the people in the image according to the sound source localization information;

[0184] A gesture detection module 305, configured to detect a gesture of the target user and determine the target smart home device corresponding to the gesture of the target user;

[0185] The device control module 306 is used to control the target smart home device according to the target voice information.

[0186] In an optional embodiment of the present application, the device further includes:

[0187] A voiceprint extraction module, used to extract target voiceprint information corresponding to the target voice information;

[0188] A voiceprint matching module, used to match the target voiceprint information with pre-stored voiceprint feature information to determine the target voiceprint feature information;

[0189] The identity determination module is used to determine the target user identity of the target user according to the target voiceprint feature information.

[0190] In an optional embodiment of the present application, the target user determination module 304 includes:

[0191] A facial feature extraction submodule, used to obtain pre-stored target facial feature information of the target user according to the target user identity of the target user;

[0192] A facial recognition submodule, configured to perform facial recognition on a person at a position corresponding to the sound source localization information in the image, and obtain facial recognition information corresponding to each person at the position corresponding to the sound source localization information in the image;

[0193] The facial matching submodule is used to match the target facial feature information with the facial recognition information corresponding to each person to determine the target user in the image.

[0194] In an optional embodiment of the present application, the target user determination module 304 includes:

[0195] A gesture pointing detection submodule, for performing gesture detection on a person at a position corresponding to the sound source positioning information in the image, and determining the smart home device corresponding to the gesture of the person at the position corresponding to the sound source positioning information in the image, if the target user has not pre-stored the target voiceprint information and / or the target facial feature information;

[0196] A first determination submodule is configured to use, if the number of smart home devices corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image is one, the smart home device corresponding to the gesture of the person in the image as the target smart home device;

[0197] The second determination submodule is used to randomly select a smart home device corresponding to the gesture of the person in the image if the number of smart home devices corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image is greater than one; and use the randomly selected smart home device as the target smart home device.

[0198] In an optional embodiment of the present application, the smart home device has a unique device name, and the apparatus further includes:

[0199] An identification image acquisition module, used to acquire the device identification image taken by the camera;

[0200] A device identification module, used to identify the smart home device in the device identification image;

[0201] a first touch operation module, configured to receive a first touch operation of a user on the device name if the smart home device is identified from the device identification image, wherein the first touch operation is used to match the device name with the smart home device in the device identification image;

[0202] a second touch operation module, configured to receive a second touch operation of the user on the device name if the smart home device is not recognized from the device recognition image, wherein the second touch operation is used to match the device name with a corresponding position of the smart home device in the device recognition image;

[0203] The target device recognition image acquisition module is used to acquire the target device recognition image containing the matched device name.

[0204] In an optional embodiment of the present application, the gesture detection module 305 includes:

[0205] A target position acquisition submodule, used to perform gesture detection on the target user to obtain a target position corresponding to the gesture of the target user;

[0206] An image device detection submodule, used to detect the smart home device in the image captured by the camera;

[0207] A third determination submodule is used for determining the device name of the smart home device according to the target device recognition image if the smart home device exists in the target orientation of the image taken by the camera; and determining the target smart home device from the image according to the device name of the smart home device;

[0208] The first prompt submodule is used to remind the target user to add the target smart home device if there is no smart home device in the target direction of the image taken by the camera.

[0209] In an optional embodiment of the present application, the smart home system stores a device list, the smart home device has a unique device name, the smart home device has a corresponding device type, and the apparatus further includes:

[0210] a device screening module, configured to screen the smart home devices of the target device type from the device list if the target voice information supports determining the target device type of the target smart home device and the device name in the target voice information is missing;

[0211] A second prompting module, configured to remind the user to add the target smart home device if the smart home device of the target device type is not obtained through screening;

[0212] a fourth determination module, configured to, if a smart home device of the target device type is obtained through screening, use the smart home device of the target device type as the target smart home device;

[0213] The target voice information determination module is used to determine that the target voice information does not support the determination of the target smart home device if more than one smart home device of the target device type is screened.

[0214] In an optional embodiment of the present application, the gesture detection module 305 includes:

[0215] The third prompt submodule is used to remind the target user to voice input the device name of the target smart home device if more than one smart home device of the target device type is screened and there is more than one smart home device of the target device type in the target position corresponding to the gesture of the target user.

[0216] In an optional embodiment of the present application, the smart home system stores a device function map, the smart home device has a unique device name, the smart home device has a corresponding device type, and the gesture detection module 305 includes:

[0217] A graph screening submodule, for screening the smart home device having the target device function in the device function graph according to the target voice information, if the target device function of the target smart home device is supported to be determined according to the target voice information, and the device name and the device type in the target voice information are defaulted;

[0218] a fifth determining submodule, configured to use the smart home device at the target location corresponding to the gesture of the target user as the target smart home device if the smart home device having the target device function exists at the target location corresponding to the gesture of the target user;

[0219] The fourth prompting submodule is used to remind the target user to add the target smart home device if the smart home device having the target device function does not exist in the target position corresponding to the gesture of the target user.

[0220] In an embodiment of the present application, the target voice information of the target user collected by the microphone is obtained; if the target voice information does not support the determination of the target smart home device, the sound source positioning information of the target user is obtained according to the target voice information; the person in the image taken by the camera is detected; according to the sound source positioning information, the target user is determined from the people contained in the image; gesture detection is performed on the target user to determine the target smart home device corresponding to the gesture of the target user; according to the target voice information, the target smart home device is controlled. In view of the situation where there is a default target voice information in the control of the target smart home device, the embodiment of the present application determines the specific target user by combining the sound source positioning information and image recognition technology, and identifies the target smart home device in combination with the gesture of the target user. Through this multimodal approach, the target smart home device pointed to by the default target voice information can be quickly determined, avoiding entering multiple rounds of dialogue selection or operation failure due to the inability to determine the target device, thereby significantly improving the user experience.

[0221] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0222] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0223] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 4 An exemplary apparatus 400 that may be used to implement various embodiments described in this application is schematically illustrated.

[0224] For one embodiment, Figure 4 An exemplary apparatus 400 is shown having one or more processors 402, a control module (chip set) 404 coupled to at least one of the processor(s) 402, a memory 406 coupled to the control module 404, a non-volatile memory (NVM) / storage device 408 coupled to the control module 404, one or more input / output devices 410 coupled to the control module 404, and a network interface 412 coupled to the control module 404.

[0225] The processor 402 may include one or more single-core or multi-core processors, and the processor 402 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 400 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0226] In some embodiments, the apparatus 400 may include one or more computer-readable media (e.g., memory 406 or NVM / storage device 408) having instructions 414 and one or more processors 402 combined with the one or more computer-readable media and configured to execute the instructions 414 to implement a module to perform the actions described in the present disclosure.

[0227] For one embodiment, control module 404 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 402 and / or any suitable device or component in communication with control module 404 .

[0228] The control module 404 may include a memory controller module to provide an interface to the memory 406. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0229] The memory 406 may be used, for example, to load and store data and / or instructions 414 for the device 400. For one embodiment, the memory 406 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 406 may include a double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0230] For one embodiment, control module 404 may include one or more input / output controllers to provide interfaces to NVM / storage device 408 and input / output device(s) 410 .

[0231] For example, NVM / storage 408 may be used to store data and / or instructions 414. NVM / storage 408 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0232] NVM / storage device 408 may include storage resources that are physically part of the device on which apparatus 400 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 408 may be accessed via input / output device(s) 410 over a network.

[0233] (One or more) input / output devices 410 may provide an interface for the apparatus 400 to communicate with any other appropriate device, and the input / output device 410 may include a communication component, an audio component, a sensor component, etc. The network interface 412 may provide an interface for the apparatus 400 to communicate through one or more networks, and the apparatus 400 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0234] For one embodiment, at least one of the processor(s) 402 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 404. For one embodiment, at least one of the processor(s) 402 may be packaged together with the logic of one or more controllers of the control module 404 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 402 may be integrated on the same die with the logic of one or more controllers of the control module 404. For one embodiment, at least one of the processor(s) 402 may be integrated on the same die with the logic of one or more controllers of the control module 404 to form a system-on-chip (SoC).

[0235] In various embodiments, the device 400 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 400 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 400 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0236] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0237] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0238] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0239] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable smart home device control terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable smart home device control terminal device are used to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0240] These computer program instructions may also be stored in a computer-readable memory that can guide a computer or other programmable smart home device to control a terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0241] These computer program instructions can also be loaded onto a computer or other programmable smart home device control terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0242] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0243] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0244] The above is a detailed introduction to a smart home device control method and apparatus, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a general technician in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A smart home device control method, characterized in that: Applied to a smart home system, the smart home system includes smart home devices, a microphone and a camera, and the method includes: Acquire target voice information of a target user collected by the microphone; If the target voice information does not support determining the target smart home device, acquiring the sound source localization information of the target user according to the target voice information; Detecting a person in an image captured by the camera; Determine the target user from the persons included in the image according to the sound source localization information; Performing gesture detection on the target user to determine the target smart home device corresponding to the gesture of the target user; The target smart home device is controlled according to the target voice information.

2. The method according to claim 1, characterized in that After acquiring the target voice information of the target user collected by the microphone, the method further includes: Extracting target voiceprint information corresponding to the target voice information; Matching the target voiceprint information with pre-stored voiceprint feature information to determine the target voiceprint feature information; The target user identity of the target user is determined according to the target voiceprint feature information.

3. The method according to claim 2, characterized in that The step of determining the target user from persons included in the image according to the sound source localization information includes: According to the target user identity of the target user, obtaining pre-stored target facial feature information of the target user; Performing facial recognition on people at the position corresponding to the sound source localization information in the image, to obtain facial recognition information corresponding to each person at the position corresponding to the sound source localization information in the image; The target facial feature information is matched with the facial recognition information corresponding to each person to determine the target user in the image.

4. The method according to claim 3, characterized in that The step of determining the target user from persons included in the image according to the sound source localization information includes: If the target user has not pre-stored the target voiceprint information and / or the target facial feature information, performing gesture detection on the person at the position corresponding to the sound source localization information in the image, and determining the smart home device corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image; If the number of the smart home devices corresponding to the gesture of the person at the position corresponding to the sound source localization information in the image is one, taking the smart home device corresponding to the gesture of the person in the image as the target smart home device; If the number of smart home devices corresponding to the gestures of the person at the position corresponding to the sound source localization information in the image is greater than one, randomly selecting a smart home device corresponding to the gesture of the person in the image; The randomly selected smart home device is used as the target smart home device.

5. The method according to claim 1, characterized in that The smart home device has a unique device name, and the method further includes: Acquire the device recognition image captured by the camera; Identifying the smart home device in the device recognition image; If the smart home device is identified from the device identification image, receiving a first touch operation of the user on the device name, wherein the first touch operation is used to match the device name with the smart home device in the device identification image; If the smart home device is not recognized from the device recognition image, receiving a second touch operation of the user on the device name, wherein the second touch operation is used to match the device name with a corresponding position of the smart home device in the device recognition image; A target device recognition image including the matched device name is obtained.

6. The method according to claim 5, characterized in that The performing gesture detection on the target user to determine the target smart home device corresponding to the gesture of the target user includes: Performing gesture detection on the target user to obtain a target position corresponding to the gesture of the target user; Detecting the smart home device in the image captured by the camera; If the smart home device exists at the target position of the image captured by the camera, determining the device name of the smart home device according to the target device recognition image; Determining the target smart home device from the image according to the device name of the smart home device; If there is no smart home device at the target location of the image taken by the camera, the target user is reminded to add the target smart home device.

7. The method according to claim 1, characterized in that The smart home system stores a device list, the smart home device has a unique device name, and the smart home device has a corresponding device type. After acquiring the target voice information of the target user collected by the microphone, the method further includes: If the target device type of the target smart home device is supported to be determined according to the target voice information, and the device name in the target voice information is missing, then filtering the smart home device of the target device type from the device list; If the smart home device of the target device type is not obtained through screening, prompting the user to add the target smart home device; If a smart home device of the target device type is obtained through screening, the smart home device of the target device type is used as the target smart home device; If more than one smart home device of the target device type is obtained through screening, it is determined that the target voice information does not support the determination of the target smart home device.

8. The method according to claim 7, characterized in that The performing gesture detection on the target user to determine the target smart home device corresponding to the gesture of the target user includes: If more than one smart home device of the target device type is obtained through screening, and there is more than one smart home device of the target device type in the target position corresponding to the gesture of the target user, the target user is prompted to voice input the device name of the target smart home device.

9. The method according to claim 1, characterized in that: The smart home system stores a device function map, the smart home device has a unique device name, the smart home device has a corresponding device type, and the gesture detection of the target user is performed to determine the target smart home device corresponding to the gesture of the target user, including: If the target device function of the target smart home device is supported to be determined according to the target voice information, and the device name and the device type in the target voice information are missing, then, according to the target voice information, the smart home device having the target device function is selected in the device function map; If the smart home device having the target device function exists at the target location corresponding to the gesture of the target user, the smart home device at the target location corresponding to the gesture of the target user is used as the target smart home device; If the smart home device having the target device function does not exist in the target location corresponding to the gesture of the target user, the target user is reminded to add the target smart home device.

10. A smart home equipment control device, characterized in that: Applied to a smart home system, the smart home system includes smart home devices, a microphone and a camera, and the device includes: A voice information acquisition module, used to acquire the target voice information of the target user collected by the microphone; A sound source localization module, configured to obtain sound source localization information of the target user according to the target voice information if the target voice information does not support determining the target smart home device; An image detection module, used to detect people in the image captured by the camera; A target user determination module, configured to determine the target user from the people in the image according to the sound source localization information; A gesture detection module, used to detect the gesture of the target user and determine the target smart home device corresponding to the gesture of the target user; The device control module is used to control the target smart home device according to the target voice information.

11. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, enables the processor to execute the smart home device control method as described in one or more of claims 1-9.

12. One or more machine-readable media having executable codes stored thereon, which, when executed, enable a processor to execute the smart home device control method as described in one or more of claims 1-9.

Images