Voice control method of smart home device and electronic device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GREE ELECTRIC APPLIANCE INC OF ZHUHAI
- Filing Date
- 2026-07-09
- Publication Date
- 2026-08-07
AI Technical Summary
[0004]本申请实施例的目的是提供一种智能家居设备的语音控制方法和电子设备及介质，能够解决现有语音交互系统无法应用于多人并发语音场景问题
Smart Images
Figure CN122531381A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of equipment control technology, specifically relating to a voice control method for a smart home device and an electronic device. Background Technology
[0002] Currently, mainstream smart home voice interaction systems generally adopt the following technical approach: after a single speaker wakes up the voice interaction system, the corresponding mode is executed globally. For example, after a smart home voice interaction system is woken up by preset keywords, it will treat subsequent voice as a single user command, directly parse and execute it.
[0003] However, in scenarios with multiple concurrent voice commands, the aforementioned voice interaction methods cannot distinguish between different commands from multiple speakers, nor can they determine whether the voice command comes from an authorized person, thus leading to confusion in the voice control of smart home devices. Summary of the Invention
[0004] The purpose of this application is to provide a voice control method, electronic device, and medium for smart home devices, which can solve the problem that existing voice interaction systems cannot be applied to scenarios with multiple concurrent voice users.
[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a voice control method for a smart home device, the method comprising: Acquire multiple audio data points synchronously collected by a time-synchronized audio acquisition device at multiple preset acquisition locations, wherein the audio data is audio collected when multiple speakers are speaking; The multiple audio data are processed by a pre-trained first model for performing speech separation to obtain target speech data for each speaker. Based on the target speech data and the acquisition location, determine the coordinate data of each speaker; Each speaker's role label is determined based on the target voice data and the coordinate data; each role label is assigned different device control permissions. Generate first intent information for each speaker based on the target speech data; When the first intent information matches a preset sensitive whitelist and the role tag is not a preset tag, a second intent information is generated based on the first intent information; the first intent information is the true intent of the target voice, and the second intent information is a vague intent used to authorize control of the target smart home device after preset privacy processing of the first intent information; Based on the second intent information, a first notification message is generated to request authorization to control the target home device of the speaker, and the notification message is sent to the administrator corresponding to the target home device; In response to receiving a second notification message for agreeing to authorize the target smart home device, the first intent information is sent to the target smart home device; The target smart home device is controlled according to the first intent information.
[0006] Optionally, generating first intent information for each speaker based on the target speech data includes: The target speech data is input into a second model used to convert speech into text, and the target text information is output. The target text information is input into a third model used to convert the text into intent, and the first intent information of each speaker is output.
[0007] Optionally, generating the second intent information based on the first intent information includes: The preset fields in the first intent information are used to generate replacement values centered on the real values to obtain the second intent data; Alternatively, the preset fields in the first intent information can be semantically generalized according to a predefined generalization hierarchy to obtain the second intent data.
[0008] Optionally, determining the coordinate data of each speaker based on the target speech data and the acquisition location includes: Determine the time difference between the target speech data and each audio acquisition device; The direction of the sound source is determined based on the time difference; Determine the spatial geometric model corresponding to the acquisition location; The coordinate data of each speaker are determined based on the direction of the sound source and the spatial geometric model.
[0009] Optionally, determining the role label for each speaker based on the target speech data and the coordinate data includes: Convert the target speech data into a first numerical vector; Determine the second numerical vector corresponding to each registered user in the preset registered user voiceprint database; Determine the cosine similarity between the first numerical vector and the second numerical vector; The role label for each speaker is determined based on the cosine similarity and the coordinate data.
[0010] Optionally, determining the role label for each speaker based on the cosine similarity and the coordinate data includes: When the cosine similarity is less than a preset value, the probability of the first role corresponding to the behavioral habits of each speaker is determined based on the coordinate data; Determine the probability of the second role based on the target speech data; Acquire device behavior data of associated devices of the target smart home device within a preset time period, and determine the probability of a third role based on the device behavior data; The role label for each speaker is determined based on the first role probability, the second role probability, and the third role probability.
[0011] Optionally, determining the role label for each speaker based on the first role probability, the second role probability, and the third role probability includes: Obtain the first weight value corresponding to the behavioral habit, the second weight value of the target voice data, and the third weight value of the device behavior; The probability scores of the candidate character tags are determined by weighting the first character probability, the second character probability, and the third character probability using the first weight value, the second weight value, and the third weight value, respectively. Each speaker's role label is determined based on the probability score.
[0012] Secondly, embodiments of this application provide a voice control device for a smart home device, the device comprising: The audio data acquisition module is used to acquire multiple audio data points synchronously collected by a time-synchronized audio acquisition device at multiple preset acquisition locations. The audio data is audio collected when multiple speakers are speaking. The target speech data determination module is used to perform speech separation processing on the plurality of audio data using a pre-trained first model for performing speech separation, to obtain the target speech data for each speaker; The coordinate data determination module is used to determine the coordinate data of each speaker based on the target voice data and the acquisition location; The role label determination module is used to determine the role label of each speaker based on the target voice data and the coordinate data; the role label is set with different corresponding device control permissions. The first intent information generation module is used to generate first intent information for each speaker based on the target speech data; The second intent information determination module is used to generate second intent information based on the first intent information when the first intent information matches a preset sensitive whitelist and the role tag is not a preset tag; the first intent information is the true intent of the target voice, and the second intent information is a vague intent used to authorize control of the target smart home device after preset privacy processing of the first intent information. The first notification information generation module is used to generate a first notification information based on the second intent information for requesting control authorization for the target home device of the speaker, and send the notification information to the administrator corresponding to the target home device; The second notification message generation module is used to send the first intent information to the target smart home device in response to receiving a second notification message for agreeing to authorize the target home device; A home appliance control module is used to control the target smart home appliance according to the first intent information.
[0013] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.
[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.
[0015] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.
[0016] In this embodiment, a role label for each speaker is determined based on the analysis of the target voice data of each speaker. Different role labels correspond to different device control permissions, thereby enabling voice control of smart home devices according to the role labels. This effectively solves the problem of controlling home devices in scenarios with multiple speakers. Furthermore, the user's first intent information can be extracted and combined with this information to control smart home devices. This allows for source isolation, privacy protection, and controllable execution of high-risk voice commands in complex home acoustic environments, achieving a balance between security, compliance, and user experience. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating the steps of a voice control method for a smart home device according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a voice control device for a smart home device according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0020] Currently, mainstream smart home voice interaction systems generally adopt the following technical approach: a single speaker wakes up the voice interaction system, and then the corresponding mode is executed globally. For example, after a smart home voice interaction system is woken up by a preset keyword, it treats subsequent voice as a single user command, directly parses and executes it, without distinguishing the speaker's identity. Meanwhile, some high-end devices support coarse-grained user binding based on voiceprint recognition, that is, collecting family member voiceprint templates during the registration phase and comparing whether the speaker is a "known user" before execution. However, this function is only used for personalized services (such as playing a personal playlist), not for access control, and cannot handle scenarios where multiple people speak simultaneously. Furthermore, for high-risk commands such as "open the door" and "pay," the voice interaction system requires the user to confirm through an application (App) or enter a password. However, this mechanism relies on the user's active response and cannot determine whether the command comes from an authorized person in a multi-person environment.
[0021] In one publicly available voice control method, voice information sent by a user for controlling a device can be acquired; the user's identity can be identified based on the voice information; and the device can be controlled to execute the voice information based on the user's identity. This solution can address the issue of poor security in voice control, thereby improving its security.
[0022] In another publicly available voice control method, in response to the detection of sound, it can identify whether the sound is speech; if the sound is speech, it can identify the identity information of the user who made the sound, and determine whether the user is a legitimate user based on the identity information; in response to determining that the user is a legitimate user, it can perform speech recognition on the sound to obtain control commands; in response to determining that the control command is a command that matches the identity information, it can execute the control command, thereby realizing the secure management and control of smart home devices, speaker security management and security protection.
[0023] Analysis of current voice control technology reveals the following problems: Currently, user authentication is mainly achieved by performing keyword wake-up and voiceprint recognition on a single-channel enhanced speech stream. However, the microphone arrays in current voice interaction systems are only used for beamforming to enhance the main direction of speech, lacking the ability to separate multiple speakers in real time. This results in the inability to decouple mixed speech at the speaker level. Even with voiceprint recognition, it only works under the assumption of a single speaker. Therefore, it is difficult to meet the needs of handling overlapping speech. Consequently, it is unable to accurately identify and isolate sensitive commands from unauthorized users in multi-person concurrent speech scenarios. For example, when a child, a visitor, and a parent speak simultaneously, such as the child saying "open the door" and the father saying "turn down the air conditioner," the voice interaction system may mistakenly execute the child's high-risk command.
[0024] In this embodiment of the application, in this multi-person speaking scenario, audio data collected from different collection locations are processed by speech separation to obtain target speech data for each speaker; and role tags for each speaker can be determined based on the analysis of the target speech data for each speaker. Different role tags can correspond to different device control permissions, thereby realizing voice control of smart home devices according to role tags, effectively solving the problem of home device control in multi-person speaking scenarios.
[0025] The voice control of the smart home device provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.
[0026] Reference Figure 1 This diagram illustrates a step-by-step flowchart of a voice control method for a smart home device according to an embodiment of this application. Specifically, it may include the following steps: Step S101: Acquire multiple audio data points synchronously collected by a time-synchronized audio acquisition device at multiple preset acquisition locations. The audio data points are audio collected when multiple speakers are speaking. Audio acquisition devices can be used to collect audio data within an environment. When only a single speaker is present in the surrounding environment, the audio acquisition device collects the audio data of that single speaker. When multiple speakers are present in the surrounding environment, the audio acquisition device can collect the audio data of all speakers. In this embodiment, the audio acquisition device can be a speaker or similar device. This audio acquisition device can be connected to a central edge computing node to upload the collected audio data to the central edge computing node for processing.
[0027] To enable voice interaction in multi-person speaking scenarios, multiple audio acquisition devices can be set up in the environment. Each audio acquisition device can be assigned a corresponding acquisition location to capture audio at that location. Multiple audio acquisition devices can be pre-synchronized in time to simultaneously acquire audio data. Multiple audio data can be time-aligned based on timestamps. By analyzing audio data from different acquisition locations at the same time, precise voice interaction control can be achieved.
[0028] For example, smart speakers supporting time synchronization can be deployed in at least three rooms of a home, such as the living room, bedroom, and kitchen. These smart speakers can have built-in IEEE 1588 PTP protocol or Wi-Fi timestamp alignment, and each device is equipped with at least two microphones forming a small array. All smart speaker devices synchronously collect audio at a 48 kHz sampling rate and transmit the raw Pulse Code Modulation (PCM) stream in real time to a central edge computing node (such as a smart gateway) via the home LAN.
[0029] In one example, after acquiring multiple audio data streams, the audio data streams can be preprocessed, and the preprocessing methods can include, but are not limited to, noise reduction processing, echo cancellation processing, etc.
[0030] Step S102: The pre-trained first model for performing speech separation is used to perform speech separation processing on the multiple audio data to obtain the target speech data for each speaker. In practical applications, a first model for performing speech separation can be configured in the central edge computing node. This first model is a speech processing model, specifically a multi-channel time-frequency masking separation model, such as a dual-path recurrent neural network (DPRNN with Transform-Average-Concatenate, DPRNN-TAC) with a transform-average-concatenation module. The output data of the first model can be audio data. Within the first model, the audio data in a multi-speaker scenario can be separated into individual speech data for each speaker, and then the output data is the target speech data for each speaker.
[0031] The specific data processing procedure within DPRNN-TAC is as follows: In the voice interaction system, at least three smart speakers (each with ≥2 microphones) are deployed to synchronously acquire audio at a sampling rate of 48kHz. The central edge node sequentially performs frame segmentation, windowing, and short-time Fourier transform (STFT) on the N signals to convert the time-domain waveform into a time-spectrum graph, which is then input into the DPRNN-TAC model.
[0032] DPRNN-TAC is a deep neural network for speech separation, consisting of a dual-path recurrent neural network (DPRNN) and a temporal attention convolutional module (TAC) connected in series. DPRNN captures the spatiotemporal dependencies of speech through two parallel paths: the intra-frame path models the correlation between different frequency points within the same time frame; the inter-frame path models the evolution of the same feature across different time frames. Each path uses a bidirectional long short-term memory (BLSTM) network as its core processing unit. A BLSTM can contain two long short-term memories (LSTMs)—one forward and one backward—reading sequences sequentially from front to back and from back to front, respectively. The outputs from both directions are concatenated at each time step, thus utilizing both past and future contextual information simultaneously.
[0033] Subsequently, the TAC module applies attention weighting to the time-series features output by the DPRNN, highlighting keyframes that contribute more to the separation task and suppressing non-speech interference. The DPRNN-TAC model ultimately estimates the time-frequency masks for each of the K speakers. These time-domain masks are two-dimensional matrices, where each element represents the probability or proportion of a given time-frequency point belonging to that speaker. Each mask is then multiplied point-by-point by the STFT spectrum of the original mixed signal to obtain the frequency domain representation for each speaker. Finally, K independent time-domain speech signals, i.e., the target speech data, are reconstructed using inverse STFT, where K is dynamically estimated by the number of active speakers.
[0034] Step S103: Determine the coordinate data of each speaker based on the target speech data and the acquisition location; After determining the target speech data, which is the speech data of each speaker, the acquisition location is the actual location where the speech of each speaker can be acquired. By combining the two, the speaking characteristics of each speaker are analyzed to determine the actual location of each speaker when speaking, i.e., the coordinate data of the speaker.
[0035] In one optional embodiment, step S103 may specifically include the following sub-steps: Sub-step S11: Determine the time difference between the target speech data and each audio acquisition device; After determining the target speech data for each speaker, timestamp information can be carried in each target speech data, thereby determining the time difference between each speaker and each audio acquisition device.
[0036] Sub-step S12: Determine the direction of the sound source based on the time difference; After determining the time difference, the direction of the sound source can be deduced based on the time difference. Specifically, the direction of the sound source can be calculated using the Time Difference of Arrival (TDOA) of the same speech segment between each speaker, through the Generalized Cross-Correlation with Phase Transform (GCC-PHAT) algorithm.
[0037] The cross power spectrum is calculated by selecting two smart speakers at different locations and transforming the two signals to the frequency domain. Only the phase information is retained (i.e., divided by the modulus of the cross power spectrum) to suppress reverberation and noise interference. Then, the time delay function is obtained by inverse Fourier transform. The peak position of the time delay function is the estimated value of the time difference of arrival (TDOA) between the two speakers.
[0038] The time difference, combined with the speed of sound (approximately 340 m / s), can be converted into a distance difference. Finally, using the distance difference information of at least three pairs of speakers, the two-dimensional coordinates of the sound source relative to the reference point are calculated through geometric hyperbolic localization (such as the least squares method or the Chan algorithm), thereby obtaining the direction of the sound source.
[0039] Sub-step S13: Determine the spatial geometric model corresponding to the acquisition location; Each audio acquisition device is set with an acquisition location. Based on the acquisition locations of multiple audio acquisition devices, the spatial layout of the area can be determined. Then, based on the spatial layout, the corresponding spatial geometric model can be obtained. For example, if the location of the speaker is determined, the floor plan of the house corresponding to the location of the speaker (i.e., the spatial layout) can be obtained. Then, a spatial geometric model can be constructed based on the floor plan of the house.
[0040] Sub-step S14: Determine the coordinate data of each speaker based on the direction of the sound source and the spatial geometric model.
[0041] By combining the direction of the sound source and the spatial geometric model, the actual coordinates of each speaker can be calculated.
[0042] Step S104: Determine the role label for each speaker based on the target voice data and the coordinate data; the role label is set with different device control permissions. Target speech data can include the speaker's vocal characteristics, while coordinate data can reflect the speaker's behavioral characteristics. Combining these two can reasonably infer the speaker's role label. Role labels can include, but are not limited to, visitors, family members, etc.
[0043] In one embodiment of this application, the registered user data stored in the registered user voiceprint database can be obtained within the central edge computing node. Then, by comparing the obtained registered user data with the target voice data, it can be determined whether the speaker is a registered user. When it is determined to be a registered user, the role label can be directly output. When it is not a registered user, the role label of the speaker can be determined by combining the coordinate data.
[0044] Specifically, the target speech data can be converted into a first numerical vector; a second numerical vector corresponding to each registered user in a preset registered user voiceprint database can be determined; the cosine similarity between the first numerical vector and the second numerical vector can be determined; and the role label of each speaker can be determined based on the cosine similarity and the coordinate data.
[0045] When the cosine similarity is greater than a preset value, the user-preset role is directly used as the speaker's role label. When the cosine similarity is less than the preset value, the role label of each speaker is determined based on the cosine similarity and coordinate data. The preset value can be set according to the actual scenario, and no restrictions are imposed on it in this embodiment.
[0046] For example, extracting x-vector embeddings (using a pre-trained ResNet-34 speaker encoder) from each separated target speech data is to convert a speech segment into a fixed-dimensional numerical vector (usually 512-dimensional or 1024-dimensional), effectively capturing the speaker's acoustic personality features (such as timbre, formant structure, etc.), and exhibiting strong robustness to changes in speech content, speech rate, noise, etc.
[0047] Specifically, the process of processing raw speech into a fixed-dimensional speaker vector first involves pre-emphasizing, framing, windowing, and extracting Mel filter bank (FBank) features to obtain a time-varying acoustic feature sequence. This sequence is then input into a deep neural network (such as TDNN or ResNet-34). The network first outputs frame-level embeddings, and then calculates the mean and standard deviation of all frame embeddings through statistical pooling layers, thereby mapping speech of arbitrary length into a fixed-length statistical vector. Finally, the vector is reduced in dimensionality by several fully connected layers and normalized by L2, outputting a 512-dimensional or 1024-dimensional x-vector. This vector compactly encodes the speaker's unique timbre, formants, and other acoustic characteristics.
[0048] The numerical vectors of acoustic individual characteristics are compared with the locally stored registered user voiceprint database (encrypted and stored in the secure element SE) using cosine similarity. The cosine similarity between two vectors A and B is (dot product of A and B) / (magnitude of A * magnitude of B). The similarity value ranges from -1 to 1, and the closer it is to 1, the more similar they are.
[0049] In an optional embodiment of this application, when the cosine similarity is less than a preset value, role determination can be performed by combining multi-dimensional context. The multi-dimensional context can include, but is not limited to, any combination of behavioral habits, voice semantics, and IoT device context.
[0050] In one example, the role labels of each speaker can be analyzed by combining behavioral habits, voice semantics, and IoT device context.
[0051] (1) Behavioral habit analysis: Based on historical sound source localization data, a spatiotemporal activity heatmap of each user can be constructed, and then it can be determined whether the current speaking coordinate data conforms to their regular behavior. When the speaker conforms to a user's regular behavior, the probability of the first role corresponding to each speaker's behavioral habit can be determined.
[0052] (2) Speech semantic analysis: After obtaining the target speech data of each speaker, a lightweight NLU model can be used to identify keywords and pragmatic features (such as "homework" and "password") to assist in the judgment of role orientation, and then determine the probability of the second role.
[0053] (3) IoT device context analysis: Obtain device behavior data of the associated devices of the target smart home device within a preset time period, and determine the probability of a third role based on the device behavior data, such as associating door lock status, wearable device heart rate, Bluetooth connection records, etc. during the same time period to verify the rationality of each speaker's identity.
[0054] Through the above three-way role analysis, we can obtain the probability of the first role, the probability of the second role, and the probability of the third role. Then, we can combine the probability of the first role, the probability of the second role, and the probability of the third role to determine the final role label of each speaker.
[0055] In one example, different types of context have varying degrees of influence on the final role label. Therefore, weight values for different types of context can be pre-set, and the final role label can be determined through weighted processing. The specific process is as follows: Obtain the first weight value corresponding to the behavioral habit, the second weight value of the target voice data, and the third weight value of the device behavior; use the first weight value, the second weight value, and the third weight value to weight the first role probability, the second role probability, and the third role probability, respectively, to determine the probability score of the candidate role label; determine the role label for each speaker based on the probability score. Wherein, the first weight value + the second weight value + the third weight value = 1.
[0056] The following example illustrates the process of determining speaker role labels using the weighted approach: For example, the weight of behavioral habits can be set to 0.5, the weight of voice semantics to 0.3, and the weight of Internet of Things (IoT) device context to 0.2.
[0057] Behavioral Habit Model (Weight 0.5): Historical system data shows that the registered user "Father" is usually not home on weekday afternoons, while "Mother" is often in the kitchen at this time. The living room heatmap shows that few family members stay during this period. However, the current sound source location coordinates are in the living room entrance area, a location that has a high probability of being used by "Visitors" or "Delivery Personnel" in the past week (high frequency on the heatmap). The model outputs a probability of 0.8 for "Visitors" and 0.1 for "Father". Speech-semantic analysis (weight 0.3): The statement "open the door" is a sensitive operation. The semantics lack commonly used family terms (such as "husband" or "baby"), and the tone is relatively formal, consistent with that of a visitor or service personnel. The lightweight NLU model identifies the "help / request" intent and outputs a probability of 0.75 for "non-family member".
[0058] IoT device context (weight 0.2): At the same time, the door magnetic sensor shows that the door is closed but unlocked; the doorbell camera detects a person in a delivery uniform outside the door; the indoor smart door lock has no user Bluetooth key approach record. The system determines that the current request is from an external person and outputs "Visitor" probability 0.9. Combined Calculation: Overall Score = 0.5×0.8 + 0.3×0.75 + 0.2×0.9 = 0.4 + 0.225 + 0.18 = 0.805. Exceeding the preset threshold of 0.75, the system outputs the role tag as "Suspected Visitor" (non-administrator).
[0059] After the role tags are determined, different role tags correspond to different device control permissions, and the target smart home devices can be controlled according to the role tags.
[0060] For example, it can be pre-set that family members have door lock control permissions while visitors do not. When a family member enters the voice command "open the door," they can control the door lock to open. However, when a visitor enters the voice command "open the door," they cannot directly control the door lock to open. Instead, a "open the door" request can be generated and sent to the user who can control the door lock. The door lock can only be opened after the user confirms that the door lock is open.
[0061] Step S105: Generate first intent information for each speaker based on the target speech data; The target voice data can contain the user's intent. By analyzing the target voice data, the first intent information of each speaker can be obtained. The first intent information is the intent information associated with the control of smart home devices.
[0062] In the smart gateway, a second model and a third model can also be set up. The second model is used to convert speech to text, and the third model is used to perform intent analysis of the text. After obtaining the target speech data output by the first model, the target speech data can be input into the second model used to convert speech to text, and the target text information is output. Then, the target text information is input into the third model used to convert text to intent, and the first intent information of each speaker is output.
[0063] For example, local ASR (using the edge-optimized Whisper-tiny model) and NLU (based on BERT-mini intent classifier) can be performed on each voice data stream to output structured instructions (i.e., first intent information): {intent: "unlock_door", slots: {"location": "front", "method": "voice"}}.
[0064] Step S106: When the first intent information matches a preset sensitive whitelist and the role tag is not a preset tag, generate second intent information based on the first intent information; Wherein, the first intent information is the true intent of the target voice, and the second intent information is a vague intent used to authorize control of the target smart home device after preset privacy processing of the first intent information; The primary intent information characterizes the speaker's specific control intention towards smart home devices, while the role label determines the speaker's control permissions. Combining the primary intent information and the role label enables control over smart home devices.
[0065] In practical applications, there are instances of missing or delayed privacy protection and access control mechanisms for sensitive commands: when the system uploads or processes voice, the original audio or complete text may contain the biometric features (voiceprints) or behavioral intentions of unauthorized users, thus posing a risk of privacy leakage; and permission judgment occurs after command parsing, lacking a pre-interception mechanism.
[0066] To ensure the security of voice control for smart home devices, a sensitive intent whitelist can be set up. After obtaining the first intent information, it can be matched with the sensitive whitelist. If the match is successful and the role label is not a preset label, the first intent information needs to be privatized to obtain the second intent information. When the match fails or the role label is a preset label (such as administrator), no privatization is required, and the first intent information can be directly output to execute the control of smart home devices.
[0067] In one example, the process of generating second intent information based on the first intent information is as follows: generating alternative values for the preset fields in the first intent information with the real values as the center, to obtain second intent data; or semantically generalizing the preset fields in the first intent information according to a predefined generalization hierarchy to obtain second intent data.
[0068] The first intent information may include a first field representing the control intent and a second field representing semantic parameters. When privacy processing of the first intent information is required, a substitute value can be generated based on the true value of the second field to replace the original second field. The new second field is then combined with the first field to obtain the second intent information.
[0069] In addition, the first field and the second field can be semantically generalized to obtain a new first field and a new second field. The combination of the new first field and the second field yields the second intent information.
[0070] For example, a voice interaction system maintains a whitelist of sensitive intents (such as unlock, pay, delete, reset). If the current speaker's role is not equal to "administrator" and the intent is on the whitelist, privacy protection is triggered: an exponential mechanism is applied to key fields in slots (such as "location"): centering on the true value, alternative values are sampled probabilistically from the candidate set {"front", "back", "garage"} according to a preset privacy budget of ε=1.0. The true value has the highest probability of being selected, but other options also have a certain chance of being output; the specific probability is controlled by the privacy budget ε (e.g., when ε=1.0, the true value has approximately a 45% probability of being retained, and the remaining 55% is randomly assigned to other options). This perturbation method effectively prevents attackers from accurately inferring the user's true intent while ensuring that the obfuscated command still has semantic validity, thus supporting subsequent authorization decisions. Alternatively, the entire intent-slot structure can be semantically generalized through a predefined hierarchical structure to generate an obfuscated intent: {intent: "security_action", slots: {"type": "door_control"}}. The original ASR text and the precise slots are stored securely only locally. Step S107: Generate a first notification message based on the second intent information to request control authorization for the target home device of the speaker, and send the notification message to the administrator corresponding to the target home device; Step S108: In response to receiving a second notification message for agreeing to authorize the target home device, the first intent information is sent to the target smart home device; For example, obfuscated secondary intent information can be uploaded to the home control center via TLS 1.3 encryption. The home control center then pushes a notification to the administrator's mobile phone: "A visitor has requested access control operation. Do you wish to authorize?" After the administrator clicks "Allow," the home control center sends the actual command (including a digital signature) to the door lock. If there is no response within 60 seconds, the request is automatically rejected.
[0071] Step S109: Control the target smart home device according to the first intent information.
[0072] In this embodiment, a role label for each speaker is determined based on the analysis of the target voice data of each speaker. Different role labels correspond to different device control permissions, thereby enabling voice control of smart home devices according to the role labels. This effectively solves the problem of controlling home devices in scenarios with multiple speakers. Furthermore, the user's first intent information can be extracted and combined with this information to control smart home devices. This allows for source isolation, privacy protection, and controllable execution of high-risk voice commands in complex home acoustic environments, achieving a balance between security, compliance, and user experience.
[0073] It should be noted that the voice control method for smart home devices provided in this application embodiment can be executed by a voice control device of the smart home device, or a control module in the voice control device of the smart home device for executing the method of loading voice control for the smart home device. This application embodiment uses the execution of the method of loading voice control for the smart home device by the voice control device of the smart home device as an example to illustrate the voice control method for smart home devices provided in this application embodiment.
[0074] Reference Figure 2 This diagram illustrates the structure of a voice control device for a smart home device according to an embodiment of this application, which may specifically include the following modules: The audio data acquisition module 201 is used to acquire multiple audio data synchronously acquired by a time-synchronized audio acquisition device at multiple preset acquisition locations, wherein the audio data is audio acquired when multiple speakers are speaking; The target speech data determination module 202 is used to perform speech separation processing on the plurality of audio data using a pre-trained first model for performing speech separation, to obtain the target speech data for each speaker; The coordinate data determination module 203 is used to determine the coordinate data of each speaker based on the target voice data and the acquisition location; The role label determination module 204 is used to determine the role label of each speaker based on the target voice data and the coordinate data; the role label is set with different device control permissions. The first intent information generation module 205 is used to generate first intent information for each speaker based on the target speech data; The second intent information determination module 206 is used to generate second intent information based on the first intent information when the first intent information matches a preset sensitive whitelist and the role tag is not a preset tag; the first intent information is the true intent of the target voice, and the second intent information is a vague intent used to authorize the control of the target smart home device after preset privacy processing of the first intent information. The first notification information generation module 207 is used to generate a first notification information based on the second intent information for requesting control authorization for the target home device of the speaker, and send the notification information to the administrator corresponding to the target home device; The second notification message generation module 208 is used to send the first intent information to the target smart home device in response to receiving a second notification message for agreeing to authorize the target home device; Home appliance control module 209 is used to control the target smart home appliance according to the first intent information.
[0075] In one embodiment of this application, the first intent information generation module may include: The target text information determination submodule is used to input the target speech data into a second model for converting speech into text, and output the target text information. The first intent information determination submodule is used to input the target text information into a third model for converting text into intent, and output the first intent information of each speaker.
[0076] In one embodiment of this application, the second intent information determination module may include: The first generation submodule is used to generate replacement values for the preset fields in the first intent information with the real values as the center, so as to obtain the second intent data; The second generation submodule is used to semantically generalize the preset fields in the first intent information according to a predefined generalization hierarchy to obtain the second intent data.
[0077] In one embodiment of this application, the coordinate data determination module 203 may include: A time difference determination unit is used to determine the time difference between the target speech data and each audio acquisition device; A sound source direction determination unit is used to determine the sound source direction based on the time difference; A spatial geometric model determination unit is used to determine the spatial geometric model corresponding to the acquisition location; The coordinate data determination unit is used to determine the coordinate data of each speaker based on the direction of the sound source and the spatial geometric model.
[0078] In one embodiment of this application, the role tag determination module 204 may include: The first numerical vector determination submodule is used to convert the target speech data into a first numerical vector. The second numerical vector determination submodule is used to determine the second numerical vector corresponding to each registered user in the preset registered user voiceprint database; The cosine similarity determination submodule is used to determine the cosine similarity between the first numerical vector and the second numerical vector. The role label determination submodule is used to determine the role label of each speaker based on the cosine similarity and the coordinate data.
[0079] In one embodiment of this application, the role tag determination submodule may include: The first role probability determination unit is used to determine the first role probability corresponding to the behavioral habits of each speaker based on the coordinate data when the cosine similarity is less than a preset value. The second role probability determination unit is used to determine the second role probability based on the target speech data. The third role probability determination unit is used to acquire device behavior data of the associated devices of the target smart home device within a preset time period, and determine the third role probability based on the device behavior data. The role label determination unit is used to determine the role label of each speaker based on the first role probability, the second role probability, and the third role probability.
[0080] In one embodiment of this application, the role tag determination unit may include: The weight information determination subunit is used to obtain the first weight value corresponding to the behavior habit, the second weight value of the target voice data, and the third weight value of the device behavior; The probability score determination subunit is used to perform weighted processing on the first role probability, the second role probability, and the third role probability using the first weight value, the second weight value, and the third weight value, respectively, to determine the probability score of the candidate role tag; The role label determination subunit is used to determine the role label of each speaker based on the probability score.
[0081] In this embodiment, a role label for each speaker is determined based on the analysis of the target voice data of each speaker. Different role labels correspond to different device control permissions, thereby enabling voice control of smart home devices according to the role labels. This effectively solves the problem of controlling home devices in scenarios with multiple speakers. Furthermore, the user's first intent information can be extracted and combined with this information to control smart home devices. This allows for source isolation, privacy protection, and controllable execution of high-risk voice commands in complex home acoustic environments, achieving a balance between security, compliance, and user experience.
[0082] The voice control device for smart home devices in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.
[0083] The voice control device for smart home devices in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.
[0084] The voice control device for smart home devices provided in this application embodiment can achieve… Figure 1 The various processes implemented by the voice control device of the smart home device in the method embodiment will not be described again here to avoid repetition.
[0085] Optionally, this application embodiment also provides an electronic device, including a processor 1010, a memory 1009, and a program or instructions stored in the memory 1009 and executable on the processor 1010. When the program or instructions are executed by the processor 1010, they implement the various processes of the above-described smart home device voice control method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0086] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.
[0087] Figure 3 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. The electronic device 1000 includes, but is not limited to, the following components: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.
[0088] The memory 1009 includes applications and an operating system; the user input unit 1007 may include a touch panel 10071 and other input devices 10072; the input unit 1004 may include an image processor 10041 and a microphone 10042; and the display unit 1006 may include a display panel 10061.
[0089] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 3 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here. This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described smart home device voice control method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0090] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0091] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described smart home device voice control method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0092] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0093] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0095] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A voice control method for a smart home device, characterized in that, The method includes: Acquire multiple audio data points synchronously collected by a time-synchronized audio acquisition device at multiple preset acquisition locations, wherein the audio data is audio collected when multiple speakers are speaking; The multiple audio data are processed by a pre-trained first model for performing speech separation to obtain target speech data for each speaker. Based on the target speech data and the acquisition location, determine the coordinate data of each speaker; Each speaker's role label is determined based on the target voice data and the coordinate data; each role label is assigned different device control permissions. Generate first intent information for each speaker based on the target speech data; When the first intent information matches a preset sensitive whitelist and the role tag is not a preset tag, a second intent information is generated based on the first intent information; the first intent information is the true intent of the target voice, and the second intent information is a vague intent used to authorize control of the target smart home device after preset privacy processing of the first intent information; Based on the second intent information, a first notification message is generated for requesting authorization to control the target smart home device of the speaker, and the notification message is sent to the administrator corresponding to the target smart home device; In response to receiving a second notification message for agreeing to authorize the target smart home device, the first intent information is sent to the target smart home device; The target smart home device is controlled according to the first intent information.
2. The method according to claim 1, characterized in that, The step of generating first intent information for each speaker based on the target speech data includes: The target speech data is input into a second model used to convert speech into text, and the target text information is output. The target text information is input into a third model used to convert the text into intent, and the first intent information of each speaker is output.
3. The method according to claim 1, characterized in that, The step of generating second intent information based on the first intent information includes: The preset fields in the first intent information are used to generate replacement values centered on the real values to obtain the second intent data; Alternatively, the preset fields in the first intent information can be semantically generalized according to a predefined generalization hierarchy to obtain the second intent data.
4. The method according to claim 1, characterized in that, The step of determining the coordinate data of each speaker based on the target speech data and the acquisition location includes: Determine the time difference between the target speech data and each audio acquisition device; The direction of the sound source is determined based on the time difference; Determine the spatial geometric model corresponding to the acquisition location; The coordinate data of each speaker are determined based on the direction of the sound source and the spatial geometric model.
5. The method according to claim 1, characterized in that, The process of determining the role label for each speaker based on the target speech data and the coordinate data includes: Convert the target speech data into a first numerical vector; Determine the second numerical vector corresponding to each registered user in the preset registered user voiceprint database; Determine the cosine similarity between the first numerical vector and the second numerical vector; The role label for each speaker is determined based on the cosine similarity and the coordinate data.
6. The method according to claim 5, characterized in that, The process of determining the role label for each speaker based on the cosine similarity and the coordinate data includes: When the cosine similarity is less than a preset value, the probability of the first role corresponding to the behavioral habits of each speaker is determined based on the coordinate data; Determine the probability of the second role based on the target speech data; Acquire device behavior data of associated devices of the target smart home device within a preset time period, and determine the probability of a third role based on the device behavior data; The role label for each speaker is determined based on the first role probability, the second role probability, and the third role probability.
7. The method according to claim 6, characterized in that, The step of determining the role label for each speaker based on the first role probability, the second role probability, and the third role probability includes: Obtain the first weight value corresponding to the behavioral habit, the second weight value of the target voice data, and the third weight value of the device behavior; The probability scores of the candidate character tags are determined by weighting the first character probability, the second character probability, and the third character probability using the first weight value, the second weight value, and the third weight value, respectively. Each speaker's role label is determined based on the probability score.
8. An electronic device, characterized in that, The device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the voice control method for a smart home device as described in any one of claims 1 to 7.