Voice control method, system, device and computer readable storage medium

Abstract

The application discloses a voice control method, system, device and computer readable storage medium, relates to the technical field of voice control, and the method comprises the following steps: acquiring voice data and at least one sensor data, determining the scene type of the current scene of the to-be-controlled device according to each sensor data; generating a target control instruction based on the scene type and the voice data, and controlling the to-be-controlled device according to the target control instruction. The application improves the recognition accuracy of ambiguous voice instructions and improves the robustness of voice interaction.

Description

Technical Field

[0001] This application relates to the field of voice control technology, and in particular to a voice control method, system, device and computer-readable storage medium. Background Technology

[0002] With the rapid development of smart devices, voice command control has gradually become one of the important ways for people to interact with devices. From smart homes to smart wearable devices, voice interaction control provides users with a more convenient and natural interactive experience. Voice interaction control, through acoustic signal acquisition, semantic feature extraction, and command mapping, transforms user voice into device control commands, and has become a core technology for improving human-computer interaction efficiency. Its application in scenarios such as health monitoring and environmental adjustment frees users from the constraints of physical buttons, allowing them to invoke device functions through natural language. The fluency and accuracy of voice interaction directly affect the user experience and the level of device intelligence.

[0003] Traditional voice control methods are mostly based on preset syntax trees and fixed sentence templates. Their core logic lies in parsing commands through keyword matching. For example, the device pre-builds a syntax database containing standardized commands such as "turn up the volume" and "turn on heart rate monitoring," and then triggers preset control commands after identifying keywords through acoustic model matching. This mode performs well when the grammatical structure is regular and can accurately recognize voice commands with standard sentence structures, such as "reduce screen brightness by 20%."

[0004] However, the limitations of this voice control method become apparent when faced with ambiguous language expressions prevalent in real-world scenarios. For example, ambiguous voice commands often used by users in dynamic usage scenarios (such as "too dark" or "add more power") suffer from structural mismatch between natural language expressions and preset grammatical templates. This results in an inability to establish an accurate association between voice data and device control commands, ultimately leading to an increased control error rate. It is evident that this rigid parsing mechanism based on fixed templates severely impacts the robustness of voice interaction.

[0005] Therefore, how to improve the recognition accuracy of ambiguous voice commands in order to enhance the robustness of voice interaction has become an urgent technical problem to be solved in this field. Summary of the Invention

[0006] The main objective of this application is to provide a voice control method, system, device, and computer-readable storage medium, aiming to solve the technical problem of how to improve the recognition accuracy of ambiguous voice commands in order to improve the robustness of voice interaction.

[0007] To achieve the above objectives, this application provides a voice control method, which includes the following steps:

[0008] Acquire voice data and at least one sensor data, and determine the scene type of the current scene of the device to be controlled based on the sensor data;

[0009] Based on the scenario type and the voice data, a target control command is generated, and the device to be controlled is controlled according to the target control command.

[0010] In one possible implementation, the step of generating target control commands based on the scene type and the voice data includes:

[0011] Obtain the weight percentage of each sensor data according to the scene type;

[0012] Target control commands are generated based on the data from each sensor, the weighting of each sensor data, and the voice data.

[0013] In one possible implementation, the step of generating target control commands based on the sensor data, the weight ratio of each sensor data, and the voice data includes:

[0014] The speech data is subjected to semantic recognition processing to obtain semantic understanding results;

[0015] The sensor data, the weight ratio of each sensor data, and the semantic understanding results are input into the pre-trained intent decision model to obtain the target control command.

[0016] In one possible implementation, the step of performing semantic recognition processing on the speech data to obtain a semantic understanding result includes:

[0017] The speech data is preprocessed to obtain enhanced speech data, wherein the preprocessing includes noise reduction and / or speech enhancement.

[0018] The enhanced speech data is subjected to speech recognition to obtain text recognition results, and the text recognition results are subjected to semantic understanding through a pre-trained semantic understanding model to obtain semantic understanding results.

[0019] In one possible implementation, the device to be controlled is connected to a cloud server. After the step of obtaining a semantic understanding result by performing semantic understanding on the text recognition result using a pre-trained semantic understanding model, the method further includes:

[0020] When the updated vocabulary is received from the cloud server, the semantic understanding model is iteratively updated based on the updated vocabulary.

[0021] In one possible implementation, after the step of generating target control commands based on the scene type and the voice data, the method further includes:

[0022] Obtain the execution status feedback type for the target control command, and determine the vibration mode corresponding to the execution status feedback type as the target vibration mode, wherein different execution status feedback types correspond to different vibration modes;

[0023] The device to be controlled is controlled to vibrate in the target vibration mode.

[0024] In one possible implementation, the sensor data includes physiological data, and after the step of generating target control commands based on the scene type and the voice data, the method further includes:

[0025] The user's physiological state is identified based on the physiological data, and a conflict detection is performed between the physiological state and the target control command to obtain the detection result;

[0026] If the detection result indicates that the physiological state conflicts with the target control command, then the device to be controlled is controlled to enter the abnormal command processing mode;

[0027] If the detection result indicates that the physiological state is coordinated with the target control command, then the step of controlling the device to be controlled according to the target control command is performed.

[0028] Furthermore, to achieve the above objectives, this application also provides a voice control system, the voice control system comprising:

[0029] The acquisition module is used to acquire voice data and at least one sensor data, and determine the scene type of the current scene of the device to be controlled based on the sensor data.

[0030] The decision module is used to generate target control instructions based on the scenario type and the voice data, and to control the device to be controlled according to the target control instructions.

[0031] In addition, to achieve the above objectives, this application also provides a voice control device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the voice control method as described above.

[0032] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of the voice control method described above.

[0033] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the voice control method described above.

[0034] One or more technical solutions proposed in this application have at least the following technical effects:

[0035] This application acquires voice data and at least one sensor data, determines the scene type of the current scene of the device to be controlled based on the sensor data, generates a target control command based on the scene type and the voice data, and controls the device to be controlled according to the target control command. Thus, in this embodiment, by introducing a collaborative analysis mechanism of sensor data and voice data, sensor data assists voice data in voice interaction during the voice control process. Specifically, it can be understood that sensor data (such as light intensity, motion acceleration, etc.) can objectively characterize the physical environment state of the device (such as indoor lighting conditions, user movement state, etc.), thereby determining the current scene type of the device to be controlled (such as "night reading scene," "outdoor sports scene," etc.). This allows the scene type to provide dynamic contextual constraints for semantic parsing when a voice command containing ambiguous expressions (such as "adjust brightness") is received. For example, in a "night reading scene," "adjust brightness" is preferentially mapped to a screen brightness adjustment command, while in "outdoor sports mode," it may be preferentially associated with the device's warning light activation operation. This scene-type-based semantic ambiguity resolution mechanism allows speech expressions that originally mismatched with preset grammatical templates to be re-anchored to the device control logic of a specific scene, thus overcoming the semantic vacuum caused by the isolated parsing of speech data in traditional methods. By fusing scene type features represented by speech data and sensor data in a multi-dimensional manner, the sensor data is used to fill in the contextual gaps in voice interaction, achieving a reliable mapping from ambiguous commands to precise control commands. This effectively reduces the probability of control errors caused by command ambiguity, thereby improving the recognition accuracy of ambiguous speech commands and enhancing the robustness of voice interaction. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the first embodiment of the voice control method of this application;

[0039] Figure 2 This is a schematic diagram of the working principle architecture of the voice control method according to an embodiment of the present application;

[0040] Figure 3 This is a schematic diagram illustrating the working principle of voice control in one embodiment of the voice control method of this application.

[0041] Figure 4 This is a schematic diagram of the voice control process according to an embodiment of the voice control method of this application;

[0042] Figure 5 This is a schematic diagram of the hardware operating environment of the voice control method device in the embodiments of this application.

[0043] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] With the increasing popularity of smart wearable devices, users' demand for controlling these devices via voice commands is growing. However, traditional voice recognition systems have limitations in handling ambiguous or imprecise voice commands, leading to a poor user experience. Existing devices rely on precise voice commands (such as fixed phrases) and have low recognition rates for vague expressions ("it's a bit hot"), grammatical errors, and accents. They also lack multi-dimensional contextual understanding capabilities (e.g., in a sports scenario, "turn it up" might refer to heart rate monitoring frequency rather than volume). Furthermore, background noise and environmental interference also reduce the accuracy of voice recognition.

[0046] Based on this, the main solution of this application is: to acquire voice data and at least one sensor data, to determine the scene type of the current scene of the device to be controlled based on the sensor data; to generate a target control command based on the scene type and the voice data, and to control the device to be controlled according to the target control command.

[0047] This application introduces a collaborative analysis mechanism between sensor data and voice data, using sensor data to assist voice data in voice interaction during voice control. Specifically, sensor data (such as light intensity and motion acceleration) can objectively characterize the physical environment of the device (such as indoor lighting conditions and user movement), thereby determining the current scenario type of the device to be controlled (such as "nighttime reading scenario" or "outdoor sports scenario"). When a voice command containing ambiguous expressions (such as "adjust brightness") is received, the scenario type provides dynamic contextual constraints for semantic parsing. For example, in a "nighttime reading scenario," "adjust brightness" is preferentially mapped to a screen brightness adjustment command, while in "outdoor sports mode," it may be preferentially associated with turning on the device's warning light. This scenario-type-based semantic ambiguity resolution mechanism allows voice expressions that originally mismatched with the preset grammatical template to be re-anchored to the device control logic within a specific scenario, thus overcoming the semantic vacuum caused by the isolated parsing of voice data in traditional methods. By fusing scene type features represented by voice data and sensor data in a multi-dimensional manner, the missing context in voice interaction is filled in using sensor data, and a reliable mapping from fuzzy commands to precise control commands is achieved. This effectively reduces the probability of control errors caused by command ambiguity, thereby improving the recognition accuracy of fuzzy voice commands and enhancing the robustness of voice interaction.

[0048] It should be noted that the executing entity of the various embodiments of the voice control method of this application can be a computing service device with data processing, network communication, and program execution functions, such as a server, tablet computer, personal computer, mobile phone, etc., or a voice control device capable of realizing the above functions, such as a smart wearable device, smart treadmill, smart speaker, smart TV, smart lighting device, etc. The various embodiments of the voice control method of this application do not impose specific limitations in this regard. For example, a smart wearable device is used as the executing entity to illustrate and explain the various embodiments of this application.

[0049] Based on this, this application proposes a voice control method according to a first embodiment, please refer to... Figure 1 As shown, the voice control method includes the following steps S10 to S20:

[0050] Step S10: Acquire voice data and at least one sensor data, and determine the scene type of the current scene of the device to be controlled based on the sensor data.

[0051] It should be noted that the device to be controlled refers to the device with which voice interaction is currently performed and which voice control is to be performed. Specifically, it can be the smart wearable device itself as the executing entity, or other devices connected to the smart wearable device. The following descriptions of various embodiments use the smart wearable device itself as the device to be controlled.

[0052] The voice data can be the raw sound signal picked up by a smart wearable device, such as a user-inputted voice command. Furthermore, a microphone array can be installed within the smart wearable device to pick up sound signals from all directions. After acquiring the voice data, preprocessing can be performed on the raw voice data, such as noise reduction (beamforming algorithms can be used to suppress noise in voice data acquired using a microphone array) and voice enhancement, to improve the signal quality of the voice data. Subsequent processing can then be performed based on the preprocessed voice data to reduce the impact of background noise and environmental interference on the accuracy of speech recognition.

[0053] Specifically, the sensor data can be data collected by various sensors in smart wearable devices (such as physiological sensors, environmental sensors, motion sensors, etc.). Among them, physiological sensors refer to sensors used to monitor the user's vital signs or physical state parameters, such as heart rate sensors, body temperature sensors, and PPG (Photo Plethysmo Graph) sensors; environmental sensors refer to sensors used to sense physical quantities or spatial information of the external environment, such as GPS (Global Positioning System) sensors, temperature and humidity sensors, barometric pressure sensors, and ambient light sensors; motion sensors refer to sensors used to detect the user's limb movement state, posture changes, or acceleration characteristics, such as IMU (Inertial Measurement Unit), gyroscopes, and accelerometers.

[0054] Furthermore, smart wearable devices can simultaneously collect voice data and sensor data. Specifically, a timestamp alignment mechanism can be adopted, such as the IEEE (Institute of Electrical and Electronics Engineers) 1588 Precision Time Protocol, to ensure that the time deviation between voice and sensor data is small, thus ensuring that the scene type determined based on the sensor data is the same scene type in which the smart wearable device was in when the voice data was collected.

[0055] Based on sensor data, the scene type of the device under control can be determined. The scenes that the smart wearable device may be in can be classified in advance according to actual needs. For example, in one specific implementation, the scene of the smart wearable device can be divided into scene types such as home scene, sports scene, medical scene, and night reading scene. Understandably, the data collected by sensors when users use smart wearable devices in different scenarios will typically differ. For example, in a home setting, motion sensors (such as accelerometers and gyroscopes) usually collect data with gradual changes (reflecting static or low-intensity activity), environmental sensors (such as temperature and humidity sensors) can usually detect constant indoor temperature and humidity, and physiological sensors (such as heart rate sensors) typically collect data within the user's resting heart rate range. In an exercise setting, accelerometers can usually detect high-frequency periodic vibrations (such as running cadence), GPS sensors can detect movement trajectories, and physiological sensors can detect heart rate increases to the exercise threshold range. In a medical setting, physiological sensors (such as PPG sensors) typically continuously collect key indicators such as blood oxygen and heart rate variability, while motion sensors detect prolonged periods of rest or low activity. In a nighttime reading setting, ambient light sensors can usually detect low light intensity, and motion sensors can detect when the user's wrist is stationary or making slight movements (like turning pages). Based on these differences, a scenario classification method can be preset to determine the scenario type in which the smart wearable device is located.

[0056] Furthermore, to improve the accuracy and efficiency of scene classification, a scene classification model can be pre-trained. Specifically, the model is trained using sensor data as input and the scene type corresponding to the sensor data as a label, resulting in a trained scene classification model. The acquired sensor data can then be input into this pre-trained model, and the output will show the scene type of the current location of the controlled device. Specifically, this scene classification model can be any classification model, such as a random forest model, logistic regression model, support vector machine model, decision tree model, etc. This embodiment does not impose any specific limitations on it.

[0057] Furthermore, the scene classification model can be trained on a cloud server, and after the training is complete, the trained scene classification model can be deployed to a smart wearable device to use the scene classification model locally, thereby reducing the privacy leakage of local data.

[0058] Step S20: Generate a target control command based on the scene type and the voice data, and control the device to be controlled according to the target control command.

[0059] After determining the scene type of the smart wearable device's current environment, target control commands are generated based on the scene type and voice data. Specifically, a preset command generation model can be invoked. By analyzing the device's functional requirements corresponding to the scene type and the user intent contained in the voice data, target control commands matching the current scene are generated. For example, if the scene type is "sports mode" and the voice data is "start recording data," the generated target control command could be "activate the motion sensor and record motion data." As another example, if the scene type is "home scene" and the voice data is "too dark," the generated target control command could be "increase the brightness of the smart lights."

[0060] After generating the target control command, the smart wearable device is controlled according to the target control command, thereby completing the voice interaction with the smart wearable device. Specifically, the target control command can be sent to the corresponding control unit of the smart wearable device, so that the control unit can execute the target control command. If the target control command involves hardware function adjustment, such as "turn on GPS positioning", the control unit can directly drive the hardware module to execute the command; if it involves software function adjustment, such as "switch to music playback interface", a software control signal can be sent to the device's operating system to trigger the corresponding software function.

[0061] This embodiment introduces a collaborative analysis mechanism for sensor data and voice data, using sensor data to assist voice data in voice interaction during voice control. Specifically, sensor data (such as light intensity and motion acceleration) can objectively characterize the physical environment of the device (such as indoor lighting conditions and user movement), thereby determining the current scene type of the device to be controlled (such as "nighttime reading scene" or "outdoor sports scene"). When a voice command containing ambiguous expressions (such as "adjust brightness") is received, the scene type provides dynamic contextual constraints for semantic parsing. For example, in a "nighttime reading scene," "adjust brightness" is preferentially mapped to a screen brightness adjustment command, while in "outdoor sports mode," it may be preferentially associated with turning on the device's warning light. This scene-type-based semantic ambiguity resolution mechanism allows voice expressions that originally mismatched with the preset grammatical template to be re-anchored to the device control logic within a specific scene, thus overcoming the semantic vacuum caused by the isolated parsing of voice data in traditional methods. By fusing scene type features represented by voice data and sensor data in a multi-dimensional manner, the missing context in voice interaction is filled in using sensor data, and a reliable mapping from fuzzy commands to precise control commands is achieved. This effectively reduces the probability of control errors caused by command ambiguity, thereby improving the recognition accuracy of fuzzy voice commands and enhancing the robustness of voice interaction.

[0062] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of generating target control commands based on the scene type and the voice data includes:

[0063] Step A10: Obtain the weight percentage of each sensor data according to the scene type;

[0064] It should be noted that the weight ratio of each sensor data can be obtained based on a preset mapping relationship. The preset mapping relationship includes the mapping relationship between different scene types and the weight ratio of each sensor data. The mapping relationship can be preset. For example, in a specific embodiment, the preset mapping relationship may include the mapping relationship shown in Table 1 below.

[0065]

[0066] Table 1

[0067] It's important to note that the weighting of each sensor's data can be dynamically adjusted. For example, Q-learning reinforcement learning can continuously optimize the weighting of each sensor's data under different scenario types. By introducing a dynamic optimization mechanism, the weighting can adaptively adjust to changes in the actual environment and task requirements. For instance, in motion scenarios, when GPS signals are blocked, the decision weight of IMU data is automatically increased to maintain positioning accuracy; or in medical monitoring, when abnormal fluctuations in temperature data are detected, the analysis weight of PPG data is dynamically increased to cross-validate physiological states. This dynamic optimization not only overcomes the limitations of fixed-weight models in adapting to complex scenarios but also gradually establishes a more refined sensor collaboration mechanism by continuously learning from historical data and environmental characteristics. Ultimately, this achieves a simultaneous improvement in resource utilization, data fusion accuracy, and system response speed, ensuring that the multi-sensor system always maintains its optimal state in different application scenarios.

[0068] Step A20: Generate target control instructions based on the data from each sensor, the weight ratio of each sensor data, and the voice data.

[0069] After obtaining the weight percentages of each sensor data point, command decisions are made based on the sensor data, their corresponding weight percentages, and voice data to generate target control commands. By fusing multimodal sensor data through weight percentage fusion, the dominant role of high-weight sensor data and the semantic parsing of voice data complement each other during command decision-making. For example, when environmental noise interferes with voice commands (such as the user's breathing affecting voice recognition in a motion scenario), the high weight percentage (45%) of IMU data accurately captures the intention of limb movements. In medical scenarios, the weighted collaboration of PPG data (50%) and temperature sensor data (30%) can effectively filter out accidental voice mis-triggers, ensuring that the generation of key commands such as blood oxygen monitoring strictly follows physiological data characteristics, thereby improving the environmental adaptability and reliability of command generation.

[0070] In one possible implementation, the step of generating target control commands based on the sensor data, the weight ratio of each sensor data, and the voice data includes:

[0071] Step B10: Perform semantic recognition processing on the speech data to obtain semantic understanding results;

[0072] This semantic recognition processing specifically refers to recognizing the semantics of voice data in order to understand the user's intent.

[0073] Step B20: Input the sensor data, the weight ratio of each sensor data, and the semantic understanding result into the pre-trained intent decision model to obtain the target control command.

[0074] The pre-trained intent decision model can be any trained classification model, such as a decision tree model. Furthermore, a decision tree model driven by SHAP (Shapley Additive exPlanations) values ​​can be used to automatically eliminate low-contribution sensor data based on SHAP values ​​(e.g., in medical scenarios, when the SHAP value of a temperature sensor is <0.05, its instantaneous abnormal fluctuations are ignored), reducing redundant feature interference and improving the model's inference speed.

[0075] After inputting relevant data into a pre-trained intention decision model, the intention decision model can output the probability distribution of all control commands, and thus determine the target control command based on the probability distribution, such as selecting the control command with the highest probability as the target control command.

[0076] Similar to scene classification models, the intent decision model can also be trained on a cloud server. After training, the trained intent decision model can be deployed to smart wearable devices to generate target control commands locally, reducing the privacy leakage of local data.

[0077] In one possible implementation, the step of performing semantic recognition processing on the speech data to obtain a semantic understanding result includes:

[0078] Step C10: Perform signal preprocessing on the speech data to obtain enhanced speech data, wherein the signal preprocessing includes noise reduction and / or speech enhancement;

[0079] Noise reduction refers to eliminating environmental background noise (such as fan noise, traffic noise, etc.) in speech signals through digital signal processing technology to improve the speech signal-to-noise ratio. For example, beamforming algorithms and frequency domain mask estimation algorithms can be used for noise reduction processing. Speech enhancement refers to improving speech clarity and intelligibility by adjusting the spectral characteristics of the speech signal. For example, in a specific implementation, the DeepFilterNet2 (Deep Filtering Network 2) model can be used for speech enhancement processing. This model achieves real-time enhancement of low signal-to-noise ratio speech signals through a hierarchical feature extraction module and a complex domain filtering mechanism, which is especially suitable for repairing speech distortion caused by environmental wind speed interference in motion scenes.

[0080] Step C30: Perform speech recognition on the enhanced speech data to obtain text recognition results, and then perform semantic understanding on the text recognition results using a pre-trained semantic understanding model to obtain semantic understanding results.

[0081] After preprocessing the speech data to obtain enhanced speech data, speech recognition is performed on the enhanced speech data to obtain text recognition results. Specifically, a speech recognition model can be used for speech recognition. Furthermore, in order to reduce the impact of the user's accent on the speech recognition results, in a preferred embodiment, a dialect adaptive acoustic model can be used for speech recognition, and the dialect adaptive acoustic model can be trained by transfer learning plus few-sample training.

[0082] After obtaining the text recognition result, a pre-trained semantic understanding model is used to perform semantic understanding on the text recognition result to obtain the semantic understanding result, which can be the output of the semantic understanding model. The semantic understanding model can be any pre-trained natural language understanding model, such as the BERT (Bidirectional Encoder Representations from Transformers) model.

[0083] Furthermore, the semantic understanding model can be trained on a cloud server, and processing techniques such as model compression and knowledge distillation can be used to deploy a lightweight semantic understanding model in smart wearable devices, reducing the computing power requirements of smart wearable devices.

[0084] In one possible implementation, the device to be controlled is connected to a cloud server. After the step of obtaining a semantic understanding result by performing semantic understanding on the text recognition result using a pre-trained semantic understanding model, the method further includes:

[0085] Step D10: When the updated lexicon is received from the cloud server, the semantic understanding model is iteratively updated based on the updated lexicon.

[0086] Users generate a large amount of voice commands and contextual data when using smart wearable devices. These devices can collect this data to update the user's personalized vocabulary and upload it to a cloud server. Furthermore, before uploading to the cloud, differential privacy techniques can be used to protect the data; for example, noise can be added using a Laplacian mechanism or a Gaussian mechanism. For word frequency statistics in the vocabulary, the frequency of each word can be counted locally, and then noise can be added to the results to ensure that data from a single user does not significantly affect the overall statistical results. Data can be uploaded asynchronously, meaning it is not uploaded in real time but accumulates to a certain amount or is uploaded periodically. After receiving noise data from multiple users, the cloud can aggregate this data to update the global vocabulary, and the updated vocabulary is then distributed to the smart wearable devices connected to the cloud.

[0087] After receiving the updated vocabulary from the cloud, the smart wearable device updates the vocabulary and iteratively updates the semantic understanding model based on the updated vocabulary. This dynamically adapts to the user's vocabulary usage habits, enabling the semantic understanding model to personalize its understanding of the user's language habits and improve the accuracy of the semantic understanding model.

[0088] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. In addition, after the step of controlling the device to be controlled according to the target control command, the method further includes:

[0089] Step E10: Obtain the execution status feedback type for the target control command, and determine the vibration mode corresponding to the execution status feedback type as the target vibration mode. Different execution status feedback types correspond to different vibration modes.

[0090] The execution status feedback type specifically refers to the type of execution status feedback provided for the target control instruction after it has been generated and during its execution. Examples of feedback include: instruction confirmed to have started execution, system error not executed, abnormal situation not executed, instruction in execution, etc.

[0091] Vibration modes corresponding to different execution state feedback types can be preset, and the differences in vibration modes can be specifically manifested in differences in technical parameters such as vibration intensity, vibration duration, and / or vibration frequency. For example, in one specific embodiment, the correspondence between different execution state feedback types and vibration modes can be as shown in Table 2 below.

[0092]

[0093] Table 2

[0094] Step E20: Control the device to be controlled to vibrate in the target vibration mode.

[0095] By accurately matching execution status with tactile feedback methods, multi-level and highly recognizable tactile semantic expression is achieved, significantly enhancing the information transmission efficiency of human-computer interaction. The vibration-based tactile feedback method ensures a high emergency response rate (such as heart rate overload alarm during exercise) in noisy or high-load scenarios. At the same time, the differentiated vibration prompt strategy enables users to accurately identify the execution status of instructions without visual assistance. Thus, the correctness of instruction understanding and the effectiveness of operation are confirmed through multi-level tactile feedback (vibration mode), forming a tactile interaction system that balances immediacy, robustness, and low power consumption.

[0096] In one possible implementation, the sensor data includes physiological data, and after the step of generating target control commands based on the scene type and the voice data, the method further includes:

[0097] Step F10: Identify the user's physiological state based on the physiological data, and perform conflict detection between the physiological state and the target control command to obtain the detection result;

[0098] The system identifies the user's physiological state based on physiological data. This physiological state refers to abnormal states of the user's cardiovascular and cerebrovascular related physiological indicators, such as whether the user's heart rate is too high, blood pressure is abnormal, blood oxygen saturation is insufficient, respiratory rate exceeds the threshold, and body temperature is too high.

[0099] When a user's physiological state is detected, a conflict detection is performed between the physiological state and the target control command. Generally, if the execution of the target control command would further worsen the user's abnormal physiological state, the target control command is considered to be in conflict with the user's physiological state; otherwise, the target control command is considered to be in harmony with the user's physiological state. For example, if the target control command is "increase the speed of the smart treadmill," but the user's physiological state is a heart rate >150 bpm (excessively high), increasing the treadmill speed would obviously further increase the user's heart rate, thus determining that the user's physiological state conflicts with the target control command. On the other hand, if the target control command is "decrease the speed of the smart treadmill," and the user's physiological state is a heart rate >150 bpm (excessively high), decreasing the treadmill speed would not further increase the user's heart rate but might even slow it down, thus determining that the user's physiological state is in harmony with the target control command.

[0100] Step F20: If the detection result indicates that the physiological state conflicts with the target control command, then control the device to be controlled to enter the abnormal command processing mode.

[0101] If the detection results indicate a conflict between the physiological state and the target control command, the smart wearable device is controlled to enter an abnormal command processing mode. This abnormal command mode is distinct from the regular operating mode. In this mode, the execution of the target control command is blocked, and a safety protection mechanism is triggered. Furthermore, conflict event logs can be recorded and uploaded to the cloud for risk modeling and analysis to optimize subsequent command conflict detection algorithms. For example, in one specific implementation, the user's PPG signal is collected in the abnormal command processing mode, and heart rate variability is calculated. If the RMSSD (Root Mean Square of Successive Differences) is less than 30ms, it is considered an abnormal heart rate variability, and an emergency contact call can be triggered.

[0102] Furthermore, a prompt message can be output before entering the abnormal command handling mode to remind the user that the command conflicts with their physiological state, allowing the user to decide whether to enter the abnormal command handling mode based on the actual situation, thus avoiding accidental entry into the abnormal command handling mode.

[0103] Step F30: If the detection result indicates that the physiological state is coordinated with the target control command, then the step of controlling the device to be controlled according to the target control command is executed.

[0104] If the physiological state is detected to be in harmony with the target control command, the smart wearable device is controlled based on the target control command. Specifically, the smart wearable device may begin to execute the target control command to complete the voice interaction.

[0105] This embodiment analyzes the user's physiological state in real time, constructs a dynamic correlation between physiological state and device control commands, and achieves advanced perception of abnormal physiological states and prediction of command conflicts. Specifically, when the user issues a target control command, it intelligently identifies the potential risk of command execution to the user's physiological state based on physiological data, thereby proactively intercepting high-risk operations (such as preventing acceleration commands when the heart rate is too high) before command execution, effectively avoiding secondary health damage caused by erroneous device responses.

[0106] For example, to aid in understanding the voice control flow in this embodiment, a specific embodiment is now described. In this specific embodiment, refer to... Figure 2 As shown, the overall architecture of the voice control solution is divided into three layers: a user interaction layer, an edge computing layer, and a cloud collaboration layer. The user interaction layer handles voice input and provides tactile and visual feedback; the edge computing layer achieves real-time response by deploying lightweight models locally; and the cloud collaboration layer updates the model and analyzes big data to update the user's personalized vocabulary. Based on this, refer to... Figure 3 As shown, the working principle of voice control is as follows: After voice data is input, it undergoes signal preprocessing (environmental noise suppression, voice enhancement), and scene profile construction is performed by combining wearable device sensor data to determine the scene type in which the wearable device is located, achieving multimodal data fusion; then, the AI ​​model makes command decisions; finally, the device is controlled based on the decision results, and haptic feedback is used to confirm the correctness of command understanding. Specifically, refer to... Figure 4 As shown, the voice control process includes:

[0107] 1. Voice input

[0108] Microphone ring array beamforming technology can be used to pick up voice data and suppress ambient noise; an improved DeepFilterNet2 is used to efficiently reduce noise and enhance voice data.

[0109] 2. Multimodal sensor fusion

[0110] Acquire data from various sensor types, including physiological sensors (such as heart rate, body temperature, PPG), environmental sensors (such as GPS, temperature and humidity sensors), and motion sensors (9-axis IMU). Infer the current scenario type (such as sports, medical monitoring, home control) of the device by real-time analysis of multimodal sensor data (such as heart rate, motion status, and location). Then, dynamically adjust the weighting of each sensor data point based on the scenario type to optimize subsequent decision-making. Machine learning models (such as random forest models) can be used to analyze sensor data in real-time to determine the scenario type. Dynamic adjustment of sensor data weighting based on scenario type can be achieved by defining weighting rules and dynamic optimization mechanisms.

[0111] 1) Weighting rules

[0112] Predefined scene-sensor weight mapping table (updateable via cloud):

[0113]

[0114]

[0115] 2) Dynamic optimization mechanism

[0116] Q-learning reinforcement learning can be used to continuously optimize weight allocation. The reward function in Q-learning reinforcement learning is the user's satisfaction with the system response (explicit rating and / or implicit operation correction). The output is a dynamically adjusted sensor weight matrix.

[0117] 3. Semantic understanding

[0118] A three-level nested neural network architecture is adopted. The first layer is noise-resistant speech enhancement (improved DeepFilterNet2); the second layer is a dialect-adaptive acoustic model (transfer learning + few-shot training) to obtain speech recognition results; the third layer is an interpretable intent decision tree (using SHAP value-driven feature selection), which outputs the instruction probability distribution based on the semantic understanding results of the input and the context data of the sensor (sensor data and corresponding weight ratios).

[0119] After obtaining speech recognition results using a dialect-adaptive acoustic model, semantic understanding results are obtained through the BERT model. This BERT model is a lightweight model that can be deployed locally. Knowledge distillation is used to compress the BERT-base, reducing the number of layers and attention heads, and pruning the vocabulary to retain only high-frequency words in the smart wearable domain. The BERT model parses the contextual semantics of fuzzy commands, and combined with edge-cloud collaborative computing, precise control commands are ultimately generated.

[0120] Personalized word lists are updated asynchronously in the cloud (differential privacy protection). Users generate a large amount of voice commands and contextual data when using smart wearable devices. This data needs to be collected and used to update the personalized word list. Differential privacy techniques are used to process the data before it leaves the device. For example, Laplace or Gaussian mechanisms can be used to add noise. For word frequency statistics in the word list, the frequency of each word can be counted locally, and then noise can be added to the statistical results to ensure that the data from a single user does not significantly affect the global statistical results. Data is uploaded asynchronously, meaning it is not uploaded in real time but accumulates to a certain amount or is uploaded periodically. After receiving noisy data from multiple users, the cloud needs to aggregate this data to update the global word list. The updated word list is then securely distributed to all devices, enabling devices to adaptively iterate and update the BERT model based on the updated word list.

[0121] 4. Control Execution and Feedback

[0122] Multi-level haptic feedback (vibration patterns) confirms the correctness of command understanding and verifies the effectiveness of operations. This multi-level haptic feedback is implemented through the vibration module (such as a linear motor or eccentric rotor motor) and software control logic of the smart wearable device. Its core lies in dynamically adjusting the intensity, duration, and frequency of vibration to match the user interaction needs of different scenarios. The "multi-level" nature of the haptic feedback is achieved by defining different vibration modes in software and binding them to user operations or system states.

[0123] Abnormal commands automatically trigger optical pulse verification (PPG signal analysis of the user's true intent). When the system detects a conflict between a command and the user's physiological state, it activates green light (wavelength 530nm) to collect PPG signals, calculates heart rate variability (RMSSD < 30ms is considered abnormal), and triggers an emergency contact call (requiring secondary confirmation from the user).

[0124] It should be noted that the above examples are only used to help understand this embodiment and do not constitute a limitation on the voice control process of this embodiment. Any simple modifications based on this technical concept are within the protection scope of this application.

[0125] Furthermore, embodiments of this application also propose a voice control system, which includes:

[0126] The acquisition module is used to acquire voice data and at least one sensor data, and determine the scene type of the current scene of the device to be controlled based on the sensor data.

[0127] The decision module is used to generate target control instructions based on the scenario type and the voice data, and to control the device to be controlled according to the target control instructions.

[0128] In one embodiment, the decision module is further configured to:

[0129] Obtain the weight percentage of each sensor data according to the scene type;

[0130] Target control commands are generated based on the data from each sensor, the weighting of each sensor data, and the voice data.

[0131] In one embodiment, the decision module is further configured to:

[0132] The speech data is subjected to semantic recognition processing to obtain semantic understanding results;

[0133] The sensor data, the weight ratio of each sensor data, and the semantic understanding results are input into the pre-trained intent decision model to obtain the target control command.

[0134] In one embodiment, the decision module is further configured to:

[0135] The speech data is preprocessed to obtain enhanced speech data, wherein the preprocessing includes noise reduction and / or speech enhancement.

[0136] The enhanced speech data is subjected to speech recognition to obtain text recognition results, and the text recognition results are subjected to semantic understanding through a pre-trained semantic understanding model to obtain semantic understanding results.

[0137] In one embodiment, the device to be controlled is connected to a cloud server, and the decision module is further configured to:

[0138] When the updated vocabulary is received from the cloud server, the semantic understanding model is iteratively updated based on the updated vocabulary.

[0139] In one embodiment, the decision module is further configured to:

[0140] Obtain the execution status feedback type for the target control command, and determine the vibration mode corresponding to the execution status feedback type as the target vibration mode, wherein different execution status feedback types correspond to different vibration modes;

[0141] The device to be controlled is controlled to vibrate in the target vibration mode.

[0142] In one embodiment, the decision module is further configured to:

[0143] The user's physiological state is identified based on the physiological data, and a conflict detection is performed between the physiological state and the target control command to obtain the detection result;

[0144] If the detection result indicates that the physiological state conflicts with the target control command, then the device to be controlled is controlled to enter the abnormal command processing mode;

[0145] If the detection result indicates that the physiological state is in harmony with the target control command, then the device to be controlled is controlled according to the target control command.

[0146] Furthermore, this application also proposes a voice control device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the voice control method described above.

[0147] refer to Figure 5 The diagram illustrates a structural schematic suitable for implementing the voice control device in the embodiments of this application. The voice control device in the embodiments of this application may also include, but is not limited to, smartwatches, smart bracelets, smart treadmills, etc. Figure 5 The voice control device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0148] like Figure 5 As shown, the voice control device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the voice control device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the voice control device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show voice control devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0149] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0150] The voice control device provided in this application, employing the voice control method described in the above embodiments, can solve the technical problem of how to improve the recognition accuracy of ambiguous voice commands, thereby enhancing the robustness of voice interaction. Compared with the prior art, the beneficial effects of the voice control device provided in this application are the same as those of the voice control method provided in the above embodiments, and other technical features of this voice control device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0151] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0153] In addition, to achieve the above objectives, embodiments of this application also provide a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the voice control method in the above embodiments.

[0154] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0155] The aforementioned computer-readable storage medium may be included in the voice control device; or it may exist independently and not assembled into the voice control device.

[0156] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a voice control device, cause the voice control device to: acquire voice data and at least one sensor data; determine the scene type of the current scene of the device to be controlled based on the sensor data; generate a target control command based on the scene type and the voice data; and control the device to be controlled according to the target control command.

[0157] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0159] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the modules themselves.

[0160] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described voice control method. This solves the technical problem of how to improve the recognition accuracy of ambiguous voice commands, thereby enhancing the robustness of voice interaction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the voice control method provided in the above embodiments, and will not be repeated here.

[0161] Furthermore, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the steps of the voice control method described above.

[0162] The specific implementation of the computer program product in this application is basically the same as the embodiments of the above-described voice control and / or voice enhancement methods, and will not be repeated here.

[0163] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0164] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0166] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A voice control method, characterized in that, The voice control method includes the following steps: Acquire voice data and at least one sensor data, and determine the scene type of the current scene of the device to be controlled based on the sensor data; Based on the scenario type and the voice data, a target control command is generated, and the device to be controlled is controlled according to the target control command.

2. The voice control method as described in claim 1, characterized in that, The step of generating target control commands based on the scene type and the voice data includes: Obtain the weight percentage of each sensor data according to the scene type; Target control commands are generated based on the data from each sensor, the weighting of each sensor data, and the voice data.

3. The voice control method as described in claim 2, characterized in that, The step of generating target control commands based on the data from each sensor, the weighting of each sensor data, and the voice data includes: The speech data is subjected to semantic recognition processing to obtain semantic understanding results; The sensor data, the weight ratio of each sensor data, and the semantic understanding results are input into the pre-trained intent decision model to obtain the target control command.

4. The voice control method as described in claim 3, characterized in that, The step of performing semantic recognition processing on the speech data to obtain a semantic understanding result includes: The speech data is preprocessed to obtain enhanced speech data, wherein the preprocessing includes noise reduction and / or speech enhancement. The enhanced speech data is subjected to speech recognition to obtain text recognition results, and the text recognition results are subjected to semantic understanding through a pre-trained semantic understanding model to obtain semantic understanding results.

5. The voice control method as described in claim 4, characterized in that, The device to be controlled is connected to a cloud server. After the step of obtaining a semantic understanding result by performing semantic understanding on the text recognition result using a pre-trained semantic understanding model, the method further includes: When the updated vocabulary is received from the cloud server, the semantic understanding model is iteratively updated based on the updated vocabulary.

6. The voice control method according to any one of claims 1 to 5, characterized in that, After the step of controlling the device to be controlled according to the target control command, the method further includes: Obtain the execution status feedback type for the target control command, and determine the vibration mode corresponding to the execution status feedback type as the target vibration mode, wherein different execution status feedback types correspond to different vibration modes; The device to be controlled is controlled to vibrate in the target vibration mode.

7. The voice control method according to any one of claims 1 to 5, characterized in that, The sensor data includes physiological data. After the step of generating target control commands based on the scene type and the voice data, the method further includes: The user's physiological state is identified based on the physiological data, and a conflict detection is performed between the physiological state and the target control command to obtain the detection result; If the detection result indicates that the physiological state conflicts with the target control command, then the device to be controlled is controlled to enter the abnormal command processing mode; If the detection result indicates that the physiological state is coordinated with the target control command, then the step of controlling the device to be controlled according to the target control command is performed.

8. A voice control system, characterized in that, The voice control system includes: The acquisition module is used to acquire voice data and at least one sensor data, and determine the scene type of the current scene of the device to be controlled based on the sensor data. The decision module is used to generate target control instructions based on the scenario type and the voice data, and to control the device to be controlled according to the target control instructions.

9. A voice control device, characterized in that, The device includes: a memory, a processor, and a voice control program stored in the memory and executable on the processor, the voice control program being configured to implement the steps of the voice control method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium includes a computer-readable storage medium on which a voice control program is stored, and when executed by a processor, the voice control program implements the steps of the voice control method as described in any one of claims 1 to 7.

Images