Smart wearable device and response system

Abstract

The application discloses an intelligent wearable device and a response system, wherein the intelligent wearable device comprises a first response component and a second response component; the first response component and the second response component are each provided with an information acquisition unit and a lightweight inference model; one of the first response component and the second response component serves as a main response component, and the main response component is used for collecting user dynamic information through the corresponding information acquisition unit and outputting a first inference result corresponding to the user dynamic information by using the corresponding lightweight inference model. On the basis of responding to the corresponding demand of the user dynamic information, the application can improve the inference speed, reduce the demand for storage resources and computing resources, and also reduce the power consumption of the intelligent wearable device in the user demand response process.

Description

Technical Field

[0001] This application relates to the field of speech signal processing technology, specifically to an intelligent wearable device and its response system. Background Technology

[0002] AI-powered earphones and / or AI glasses, among other smart wearable devices, have been widely adopted in people's work and daily lives. These devices typically collect control data such as user voice commands and respond to the collected data to meet specific user needs. Some smart wearable devices can also load corresponding intelligent models based on application scenarios and / or specific user requirements. These models process the received control data to provide a more in-depth and intelligent response to user needs. However, the inventors have encountered a significant power consumption issue while researching how smart wearable devices respond to user demands. Summary of the Invention

[0003] In view of this, this application provides a smart wearable device and a response system to reduce the power consumption of the smart wearable device.

[0004] This application provides a smart wearable device, including a first response component and a second response component; both the first response component and the second response component are provided with an information acquisition unit and a lightweight inference model.

[0005] One of the first response component and the second response component serves as the main response component. The main response component is used to collect user dynamic information through the corresponding information collection unit and output the first inference result corresponding to the user dynamic information using the corresponding lightweight inference model.

[0006] Optionally, the other of the first response component and the second response component is a secondary response component; the primary response component is further configured to obtain the quality evaluation parameters of the first inference result, and when the evaluation parameters exceed the corresponding quality reference range, send inference instruction information to the secondary response component; the secondary response component is configured to collect user dynamic information after receiving the inference instruction information, and output the second inference result corresponding to the user dynamic information using the corresponding lightweight inference model; the first inference result and the second inference result are used to determine the final inference result.

[0007] Optionally, the quality evaluation parameters include at least one of confidence level, wind noise parameter, signal-to-noise ratio, total harmonic distortion, and amplitude linearity.

[0008] Optionally, the main response component or the secondary response component is further configured to select the one with better quality evaluation parameters from the first inference result and the second inference result as the final inference result.

[0009] Optionally, the main response component or the secondary response component is further configured to obtain a first total confidence level corresponding to multiple first inference results and a second total confidence level corresponding to multiple second inference results within a preset time period, and determine the response component corresponding to the better confidence level among the first total confidence level and the second total confidence level as the current main response component.

[0010] Optionally, the lightweight inference model includes a lightweight model obtained by knowledge distillation of the teacher model.

[0011] Optionally, the information acquisition unit includes at least one of a microphone, a camera, and a motion sensor.

[0012] Optionally, one of the first response component and the second response component is a left-side component of the smart wearable device, and the other is a right-side component of the smart wearable device.

[0013] This application also provides a response system, which includes a smart terminal and any of the above-mentioned smart wearable devices; the smart terminal is equipped with a teacher model, and the teacher model obtains a lightweight inference model loaded onto the smart wearable device through knowledge distillation.

[0014] Optionally, the smart wearable device is used to send an inference request message to the smart terminal when the first confidence level of the first inference result is less than the first confidence level threshold and the second confidence level of the second inference result is less than the first confidence level threshold; the smart terminal is used to obtain the corresponding user dynamic information after receiving the inference request message, obtain the third inference result corresponding to the user dynamic information using the teacher model, and send the third inference result to the smart wearable device.

[0015] Optionally, the smart wearable device is also used to adjust the model parameters of each lightweight inference model of the smart wearable device.

[0016] In the aforementioned smart wearable device and response system, the main response component collects user dynamic information through a corresponding information acquisition unit. This enables the collection of at least one type of information emitted by the user, such as voice signals, gesture information, head movements, and / or hand movements, providing high flexibility in information acquisition. The system also employs a corresponding lightweight inference model to output the first inference result corresponding to the user dynamic information. Based on responding to the user's dynamic information needs, the system can improve inference speed, reduce storage and computing resource requirements, and lower the power consumption of the smart wearable device during the user demand response process.

[0017] Furthermore, the main response component can also obtain the quality evaluation parameters of the first inference result. If the evaluation parameters are within the corresponding quality reference range, it indicates that the reliability of the currently obtained first inference result is high. Then, the first inference result is used as the standard, and corresponding output and / or response are made based on the first inference result. At this time, the smart wearable device only uses one response component to infer the user dynamic information, which can further reduce the power consumption in the user demand response process. If the quality evaluation parameters of the first inference result exceed the corresponding quality reference range, the first response component and the second response component are used to infer the corresponding user dynamic information to obtain multiple inference results. The inference result with the better quality evaluation parameters is used as the final inference result, which can improve the performance of the inference process, thereby improving the accuracy and practicality of AI applications. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of a smart wearable device according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of a headset according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of smart glasses according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the response system structure according to an embodiment of this application. Detailed Implementation

[0023] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. In the absence of conflict, the following embodiments and their technical features can be combined with each other.

[0024] This application provides a smart wearable device, which may include head-mounted smart terminals such as smart headphones (e.g., TWS earphones) and smart glasses. This type of smart wearable device can load AI models to respond to various user needs.

[0025] refer to Figure 1 As shown, the smart wearable device includes a first response component and a second response component; the first and second response components can communicate with each other via Wi-Fi, Ultra Wide Band (UWB), Bluetooth, and / or cellular communication. Both the first and second response components are equipped with an information acquisition unit and a lightweight inference model. Optionally, the information acquisition unit can collect user dynamic information such as voice, gestures, and / or actions.

[0026] One of the first response component and the second response component serves as the main response component. The main response component collects user dynamic information through the corresponding information collection unit and outputs a first inference result corresponding to the user dynamic information using a corresponding lightweight inference model, in order to respond to the user's needs corresponding to the dynamic information. Optionally, the first inference result includes search results, data queries, question answers, and / or related data monitoring results output by the lightweight inference model in the main response component for the user dynamic information.

[0027] Optionally, the lightweight inference model includes a lightweight model obtained by knowledge distillation from the teacher model, which can also be called a student model. The teacher model includes a large AI model trained for application scenarios and / or specific user group needs related to smart wearable devices. Knowledge distillation can transfer knowledge from the teacher model to the student model, allowing the student model to maintain a small size while approximating the performance of the teacher model. This results in faster inference speed, reduced storage and computing resource requirements, and lower power consumption when the first and second response components use the student model to obtain the first inference result corresponding to the user's dynamic information.

[0028] Optionally, the information acquisition unit includes at least one of a microphone, a camera, and a motion sensor. The microphone is used to acquire the user's voice signal (or audio information), the camera is used to acquire the user's gesture information, and the motion sensor is used to sense the user's head movements and / or hand movements, etc. The information acquisition unit, including a microphone, camera, and / or motion sensor, enables the acquired user dynamic information to include at least one of voice signals, gesture information, head movements, and hand movements. This allows the smart wearable device to output corresponding model inference results for each type of user dynamic information, improving the convenience and flexibility of the information acquisition process and enhancing the functionality of the corresponding smart wearable device.

[0029] Optionally, one of the first response component and the second response component is a left-side component of the smart wearable device, and the other is a right-side component of the smart wearable device. For example, see reference... Figure 2As shown, one of the first response component and the second response component can be located on the left ear side of the headset, and the other can be located on the right ear side of the headset; for example, refer to Figure 3 As shown, one of the first response component and the second response component can be located on the left eye side of the AI ​​glasses, and the other can be located on the right eye side of the AI ​​glasses, etc.

[0030] In the aforementioned smart wearable device, the main response component collects user dynamic information through a corresponding information acquisition unit. This enables the collection of at least one type of information emitted by the user, such as voice signals, gesture information, head movements, and / or hand movements. This provides high flexibility in information collection. The device also uses a corresponding lightweight inference model to output the first inference result corresponding to the user's dynamic information. Based on responding to the user's dynamic information needs, this component can improve the inference speed, reduce the demand for storage and computing resources, and also reduce the power consumption of the smart wearable device during the user demand response process.

[0031] In some embodiments, the other of the first response component and the second response component is a secondary response component.

[0032] The main response component is also used to obtain the quality evaluation parameters of the first inference result, and send inference instruction information to the secondary response component when the evaluation parameters exceed the corresponding quality reference range.

[0033] The secondary response component is used to obtain user dynamic information collected by its information collection unit after receiving the inference instruction information, and output the second inference result corresponding to the user dynamic information using the corresponding lightweight inference model; the first inference result and the second inference result are used to determine the final inference result, specifically, the inference result with better quality evaluation parameters can be used as the final inference result to ensure the reliability of the obtained final inference result.

[0034] Optionally, the quality evaluation parameters include at least one of confidence level, wind noise parameter, signal-to-noise ratio, total harmonic distortion, and amplitude linearity. If the quality evaluation parameters include confidence level, the corresponding quality reference range can be greater than or equal to a second confidence threshold, which can be set to values ​​such as 0.95 or 0.98. If the confidence level is less than the second confidence threshold, it indicates that the evaluation parameter exceeds the corresponding quality reference range, and the confidence level and reliability of the first inference result are low. If the confidence level is greater than or equal to the second confidence threshold, it indicates that the evaluation parameter does not exceed the corresponding quality reference range, and the confidence level and reliability of the first inference result are high. If the quality evaluation parameters include wind noise parameters, the corresponding quality reference range can be less than or equal to the wind noise threshold, and the wind noise threshold can be set to a value of 1. If the wind noise parameter is greater than the wind noise threshold, it indicates that the evaluation parameter exceeds the corresponding quality reference range. In this case, the wind noise of the first inference result is relatively large, and the effective information content is relatively low. If the wind noise parameter is less than or equal to the wind noise threshold, it indicates that the evaluation parameter does not exceed the corresponding quality reference range. In this case, the wind noise of the first inference result is relatively small, the effective information content is relatively high, and the reliability is high. Optionally, the wind noise parameter can be characterized by the following features: low-frequency components of the audio signal (e.g., components below 500Hz, 1kHz, etc.); the proportion of low-frequency components in the audio signal, such as power proportion; low-order MFCC coefficients; energy ratio of the low-frequency band (e.g., 0-1kHz) to the full frequency band; autocorrelation characteristics, etc.

[0035] In this embodiment, when the quality evaluation parameter of the first inference result exceeds the corresponding quality reference range, i.e., the reliability of the first inference result is relatively low, the first response component and the second response component are used to infer the corresponding user dynamic information to obtain multiple inference results. The inference result with the better quality evaluation parameter is used as the final inference result, which can improve the performance of the inference process, thereby improving the accuracy and practicality of AI applications. In particular, when there is a large difference in the quality of the user dynamic information collected by the two response components, the inference performance is improved to a greater extent.

[0036] Optionally, the main response component can also obtain the quality evaluation parameters of the first inference result. If the evaluation parameters are within the corresponding quality reference range, it indicates that the reliability of the first inference result obtained is high. Then, the first inference result is used as the standard, and corresponding output and / or response are made according to the first inference result. At this time, the smart wearable device only uses one response component to infer the user's dynamic information, which can further reduce the power consumption in the user demand response process.

[0037] In some examples, the primary response component or the secondary response component is further configured to select the one with the better quality evaluation parameter from the first inference result and the second inference result as the final inference result. The final inference result may include an inference result with higher confidence, an inference result with lower wind noise parameter, or an inference result with higher signal-to-noise ratio, etc.

[0038] In some embodiments, the primary response component or the secondary response component is further configured to obtain a first total confidence level corresponding to multiple first inference results and a second total confidence level corresponding to multiple second inference results within a preset time period, and determine the response component corresponding to the higher confidence level (i.e., the larger total confidence level) between the first total confidence level and the second total confidence level as the current primary response component; that is, if the first total confidence level corresponding to the primary response component is larger, then the primary response component continues to be used as the primary response component; if the second total confidence level corresponding to the secondary response component is larger, then the secondary response component is determined as the new primary response component, and the original primary response component is switched to the current secondary response component.

[0039] The preset time period includes a period preceding the current moment, such as one second or one minute prior to the current moment. The first total confidence level can be determined by the first confidence levels corresponding to multiple first inference results within the preset time period; for example, the first total confidence level can be the average or median of multiple first confidence levels within the preset time period. The second total confidence level can be determined by the second confidence levels corresponding to multiple second inference results within the preset time period; for example, the second total confidence level can be the average or median of multiple second confidence levels within the preset time period.

[0040] In this embodiment, when the total confidence of multiple inference results of the secondary response component is greater than the total confidence of the primary response component, the secondary response component is switched to the primary response component, and the secondary response component is used to perform inference work first. On the one hand, the response component with better performance can be used to perform the corresponding inference work first, thereby improving the performance of inference; on the other hand, it can also avoid both response components performing inference work and reduce the wireless interaction between the two response components, thereby reducing the power consumption of the smart wearable device and improving the battery life of the smart wearable device.

[0041] In the above-mentioned smart wearable devices, the main response component collects user dynamic information through the corresponding information collection unit, enabling the collection of at least one type of information emitted by the user, such as voice signals, gesture information, head movements and / or hand movements. This provides high flexibility in information collection. The corresponding lightweight inference model is used to output the first inference result corresponding to the user dynamic information. Based on responding to the user's dynamic information needs, the inference speed can be improved, the storage and computing resource requirements can be reduced, and the power consumption of the smart wearable device in the process of responding to user needs can also be reduced. Furthermore, the main response component can also obtain the quality evaluation parameters of the first inference result. If the evaluation parameters are within the corresponding quality reference range, it indicates that the reliability of the currently obtained first inference result is high. In this case, the first inference result is used as the standard, and corresponding output and / or response are made based on the first inference result. At this time, the smart wearable device only uses one response component to infer the user dynamic information, which can further reduce the power consumption in the user demand response process. If the quality evaluation parameters of the first inference result exceed the corresponding quality reference range, the first response component and the second response component are used to infer the corresponding user dynamic information to obtain multiple inference results. The inference result with the better quality evaluation parameters is used as the final inference result, which can improve the performance of the inference process, thereby improving the accuracy and practicality of AI applications.

[0042] A second aspect of this application provides a response system, with reference to Figure 4 As shown, the response system includes a smart terminal and a smart wearable device as described in any of the above embodiments; the smart terminal is equipped with a teacher model, and the teacher model obtains a lightweight inference model loaded onto the smart wearable device through knowledge distillation.

[0043] The smart terminal may include a mobile phone, personal computer, and / or cloud server, etc., as a control terminal that communicates with the smart wearable device. Optionally, the communication connection between the smart wearable device and the smart terminal can be achieved first through a wireless connection method, and then through multiple communication connection methods; this is not limited here. For example, the smart wearable device can connect to a wireless access point via Wi-Fi, and the wireless access point can communicate with the smart terminal, thereby realizing the communication connection between the smart wearable device and the smart terminal.

[0044] Optionally, the teacher model includes a large AI model trained for application scenarios and / or specific user group needs corresponding to smart wearable devices. The teacher model may have the following characteristics: large and complex, with high accuracy, but high computational resource consumption, making it difficult to deploy in resource-constrained environments such as smart wearable devices. The lightweight inference model loaded onto the smart wearable device can also be called the student model. Knowledge distillation can transfer knowledge from the teacher model to the student model, allowing the student model to maintain a small size while approximating the performance of the teacher model. This results in faster inference speed, reduced storage and computational resource requirements, and lower power consumption when the first and second response components use the student model to obtain the first inference result corresponding to the user's dynamic information. The student model may have the following characteristics: small and lightweight, with low computational resource consumption and easy deployment, but relatively low accuracy.

[0045] In the aforementioned response system, the teacher model on the smart terminal undergoes knowledge distillation to obtain a lightweight inference model that is loaded onto the smart wearable device. This makes the student model much smaller than the teacher model, reducing the demand for storage and computing resources on the smart wearable device and improving its inference and response speed.

[0046] In some embodiments, the smart wearable device is configured to send an inference request message to the smart terminal when the first confidence level of the first inference result is less than a first confidence threshold and the second confidence level of the second inference result is less than the first confidence threshold. The first confidence threshold can be set according to the performance requirements of the smart wearable device, for example, it can be set to a value such as 0.96 or 0.98.

[0047] The smart terminal is used to obtain corresponding user dynamic information after receiving the inference request information, obtain the third inference result corresponding to the user dynamic information using the teacher model, and send the third inference result to the smart wearable device so that the smart wearable device can make corresponding outputs and / or responses based on the third inference result.

[0048] Optionally, the smart terminal is also used to obtain a third confidence level of the third inference result, and when the third confidence level is greater than or equal to the first confidence level threshold, it sends the third inference result to the smart wearable device. At this time, the smart wearable device responds to the user's needs based on the third inference result, which can ensure the accuracy of the response process.

[0049] Optionally, the smart wearable device is also used to adjust the model parameters of each lightweight inference model of the smart wearable device when the third confidence level is greater than or equal to the first confidence level threshold. For example, the smart wearable device can adjust the model parameters of each lightweight inference model according to the third inference result and / or the teacher model that generates the third inference result, so as to make the inference performance of each lightweight inference model in the smart wearable device closer to the inference performance of the teacher model, thereby improving the inference performance of the smart wearable device.

[0050] Optionally, the smart wearable device can adjust the model parameters such as the weights and biases of each connection layer, the parameters of the normalization layer, the parameters of the intermediate feature layer, and / or the parameters of the attention mechanism in each of its lightweight inference models, so that the inference performance of the adjusted lightweight inference model is closer to the inference performance of the teacher model.

[0051] The above-described response system includes the smart wearable device described in any of the above embodiments, and has all the beneficial effects of the smart wearable device described in any of the foregoing embodiments, which will not be repeated here.

[0052] Although this application has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and drawings. This application includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components, the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this specification shown herein.

[0053] That is, the above description is only an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made using the content of this application’s specification and drawings, such as the combination of technical features between different embodiments, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of this application.

[0054] Furthermore, it should be understood that in the description of this application, the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Additionally, for structural elements with the same or similar characteristics, this application may use the same or different reference numerals for identification. Moreover, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0055] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as more preferred or advantageous than other embodiments. This application has been provided above to enable any person skilled in the art to implement and use it. Various details have been set forth in the above description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other embodiments, well-known structures and processes will not be described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

Claims

1. A smart wearable device, characterized in that, The smart wearable device includes a first response component and a second response component; both the first response component and the second response component are equipped with an information acquisition unit and a lightweight inference model; One of the first response component and the second response component serves as the main response component. The main response component is used to collect user dynamic information through the corresponding information collection unit and output the first inference result corresponding to the user dynamic information using the corresponding lightweight inference model. The other of the first response component and the second response component is a secondary response component; the primary response component is further configured to obtain the quality evaluation parameters of the first inference result, and when the evaluation parameters exceed the corresponding quality reference range, send inference instruction information to the secondary response component; the secondary response component is configured to collect user dynamic information after receiving the inference instruction information, and output the second inference result corresponding to the user dynamic information using the corresponding lightweight inference model; the first inference result and the second inference result are used to determine the final inference result.

2. The smart wearable device according to claim 1, characterized in that, The quality evaluation parameters include at least one of confidence level, wind noise parameter, signal-to-noise ratio, total harmonic distortion, and amplitude linearity.

3. The smart wearable device according to claim 1, characterized in that, The primary response component or the secondary response component is further configured to select the one with the better quality evaluation parameter from the first inference result and the second inference result as the final inference result.

4. The smart wearable device according to claim 1, characterized in that, The main response component or the secondary response component is further configured to obtain a first total confidence level corresponding to multiple first inference results and a second total confidence level corresponding to multiple second inference results within a preset time period, and determine the response component corresponding to the better confidence level among the first total confidence level and the second total confidence level as the current main response component.

5. The smart wearable device according to claim 1, characterized in that, The lightweight reasoning model includes a lightweight model obtained by knowledge distillation of the teacher model.

6. The smart wearable device according to claim 1, characterized in that, The information acquisition unit includes at least one of a microphone, a camera, and a motion sensor.

7. The smart wearable device according to claim 1, characterized in that, One of the first response component and the second response component is the left side component of the smart wearable device, and the other is the right side component of the smart wearable device.

8. A response system, characterized in that, The response system includes a smart terminal and a smart wearable device according to any one of claims 1 to 7; the smart terminal is equipped with a teacher model, and the teacher model obtains a lightweight inference model loaded onto the smart wearable device through knowledge distillation.

9. The response system according to claim 8, characterized in that, The smart wearable device is used to send inference request information to the smart terminal when the first confidence level of the first inference result is less than the first confidence level threshold and the second confidence level of the second inference result is less than the first confidence level threshold. The smart terminal is used to obtain the corresponding user dynamic information after receiving the inference request information, use the teacher model to obtain the third inference result corresponding to the user dynamic information, and send the third inference result to the smart wearable device.

10. The response system according to claim 9, characterized in that, The smart wearable device is also used to adjust the model parameters of each lightweight inference model of the smart wearable device.

Images