An interaction method and device, a wearable device, and a storage medium

Abstract

The application provides an interaction method and device, wearable equipment and storage medium, and belongs to the technical field of intelligent interaction. The method comprises the following steps: acquiring IMU data and audio data of a user wearing wearable equipment, and aligning to obtain target multi-modal data; extracting an audio feature vector through a convolutional neural network, extracting a motion feature vector through a long short-term memory network, and fusing; mapping the fused features to generate an initial interaction intention probability vector; generating a behavior preference weight based on historical interaction data of the user, and calibrating the initial probability vector; determining a target interaction intention according to the calibrated probability vector and performing a corresponding interaction operation. The application provides an interaction method and device, wearable equipment and storage medium, which can provide more accurate and personalized human-computer interaction.

Description

Technical Field

[0001] This application belongs to the field of intelligent interaction technology, and more specifically, relates to an interaction method and device, wearable device, and storage medium. Background Technology

[0002] Current human-computer interaction methods for wearable devices generally employ a generic recognition logic, relying solely on a single type of sensor data to determine user interaction intent. This type of data struggles to comprehensively cover users' actual interaction scenarios and potential needs, and the recognition process fails to incorporate patterns in users' historical interaction behaviors. In practical applications, different users have different interaction habits, and even the same user's needs may vary at different times. However, existing methods output interaction intents using a uniform standard, failing to adjust intents based on individual user characteristics. This results in a low degree of matching between the recognized interaction intent and the user's actual needs, easily leading to misjudgments or response deviations, ultimately failing to provide more accurate and personalized human-computer interaction. Summary of the Invention

[0003] The purpose of this application is to provide an interaction method and device, wearable device, and storage medium to provide more accurate and personalized human-computer interaction.

[0004] A first aspect of this application provides an interaction method, including: Acquire multimodal data of the target user and align the multimodal data to obtain target multimodal data; the target user is a user wearing a wearable device, and the multimodal data includes IMU data and audio data; Feature extraction is performed on the target audio data using a convolutional neural network to obtain an audio feature vector. Feature extraction is performed on the target IMU data using a long short-term memory network to obtain a motion feature vector. The audio feature vector and the motion feature vector are then fused to obtain a multimodal fusion feature vector. The multimodal fusion feature vector is mapped to generate an interaction intent probability vector, which is used to characterize the initial probability distribution of multiple preset interaction intents. Obtain the target user's target behavior preference weights, which are generated based on the user's historical interaction data and are used to characterize the user's preference for each interaction intent in different application scenarios; based on the target behavior preference weights, calibrate the initial interaction intent probability vector to obtain the calibrated probability vector. Based on the calibrated probability vector, the target interaction intent is determined, and the interaction operation corresponding to the target interaction intent is executed.

[0005] A second aspect of this application provides an interactive device, including: The data acquisition module is used to acquire multimodal data of the target user and align the multimodal data to obtain target multimodal data; the target user is a user wearing a wearable device, and the multimodal data includes IMU data and audio data; The feature extraction module is used to extract features from the target audio data based on a convolutional neural network to obtain an audio feature vector, and to extract features from the target IMU data based on a long short-term memory network to obtain a motion feature vector; the audio feature vector and the motion feature vector are fused to obtain a multimodal fusion feature vector. The mapping module is used to map the multimodal fusion feature vector to generate an interaction intent probability vector, which is used to characterize the initial probability distribution of multiple preset interaction intents. The calibration module is used to obtain the target user's target behavior preference weights. The target behavior preference weights are generated based on the user's historical interaction data and are used to characterize the user's preference for each interaction intent in different application scenarios. Based on the target behavior preference weights, the interaction intent probability vector is calibrated to obtain the calibrated probability vector. The interaction module is used to determine the target interaction intent based on the calibrated probability vector and execute the interaction operation corresponding to the target interaction intent.

[0006] A third aspect of this application provides a wearable device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described interaction method.

[0007] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned interactive method.

[0008] The beneficial effects of the interaction method, device, wearable device, and storage medium provided in this application are as follows: By collecting and aligning multimodal data of IMU data and audio data, this application can simultaneously capture user motion state and speech semantic information, eliminating feature deviations caused by data spatiotemporal misalignment and laying the foundation for accurate recognition; by using convolutional neural networks and long short-term memory networks to extract and fuse audio and motion features differentially, it can compensate for the one-sidedness of single-modal data information and improve the comprehensiveness of feature expression; by combining the behavioral preference weights generated by user historical interaction data to calibrate the initial probability vector, it can make the intent distribution fit the user's personalized scenario selection tendency, avoiding the problem of general recognition being out of touch with user habits; finally, by determining the target intent and executing the operation through the calibrated probability vector, it can effectively solve the defects of low accuracy and poor adaptability of traditional single-modal interaction recognition, and achieve more accurate and personalized human-computer interaction. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating an interaction method provided in an embodiment of this application; Figure 2 This is a structural block diagram of an interactive device provided in an embodiment of this application; Figure 3 This is a schematic block diagram of a wearable device provided in an embodiment of this application. Detailed Implementation

[0011] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0013] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an interaction method provided in an embodiment of this application, which can be executed by a wearable device. The method may include: S101: Acquire the multimodal data of the target user and align the multimodal data to obtain the target multimodal data; the target user is a user wearing a wearable device, and the multimodal data includes IMU data and audio data.

[0014] In this embodiment, wearable devices refer to electronic devices that can be worn on the human body and have data acquisition, processing and interaction functions. In this embodiment, the wearable device can be smart glasses, which have built-in core components such as IMU sensors, microphone arrays, processors and memory, and support multimodal data acquisition, processing and interactive operation execution.

[0015] Multimodal data refers to various data types that can reflect user status or needs, collected by different types of sensors. Specifically, it can include IMU data and audio data. These two types of data characterize user interaction scenarios and needs from different dimensions. Among them, IMU data refers to time-series data collected by the Inertial Measurement Unit, which includes three-axis acceleration data (reflecting the linear acceleration of an object) and three-axis angular velocity data (reflecting the rotational angular velocity of an object), used to capture the user's motion state. Audio data refers to sound signal data collected by acoustic sensors such as microphones. In this embodiment, it can be collected by the array of four MEMS microphones built into the smart glasses.

[0016] Target multimodal data refers to multimodal data that has undergone alignment processing to ensure its spatiotemporal consistency meets the requirements of subsequent processing. This eliminates the spatiotemporal misalignment problem of the original multimodal data and improves data quality. Target multimodal data includes target audio data and target IMU data.

[0017] In this embodiment, the wearable device collects IMU data and audio data from the target user in real time through built-in sensor components. The IMU data captures the user's motion state, while the audio data captures the user's voice commands or ambient sounds. Because the two types of data use different acquisition devices and sampling frequencies, there is a time misalignment issue. Therefore, alignment processing is required. Through hardware timestamp synchronization and algorithm compensation, the differences in data acquisition time and transmission delay are eliminated, ultimately obtaining target multimodal data with strong time consistency. This provides a high-quality data foundation for subsequent feature extraction and intent recognition.

[0018] S102: Extract features from the target audio data using a convolutional neural network to obtain an audio feature vector; extract features from the target IMU data using a long short-term memory network to obtain a motion feature vector; fuse the audio feature vector and the motion feature vector to obtain a multimodal fusion feature vector.

[0019] In this embodiment, the audio feature vector refers to a high-dimensional vector extracted from the target audio data through a convolutional neural network that can represent the semantics and features of the audio, and is a feature representation of the audio data; the motion feature vector refers to a high-dimensional vector extracted from the IMU data through a long short-term memory network that can represent the user's motion state and trend, and is a feature representation of the IMU data.

[0020] For example, when a user is running while wearing smart glasses, issuing a "pause music" voice command triggers the feature extraction process. The target audio data is collected by the four MEMS microphone arrays built into the glasses. After Wiener filtering to remove running wind noise and footstep noise, it is converted into a Mel spectrogram (corresponding to a 2-second voice segment). The target IMU data includes the triaxial acceleration (mean 0.8g) and triaxial angular velocity (mean 120° / s) during running. After Kalman filtering to remove noise, it forms time-series data points (corresponding to 2 seconds of motion data).

[0021] The convolutional neural network uses a 3-layer convolutional + 2-layer pooling structure. After global average pooling and full-connected layer mapping, it generates a 128-dimensional audio feature vector, which accurately captures the speech spectrum features of "pausing music".

[0022] The Long Short-Term Memory (LSTM) network contains two hidden layers. It takes IMU time series data as input, captures the motion trend features during running through a gating mechanism, and outputs a 32-dimensional motion feature vector after dimensionality reduction by a fully connected layer, which represents the head movement pattern during running.

[0023] In this embodiment, the multimodal fusion feature vector refers to a high-dimensional vector obtained by fusing audio feature vectors and motion feature vectors. It simultaneously contains audio semantic features and motion scene features. To fully utilize the complementary information of the two types of data, this embodiment employs a weighted fusion method after transformation using fully connected layers to generate a 64-dimensional multimodal fusion feature vector, ensuring that both types of features are expressed within a unified feature space. In this embodiment, differentiated feature extraction networks are used to address the characteristics of different types of data in the target multimodal data: Convolutional Neural Networks excel at capturing frequency domain features in audio data, extracting audio feature vectors that characterize speech semantics from the target audio data through multi-layer convolution and activation operations; Long Short-Term Memory Networks excel at capturing dependencies in temporal data, effectively extracting temporal features reflecting motion trends from IMU data to form motion feature vectors. To fully utilize the complementary information of the two types of data, the audio feature vector and motion feature vector are fused to generate a multimodal fusion feature vector that simultaneously contains speech semantic information and motion scene information, enhancing the expressive power of the features.

[0024] S103: Map the multimodal fusion feature vector to generate an interaction intent probability vector, which is used to characterize the initial probability distribution of multiple preset interaction intents.

[0025] In this embodiment, the preset interaction intent refers to the categories of possible user interaction needs defined in advance in the wearable device, such as "play music", "pause broadcast", "check the weather", "navigation guidance", etc., which is the target category set for intent recognition.

[0026] In this embodiment, the mapping process refers to the process of converting a high-dimensional multimodal fusion feature vector into a vector that matches the preset interaction intent dimension through a deep learning network. This embodiment can be implemented by two fully connected layers. The first layer (256 neurons, ReLU activation) maps the 64-dimensional fusion feature vector into a 256-dimensional vector, and the second layer (128 neurons, Softmax activation) maps it into a 128-dimensional interaction intent probability vector. Each element in the vector corresponds to the initial probability of a type of interaction intent, and the sum of all elements is 1.

[0027] In this embodiment, the multimodal fusion feature vector contains core information about user interaction needs. Through mapping processing mechanisms such as fully connected layers, the high-dimensional fusion feature vector is mapped to the dimensional space corresponding to the preset interaction intent. During the mapping process, activation functions such as Softmax are used to convert the output into probability values, ultimately generating an interaction intent probability vector. Each element in this vector corresponds to the probability of occurrence of a type of preset interaction intent, and the sum of the probabilities of all elements is 1, intuitively presenting the probability distribution of each interaction intent in the initial state.

[0028] S104: Obtain the target user's target behavior preference weight. The target behavior preference weight is generated based on the user's historical interaction data and is used to characterize the user's preference for each interaction intent in different application scenarios. Based on the target behavior preference weight, the interaction intent probability vector is calibrated to obtain the calibrated probability vector.

[0029] In this embodiment, the target behavior preference weight refers to a weight vector generated based on the user's historical interaction data (referring to interaction-related data generated by the target user in the past use of wearable devices, including triggered interaction intentions, interaction scenarios, interaction results, and user feedback, etc.). Each element corresponds to a preference coefficient of a preset interaction intention, which is used to characterize the user's preference for each interaction intention in different application scenarios. In this embodiment, it can be generated based on the user's historical interaction data over the past 7 days, and is a 128-dimensional vector, with each element corresponding to a preference coefficient of a type of interaction intention, and the sum of all elements is 1.

[0030] In this embodiment, the application scenario refers to the environment or state in which the user is using the wearable device, such as a resting scenario (sitting, standing), a light exercise scenario (walking slowly, moving around in the office), or a vigorous exercise scenario (running, cycling), etc. The user's interaction needs differ in different scenarios.

[0031] The calibrated probability vector refers to the probability vector obtained by adjusting the initial interaction intent probability vector through the target behavior preference weights. It integrates data features and user preferences and can reflect the user's real interaction needs. The calibration method can be element-wise multiplication, which multiplies each element in the initial interaction intent probability vector with the corresponding element in the target behavior preference weights, and then normalizes the result to obtain the calibrated probability vector.

[0032] For example, when a user commutes while wearing smart glasses (in a light exercise scenario), the initial interaction intent probability vector is generated by multimodal fusion feature mapping, and the probabilities of the three core intents are: "navigation guidance" 0.35, "play music" 0.42, and "check the weather" 0.23.

[0033] The target behavior preference weights are generated based on the user's historical commuting scenario data over the past 7 days: "Navigation guidance" 0.6 (must-use for daily commuting), "Play music" 0.1 (rarely used during commuting), and "Check the weather" 0.3 (occasionally used).

[0034] The calibration process uses element-wise multiplication: "Navigation guidance" 0.35 × 0.6 = 0.21, "Play music" 0.42 × 0.1 = 0.042, "Check weather" 0.23 × 0.3 = 0.069; the results are normalized (total sum 0.321) to obtain the calibrated probability vectors: "Navigation guidance" 0.654, "Play music" 0.131, "Check weather" 0.215; after calibration, the probability of the frequently requested "Navigation guidance" is increased, which is more in line with real usage habits.

[0035] In this embodiment, the target behavior preference weight is a personalized parameter obtained based on user historical interaction data mining, reflecting the user's preference for various interaction intents in different application scenarios. This weight is then used in element-wise operations with the initial interaction intent probability vector, and the result is normalized to calibrate the initial probability distribution. The calibrated probability vector amplifies the probability of user-preferred intents and reduces the probability of non-preferred intents, making the intent distribution more closely match the user's personalized needs.

[0036] S105: Based on the calibrated probability vector, determine the target interaction intent and execute the interaction operation corresponding to the target interaction intent.

[0037] In this embodiment, the target interaction intent refers to the interaction intent with the highest probability value selected from the calibrated probability vector, which is the most likely interaction need of the user as determined by the wearable device; the interaction operation refers to the specific action performed by the wearable device in response to the target interaction intent, such as playing music, pausing broadcasts, displaying weather information, or providing voice feedback navigation instructions, which is the actual form of the intent recognition result.

[0038] In this embodiment, the calibrated probability vector fully integrates data features and user preferences. By filtering the element with the highest probability value in the vector, the user's most likely target interaction intent can be determined. Based on the determined target interaction intent, the wearable device invokes the corresponding hardware components (such as a speaker, display screen, or communication module) to perform the interaction operation. For example, when the target intent is "check the weather," the device obtains weather information through the network and feeds it back to the user through the display screen or speaker, completing a full interactive loop from data collection, processing, decision-making to execution.

[0039] As can be seen from the above, this embodiment, by collecting and aligning multimodal data of IMU data and audio data, can simultaneously capture user motion state and speech semantic information, eliminating feature bias caused by spatiotemporal data misalignment and laying the foundation for accurate recognition. It employs convolutional neural networks and long short-term memory networks to differentially extract and fuse audio and motion features, compensating for the one-sidedness of single-modal data information and improving the comprehensiveness of feature expression. By combining user historical interaction data to generate behavioral preference weights to calibrate the initial probability vector, the intent distribution aligns with the user's personalized scenario selection tendencies, avoiding the problem of general recognition being out of sync with user habits. Finally, the calibrated probability vector is used to determine the target intent and execute the operation, effectively solving the shortcomings of low accuracy and poor adaptability in traditional single-modal interaction recognition, achieving more accurate and personalized human-computer interaction, and improving the user experience.

[0040] In one embodiment of this application, the multimodal data packet contains a timestamp; Align the multimodal data to obtain the target multimodal data, including: Multimodal data with a timestamp difference less than a preset time is selected from the multimodal data and used as initial multimodal data. The timestamp difference is the difference between the timestamp of the IMU data and the timestamp of the audio data. The initial multimodal data includes initial IMU data and initial audio data. For each initial IMU data, a compensation coefficient is determined based on the timestamp error between the initial IMU data and the corresponding initial audio data; Based on the timestamp and compensation coefficient of the initial IMU data, the time compensation amount corresponding to the timestamp of the initial audio data is determined by the first formula. The first formula is: ; in, This represents the time compensation amount for the initial audio data timestamp at time t. This represents the initial compensation reference value for the initial audio data. Indicates the compensation coefficient. Indicates the compensation time window, express Initial IMU data at time 10:00 Represents the mapping coefficient between motion and time; The target multimodal data is obtained by aligning the timestamps of the initial audio data based on the time compensation amount.

[0041] In this embodiment, the timestamp is a globally synchronized clock marker generated by the sensor hub built into the wearable device. The IMU data and audio data are embedded with this timestamp in real time at the moment of acquisition to mark the precise moment of data acquisition. The timestamp difference refers to the absolute value of the difference between the timestamp of the IMU data and the timestamp of the audio data under the same physical event, which is used to measure the degree of initial spatiotemporal misalignment between the two types of data. The preset time can be calibrated based on a large amount of experimental data, such as retaining only multimodal data pairs with a timestamp difference of less than 1ms and removing severely misaligned data caused by sensor failure or transmission anomalies. The initial multimodal data is the effective data set obtained after being filtered by the preset time, which includes one-to-one corresponding initial IMU data and initial audio data. The timestamp error refers to the absolute value of the difference between the timestamp of the initial IMU data and the corresponding initial audio data.

[0042] In this embodiment, the compensation coefficient The value range is 0.8~1.2, and the adjustment rule is: when the timestamp error > 0.3ms, Increase by 0.1; when the timestamp error is <0.2ms, Reduce by 0.1; when the timestamp error is in the range of 0.2ms to 0.3ms, Keep the current value unchanged to ensure that the compensation coefficient matches the actual degree of misalignment.

[0043] In the first formula of this embodiment, the initial compensation reference value The value is 0.1ms, which can be pre-calibrated through hardware testing to compensate for the fixed latency caused by asynchronous transmission of audio data due to direct memory access. Compensation Time Window The window duration is fixed at 10ms, covering the instantaneous change cycle of head movement, ensuring that the dynamic deviation caused by the movement can be fully quantified. Initial IMU data at time 1 Specifically The mean of the absolute values ​​of the three-axis angular velocities in the initial IMU data at time twentieth moment directly reflects The intensity of the user's head movements at any given moment. The mapping coefficient between motion and time. Used to convert the physical quantities of motion obtained by integrating angular acceleration into compensation quantities in the time dimension.

[0044] This embodiment uses the trapezoidal integral method to... within the time period The integration operation is performed, with the integration step size matched to the IMU data sampling interval. The total integration result is obtained by accumulating the integration values ​​of each step size interval.

[0045] The calculated time compensation amount Add the original timestamp of the initial audio data to obtain the compensated audio data timestamp. At this point, the absolute value of the difference between the compensated audio data timestamp and the corresponding initial IMU data timestamp is less than 0.5ms. Alignment is completed and the target multimodal data is output.

[0046] In this embodiment, firstly, based on the high-precision timestamps generated by the sensor center, valid data pairs with timestamp differences less than 1ms are selected, and abnormal data is removed to ensure data quality. Then, the timestamp error of the valid data pairs is calculated, and the compensation coefficient α is dynamically adjusted based on the error to adapt the compensation coefficient to the actual degree of misalignment. Next, the intensity of the user's head movement is quantified using the angular velocity value in the IMU data, and the cumulative motion amount is obtained through integration. Combined with the initial compensation reference value, compensation coefficient, and mapping coefficient, the total time compensation amount that simultaneously cancels fixed delay and dynamic motion deviation is calculated using the first formula. Finally, the time compensation amount is superimposed on the timestamp of the initial audio data to achieve spatiotemporal alignment between the IMU data and the audio data, and the target multimodal data is output.

[0047] As can be seen from the above, this embodiment effectively eliminates severely misaligned abnormal data by filtering through timestamp differences, thereby improving the quality of data participating in alignment. The compensation coefficient is dynamically adjusted based on the actual timestamp error, and combined with the dynamic motion deviation quantized by angular acceleration integral quantization, so that the time compensation amount can adapt to changes in the user's head motion state, and the alignment accuracy of multimodal data is greatly improved.

[0048] In one embodiment of this application, before extracting features from the target audio data based on a convolutional neural network to obtain an audio feature vector, the method further includes: The target user's motion acceleration variance is determined based on the target multimodal data, and the application scenario corresponding to the wearable device is determined based on the motion acceleration variance; the application scenarios include resting scenarios, light exercise scenarios, and vigorous exercise scenarios; Determine the target user's head rotation speed threshold and audio energy threshold in this application scenario; If the target user's current head rotation speed is greater than the head rotation speed threshold, or the target user's current audio energy is greater than the audio energy threshold, then the convolutional neural network and long short-term memory network are activated.

[0049] In this embodiment, the motion acceleration variance refers to a statistic calculated based on IMU acceleration data in the target multimodal data. It can be calculated using a sliding window method with a window size of 500ms, updated every 100ms. The calculation method is the average of the sum of squares of the deviations of each dimension of the acceleration data, reflecting the intensity of the user's motion.

[0050] In this embodiment, the application scenario refers to the user's state category when using wearable devices, which is divided according to the variance of motion acceleration, including resting scenario, light exercise scenario, and vigorous exercise scenario; for example: when the variance of motion acceleration is <0.1g², it is determined to be a resting scenario (such as sitting or standing); when 0.1g²≤ the variance of motion acceleration≤0.5g², it is determined to be a light exercise scenario (such as walking slowly or moving around in the office); when the variance of motion acceleration is >0.5g², it is determined to be a vigorous exercise scenario (such as running or cycling).

[0051] The head rotation speed threshold is a critical value for head rotation speed set according to different application scenarios, used to determine whether the neural network model needs to be woken up; the audio energy threshold is a critical value for audio signal energy set according to different application scenarios, which, together with the head rotation speed threshold, enables the model to be woken up on demand.

[0052] The current head rotation speed can be calculated based on the angular velocity data in the IMU data, taking the vector magnitude of the three-axis angular velocity; the current audio energy can be obtained by performing short-time energy calculation on the audio data in the target multimodal data.

[0053] In this embodiment, the scene threshold can be set as follows: Resting scene: Head rotation speed threshold 16° / s, audio energy threshold 3.6Nf (Nf is the ambient noise floor, calculated from the first 100ms of audio data); Mild motion scenarios: head rotation speed threshold 20° / s, audio energy threshold 4Nf; Intense motion scenarios: head rotation speed threshold 35° / s, audio energy threshold 4.4Nf.

[0054] In this embodiment, the convolutional neural network and long short-term memory network are initially in a dormant state, with only the sensor and data preprocessing module running. The processor monitors the current head rotation speed and audio energy in real time. When either indicator exceeds the corresponding scene threshold, a wake-up signal is sent to the model, and the model is activated within a set time. If no signal meeting the threshold is detected for 2 consecutive seconds after activation, the model automatically goes into sleep mode.

[0055] In this embodiment, firstly, the motion acceleration variance is calculated based on IMU acceleration data, and the current application scenario is divided according to the size of the variance to achieve real-time scenario perception; then, the corresponding head rotation speed threshold and audio energy threshold are called according to the scenario. The more complex the scenario (the more intense the motion), the higher the threshold is set to avoid false triggering by motion noise or environmental noise; finally, the processor continuously monitors relevant indicators, wakes up the model to start feature extraction and intent recognition when the threshold is met, and puts the model into sleep mode when there is no effective signal to reduce ineffective power consumption.

[0056] As can be seen from the above, this embodiment achieves dynamic adaptation of trigger thresholds based on scene division of motion acceleration variance, avoiding the limitations of fixed thresholds in different scenarios; the model on-demand wake-up mechanism increases the proportion of processor sleep time; and the scenario-based threshold setting effectively reduces the missed trigger rate and false trigger rate, ensuring the timeliness and reliability of interaction and improving the user experience.

[0057] In one embodiment of this application, audio feature vectors and motion feature vectors are fused to obtain a multimodal fused feature vector, including: Based on the audio feature vector, and transformed through the first fully connected layer, the first feature representation is obtained; Based on the motion feature vector, a second feature representation is obtained by transforming it through a second fully connected layer; Based on the current application scenario of the wearable device, the dynamic fusion weight parameters are queried from the preset scenario and weight mapping table. The dynamic fusion weight parameters include audio feature weight coefficients and motion feature weight coefficients. A multimodal fusion feature vector is obtained by weighted fusion based on dynamic fusion weight parameters, the first feature representation, and the second feature representation.

[0058] In this embodiment, the first fully connected layer refers to a neural network layer specifically designed for linear transformation of audio feature vectors, which can map audio feature vectors to a unified feature space for easy subsequent weighted fusion; the first feature representation refers to the feature vector obtained after the audio feature vector is transformed by the first fully connected layer, which has the same dimension as the motion feature vector; the second fully connected layer refers to a neural network layer specifically designed for linear transformation of motion feature vectors, with a function corresponding to the first fully connected layer; the second feature representation refers to the feature vector obtained after the motion feature vector is transformed by the second fully connected layer, which has the same dimension as the first feature representation.

[0059] For example, the audio feature vector is a 128-dimensional frequency domain feature extracted from the target audio data by a convolutional neural network, covering core information such as speech tone and spectral distribution; the first fully connected layer has an input dimension of 128 and an output dimension of 64, uses ReLU as the activation function, and initializes the weights using Xavier uniform initialization, through linear transformation. ( It is the 64-dimensional first feature representation output by the first fully connected layer. This is the weight matrix. For bias terms, The first feature vector is the original 128-dimensional audio feature vector extracted by the convolutional neural network. This 128-dimensional audio feature vector is mapped to a 64-dimensional first feature representation, eliminating redundant information and unifying the feature space. The motion feature vector is a 32-dimensional temporal feature extracted from the target IMU data by the long short-term memory network, representing the motion pattern of slight head rotation. The second fully connected layer has an input dimension of 32 and an output dimension of 64, with ReLU activation and Xavier uniform initialization for weights, achieved through linear transformation. ( It is the 64-dimensional second feature representation output by the second fully connected layer. This is the weight matrix. For bias terms, It is the original 32-dimensional motion feature vector extracted by the Long Short-Term Memory Network, which is then converted into a 64-dimensional second feature representation.

[0060] In this embodiment, the scene-to-weight mapping table is a pre-stored lookup table recording different application scenes and their corresponding fusion weight parameters; the dynamic fusion weight parameters are obtained from the scene-to-weight mapping table, including audio feature weight coefficients and motion feature weight coefficients; for example: In a resting scene: the weighting coefficient for audio features is 0.7, and the weighting coefficient for motion features is 0.3. For light motion scenarios: audio feature weighting coefficient 0.5, motion feature weighting coefficient 0.5; In scenes involving intense motion: audio feature weighting coefficient 0.3, motion feature weighting coefficient 0.7.

[0061] Let the first feature be represented as (64-dimensional vector), audio feature weight coefficients are The second feature is represented as (64-dimensional vector), motion feature weight coefficients are Multimodal fusion feature vector The vector operation involves multiplying corresponding elements and then adding them together, ensuring that the weights of the two types of features are contributed to the fusion result according to the scene.

[0062] In this embodiment, firstly, the first and second fully connected layers map the audio feature vector and motion feature vector to a 64-dimensional unified feature space, respectively, eliminating the differences in dimension and distribution between the two types of features. Then, based on the current application scenario, the corresponding dynamic fusion weight parameters are queried from the mapping table. The weight allocation varies depending on the scenario: audio features contribute more in a resting scenario, motion features contribute more in a vigorous motion scenario, and the contributions of the two types of features are balanced in a light motion scenario. Finally, the transformed features are weighted and summed according to the weight coefficients to obtain a multimodal fusion feature vector, highlighting the feature information that is more effective for intent recognition in the current scenario.

[0063] As can be seen from the above, the feature transformation of the fully connected layer in this embodiment achieves spatial unification of audio features and motion features, providing a foundation for weighted fusion; the scene-adaptive dynamic weight allocation enables the fused features to adapt to the differences in feature importance in different scenes, and has better scene adaptability than the fixed weight fusion scheme.

[0064] In one embodiment of this application, determining the target interaction intent based on the calibrated probability vector includes: The highest probability value and its corresponding first candidate intent are determined from the calibrated probability vector, and the second highest probability value and its corresponding second candidate intent are determined. Calculate the probability difference between the highest probability value and the second highest probability value, and generate the intention judgment confidence level based on the probability difference; If the confidence level of intent discrimination is higher than the preset discrimination threshold, then the first candidate intent is determined as the target interaction intent; If the confidence level of the intention is not higher than the preset discrimination threshold, the signal quality assessment value of the multimodal data is obtained. The signal quality assessment value is used to characterize the reliability of the IMU data and audio data. The target interaction intent is determined based on the intent discrimination confidence level and the signal quality assessment value.

[0065] In this embodiment, the highest probability value refers to the element with the largest value in the calibrated probability vector, which corresponds to the most likely user interaction intent; the second highest probability value refers to the element with the second largest value in the calibrated probability vector.

[0066] The first candidate intent is the interaction intent corresponding to the highest probability value in the calibrated probability vector; the second candidate intent is the interaction intent corresponding to the second highest probability value in the calibrated probability vector.

[0067] The probability difference refers to the numerical difference between the highest probability value and the second highest probability value, used to measure the clarity of the intent recognition result; the intent discrimination confidence score is an index calculated based on the probability difference, used to reflect the reliability of the intent recognition result; in this embodiment, the intent discrimination confidence score can be calculated as follows: ( P1 is the highest probability value, and P2 is the second highest probability value. The value ranges from 0 to 1. The larger the value, the clearer the intention recognition result.

[0068] The preset discrimination threshold refers to the pre-set confidence threshold of intent discrimination. For example, the preset discrimination threshold can be set to 0.3, that is, when the confidence of intent discrimination is >0.3, the intent recognition result is considered clear and the target interaction intent can be directly determined.

[0069] The signal quality assessment value is an indicator that comprehensively evaluates the reliability of IMU data and audio data. Its value ranges from 0 to 1, and it is calculated as follows: (Q1 is the IMU data quality score, and Q2 is the audio data quality score), used to assist in determining the target interaction intent when the confidence level of intent discrimination is low.

[0070] The IMU data quality score Q1 can be calculated based on the acceleration variance stability and data integrity of the IMU data. When the acceleration variance coefficient of variation is <0.2 and the data integrity rate is >98%, Q1=1; otherwise, Q1=0.5+0.5×(1-coefficient of variation)×data integrity rate. The audio data quality score Q2 can be calculated based on the signal-to-noise ratio and signal energy stability of the audio data. When the signal-to-noise ratio is >10dB and the energy coefficient of variation is <0.3, Q2=1; otherwise, Q2=0.5+0.5×(signal-to-noise ratio / 10)×(1-energy coefficient of variation).

[0071] In this embodiment, firstly, a first candidate intent and a second candidate intent are selected from the calibrated probability vector to identify the candidate objects. Then, the intent discrimination confidence score is calculated to determine whether the intent recognition result is clear. If the confidence score is higher than a preset threshold, it indicates that the first candidate intent has a significant advantage and is directly identified as the target interaction intent. If the confidence score does not reach the threshold, it indicates that the intent recognition result is unclear. At this time, the signal quality evaluation value of multimodal data is obtained to comprehensively measure the clarity of the recognition result and the reliability of the data, so as to avoid misjudgment due to data quality problems and finally determine the target interaction intent.

[0072] As can be seen from the above, the introduction of intent discrimination confidence can effectively distinguish the clarity of intent recognition results and avoid blindly determining the target intent when the probability difference is small; the supplementary consideration of signal quality assessment value makes the intent determination process combined with data reliability, reducing misjudgments caused by data noise or missing data.

[0073] In one embodiment of this application, determining the target interaction intent based on intent discrimination confidence and signal quality assessment value includes: A comprehensive credibility score is obtained by fusing the intent discrimination confidence score and the signal quality assessment value. If the overall credibility score is not lower than the preset credibility threshold, the calibrated probability vector is enhanced based on the overall credibility score to obtain an enhanced probability distribution. The intent with the highest probability value is selected from the enhanced probability distribution as the target interaction intent. If the overall credibility score is lower than the preset credibility threshold, the first candidate intent and the second candidate intent are combined into a candidate intent set, the preset operation risk level corresponding to each intent in the candidate intent set is obtained, and the intent with the lowest risk level is selected as the target interaction intent.

[0074] In this embodiment, the comprehensive credibility score refers to the comprehensive index obtained by fusing the intent judgment confidence level and the signal quality assessment value, and the formula is as follows: (C represents the confidence level of intent discrimination, and Q represents the signal quality assessment value), with a value range of 0 to 1, comprehensively evaluating the reliability of intent recognition results.

[0075] The preset credibility threshold is a pre-defined critical value for the overall credibility score, used to classify the execution conditions of the intent determination strategy, and can be set based on experience.

[0076] Enhancement processing refers to the process of amplifying the differences between elements in the calibrated probability vector through mathematical operations, making the probability of the dominant intent more prominent; this embodiment can adopt an exponential enhancement strategy, assuming the calibrated probability vector is... If the overall credibility score is S, then the enhanced probability vector is... And then The enhanced probability distribution is obtained by normalization. The larger S is and the larger k is, the more obvious the enhancement effect, which can amplify the probability difference of the dominant intention.

[0077] The preset operation risk level is a risk level pre-assigned to each interaction intent, used to prioritize low-risk intents when the overall credibility score is low; in this embodiment, the preset operation risk level can be divided into three categories: low risk, medium risk, and high risk. Low risk: Only affects local device settings or information queries, with no risk of privacy leaks or property loss, such as "checking the weather" or "pausing music"; Medium risk: Involves displaying or modifying users' personal information, such as "checking emails" or "changing alarm clock"; High risk: Involves payment, uploading of private data, and changes to system settings, such as "mobile payment" and "uploading photos".

[0078] In this embodiment, firstly, the confidence score of intent discrimination and the signal quality assessment value are fused according to weights to obtain a comprehensive credibility score, which comprehensively evaluates the reliability of the intent recognition result. If the score is not lower than a preset threshold, it indicates that the data quality is acceptable. The differences between the intents in the probability vector are amplified by exponential enhancement processing to make the dominant intents more prominent, and then the intent with the highest probability is selected. If the score is lower than the preset threshold, it indicates that the reliability of the intent recognition result is low. At this time, interaction security is given priority. The first and second candidate intents are combined into a candidate intent set, the risk level of each intent is queried, and the intent with the lowest risk level is selected for execution to avoid the negative impact of high-risk operations.

[0079] As can be seen from the above, the calculation of the comprehensive credibility score achieves a comprehensive assessment of the clarity of intent recognition and data reliability; the hierarchical enhancement strategy improves the accuracy of intent recognition when the data quality is acceptable; the risk level priority selection strategy ensures interaction security in high uncertainty scenarios and avoids losses caused by misoperation. The overall solution improves the success rate of interaction in low confidence scenarios, while reducing the incidence of high-risk misoperation, thus balancing the continuity and security of interaction.

[0080] In one embodiment of this application, the method for determining the target behavior preference weight includes: A user behavior preference matrix is ​​constructed based on the target user's historical interaction data; the matrix elements of the user behavior preference matrix represent the historical frequency weights of the target user's selection of interaction intent in the corresponding application scenario. Based on the results of historical interaction operations, a correction factor is associated with each matrix element. The correction factor is calculated based on the success rate of the corresponding historical interaction operation results. Based on historical interaction data within a preset time period prior to the current moment, and by dynamically updating the matrix elements of the user behavior preference matrix using the second formula, the updated user behavior preference matrix is ​​obtained. The second formula is: ; in, This represents the matrix element updated at time t. Represents the matrix elements at time t-1. The forgetting factor represents the historical weight. This represents the normalized frequency of target users selecting interaction intent n in application scenario m within a preset time period. Indicates the correction factor. This represents a combined score of the success rate and satisfaction rating of the corresponding interactive operation within a preset time period. Based on the updated user behavior preference matrix, the row vector that matches the current application scenario of the target user is extracted, and the row vector is normalized to obtain the target behavior preference weight.

[0081] In this embodiment, the user behavior preference matrix is ​​a two-dimensional matrix constructed based on users' historical interaction data. The row dimension represents the application scenario, and the column dimension represents the interaction intent. For example, the row dimension represents 3 types of application scenarios (rest, light exercise, and vigorous exercise), and the column dimension represents 128 types of preset interaction intents. This indicates the degree of user preference for choosing interaction intent n in application scenario m.

[0082] The preset time period is a pre-defined time range used to limit the time interval of historical interaction data on which the update of the user behavior preference matrix is ​​based. In this embodiment, the preset time period can be set to 7 days, that is, the user behavior preference matrix is ​​updated based on the historical interaction data of the most recent 7 days, ensuring that the weights can track the user's recent behavioral habits.

[0083] Historical interaction results refer to the effects of actions performed by users in the past, including whether the action was successful or not, or information such as user satisfaction.

[0084] In this embodiment, the correction factor Success rate based on historical interaction results The calculated parameters are given by the following formula: ( =Number of successful executions / Total number of times this intent was selected), with a value ranging from 0.3 to 1, used to correct frequency statistics results and highlight interactive intents with better performance; the forgetting factor for historical weights. This is used to balance the influence of historical preferences and recent user behavior on the preference matrix. The closer it is to 1, the greater the influence of historical weights; The closer to 0, the greater the influence of recent data; the forgetting factor of historical weights. It is a fixed constant between 0 and 1, and its value can be obtained by calibrating based on measured data of users' historical interaction behavior; frequency statistics normalized value This refers to the normalized value of the number of times a user selects a specific intention in a certain scenario within a preset time period, reflecting the user's basic preferences; comprehensive score. This refers to an indicator obtained by combining the success rate of interactive operations with user satisfaction ratings, and the formula is: ( The user satisfaction score is set (positive feedback = 1, negative feedback = 0, no feedback = 0.5), with a value range of 0 to 1, and is used to correct the frequency statistics normalization value.

[0085] The updated user behavior preference matrix refers to the user behavior preference matrix obtained after updating through the second formula, which can dynamically reflect changes in user behavior habits.

[0086] The row vector refers to the vector corresponding to the current application scenario extracted from the updated user behavior preference matrix, which contains the preference coefficients of each interaction intent in the scenario; normalization refers to the process of performing mathematical transformation on the row vector so that the sum of all elements of the vector is 1. In this embodiment, the extracted row vector can be L1 normalized so that the sum of all elements of the row vector is 1, thereby obtaining the target behavior preference weight, ensuring that the weight can be directly used for the calibration of the initial probability vector.

[0087] In this embodiment, the target behavior preference weights are adapted in real time through the construction, correction, and dynamic updating of the user behavior preference matrix. First, based on the user's historical interaction data for the past 7 days, an initial preference matrix is ​​constructed according to the application scenario and interaction intent. The matrix elements are frequency statistics normalized values, reflecting the user's basic preferences. Then, combining the execution success rate of each interaction intent and the user satisfaction score, a correction factor is calculated to correct the frequency statistics results, highlighting the interaction intents with better performance. Next, the matrix elements are dynamically updated using a second formula, balancing the influence of historical preferences and recent behaviors through a forgetting factor, so that the matrix can be updated according to changes in user habits. Finally, based on the current application scenario, the corresponding row vectors are extracted from the updated matrix and normalized to obtain the target behavior preference weights, which are used to calibrate the initial interaction intent probability vector.

[0088] As can be seen from the above, the preset time period and forgetting factor settings in this embodiment enable dynamic tracking of user behavior preferences and can quickly adapt to changes in user habits; the introduction of correction factors and comprehensive scores makes the preference weights not only reflect the frequency of interaction but also take into account the interaction effect, thus improving the rationality of the weights; the matching of the target behavior preference weights with the current application scenario ensures that the calibrated probability vector is more in line with the user's needs in a specific scenario.

[0089] In one embodiment of this application, after fusing audio feature vectors and motion feature vectors to obtain a multimodal fused feature vector, and before mapping the multimodal fused feature vector to generate an interaction intent probability vector, the method further includes: performing robust enhancement processing on the multimodal fused feature vector to obtain an enhanced multimodal fused feature vector; the robust enhancement processing includes: calculating the self-attention weight distribution of the multimodal fused feature vector; determining a feature attenuation factor based on the signal-to-noise ratio of the audio data and the motion signal-to-noise ratio of the IMU data; suppressing the weights corresponding to low signal-to-noise ratio time segments in the self-attention weight distribution and enhancing the weights corresponding to high signal-to-noise ratio time segments according to the feature attenuation factor, generating adaptive attention weights; and performing weighted reconstruction on the multimodal fused feature vector based on the adaptive attention weights to obtain the enhanced multimodal fused feature vector; The process involves mapping the multimodal fusion feature vectors to generate an interaction intent probability vector, including: The enhanced multimodal fusion feature vector is mapped to generate an interaction intent probability vector.

[0090] In this embodiment, robustness enhancement processing refers to the process of adaptively adjusting the weights of different time segments in the fused features through an attention mechanism to suppress noisy features, strengthen effective features, and improve the anti-interference ability of the fused features.

[0091] The self-attention weight distribution can be calculated using a scaled dot product self-attention mechanism. Specifically, the 64-dimensional multimodal fusion feature vector is mapped to a query vector O (64-dimensional), a key vector K (64-dimensional), and a value vector V (64-dimensional), respectively, using the formula... Calculate the self-attention weights, where The key vector dimension is (64), and finally the weight distribution (8 weight values, with a total of 1) is obtained, which corresponds one-to-one with the time segments of the fused feature vector, representing the importance of each time segment feature.

[0092] The signal-to-noise ratio of audio data can be calculated for each time segment of the audio data, using the following formula: ,in This represents the effective power of the audio signal in that segment. The environmental noise power of this segment (calibrated by collecting audio data during periods without speech) is expressed in dB; SNR ≥ 15dB is considered high signal-to-noise ratio, SNR < 10dB is considered low signal-to-noise ratio, and 10dB ≤ SNR < 15dB is considered medium signal-to-noise ratio.

[0093] The motion signal-to-noise ratio (SNR) of IMU data can be calculated for each time segment of the IMU data. It represents the ratio of effective motion signal to jitter noise, and the formula is as follows: ,in This represents the variance of the effective motion signal in this segment of IMU data. The noise variance of the IMU data in a static state (pre-calibrated) is expressed in dB; SNR ≥ 20 dB indicates a high signal-to-noise ratio, SNR < 12 dB indicates a low signal-to-noise ratio, and 12 dB ≤ SNR < 20 dB indicates a medium signal-to-noise ratio.

[0094] The characteristic attenuation factor ranges from 0.3 to 1.2, and is calculated based on a weighted average of the audio data signal-to-noise ratio and the motion signal-to-noise ratio of the IMU data. The formula is as follows: ;in This is the corresponding coefficient for audio signal-to-noise ratio (1.2 for high signal-to-noise ratio, 1.0 for medium signal-to-noise ratio, and 0.3 for low signal-to-noise ratio). The corresponding coefficient for motion signal-to-noise ratio (1.2 for high signal-to-noise ratio, 1.0 for medium signal-to-noise ratio, and 0.3 for low signal-to-noise ratio) is used to adjust the weight intensity of segments with different signal-to-noise ratios.

[0095] The weights of each time segment in the self-attention weight distribution are multiplied by the corresponding feature decay factor λ to achieve weight adjustment: the weights of low signal-to-noise ratio segments are multiplied by a decay factor of 0.3 (suppression), the weights of high signal-to-noise ratio segments are multiplied by an enhancement factor of 1.2 (enhancement), and the weights of medium signal-to-noise ratio segments remain unchanged; after adjustment, the weights are renormalized to obtain adaptive attention weights.

[0096] The adaptive attention weights are summed element-wise with the feature sub-vectors of each time segment of the multimodal fusion feature vector, as shown in the formula: (in Let be the adaptive attention weights for the i-th time segment. (where N is the total number of time segments), resulting in a 64-dimensional enhanced multimodal fusion feature vector.

[0097] The network structure for subsequent mapping processing (two fully connected layers) is the same as above, except that the input is replaced by an enhanced multimodal fusion feature vector instead of the original multimodal fusion feature vector, ensuring that the effective features after noise interference is suppressed are used for generating the interaction intent probability vector.

[0098] In this embodiment, firstly, the importance of each time segment in the multimodal fusion feature vector is initially determined through a self-attention mechanism, resulting in a self-attention weight distribution. Then, the signal-to-noise ratio (SNR) of each time segment of the audio data and IMU data is evaluated, and a feature attenuation factor is calculated to quantify the noise interference level of different segments. Next, the self-attention weights are adaptively adjusted using the attenuation factor to suppress the noise feature weights of low SNR segments and enhance the effective feature weights of high SNR segments. Finally, an enhanced multimodal fusion feature vector is generated through weighted reconstruction to ensure that the subsequent mapping process uses effective features with stronger anti-interference capabilities.

[0099] As can be seen from the above, this embodiment achieves adaptive noise reduction and enhancement of fused features by combining self-attention mechanism and signal-to-noise ratio evaluation. Noise interference in low signal-to-noise ratio segments is effectively suppressed, while effective features in high signal-to-noise ratio segments are enhanced. The signal-to-noise ratio of multimodal fused features is improved in complex scenarios (such as noisy streets and violent movements), and the accuracy of interactive intent recognition is improved compared with the unenhanced solution. The robust enhancement processing does not require a large amount of additional computation, adapts to the low power consumption requirements of wearable devices, and is compatible with the original mapping processing flow, thus possessing good technical compatibility and feasibility for implementation.

[0100] Corresponding to one interaction method in the above embodiments, Figure 2 This is a structural block diagram of an interactive device provided according to an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 2The interactive device 20 includes: a data acquisition module 21, a feature extraction module 22, a mapping module 23, a calibration module 24, and an interaction module 25.

[0101] The data acquisition module 21 is used to acquire multimodal data of the target user and align the multimodal data to obtain target multimodal data; the target user is a user wearing a wearable device, and the multimodal data includes IMU data and audio data; Feature extraction module 22 is used to extract features from target audio data based on convolutional neural network to obtain audio feature vector, extract features from target IMU data based on long short-term memory network to obtain motion feature vector, and fuse audio feature vector and motion feature vector to obtain multimodal fusion feature vector; The mapping module 23 is used to map the multimodal fusion feature vector to generate an interaction intent probability vector, which is used to characterize the initial probability distribution of multiple preset interaction intents. The calibration module 24 is used to obtain the target user's target behavior preference weights. The target behavior preference weights are generated based on the user's historical interaction data and are used to characterize the user's preference for each interaction intent in different application scenarios. Based on the target behavior preference weights, the initial interaction intent probability vector is calibrated to obtain the calibrated probability vector. The interaction module 25 is used to determine the target interaction intent based on the calibrated probability vector and execute the interaction operation corresponding to the target interaction intent.

[0102] In one embodiment of this application, the multimodal data packet contains a timestamp; The data acquisition module 21 is specifically used for: Multimodal data with a timestamp difference less than a preset time is selected from the multimodal data and used as initial multimodal data. The timestamp difference is the difference between the timestamp of the IMU data and the timestamp of the audio data. The initial multimodal data includes initial IMU data and initial audio data. For each initial IMU data, a compensation coefficient is determined based on the timestamp error between the initial IMU data and the corresponding initial audio data; Based on the timestamp and compensation coefficient of the initial IMU data, the time compensation amount corresponding to the timestamp of the initial audio data is determined by the first formula. The first formula is: ; in, This represents the time compensation amount for the initial audio data timestamp at time t. This represents the initial compensation reference value for the initial audio data. Indicates the compensation coefficient. Indicates the compensation time window, express Initial IMU data at time 10:00 Represents the mapping coefficient between motion and time; The target multimodal data is obtained by aligning the timestamps of the initial audio data based on the time compensation amount.

[0103] In one embodiment of this application, an interactive device 20 further includes: a wake-up module; specifically used for: The target user's motion acceleration variance is determined based on the target multimodal data, and the application scenario corresponding to the wearable device is determined based on the motion acceleration variance; the application scenarios include resting scenarios, light exercise scenarios, and vigorous exercise scenarios; Determine the target user's head rotation speed threshold and audio energy threshold in this application scenario; If the target user's current head rotation speed is greater than the head rotation speed threshold, or the target user's current audio energy is greater than the audio energy threshold, then the convolutional neural network and long short-term memory network are activated.

[0104] In one embodiment of this application, the feature extraction module 22 is specifically used for: Based on the audio feature vector, and transformed through the first fully connected layer, the first feature representation is obtained; Based on the motion feature vector, a second feature representation is obtained by transforming it through a second fully connected layer; Based on the current application scenario of the wearable device, the dynamic fusion weight parameters are queried from the preset scenario and weight mapping table. The dynamic fusion weight parameters include audio feature weight coefficients and motion feature weight coefficients. A multimodal fusion feature vector is obtained by weighted fusion based on dynamic fusion weight parameters, the first feature representation, and the second feature representation.

[0105] In one embodiment of this application, the calibration module 24 is specifically used for: The highest probability value and its corresponding first candidate intent are determined from the calibrated probability vector, and the second highest probability value and its corresponding second candidate intent are determined. Calculate the probability difference between the highest probability value and the second highest probability value, and generate the intention judgment confidence level based on the probability difference; If the confidence level of intent discrimination is higher than the preset discrimination threshold, then the first candidate intent is determined as the target interaction intent; If the confidence level of the intention is not higher than the preset discrimination threshold, the signal quality assessment value of the multimodal data is obtained. The signal quality assessment value is used to characterize the reliability of the IMU data and audio data. The target interaction intent is determined based on the intent discrimination confidence level and the signal quality assessment value.

[0106] In one embodiment of this application, the calibration module 24 is further configured to: A comprehensive credibility score is obtained by fusing the intent discrimination confidence score and the signal quality assessment value. If the overall credibility score is not lower than the preset credibility threshold, the calibrated probability vector is enhanced based on the overall credibility score to obtain an enhanced probability distribution. The intent with the highest probability value is selected from the enhanced probability distribution as the target interaction intent. If the overall credibility score is lower than the preset credibility threshold, the first candidate intent and the second candidate intent are combined into a candidate intent set, the preset operation risk level corresponding to each intent in the candidate intent set is obtained, and the intent with the lowest risk level is selected as the target interaction intent.

[0107] In one embodiment of this application, the calibration module 24 is specifically used for: A user behavior preference matrix is ​​constructed based on the target user's historical interaction data; the matrix elements of the user behavior preference matrix represent the historical frequency weights of the target user's selection of interaction intent in the corresponding application scenario. Based on the results of historical interaction operations, a correction factor is associated with each matrix element; Based on historical interaction data within a preset time period prior to the current moment, and by dynamically updating the matrix elements of the user behavior preference matrix using the second formula, the updated user behavior preference matrix is ​​obtained. The second formula is: ; in, This represents the matrix element updated at time t. Represents the matrix elements at time t-1. The forgetting factor represents the historical weight. This represents the normalized frequency of target users selecting interaction intent n in application scenario m within a preset time period. Indicates the correction factor. This represents a combined score of the success rate and satisfaction rating of the corresponding interactive operation within a preset time period. Based on the updated user behavior preference matrix, the row vector that matches the current application scenario of the target user is extracted, and the row vector is normalized to obtain the target behavior preference weight.

[0108] See Figure 3 , Figure 3 This is a schematic block diagram of a wearable device provided in one embodiment of this application. Figure 3The wearable device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules in the aforementioned device embodiments, for example... Figure 2 The functions of the data acquisition module 21, feature extraction module 22, mapping module 23, calibration module 24, and interaction module 25 are shown.

[0109] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0110] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0111] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store information such as convolutional neural networks and long short-term memory networks.

[0112] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the interaction method provided in the embodiments of this application, or they can execute the implementation method of the wearable device described in the embodiments of this application, which will not be elaborated here.

[0113] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0114] The computer-readable storage medium can be an internal storage unit of the wearable device in any of the foregoing embodiments, such as a hard drive or memory of the wearable device. The computer-readable storage medium can also be an external storage device of the wearable device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the wearable device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the wearable device. The computer-readable storage medium is used to store computer programs and other programs and data required by the wearable device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0115] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0116] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the wearable device and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0117] In the several embodiments provided in this application, it should be understood that the disclosed wearable devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0119] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0120] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An interaction method, characterized in that, include: Acquire multimodal data of the target user and align the multimodal data to obtain target multimodal data; The target user is a user wearing a wearable device, and the multimodal data includes IMU data and audio data; Feature extraction is performed on the target audio data using a convolutional neural network to obtain an audio feature vector. Feature extraction is performed on the target IMU data using a long short-term memory network to obtain a motion feature vector. The audio feature vector and the motion feature vector are then fused to obtain a multimodal fusion feature vector. The multimodal fusion feature vector is mapped to generate an interaction intent probability vector, which is used to characterize the initial probability distribution of multiple preset interaction intents. Obtain the target user's target behavior preference weights, which are generated based on the user's historical interaction data and are used to characterize the user's preference for each interaction intent in different application scenarios; Based on the target behavior preference weights, the interaction intent probability vector is calibrated to obtain the calibrated probability vector; Based on the calibrated probability vector, the target interaction intent is determined, and the interaction operation corresponding to the target interaction intent is executed.

2. The interaction method as described in claim 1, characterized in that, The multimodal data packet contains a timestamp; The step of aligning the multimodal data to obtain the target multimodal data includes: Multimodal data with a timestamp difference less than a preset time is selected from the multimodal data and used as initial multimodal data. The timestamp difference is the difference between the timestamp of the IMU data and the timestamp of the audio data. The initial multimodal data includes initial IMU data and initial audio data. For each initial IMU data, a compensation coefficient is determined based on the timestamp error between the initial IMU data and the corresponding initial audio data; Based on the timestamp of the initial IMU data and the compensation coefficient, the time compensation amount corresponding to the timestamp of the initial audio data is determined by the first formula. The first formula is: ； in, This represents the time compensation amount for the initial audio data timestamp at time t. This represents the initial compensation reference value for the initial audio data. Indicates the compensation coefficient. Indicates the compensation time window, express Initial IMU data at time 10:00 Represents the mapping coefficient between motion and time; The timestamps of the initial audio data are aligned based on the time compensation amount to obtain the target multimodal data.

3. The interaction method as described in claim 1, characterized in that, Before performing feature extraction on the target audio data based on a convolutional neural network to obtain the audio feature vector, the method further includes: Based on the target multimodal data, the variance of the target user's motion acceleration is determined, and based on the variance of the motion acceleration, the application scenario corresponding to the wearable device is determined; the application scenario includes resting scenario, light exercise scenario, and vigorous exercise scenario; Determine the head rotation speed threshold and audio energy threshold of the target user in this application scenario; If the target user's current head rotation speed is greater than the head rotation speed threshold, or the target user's current audio energy is greater than the audio energy threshold, then the convolutional neural network and the long short-term memory network are activated.

4. The interaction method as described in claim 3, characterized in that, The process of fusing the audio feature vector and the motion feature vector to obtain a multimodal fused feature vector includes: Based on the audio feature vector, and after transformation through the first fully connected layer, a first feature representation is obtained; Based on the motion feature vector, and after transformation through the second fully connected layer, a second feature representation is obtained; Based on the current application scenario of the wearable device, the dynamic fusion weight parameters are queried from the preset scenario-weight mapping table. The dynamic fusion weight parameters include audio feature weight coefficients and motion feature weight coefficients. The multimodal fusion feature vector is obtained by weighted fusion based on the dynamic fusion weight parameters, the first feature representation, and the second feature representation.

5. The interaction method as described in claim 1, characterized in that, Determining the target interaction intent based on the calibrated probability vector includes: The highest probability value and its corresponding first candidate intent are determined from the calibrated probability vector, and the second highest probability value and its corresponding second candidate intent are determined. Calculate the probability difference between the highest probability value and the second highest probability value, and generate an intent discrimination confidence level based on the probability difference; If the confidence level of the intent determination is higher than the preset determination threshold, then the first candidate intent is determined as the target interaction intent; If the confidence level of the intent determination is not higher than the preset determination threshold, then the signal quality assessment value of the multimodal data is obtained, and the signal quality assessment value is used to characterize the reliability of the IMU data and the audio data; The target interaction intent is determined based on the intent discrimination confidence level and the signal quality evaluation value.

6. The interaction method as described in claim 5, characterized in that, Determining the target interaction intent based on the intent discrimination confidence level and the signal quality assessment value includes: A comprehensive credibility score is obtained by fusing the intent discrimination confidence score with the signal quality assessment value. If the overall credibility score is not lower than the preset credibility threshold, then the calibrated probability vector is enhanced based on the overall credibility score to obtain an enhanced probability distribution, and the intent with the highest probability value is selected from the enhanced probability distribution as the target interaction intent. If the overall credibility score is lower than the preset credibility threshold, the first candidate intent and the second candidate intent are combined into a candidate intent set, the preset operation risk level corresponding to each intent in the candidate intent set is obtained, and the intent with the lowest risk level is selected as the target interaction intent.

7. The interaction method as described in claim 1, characterized in that, The method for determining the target behavior preference weights includes: A user behavior preference matrix is ​​constructed based on the target user's historical interaction data; the matrix elements of the user behavior preference matrix represent the historical frequency weights of the target user's selection of interaction intentions in the corresponding application scenarios. Based on the results of historical interaction operations, a correction factor is associated with each matrix element; Based on historical interaction data within a preset time period prior to the current moment, and by dynamically updating the matrix elements of the user behavior preference matrix using the second formula, the updated user behavior preference matrix is ​​obtained. The second formula is: ； in, This represents the matrix element updated at time t. Represents the matrix elements at time t-1. The forgetting factor represents the historical weight. This represents the normalized frequency of target users selecting interaction intent n in application scenario m within the preset time period. Indicates the correction factor. This represents a comprehensive score that combines the success rate and satisfaction rating of the corresponding interactive operation within the preset time period. Based on the updated user behavior preference matrix, a row vector matching the current application scenario of the target user is extracted, and the row vector is normalized to obtain the target behavior preference weight.

8. An interactive device, characterized in that, include: The data acquisition module is used to acquire multimodal data of the target user and align the multimodal data to obtain target multimodal data; the target user is a user wearing a wearable device, and the multimodal data includes IMU data and audio data; The feature extraction module is used to extract features from the target audio data based on a convolutional neural network to obtain an audio feature vector, and to extract features from the target IMU data based on a long short-term memory network to obtain a motion feature vector; the audio feature vector and the motion feature vector are fused to obtain a multimodal fusion feature vector. The mapping module is used to map the multimodal fusion feature vector to generate an interaction intent probability vector, which is used to characterize the initial probability distribution of multiple preset interaction intents. The calibration module is used to obtain the target user's target behavior preference weights, which are generated based on the user's historical interaction data and are used to characterize the user's preference for each interaction intent in different application scenarios. Based on the target behavior preference weights, the interaction intent probability vector is calibrated to obtain the calibrated probability vector; The interaction module is used to determine the target interaction intent based on the calibrated probability vector and execute the interaction operation corresponding to the target interaction intent.

9. A wearable device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images