Multimodal voice interaction method, electronic device and storage medium for AI headset

Abstract

The present invention provides a multimodal voice interaction method, electronic device and storage medium for an AI headset, including: collecting a user's voice signal, identifying the starting and ending points in the voice signal based on a hidden Markov model and the probability distribution characteristics of the voice signal, and intercepting available voice segments; performing voice recognition on the voice segments to obtain corresponding text information; collecting the motion data of the user's head based on the motion sensor built into the AI ​​headset; inputting the text information and the motion data into a fusion model to obtain interaction intention features by fusion; parsing the interaction intention features to obtain the interaction intention of the user; and driving the AI ​​headset to execute corresponding functions based on the interaction intention. In the present invention, the starting and ending points in the voice signal can be accurately identified; at the same time, the defect that the single-modal interaction method cannot accurately parse user needs is overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing, and in particular to a multimodal speech interaction method of an AI headset, an electronic device, and a storage medium. Background Art

[0002] Smart wearable devices, especially AI headsets, have been widely integrated into people's daily lives. As users' expectations for their interactive experience continue to rise, traditional single-modal voice interaction methods are increasingly exposed to many limitations.

[0003] On the one hand, in complex real-world usage scenarios, environmental noise is everywhere. Whether it is the hustle and bustle of traffic on busy streets, noisy voices in public places, or the sound of electrical appliances running at home, they all seriously interfere with the collection and recognition of voice signals. In addition, when detecting endpoints, they often rely on simple methods such as energy thresholds, which makes it difficult to accurately determine the true start and end of speech, resulting in inaccurate interception of effective speech segments.

[0004] On the other hand, it is difficult to fully and accurately grasp the user's interaction intentions by relying solely on text information obtained through voice recognition. For example, when a user is in a sports scene and cannot operate the headphones with both hands, they can only use voice commands such as "play music" but cannot clearly tell the specific music type and playlist.

[0005] Therefore, the single-modal interaction method cannot accurately analyze user needs, which ultimately affects the effective execution of the AI ​​headset functions and makes it difficult to provide users with smooth and efficient interactive services. Summary of the invention

[0006] The main purpose of the present invention is to provide a multimodal voice interaction method, electronic device and storage medium for AI headsets, aiming to overcome the defect that it is difficult to accurately determine the start and end of voice, and to overcome the defect that a single-modal interaction method cannot accurately analyze user needs.

[0007] To achieve the above object, the present invention provides a multimodal voice interaction method of an AI headset, comprising the following steps:

[0008] Collect the user's voice signal, identify the start and end points of the voice signal based on the hidden Markov model and the probability distribution characteristics of the voice signal, and extract the usable voice segment;

[0009] Performing speech recognition on the speech segment to obtain corresponding text information;

[0010] Based on the built-in motion sensor of the AI ​​headset, the user's head motion data is collected;

[0011] The text information and the motion data are input into a fusion model to obtain interaction intention features through fusion; the fusion model is based on a multimodal attention fusion strategy, intelligently determines the importance of the text information and the motion data according to the user's current interaction scenario, adaptively allocates the weights when the two are fused, and performs deep fusion;

[0012] The interaction intention feature is analyzed to obtain the interaction intention of the user; based on the interaction intention, the AI ​​headset is driven to execute a corresponding function.

[0013] Furthermore, based on the hidden Markov model, the start and end points of the speech signal are identified in combination with the probability distribution characteristics of the speech signal, including:

[0014] Performing pre-emphasis processing on the speech signal to obtain a standard speech frame sequence;

[0015] Based on fast Fourier transform, the standard speech frame sequence is converted from the time domain to the frequency domain to obtain a speech signal representation in the frequency domain, and the power spectrum density is calculated to obtain a power spectrum density sequence;

[0016] The power spectral density sequence is input as an observation sequence into a pre-trained hidden Markov model; after the pre-training of the hidden Markov model, the Viterbi algorithm is used to quickly search for the optimal state path in the state network of the hidden Markov model, and the start and end points in the speech signal are identified in combination with the preset speech start and end feature markers in the hidden Markov model.

[0017] Further, the speech signal is pre-emphasized to obtain a standard speech frame sequence, including:

[0018] Filtering the speech signal through a first-order high-pass filter to obtain a filtered speech signal;

[0019] The filtered speech signal is divided into a plurality of short-time frames according to a fixed time length, and each of the short-time frames is windowed based on a Hamming window function to obtain a standard speech frame sequence.

[0020] Furthermore, the text information and the motion data are input into a fusion model to obtain interaction intention features by fusion, including:

[0021] Input the text information and the motion data into a fusion model, and obtain the real-time situational elements of the AI ​​headset based on the situational insight submodule embedded in the front end of the fusion model;

[0022] Based on a pre-built context-modality association knowledge base, determine the importance ratio of the text information and the motion data in the current context;

[0023] Importing the text information and motion data into a processing layer based on a multimodal attention fusion architecture, focusing on the semantic focus of the text and the key nodes of the motion data simultaneously, and calculating the multimodal interaction weight;

[0024] Based on the importance ratio and the multimodal interaction weight, a weight when the text information and the motion data are fused is calculated;

[0025] The text information and motion data are fused through a deep neural network based on a skip connection structure to obtain interaction intention features.

[0026] Furthermore, the real-time situational elements include currently activated functional applications, network connection status, ambient noise decibels, and light illumination.

[0027] Furthermore, after driving the AI ​​headset to execute the corresponding function, the method further includes:

[0028] Establishing a mapping relationship between the text information, the motion data and the interaction intention;

[0029] Based on the mapping relationship, generate a wake-up word;

[0030] Configure the correspondence between the wake-up word and the interaction intention so that when the wake-up word is recognized, the AI ​​headset is driven again to execute the function corresponding to the interaction intention.

[0031] Furthermore, after generating a wake-up word based on the mapping relationship, the method further includes:

[0032] Encoding the text information, the motion data and the interaction intention to generate coded characters;

[0033] Acquire characteristics of multiple dimensions of the wake-up word, and generate a graph based on the characteristics of each dimension;

[0034] Setting a plurality of key points on each graph, and sequentially adding each character in the coded character to each key point to obtain a key point graph;

[0035] Based on the positional relationship between each of the key point graphics, a joint agreed key is generated, the mapping relationship is encrypted, and transmitted to the management server.

[0036] Further, based on the positional relationship of each of the key point graphics, a joint agreed key is generated, including:

[0037] Arranging each of the key point graphics side by side in a preset layer, rotating each of the key point graphics according to a preset rule so that the characters located at the top key points in each of the key point graphics are characters of the same type, and obtaining a rotated graphic;

[0038] All characters located at the bottom key points in each rotating figure are obtained, and are combined in sequence, and the obtained character combination is used as the joint agreed key.

[0039] The present invention also provides a multimodal voice interaction device for an AI headset, comprising:

[0040] The interception unit is used to collect the user's voice signal, identify the start and end points in the voice signal based on the hidden Markov model and the probability distribution characteristics of the voice signal, and intercept the usable voice segment;

[0041] A recognition unit, used to perform speech recognition on the speech segment to obtain corresponding text information;

[0042] The acquisition unit is used to collect the motion data of the user's head based on the motion sensor built into the AI ​​headset;

[0043] A fusion unit is used to input the text information and the motion data into a fusion model to obtain interaction intention features by fusion; the fusion model is based on a multimodal attention fusion strategy, intelligently determines the importance of text information and motion data according to the user's current interaction scenario, adaptively allocates weights when the two are fused, and performs deep fusion;

[0044] The interaction unit is used to analyze the interaction intention feature to obtain the interaction intention of the user; based on the interaction intention, drive the AI ​​headset to execute the corresponding function.

[0045] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0046] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0047] The multimodal voice interaction method, electronic device and storage medium of the AI ​​headset provided by the present invention include: collecting the user's voice signal, based on the hidden Markov model, combining the probability distribution characteristics of the voice signal to identify the starting and ending points in the voice signal, and intercepting the available voice segment; performing voice recognition on the voice segment to obtain the corresponding text information; based on the built-in motion sensor of the AI ​​headset, collecting the motion data of the user's head; inputting the text information and the motion data into the fusion model, and fusing to obtain the interaction intention feature; the fusion model is based on the multimodal attention fusion strategy, and intelligently judges the importance of text information and motion data according to the user's current interaction scene, adaptively assigns the weight when the two are fused, and performs deep fusion; analyzes the interaction intention feature to obtain the user's interaction intention; based on the interaction intention, drives the AI ​​headset to perform the corresponding function. In the present invention, based on the hidden Markov model, combined with the probability distribution characteristics of the voice signal, the starting and ending points in the voice signal can be accurately identified; at the same time, combined with the multimodal interaction mode of voice recognition and motion data, the user's interaction intention is fused and analyzed, overcoming the defect that the single-modal interaction mode cannot accurately analyze user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 1 is a schematic diagram of the steps of a multimodal voice interaction method of an AI headset in one embodiment of the present invention;

[0049] Figure 2 is a structural block diagram of a multimodal voice interaction device of an AI headset in one embodiment of the present invention;

[0050] Figure 3 It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present invention.

[0051] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] Reference Figure 1 In one embodiment of the present invention, a multimodal voice interaction method of an AI headset is provided, comprising the following steps:

[0054] Step S1, collecting the user's voice signal, identifying the start and end points of the voice signal based on the hidden Markov model and the probability distribution characteristics of the voice signal, and extracting the usable voice segment;

[0055] Step S2, performing speech recognition on the speech segment to obtain corresponding text information;

[0056] Step S3, collecting the motion data of the user's head based on the motion sensor built into the AI ​​headset;

[0057] Step S4, inputting the text information and the motion data into a fusion model, and fusing them to obtain interaction intention features; the fusion model is based on a multimodal attention fusion strategy, intelligently determines the importance of the text information and the motion data according to the user's current interaction scenario, adaptively allocates the weight when the two are fused, and performs deep fusion;

[0058] Step S5, analyzing the interaction intention feature to obtain the user's interaction intention; based on the interaction intention, driving the AI ​​headset to execute a corresponding function.

[0059] In this embodiment, as described in step S1 above, the AI ​​headset has a built-in high-sensitivity microphone that can capture the sound emitted by the user in all directions. After the voice interaction function is turned on, the microphone converts the acoustic signals in the surrounding environment into electrical signals in real time as the original voice data for subsequent processing. This process needs to deal with various complex environmental noises, such as outdoor traffic noise, indoor electrical appliance hum, etc., so the noise reduction performance of the microphone and the ability to capture sound details are crucial.

[0060] The above hidden Markov model is a dynamic model based on probability statistics, which is suitable for processing time-varying sequence signals such as speech. The generation of speech signals has certain randomness and regularity, and there is a transition probability between different utterance states (such as vowels, consonants, silence, etc.). Combined with the probability distribution characteristics of speech signals, the collected speech signals are first preprocessed, such as pre-emphasis to enhance high-frequency components, frame processing to divide continuous speech into small segments for analysis, and then fast Fourier transform to obtain frequency domain information to calculate power spectrum density. These processed signal features are input into the hidden Markov model as observation sequences. The model uses the Viterbi algorithm to search for the optimal path in the complex state network based on the pre-trained state transition probability matrix, observation probability matrix and initial state probability distribution, and accurately locates the starting and ending points of the speech, thereby intercepting pure and usable speech segments, laying the foundation for subsequent accurate speech recognition.

[0061] As described in step S2 above, for the intercepted speech segment, a speech feature extraction algorithm is applied, such as the common Mel-frequency cepstral coefficient (MFCC) algorithm. This algorithm simulates the human ear's perception of sound frequency and converts the speech signal into a set of vectors that can characterize its key features. The speech segment is subjected to a short-time Fourier transform to obtain spectral information, and then the spectrum is filtered by combining the Mel-frequency filter group, and a series of operations such as logarithm and discrete cosine transform are performed to finally obtain the MFCC feature vector. Compared with the original speech signal, these feature vectors have reduced dimensions and retain the key semantic features of the speech, which is convenient for subsequent model processing.

[0062] The above-mentioned speech recognition model using a deep neural network architecture, such as a model based on a long short-term memory network (LSTM) or a convolutional neural network (CNN) combined with LSTM. The model is trained with massive speech data in advance. These data cover a variety of factors such as different ages, genders, regional accents, speaking speeds, and language styles, giving the model a strong generalization ability. After the extracted MFCC feature vector is input into the model, the model, based on the mapping relationship between the learned speech features and the text, calculates and transmits layers of neurons, and finally outputs the corresponding text information to achieve the conversion from speech to text.

[0063] As described in step S3 above, AI headphones usually have built-in accelerometers and gyroscopes, which work together to sense the movement of the user's head. The accelerometer can measure the acceleration changes of the head in the three coordinate axes, reflecting the translational movement of the head, such as nodding, shaking the head, etc.; the gyroscope is used to measure the angular velocity of the head, accurately capturing the direction and speed of the head rotation, such as turning the head to check the surrounding environment. The above sensors continuously collect data at a certain sampling frequency, converting physical movement into electrical signal data, and providing key auxiliary information for multimodal interaction.

[0064] The collected raw motion data will inevitably be disturbed by factors such as sensor noise and slight unintentional shaking of the user. Therefore, it is necessary to filter and denoise it, such as using the Kalman filter algorithm, which is based on the principle of combining the prediction of motion state with measurement update, and can effectively remove noise and restore a more realistic head motion trajectory. At the same time, in order to facilitate the subsequent fusion with text information, the data needs to be normalized to map data of different dimensions and magnitudes to a unified numerical range. The commonly used method is Min-Max normalization to ensure the compatibility of the data in the fusion model.

[0065] As in step S4 above: before entering the fusion model, the text information needs to be cleaned and optimized to remove redundant words and correct spelling errors. Keywords and key phrases can also be annotated based on semantic analysis to enhance the clarity of the text. The motion data is further filtered by dynamic thresholds to remove noise glitches caused by short-term sensor anomalies to ensure data quality.

[0066] The core of the above fusion model lies in the multimodal attention mechanism. When the user initiates an interaction, the model monitors the current interaction scene in real time, including but not limited to the applications that the AI ​​headset is running (such as music playback, navigation, calls, etc.), the environmental information (noise level, light brightness, geographic location, etc.) and the user's recent interaction history. Based on these scene elements, the model intelligently judges the relative importance of text information and motion data in the current situation. For example, when using a navigation application in a noisy environment, if the motion data shows that the head turns frequently, the model will tend to give the motion data a higher weight, because this may mean that the user is checking the surrounding road conditions through head movements; while in a quiet environment, voice search, text information may occupy a dominant weight. Through this adaptive weight distribution, the model uses the attention mechanism to focus on key information, deeply fuses text features and motion features, and extracts interaction intention features that can accurately reflect the user's interaction intentions, avoiding information loss or misjudgment caused by simple weighting.

[0067] As described in step S5 above: an intent parsing system that integrates semantic knowledge graphs and logical reasoning rules can be constructed. The knowledge graph integrates knowledge from multiple fields, covering everyday language, professional terms, product function names, etc. with corresponding meanings and operational associations, providing a rich semantic background for understanding interactive intent. Combined with logical reasoning rules, the input interactive intent features are analyzed. For example, if the interactive intent features include "music", "play" and nodding motion features, it can be seen from the knowledge graph that "playing music" is a common demand, and the nodding action can be used as a confirmation instruction, thereby accurately parsing the user's intention to confirm playing music.

[0068] Once the user's interaction intention is accurately analyzed, the control system of the AI ​​headset immediately calls the corresponding functional module according to the intention. If the user's intention is to play music, the headset activates the music player program, retrieves the appropriate tracks from the local storage or cloud music library and plays them; if it is to query information, it starts the search engine module, sends a query request to the specified data source, and feeds back the results to the user in the form of voice or display (if screen display is supported), so as to achieve a quick response to the user's interaction needs and provide a convenient and efficient interactive experience.

[0069] In one embodiment, based on the hidden Markov model, the start and end points of the speech signal are identified in combination with the probability distribution characteristics of the speech signal, including:

[0070] Performing pre-emphasis processing on the speech signal to obtain a standard speech frame sequence;

[0071] Based on fast Fourier transform, the standard speech frame sequence is converted from the time domain to the frequency domain to obtain a speech signal representation in the frequency domain, and the power spectrum density is calculated to obtain a power spectrum density sequence;

[0072] The power spectral density sequence is input as an observation sequence into a pre-trained hidden Markov model; after the pre-training of the hidden Markov model, the Viterbi algorithm is used to quickly search for the optimal state path in the state network of the hidden Markov model, and the start and end points in the speech signal are identified in combination with the preset speech start and end feature markers in the hidden Markov model.

[0073] In this embodiment, the representation of the speech signal in the time domain has certain limitations for directly analyzing its speech features, and through the fast Fourier transform, the speech frame sequence in the time domain can be converted into a frequency domain representation. This is because the speech signal can more clearly show the distribution of different frequency components in the frequency domain. For example, different speech units such as vowels and consonants have their own specific spectral characteristics in the frequency domain. After FFT conversion, we can intuitively see the amplitude of each frequency component in the speech signal and other information, which is convenient for extracting more distinguishing speech features and providing a basis for subsequent accurate judgment of the start and end points of the speech.

[0074] After obtaining the representation of the speech signal in the frequency domain, calculating the power spectrum density is a key step to further quantify the distribution of speech signal energy at different frequencies. The specific calculation method is to sum the squares of the amplitudes of each frequency component of the frequency domain signal and divide it by the frame length based on the power calculation formula. By performing such calculation operations frame by frame, a complete power spectrum density sequence can be constructed. The power spectrum density can reflect the energy strength of the speech signal at different frequencies. The power spectrum density under different speech states (such as voicing, pause, etc.) has different characteristics. By analyzing this sequence, the characteristics of speech signals changing over time can be better captured, thereby helping to identify the starting and ending points of the speech signal.

[0075] Hidden Markov model is a model based on probability statistics. It assumes that the generation of speech signals is a Markov process, that is, the current state of speech depends only on the state of the previous moment, and there are some hidden states (such as different voicing states, pause states, etc.). These hidden states are inferred through features such as observed power spectrum density sequences. In the training stage, a large number of speech samples covering various noisy environments, different speech speeds, different intonations, etc. are used. By statistically analyzing the probability of speech transitions between different states in these samples, a state transition probability matrix is ​​constructed. This matrix accurately describes the transition probability of speech signals between different voicing states and pause states; at the same time, the observation probability matrix accurately reflects the probability of generating the current observation sequence (i.e., power spectrum density sequence) under a given voicing or pause state; the initial state probability distribution scientifically describes the possibility of speech starting from various states. After training with such a large number of samples, the model learns the inherent probability distribution law of speech signals and prepares for subsequent recognition.

[0076] The Viterbi algorithm is a dynamic programming algorithm used to find the optimal state path in the complex state network of the hidden Markov model. When the power spectral density sequence is input as the observation sequence into the trained hidden Markov model, the Viterbi algorithm will quickly search for an optimal state path with the highest probability from many possible state transition paths based on the state transition probability matrix, observation probability matrix and initial state probability distribution in the model. In the hidden Markov model, each state is pre-labeled with speech start and end feature markers, such as some states corresponding to the start of speech, and some corresponding to the end. Once the optimal state path is found through the Viterbi algorithm, the start and end points in the speech signal can be accurately identified based on the speech start and end feature markers corresponding to the states passed on this path, thereby accurately intercepting the available speech segments and providing high-quality speech input data for subsequent speech recognition and other steps.

[0077] In one embodiment, the speech signal is pre-emphasized to obtain a standard speech frame sequence, including:

[0078] Filtering the speech signal through a first-order high-pass filter to obtain a filtered speech signal;

[0079] The filtered speech signal is divided into a plurality of short-time frames according to a fixed time length, and each of the short-time frames is windowed based on a Hamming window function to obtain a standard speech frame sequence.

[0080] In this embodiment, in an actual voice collection environment, the high-frequency part of the voice signal has relatively low energy and is easily affected by environmental noise and other factors and becomes unclear. The operation of filtering the voice signal through a first-order high-pass filter is mainly aimed at improving the high-frequency component of the voice signal and enhancing the detailed features and recognition of the voice. This allows the voice signal to more clearly show the differences between different voice units (such as vowels, consonants, etc.) during subsequent processing, which helps to more accurately extract voice features and identify the start and end points of the voice.

[0081] The first-order high-pass filter is a simple and commonly used signal processing filter. It implements the screening of signal frequency based on circuit principles or differential equations in digital signal processing. From the frequency domain perspective, it can attenuate low-frequency signals and allow high-frequency signals to pass relatively smoothly. Its transfer function determines the degree of attenuation and amplification of different frequency components. When filtering the speech signal, for each sampling point, the filter will calculate according to the established filter coefficient (which determines the filter's cutoff frequency and other characteristics) and the current sampling value of the speech signal and the output value of the previous moment, thereby changing the amplitude of each frequency component of the speech signal, achieving the effect of enhancing the high-frequency component, and finally outputting the filtered speech signal, so that the spectral distribution of the speech signal after this processing is more in line with the requirements of subsequent speech analysis.

[0082] The speech signal itself is a signal that changes continuously over time, but if you want to perform effective feature analysis on it, it is difficult to directly process the entire continuous signal. Dividing it into multiple short time frames is equivalent to decomposing a long and complex speech process into relatively short, local time periods for analysis. This can better capture the feature changes of speech in different local time periods, because different speech units (such as syllables) have relatively stable features in a short time, which facilitates the subsequent frame-by-frame extraction of key information such as spectral features.

[0083] The choice of fixed duration is usually determined based on the range of syllable duration at a normal speaking speed. Generally speaking, the common frame length is set at around 20-30 milliseconds. This duration range can better cover the local characteristics of a syllable or speech, ensuring that each short-time frame can effectively reflect the characteristics of the speech in this short period of time. At the same time, a certain frame shift (usually around 10-15 milliseconds) will be set between adjacent frames, that is, there is a partial overlap between two adjacent frames. This setting can avoid losing some transition features of the speech during the framing process, making the segmentation of the entire speech signal more reasonable, and facilitating the subsequent comprehensive analysis of the time-frequency characteristics of the speech.

[0084] When performing frame operations on speech signals, since the continuous signal is truncated into short time frames, this truncation process will cause spectrum leakage in the frequency domain. Simply put, the originally discrete spectrum components will expand to adjacent frequencies in the frequency domain, resulting in errors in spectrum analysis and the inability to accurately obtain the true frequency characteristics of the speech signal.

[0085] The Hamming window function is a commonly used window function. It changes the distribution of the speech signal in the time domain by multiplying each frame of the speech signal by the Hamming window function. The Hamming window function has the characteristics of being high in the middle and gradually transitioning to zero on both sides in the time domain. When multiplied by the short-time frame signal, the values ​​at both ends of each frame signal will gradually approach zero, which is equivalent to smoothing the signal in the time domain, thereby effectively reducing the impact of spectrum leakage. After such windowing processing, the spectral characteristics of each short-time frame in the frequency domain can more accurately reflect the true frequency characteristics of the speech signal. The final standard speech frame sequence provides a high-quality data foundation that meets the analysis requirements for subsequent frequency-domain-based analysis (such as fast Fourier transform and other operations), which helps to perform more accurate speech processing and speech start and end point recognition and other related work.

[0086] In one embodiment, the text information and the motion data are input into a fusion model to obtain interaction intention features by fusion, including:

[0087] Input the text information and the motion data into a fusion model, and obtain the real-time situational elements of the AI ​​headset based on the situational insight submodule embedded in the front end of the fusion model;

[0088] Based on a pre-built context-modality association knowledge base, determine the importance ratio of the text information and the motion data in the current context;

[0089] Importing the text information and motion data into a processing layer based on a multimodal attention fusion architecture, focusing on the semantic focus of the text and the key nodes of the motion data simultaneously, and calculating the multimodal interaction weight;

[0090] Based on the importance ratio and the multimodal interaction weight, a weight when the text information and the motion data are fused is calculated;

[0091] The text information and motion data are fused through a deep neural network based on a skip connection structure to obtain interaction intention features.

[0092] In this embodiment, the following is a detailed description of each step in the technical solution:

[0093] In the process of multimodal voice interaction, the context of the AI ​​headset is crucial to accurately understand the user's intention. The context insight submodule is like an "antenna" for information collection and analysis. It can capture various aspects of information such as the environment in which the AI ​​headset is located and its own operating status in real time to determine various real-time context elements. For example, it will access the system log of the device to which the AI ​​headset is connected to understand what application is currently running (such as playing music, using navigation functions, or being in a call state, etc.), and understand the network connection status (whether it is in a stable Wi-Fi environment or relying on mobile data and how strong the signal is); at the same time, it will also obtain external environmental information, such as using sensors to detect the intensity of surrounding light (bright indoor environment or dim outdoor night, etc.), the degree of environmental noise (the amount of background noise collected by the microphone can be used to determine whether it is a quiet environment or a noisy street market, etc.), and even some status information of the headset itself, such as the remaining power. By integrating these different dimensions of information, a comprehensive depiction of the current real-time context of the AI ​​headset is provided, providing a basic basis for subsequent judgments on the importance of text information and motion data.

[0094] Different real-time situational factors will greatly affect the judgment of user interaction intentions. For example, when a user is in a noisy outdoor environment and is using the navigation function, the movement data of his head (such as frequently turning his head to check the surrounding road conditions) may be relatively more important, because in this environment, voice commands may be interfered with and difficult to hear, and users are more inclined to use head movements to assist in expressing their intentions; while in a quiet indoor environment, when users perform operations such as information query, clear and accurate text information often occupies a more dominant position and can convey needs more accurately. Therefore, obtaining these situational factors in real time is the first step to achieve accurate multimodal fusion.

[0095] The pre-built context-modality association knowledge base is constructed through analysis, induction and organization based on a large amount of actual interaction scenario data. It stores the association between different contextual elements and the importance of text information and motion data. For example, the knowledge base will record that in a context such as "the navigation application is turned on, the environment is noisy and the light is dim", the importance of motion data may be set to 70%, and the importance of text information is 30%; while in the context of "the music playback application is turned on, the environment is quiet and the light is bright", the importance of text information may reach 80%, and motion data is 20%, etc. It covers the importance of text and motion data under various possible application scenarios, environmental conditions and headphone status combinations, forming a comprehensive reference basis to help the system reasonably judge the importance ratio of the two according to the current real-time context.

[0096] After obtaining the current real-time context elements, a match search will be performed in the context-modality association knowledge base, and the importance ratio of text information and motion data in the current context will be determined based on the existing association relationship. This judgment method allows the fusion model to flexibly pay different attention to data of different modalities according to actual conditions, avoiding the limitations of the traditional fixed weight allocation method, making the subsequent fusion process more in line with the user's real intention and improving the accuracy of interaction intention feature extraction.

[0097] The multimodal attention fusion architecture aims to extract the most critical parts that best reflect user intentions from text information and motion data. By simulating the human attention mechanism, it can simultaneously focus on the semantic focus of the text (such as the core meaning expressed by keywords and key phrases in the text) and the key nodes of the motion data (such as specific movements in head movements that have clear meanings, such as nodding to confirm and shaking your head to deny). In this processing layer, the text and motion data will be deeply analyzed to identify which parts are more critical in the current interaction based on their respective internal structures and characteristics, and then these key parts will be assigned corresponding weights, that is, the multimodal interaction weights will be calculated. These weights reflect the relative importance of different elements in the text and motion data in the fusion process, which can highlight important information and weaken secondary information, making the fusion process more targeted.

[0098] The operation of simultaneously focusing on the semantic key points of text and the key nodes of motion data breaks the traditional mode of processing text and motion data separately and then simply splicing them together. It fully considers the correlation and complementarity between the two modal data. By calculating the multimodal interaction weight, it can more accurately capture the collaborative expression of text and motion data in the interaction process, dig out the deep user intention, and provide a more detailed and practical weight reference for the subsequent accurate fusion of the two, so that the fusion result can better reflect the user's real interaction ideas.

[0099] Based on the importance ratio and the multimodal interaction weight, the weight of the text information and motion data when fused is calculated. The final fusion weight is determined by comprehensive calculation based on the importance ratio and multimodal interaction weight in the real-time context obtained in the first two steps. Specifically, the importance ratio of the text information and motion data obtained from the context-modality association knowledge base in the current context is organically combined with the multimodal interaction weight calculated by the multimodal attention fusion architecture. For example, assuming that the context-modality association knowledge base determines that the importance ratio of text information in the current context is 40%, and that of motion data is 60%, and the multimodal attention fusion architecture calculates that the multimodal interaction weight of a keyword in the text information is 0.3, and the multimodal interaction weight of a key action in the motion data is 0.7, then after comprehensive consideration, the weight of this keyword in the final fusion is 40%×0.3, and the weight of the key action in the fusion is 60%×0.7. By analogy, such comprehensive calculations are performed on each key element in the text information and motion data to obtain their specific weights when fused.

[0100] The fusion weight obtained through such comprehensive calculation fully integrates the macro-context factors and the micro-characteristic factors of text and motion data, so that in the fusion process, each key part of text information and motion data can be given the most reasonable and accurate weight distribution. This not only takes into account the impact of the overall context on different modal data, but also takes into account the specific significance of the data itself in the interaction, thus ensuring that the fusion process is more scientific and accurate, and can extract the interaction intention features that meet the user's true intentions to the greatest extent.

[0101] The above-mentioned deep neural network based on the skip connection structure is an innovative network architecture design. In traditional deep neural networks, information is gradually transmitted from the input layer to the output layer through multiple hidden layers. As the number of network layers increases, problems such as gradient vanishing or gradient explosion will occur, causing the underlying features to be gradually lost or difficult to be effectively used during the transmission process. The skip connection structure allows low-level features to be directly transmitted across layers to high levels and re-integrated with high-level features processed by the intermediate layers. For example, when fusing text information and motion data, the original original features of the text information and motion data can be directly combined with high-level semantic features and motion features processed by the deep network through skip connections, skipping some intermediate hidden layers. This avoids excessive loss of feature information, retains more key details in the original data, and accelerates the convergence of the network, allowing the model to learn the effective feature representation after the fusion of text and motion data more quickly.

[0102] After inputting text information and motion data into this deep neural network with a skip connection structure, the neurons in the network will perform layer-by-layer calculations and feature transformations on the input data according to the set activation function and connection weights. During this process, the features of the text information and motion data will be fused at different levels, and the feature information will be supplemented and strengthened through skip connections. As the network continuously learns and adjusts the weights, it gradually extracts deep features that can highly summarize the user's interaction intention, and finally outputs the interaction intention features. These interaction intention features condense the comprehensive meaning of the text information and motion data in the current context, providing a key basis for accurately analyzing the user's intention and driving the AI headset to execute corresponding functions later, and realizing an important transformation from multi-modal data to the extraction of user interaction intention.

[0103] In one embodiment, the real-time context elements include the currently activated function application, network connection status, environmental noise decibels, and light illuminance.

[0104] In one embodiment, after driving the AI headset to execute the corresponding function, it includes:

[0105] Establish a mapping relationship between the text information, motion data, and the interaction intention;

[0106] Generate a wake word based on the mapping relationship;

[0107] Configure the corresponding relationship between the wake word and the interaction intention so that when the wake word is recognized, the AI headset is driven to execute the function corresponding to the interaction intention again.

[0108] In this embodiment, during the multi-modal voice interaction process, there is an internal logical relationship among the user's text information, motion data, and the finally parsed interaction intention. Establishing the mapping relationship is to clearly sort out and record these relationships so that more intelligent operations can be carried out based on these relationships later, such as generating appropriate wake words and realizing the quick reproduction of corresponding functions based on wake words.

[0109] Based on the previously established mapping relationship among text information, motion data, and interaction intention, algorithms and rules are used to generate a wake word. The generation of this wake word aims to find a concise and representative word or phrase that can highly summarize the corresponding interaction intention and the associated text and motion data features. For example, for the above-mentioned interaction intention of confirming music playback, a wake word such as "Music Start" may be generated, which is both concise and can be related to the music playback function to a certain extent. The generation process may consider factors such as the memorability of the word, the convenience of pronunciation, and the fit with the overall interaction context, and at the same time ensure that there is sufficient differentiation between different wake words to avoid confusion, so as to accurately identify and trigger the corresponding function later.

[0110] Generating wake-up words provides users with a more convenient and efficient way of interaction. Users do not need to repeat the complete text instructions and make corresponding movements. As long as the wake-up word is spoken, the system can quickly identify and know the function the user wants to perform. Especially in some scenarios where specific functions are frequently used or it is inconvenient to operate the headset with both hands, it can greatly improve the interaction efficiency and enhance the user experience. Moreover, the wake-up word is extracted from the existing mapping relationship and is closely linked to the specific interaction intention, ensuring the accuracy and pertinence of its meaning.

[0111] Configure the correspondence between the wake-up word and the interaction intent, so that when the wake-up word is recognized, the AI ​​headset is driven again to execute the function corresponding to the interaction intent, and a special configuration table or database structure is established internally to explicitly associate and store the generated wake-up word with the corresponding interaction intent. For example, in this configuration table, the wake-up word "Yin Start" will be associated with the interaction intent "Confirm to play music". Figure 1 One corresponding record. When the user says the wake-up word, the speech recognition module will quickly recognize the wake-up word, and then the system will search and match it in this configuration table to find the corresponding interaction intent, and then call the corresponding function module according to the interaction intent to drive the AI ​​headset to execute the previous corresponding function again, such as starting the music player to start playing music.

[0112] During subsequent use, as long as the user remembers the corresponding wake-up word, they can quickly let the AI ​​headset execute functions that have been performed before and may need to be operated again. There is no need to fully express text instructions and cooperate with motion data to convey intentions as when using it for the first time. This greatly simplifies the operation process and improves the timeliness and convenience of interaction.

[0113] In one embodiment, after generating a wake-up word based on the mapping relationship, the method further includes:

[0114] Encoding the text information, the motion data and the interaction intention to generate coded characters;

[0115] Acquire characteristics of multiple dimensions of the wake-up word, and generate a graph based on the characteristics of each dimension;

[0116] Setting a plurality of key points on each graph, and sequentially adding each character in the coded character to each key point to obtain a key point graph;

[0117] Based on the positional relationship between each of the key point graphics, a joint agreed key is generated, the mapping relationship is encrypted, and transmitted to the management server.

[0118] In this embodiment, in the multimodal voice interaction system, text information, motion data and interaction intentions contain a lot of sensitive content related to user interaction, and they are encoded mainly for data security and privacy protection considerations. Through the encoding operation, these information that originally has clear meanings and may be easily interpreted by the outside world can be converted into a coded character in a specific format, so that even if it is obtained during the transmission and storage process, it is difficult to be directly understood, thereby ensuring the confidentiality of user interaction data and preventing the leakage of privacy information such as user usage habits and intentions.

[0119] A variety of encoding algorithms can be used to implement this process. For example, for text information, common character encoding standards such as ASCII or Unicode can be used to convert the characters in the text into corresponding digital codes, and then these digital code sequences can be further processed through encrypted hash functions (such as SHA-256, etc.) to generate a fixed-length hash value as part of the encoded character; for motion data, since it is numerical data collected by sensors (such as acceleration, angular velocity, etc.), these values ​​can be quantized according to certain accuracy requirements, and then specific encoding rules (such as binary-based encoding methods, converting quantized values ​​into binary strings) can be used, and then combined with other processed text information encoding parts; for interaction intent, it can also be represented by a specific digital identifier or semantic code, and then integrated into the overall encoding sequence. Finally, a string of unique and confidential encoded characters is generated, which can uniquely correspond to the combination of text, motion data and interaction intent in a specific interaction situation.

[0120] The wake-up word has multiple dimensions of characteristics. Common dimensional characteristics can include phonetic characteristics (such as the number of syllables, tone changes, pronunciation duration, etc. of the wake-up word), which can be used to analyze its uniqueness from the phonetic perspective; character characteristics (such as the letters contained in the wake-up word, the stroke structure of Chinese characters, font form, etc.), which can be considered from the perspective of writing; in addition, semantic characteristics cannot be ignored, such as the semantic category covered by the wake-up word, the degree of correlation with related functions, etc. Analyzing these different dimensional characteristics can fully and deeply grasp the characteristics of the wake-up word and provide rich materials for graphic generation.

[0121] Taking the number of syllables in the phonetic characteristics as an example, if the wake-up word has three syllables, then based on the pre-set graphic generation rules, different types of polygons can be generated for different numbers of syllables (three syllables correspond to triangles), and their side lengths, angles and other geometric parameters can be further adjusted and determined based on factors such as pronunciation duration and tone changes, so that the generated graphics can reflect the characteristics of the wake-up word in the voice dimension to a certain extent. Similarly, for character characteristics, a graphics drawing algorithm can be used to generate corresponding graphic outlines based on the stroke structure, such as the complexity of the strokes determines the complexity of the graphics; starting from the semantic characteristics, different colors can be selected to fill the graphics according to the functional categories of semantic associations. By comprehensively using the characteristics of multiple dimensions in this way, a unique graphic that can characterize its characteristics to a certain extent is generated for each wake-up word, laying the foundation for subsequent encryption operations.

[0122] The purpose of setting multiple key points on the generated graphics is to integrate the coded characters into the graphics in an orderly manner, to achieve an organic combination of data and graphics, and to facilitate subsequent operations such as key generation based on the positional relationship of the graphics. The setting of key points can be determined based on the geometric features of the graphics. For example, for polygonal graphics, key points can be set according to regular positions such as vertices and midpoints of edges; for irregular graphics, the center of gravity of the graphics, the position where the curvature of the boundary changes greatly, etc. can be used as the basis for key point selection. The number of key points to be set needs to comprehensively consider factors such as the length of the coded characters and the complexity requirements of subsequent encryption operations to ensure that the coded characters can be added completely and reasonably.

[0123] Add each character in the coded character to each key point in sequence, so that the coded character exists in a visual way that is closely related to the graphics. For example, if the coded character is a combination of numbers and letters, place each character in the corresponding key point position in sequence. This process can be implemented using the text annotation function in the graphics drawing software or a custom graphics data structure, so that each key point carries part of the information in the coded character. The key point graphics obtained in this way deeply integrate the originally independent coded characters with the graphics generated based on the wake-up word characteristics, which not only retains the information of the coded characters, but also gives them a new form of expression through graphics, providing a unique data carrier for the subsequent generation of encryption keys.

[0124] The positional relationship of each key point graphic in space is unique and complex, and this positional relationship can be used as a key material for generating encryption keys. For example, by calculating the distance between any two key point graphics (using geometric distance calculation methods such as Euclidean distance), relative angle (determining the angle relationship between two graphics through vector operations), coordinate offset between the center of gravity of the graphics and other positional parameters, and then combining these parameters in a certain order, and then further processing them through encryption algorithms (such as the AES algorithm in symmetric encryption algorithms or the RSA algorithm in asymmetric encryption algorithms, etc., which can be selected according to system security requirements), the data related to these positional relationships can be converted into a joint agreed key with confidentiality. This key generation process makes full use of the complexity and randomness of the positional relationship of key point graphics, so that the generated key has high security and uniqueness.

[0125] The purpose of encrypting the mapping relationship is to ensure that the core text information, the relationship between motion data and interaction intent in the entire interactive system is not leaked, and to protect the privacy of users and the security of the system. Then, the encrypted mapping relationship is transmitted to the management server, which usually has a higher security protection mechanism (such as firewalls, intrusion detection systems, etc.) and can better store and manage these encrypted data.

[0126] In one embodiment, generating a joint agreed key based on the positional relationship of each of the key point graphics includes:

[0127] Arranging each of the key point graphics side by side in a preset layer, rotating each of the key point graphics according to a preset rule so that the characters located at the top key points in each of the key point graphics are characters of the same type, and obtaining a rotated graphic;

[0128] All characters located at the bottom key points in each rotating figure are obtained, and are combined in sequence, and the obtained character combination is used as the joint agreed key.

[0129] In this embodiment, the key point graphics are arranged side by side in a preset layer mainly to establish a unified spatial layout that is convenient for subsequent operation and analysis. The preset layer can be a canvas created in a drawing software, and all the key point graphics are placed on this plane and arranged in a certain order (such as from left to right or according to the order in which the graphics are generated, etc.) so that they are in a relatively regular and orderly state. Doing so helps to clearly observe and process the relationship between the various graphics, and provides a convenient basic condition for subsequent key generation operations based on graphic features.

[0130] Each key point graphic is rotated according to the preset rules, with the aim of making the characters at the top key point in each key point graphic the same type of characters. The preset rules here are usually formulated based on the definition of the coded character type and the characteristics of the graphic structure. For example, the coded characters include different types such as numbers, letters (which can be further divided into uppercase letters and lowercase letters), and specific symbols. The preset rules can stipulate that the graphics are rotated so that the characters corresponding to the top key points of each graphic are all numbers or uppercase letters, etc. Through such a rotation operation, a standardization and alignment is achieved, so that each graphic has the same type of character identification at a specific position (i.e., the top key point), which facilitates the subsequent extraction of key information for key generation. This unified processing method can find patterns in seemingly disordered graphic combinations, dig out key materials that can be used to generate consistent and related key materials, and enhance the logic and controllability of key generation.

[0131] The reason for choosing to obtain all characters at the bottom key points in the rotated graphics is that after the previous rotation operation of unifying the characters at the top key points, the characters at the bottom key points become a relatively unique information set that is associated between various graphics.

[0132] The characters of the key points at the bottom of each rotating graphic are combined in sequence to form a character combination, which is used as a joint agreed key. This method utilizes the unique relevance and order contained in the key point characters at the bottom of the graphic after specific processing (rotation and arrangement, etc.). Since the key point characters at the bottom of each graphic are extracted under a unified rule, the combination order between them reflects the arrangement order of each key point graphic in the preset layer and the relative position relationship after rotation adjustment, which indirectly reflects the complex correspondence between text information, motion data, and interaction intentions in the entire multimodal voice interaction system. The joint agreed key generated in this way is unique and confidential, and can be used to encrypt key mapping relationships and other data to ensure that these sensitive information is not obtained and interpreted by unauthorized third parties during transmission and storage, thereby ensuring the information security and user privacy of the entire interactive system.

[0133] In one embodiment, generating a joint agreed key based on the positional relationship of each of the key point graphics includes:

[0134] For each key point figure, the contour extraction algorithm is used to identify the contour of the figure, and the geometric properties of the figure are determined by statistics and analysis of the contour pixels;

[0135] The deep learning image classification model is used to determine the type information of each key point graphic, and the geometric attributes and type information are used as the shape dimension characteristics of the graphic;

[0136] Construct a three-dimensional space coordinate system with the initial calibration position of the built-in sensor when the headset is first started as the origin, locate the coordinates of each key point graphic according to its actual position in the virtual space, and use the spatial vector calculation method to accurately calculate the spatial distance and direction vector between any two key point graphics, record them in the form of vector coordinates, and construct a complete set of spatial position relationships between graphics;

[0137] The acquired graphic shape dimensional characteristics and spatial position relationship set are input into a pre-designed key generator; the key generator adopts a hybrid encryption architecture, first one-way encrypts the graphic shape dimensional characteristics and the spatial position relationship set based on a hash function to obtain a hash value of a fixed length; a chaotic encryption algorithm is introduced, and the environmental parameters of the current environment of the AI ​​headset are used as the initial value of the chaotic sequence, and an iterative encryption operation is performed in combination with the hash value to generate a joint agreed key; during the operation, the encryption parameters are dynamically adjusted according to the remaining power of the headset and the device status information.

[0138] In one embodiment, generating a joint agreed key based on the positional relationship of each of the key point graphics includes:

[0139] Construct a three-dimensional virtual space coordinate system, map each key point graphic one by one into the space, and assign each key point graphic a unique initial spatial coordinate according to the type of graphic and the number of characters it carries. For example, the starting coordinates of the triangular key point graphic are located in the first quadrant of the space, and the depth level in the quadrant is determined by the number of characters it carries; the quadrilateral key point graphic starts from the second quadrant, and the coordinate position is also finely adjusted according to the number of characters.

[0140] Start the dynamic graphics transformation engine to collect the physical parameters of the current environment of the AI ​​headset in real time, such as temperature, humidity, magnetic field strength, and acceleration and angular velocity change data of the user's head movement;

[0141] The physical parameters are converted into graphic transformation control instructions; based on the graphic transformation control instructions, each key point graphic is transformed, including multi-dimensional rotation, scaling and translation operations. For example, when the ambient temperature rises, the graphic is moderately stretched in the vertical axis; when the user's head turns left quickly, the graphic is translated to the left by a certain unit, so that the graphic position relationship is adjusted dynamically in real time, greatly increasing the complexity and randomness of the position change.

[0142] Based on the precise position of the transformed key point graphics in three-dimensional space, an asymmetric encryption algorithm combined with a hash function is used to calculate and generate a joint agreed key. Specifically, the key position parameters of the center of gravity coordinates, relative distances between vertices, and spatial orientation vectors of each graphic are first one-way encrypted through a hash function to obtain a hash value fragment of a fixed length; using an asymmetric encryption algorithm, the unique hardware identification code built into the AI ​​headset when it is first activated is used as the private key to encrypt and convert the hash value fragment to generate the final joint agreed key. The key generated in this way not only integrates the highly complex information of the dynamic position relationship of the graphics, but also uses quantum encryption and asymmetric encryption to ensure the super security of the key generation process, effectively improving the reliability of encryption of the mapping relationship.

[0143] Reference Figure 2 In another embodiment of the present invention, a multimodal voice interaction device of an AI headset is provided, comprising:

[0144] The interception unit is used to collect the user's voice signal, identify the start and end points in the voice signal based on the hidden Markov model and the probability distribution characteristics of the voice signal, and intercept the usable voice segment;

[0145] A recognition unit, used to perform speech recognition on the speech segment to obtain corresponding text information;

[0146] The acquisition unit is used to collect the motion data of the user's head based on the motion sensor built into the AI ​​headset;

[0147] A fusion unit is used to input the text information and the motion data into a fusion model to obtain interaction intention features by fusion; the fusion model is based on a multimodal attention fusion strategy, intelligently determines the importance of text information and motion data according to the user's current interaction scenario, adaptively allocates weights when the two are fused, and performs deep fusion;

[0148] The interaction unit is used to analyze the interaction intention feature to obtain the interaction intention of the user; based on the interaction intention, drive the AI ​​headset to execute the corresponding function.

[0149] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0150] Reference Figure 3 In an embodiment of the present invention, an electronic device is also provided, the internal structure of the electronic device can be as follows Figure 3As shown. The electronic device includes a processor, a memory, a display screen, an input device, a network interface and a database connected via a system bus. Among them, the computer-designed processor is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store the corresponding data in this embodiment. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0151] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the electronic device to which the solution of the present invention is applied.

[0152] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0153] In summary, the multimodal voice interaction method, electronic device and storage medium of the AI ​​headset provided in the embodiment of the present invention include: collecting the user's voice signal, based on the hidden Markov model, combining the probability distribution characteristics of the voice signal to identify the starting and ending points in the voice signal, and intercepting the available voice segment; performing voice recognition on the voice segment to obtain the corresponding text information; based on the built-in motion sensor of the AI ​​headset, collecting the motion data of the user's head; inputting the text information and the motion data into the fusion model, and fusing to obtain the interaction intention feature; the fusion model is based on the multimodal attention fusion strategy, according to the user's current interaction scene, intelligently judges the importance of text information and motion data, adaptively allocates the weight when the two are fused, and performs deep fusion; analyzes the interaction intention feature to obtain the user's interaction intention; based on the interaction intention, drives the AI ​​headset to perform the corresponding function. In the present invention, based on the hidden Markov model, combined with the probability distribution characteristics of the voice signal, the starting and ending points in the voice signal can be accurately identified; at the same time, combined with the multimodal interaction mode of voice recognition and motion data, the user's interaction intention is fused and analyzed, overcoming the defect that the single-modal interaction mode cannot accurately analyze user needs.

[0154] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0155] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0156] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal voice interaction method for an AI headset, characterized in that: The following steps are involved: Collect the user's voice signal, identify the start and end points of the voice signal based on the hidden Markov model and the probability distribution characteristics of the voice signal, and extract the usable voice segment; Performing speech recognition on the speech segment to obtain corresponding text information; Based on the motion sensor built into the AI ​​headset, the user's head motion data is collected; The text information and the motion data are input into a fusion model to obtain interaction intention features through fusion; the fusion model is based on a multimodal attention fusion strategy, intelligently determines the importance of the text information and the motion data according to the user's current interaction scenario, adaptively allocates the weights when the two are fused, and performs deep fusion; Parsing the interaction intention feature to obtain the interaction intention of the user; Based on the interaction intention, drive the AI ​​headset to perform corresponding functions; The text information and the motion data are input into a fusion model, and the fusion obtains interaction intention features, including: Input the text information and the motion data into a fusion model, and obtain the real-time situational elements of the AI ​​headset based on the situational insight submodule embedded in the front end of the fusion model; Based on a pre-built context-modality association knowledge base, determine the importance ratio of the text information and the motion data in the current context; Importing the text information and motion data into a processing layer based on a multimodal attention fusion architecture, focusing on the semantic focus of the text and the key nodes of the motion data simultaneously, and calculating the multimodal interaction weight; Based on the importance ratio and the multimodal interaction weight, a weight when the text information and the motion data are fused is calculated; The text information and motion data are fused through a deep neural network based on a skip connection structure to obtain interaction intention features.

2. The multimodal voice interaction method of the AI ​​headset according to claim 1, characterized in that: Based on the Hidden Markov Model, combined with the probability distribution characteristics of the speech signal, the start and end points of the speech signal are identified, including: Performing pre-emphasis processing on the speech signal to obtain a standard speech frame sequence; Based on fast Fourier transform, the standard speech frame sequence is converted from the time domain to the frequency domain to obtain a speech signal representation in the frequency domain, and the power spectrum density is calculated to obtain a power spectrum density sequence; The power spectral density sequence is input as an observation sequence into a pre-trained hidden Markov model; after the pre-training of the hidden Markov model, the Viterbi algorithm is used to quickly search for the optimal state path in the state network of the hidden Markov model, and the start and end points in the speech signal are identified in combination with the preset speech start and end feature markers in the hidden Markov model.

3. The multimodal voice interaction method of the AI ​​headset according to claim 2, characterized in that: The speech signal is pre-emphasized to obtain a standard speech frame sequence, including: Filtering the speech signal through a first-order high-pass filter to obtain a filtered speech signal; The filtered speech signal is divided into a plurality of short-time frames according to a fixed time length, and each of the short-time frames is windowed based on a Hamming window function to obtain a standard speech frame sequence.

4. The multimodal voice interaction method of the AI ​​headset according to claim 1, characterized in that: The real-time contextual elements include currently activated functional applications, network connection status, ambient noise decibels, and light illumination.

5. The multimodal voice interaction method of the AI ​​headset according to claim 1, characterized in that: After driving the AI ​​headset to execute the corresponding function, the method includes: Establishing a mapping relationship between the text information, the motion data and the interaction intention; Based on the mapping relationship, generate a wake-up word; Configure the correspondence between the wake-up word and the interaction intention so that when the wake-up word is recognized, the AI ​​headset is driven again to execute the function corresponding to the interaction intention.

6. The multimodal voice interaction method of the AI ​​headset according to claim 5, characterized in that: After generating a wake-up word based on the mapping relationship, the method further includes: Encoding the text information, the motion data and the interaction intention to generate coded characters; Acquire characteristics of multiple dimensions of the wake-up word, and generate a graph based on the characteristics of each dimension; Setting a plurality of key points on each graph, and sequentially adding each character in the coded character to each key point to obtain a key point graph; Based on the positional relationship between each of the key point graphics, a joint agreed key is generated, the mapping relationship is encrypted, and transmitted to the management server.

7. The multimodal voice interaction method of the AI ​​headset according to claim 6, characterized in that: Based on the positional relationship of each of the key point graphics, a joint agreed key is generated, including: Arranging each of the key point graphics side by side in a preset layer, rotating each of the key point graphics according to a preset rule so that the characters located at the top key points in each of the key point graphics are characters of the same type, and obtaining a rotated graphic; All characters located at the bottom key points in each rotating figure are obtained, and are combined in sequence, and the obtained character combination is used as the joint agreed key.

8. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images