Intelligent remote controller and control method thereof

Abstract

The invention provides an intelligent remote controller and a control method thereof, and relates to the technical field of intelligent remote control, and the intelligent remote controller comprises a voice acquisition module which is used for capturing a voice signal sent by a user and converting the voice signal into a digital signal; the endpoint detection module is used for distinguishing a voice segment and a noise segment by calculating the spectrum entropy of the digital signal so as to obtain voice data of endpoint detection; the preprocessing module is used for performing pre-emphasis, framing and windowing processing on the voice data subjected to endpoint detection to obtain preprocessed voice data; and the touchpad module and the voice acquisition module jointly form interactive multi-mode input, and the touchpad module is used for detecting a touch position, outputting a corresponding coordinate value and taking the output coordinate value as original data of gesture recognition. The voice and gesture data are processed and classified in real time, the user instruction can be quickly recognized, quick and accurate control response is achieved through template matching, and the intelligent remote control efficiency and the user experience are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent remote control, and in particular to an intelligent remote control and a control method thereof. Background Art

[0002] Traditional remote controls usually rely on simple button operations, with each button corresponding to a specific function. This single operation mode is not flexible enough when faced with complex function settings. For example, on a smart TV, users may need to control multiple input sources, adjust picture settings, access applications, etc. at the same time. Traditional button remote controls may require users to complete these operations through a combination of multiple buttons, which is not only cumbersome but also prone to errors.

[0003] For example, in a large living room, if the signal of the remote control cannot cover the entire area, the user may need to readjust the position of the remote control to achieve control. This not only affects the convenience of operation, but also limits the user's freedom. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an intelligent remote control and a control method thereof, which can quickly identify user commands by real-time processing and classification of voice and gesture data, and achieve fast and accurate control response through template matching, thereby improving the efficiency of the intelligent remote control and user experience.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] In a first aspect, a smart remote controller includes:

[0007] A voice acquisition module is used to capture the voice signal sent by the user and convert the voice signal into a digital signal;

[0008] An endpoint detection module, used to distinguish between speech segments and noise segments by calculating the spectrum entropy of digital signals to obtain speech data for endpoint detection;

[0009] A preprocessing module, used for performing pre-emphasis, framing and windowing processing on the voice data detected by the endpoint to obtain preprocessed voice data;

[0010] The touch panel module, together with the voice acquisition module, constitutes an interactive multi-modal input, which is used to detect the touch position and output the corresponding coordinate value, and the output coordinate value is used as the raw data for gesture recognition;

[0011] The six-axis sensor module is used to detect the motion data, motion state and direction information of the device, which complements the raw data of gesture recognition to obtain gesture data;

[0012] The classification module is used to extract features from the pre-processed speech data and classify and identify the gesture data to obtain speech feature data and gesture classification results;

[0013] A template matching module is used to construct multiple groups of voice templates and gesture templates according to the voice feature data and gesture classification results to obtain template matching data;

[0014] The wireless communication module is used to wirelessly transmit the template matching data to the receiving end to realize intelligent remote control.

[0015] Furthermore, the speech segment and the noise segment are distinguished by calculating the spectrum entropy of the digital signal to obtain the speech data for endpoint detection, including:

[0016] According to the speech signal, a frequency spectrum of the speech signal frame is obtained, and an amplitude spectrum is extracted from the frequency spectrum to obtain an amplitude spectrum;

[0017] Normalize the amplitude spectrum and calculate to obtain the spectrum entropy;

[0018] According to the spectral entropy characteristics, the threshold is set by analyzing the pure noise signal, and the spectral entropy of each frame is traversed to distinguish the speech segment and the noise segment to obtain the speech data for endpoint detection. If the spectral entropy of a frame is lower than the set threshold, the frame is considered to belong to the speech segment; otherwise, the frame is considered to belong to the noise segment.

[0019] Further, the speech data detected by the endpoint is pre-emphasized, framed and windowed to obtain pre-processed speech data, including:

[0020] Design a first-order high-pass filter to process the speech data detected by endpoints to obtain a pre-emphasized signal;

[0021] The pre-emphasized speech signal is divided into multiple short-time stable signal frames to obtain a framed signal; according to the characteristics of the speech signal, starting from the starting point of the speech signal, the segmentation is performed according to the determined frame length and frame shift, starting from the first sampling point, taking continuous frame length sampling points as the first frame, skipping frame shift sampling points, and taking continuous frame length sampling points from the next position as the second frame, and so on, until the end of the signal;

[0022] According to the framed signal, windowing processing is performed to obtain pre-processed speech data.

[0023] Furthermore, the touch panel module includes a touch panel sensor and a touch control chip; detecting the touch position and outputting the corresponding coordinate value, and using the output coordinate value as the original data for gesture recognition, including:

[0024] The touch control chip obtains the touch detection result by monitoring the capacitance change on the touch panel sensor; based on the touch detection result, the touch position is sampled to obtain the electrical signal, and the electrical signal change is mapped to the two-dimensional coordinate to obtain the coordinate value;

[0025] The main control chip performs feature recognition on the received coordinate value sequence to obtain raw data for gesture recognition.

[0026] Furthermore, the six-axis sensor module includes a three-axis accelerometer and a three-axis gyroscope, and the motion data includes motion state and direction information, wherein the accelerometer is used to measure the acceleration change of the object in the three axes of X, Y, and Z, and the gyroscope is used to measure the angular velocity change of the object around these three axes; the motion data motion state and direction information of the detection device are complementary to the original data of gesture recognition to obtain gesture data, including:

[0027] The three-axis accelerometer can capture the motion status of the device in real time, including movement speed, acceleration and direction changes;

[0028] Get the device's rotation state and direction information through the gyroscope;

[0029] The gesture data is obtained by fusing the six-axis sensor data with the data source of gesture recognition.

[0030] Furthermore, the gesture data is classified and identified to obtain voice feature data and gesture classification results, including:

[0031] Extract gesture features from gesture data, where the gesture features include distance, speed, acceleration, and direction change of the touch point;

[0032] Each gesture feature is converted into fixed-length feature vector data, segmented, and recursively constructed subtrees to obtain the gesture classification result.

[0033] Furthermore, the template matching module constructs multiple groups of speech templates and gesture templates according to the speech feature data and the gesture classification results to obtain template matching data, including:

[0034] According to the speech feature data and the gesture classification results, a speech feature vector and a feature vector after gesture classification are obtained;

[0035] The initial codebook is constructed by randomly selecting feature vectors of the training set and iteratively optimizing to construct a template set;

[0036] According to the template set, the feature vector is mapped to a low-dimensional codebook space and compared with each template in the template set to obtain a comparison result;

[0037] The comparison result is matched with the control instruction to obtain template matching data.

[0038] Furthermore, the template matching data is wirelessly transmitted to a receiving end to realize intelligent remote control, including:

[0039] The wireless communication module obtains the processed data from the template matching module, and performs data packaging and modulation processing to achieve wireless transmission;

[0040] The receiving end receives the wireless signal, converts it into a baseband signal, and identifies the specific control instructions after demodulation and analysis;

[0041] According to the control instructions, perform corresponding operations to realize intelligent remote control.

[0042] In a second aspect, a control method of an intelligent remote controller is provided, the control method comprising the following steps:

[0043] The voice acquisition module captures the voice signal emitted by the user and converts the voice signal into a digital signal;

[0044] Calculating the spectrum entropy of the digital signal using an endpoint detection module to distinguish between speech segments and noise segments, thereby obtaining speech data for endpoint detection;

[0045] Pre-emphasize, frame and window the voice data detected by the endpoint to optimize the signal quality and obtain pre-processed voice data;

[0046] Detecting the user's touch position through the touch panel module and outputting corresponding coordinate values, and using the coordinate values ​​as raw data for gesture recognition;

[0047] The six-axis sensor module is used to detect the motion data, motion state and direction information of the device, which complements the original data of gesture recognition and is processed together to obtain gesture data;

[0048] In the classification module, the pre-processed speech data is subjected to feature extraction, and the gesture data is subjected to classification and recognition, so as to obtain speech feature data and gesture classification results respectively;

[0049] According to the extracted voice feature data and gesture classification results, multiple groups of voice templates and gesture templates are constructed in the template matching module, and the voice signal and gesture data to be recognized are matched with the templates to obtain template matching data;

[0050] The wireless communication module is used to wirelessly transmit the template matching data to the receiving end, and the receiving end performs corresponding control operations according to the received data, thereby realizing the intelligent remote control function.

[0051] The above solution of the present invention includes at least the following beneficial effects:

[0052] The smart remote control combines voice, touch and gesture control input methods, providing rich and flexible user interaction methods; through digital processing of voice signals, spectrum entropy calculation and endpoint detection, it can effectively distinguish between voice segments and noise segments, reduce background noise interference, and improve the accuracy of voice recognition. Pre-emphasis, framing and windowing processing further optimize the voice signal and enhance recognition sensitivity; combined with the touchpad module and six-axis sensor module, the smart remote control can recognize the user's touch gestures and the movement status of the device; the classification module converts voice and gesture data into recognizable feature information through feature extraction and classification recognition.

[0053] The template matching module achieves efficient command recognition and execution and speeds up the response speed by constructing voice and gesture templates and matching the input data with the templates; the wireless communication module can wirelessly transmit the processed template matching data to the receiving end to achieve remote control; the combined use of the accelerometer and gyroscope of the six-axis sensor module can capture and analyze the motion state and direction changes of the device in real time. Combining acceleration and angular velocity data for fusion processing, it can accurately identify fast gestures, acceleration and deceleration gestures, obtain dynamic and static posture information of the device, and improve the accuracy of gesture recognition; through the Mel frequency cepstral coefficient and other feature extraction techniques, the voice signal is deeply analyzed, so that the voice commands can be better understood and processed. At the same time, the feature vector data processing and classification of gestures also make gesture recognition more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The present invention is a flowchart of an intelligent remote controller provided by an embodiment of the present invention.

[0055] Figure 2 The present invention provides a voice template matching structure block diagram of an intelligent remote controller.

[0056] Figure 3 It is a structural block diagram of a gesture template matching of an intelligent remote controller provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in a form and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0058] like Figures 1 to 3 As shown, an embodiment of the present invention provides an intelligent remote controller, comprising:

[0059] The voice acquisition module 11 is used to capture the voice signal sent by the user and convert the voice signal into a digital signal;

[0060] An endpoint detection module 12, used to distinguish between speech segments and noise segments by calculating the spectrum entropy of the digital signal to obtain speech data for endpoint detection;

[0061] The preprocessing module 13 is used to perform pre-emphasis, frame division and windowing processing on the voice data detected by the endpoint to obtain pre-processed voice data;

[0062] The touch panel module 14, together with the voice acquisition module, constitutes an interactive multi-modal input, which is used to detect the touch position and output the corresponding coordinate value, and use the output coordinate value as the original data for gesture recognition;

[0063] The six-axis sensor module 15 is used to detect the motion data of the device, which complements the original data of gesture recognition to obtain gesture data;

[0064] The classification module 16 is used to extract features from the pre-processed speech data and classify and identify the gesture data to obtain speech feature data and gesture classification results;

[0065] The template matching module 17 is used to construct multiple groups of voice templates and gesture templates according to the voice feature data and the gesture classification results to obtain template matching data;

[0066] The wireless communication module 18 is used to wirelessly transmit the template matching data to the receiving end to realize intelligent remote control.

[0067] In the embodiment of the present invention, the voice acquisition module provides a convenient voice input method, and the digital conversion of the voice signal facilitates the subsequent processing and analysis, ensuring the accuracy and reliability of the signal; the endpoint detection module effectively improves the accuracy of voice recognition by distinguishing between voice segments and noise segments, and reduces the interference of background noise on the recognition results. Pre-emphasis, framing and windowing processing can optimize the voice signal and improve the sensitivity and accuracy of voice recognition; the touchpad provides an intuitive and natural interaction method, and the output coordinate value is used as the original data for gesture recognition; the six-axis sensor module can more accurately understand the user's gesture intention by detecting the motion state and direction information of the device. The data of the six-axis sensor provides an additional dimension for gesture recognition, enhancing the accuracy and flexibility of recognition; feature extraction and classification recognition technology enable accurate understanding of the user's voice and gesture commands, achieve efficient command parsing and execution, and improve intelligence and response speed; by constructing voice and gesture templates, the user's input can be quickly matched with predefined commands, thereby accelerating the recognition process; the wireless communication function enables the intelligent remote control to be seamlessly connected to the remote device, realizing convenient remote control.

[0068] In a preferred embodiment of the present invention, the voice collection module may include:

[0069] The voice acquisition module includes a microphone and an analog-to-digital converter. The microphone is used to capture the voice signal emitted by the user, and the analog-to-digital converter converts the analog voice signal into a digital signal.

[0070] In the embodiment of the present invention, the microphone can accurately capture the voice signal emitted by the user to ensure the integrity and clarity of the voice information; the analog-to-digital converter can convert the analog voice signal captured by the microphone into a digital signal. The digitized signal is easier to store, transmit and process, and can also reduce the attenuation and distortion of the signal during transmission. The digital signal has a stronger anti-interference ability than the analog signal. During the signal transmission process, the digital signal can better resist external noise and interference, and ensure the stability and reliability of the voice signal.

[0071] In a preferred embodiment of the present invention, distinguishing speech segments and noise segments by calculating the spectrum entropy of digital signals to obtain speech data for endpoint detection may include:

[0072] According to the speech signal, a frequency spectrum of the speech signal frame is obtained, and an amplitude spectrum is extracted from the frequency spectrum to obtain an amplitude spectrum;

[0073] Normalize the amplitude spectrum and pass To obtain the spectral entropy, where p(k0) is the proportion of each frequency component in the total energy of the signal, k0 is the total number of frequency components, ∈ is a positive number representing the smoothing parameter, γ(k0) is the weighting parameter of the frequency component, i is the index, and H is the spectral entropy;

[0074] According to the spectral entropy characteristics, the threshold is set by analyzing the pure noise signal, and the spectral entropy of each frame is traversed to distinguish the speech segment and the noise segment to obtain the speech data for endpoint detection. If the spectral entropy of a frame is lower than the set threshold, the frame is considered to belong to the speech segment; otherwise, the frame is considered to belong to the noise segment.

[0075] In the embodiment of the present invention, since the speech signal is non-stationary, it is divided into multiple short frames, and the length of each frame is usually between 10-30 milliseconds. The non-stationary speech signal is approximated as a stationary signal. After framing, a window function needs to be applied to each frame signal, such as the Hamming window applied in step 13; the signal after each frame of windowing is subjected to a fast Fourier transform to obtain a frequency spectrum representation of each frame signal. The result of FFT is a set of complex numbers, in which the real part represents the cosine component of the spectrum, and the imaginary part represents the sine component of the spectrum. The absolute value of the complex number represents the amplitude of the corresponding frequency component to obtain an amplitude spectrum.

[0076] The energy of each frequency component is calculated, and then the energy of each frequency component is divided by the total energy to obtain the proportion of each frequency component in the total energy of the signal. The spectral entropy is calculated using the normalized amplitude spectrum. The spectral entropy characteristics of the pure noise signal are analyzed, and the spectral entropy of multiple frames of the pure noise signal is calculated. The distribution characteristics of the spectral entropy, such as the mean and variance, are statistically obtained. A threshold is set according to the distribution characteristics. For each frame, its spectral entropy is calculated and compared with the set threshold. If the spectral entropy of a frame is lower than the set threshold, the frame is considered to be a speech segment. If the spectral entropy of a frame is higher than the set threshold, the frame is considered to be a noise segment.

[0077] In a preferred embodiment of the present invention, the speech data detected by the endpoint is pre-emphasized, framed and windowed to obtain pre-processed speech data, which may include:

[0078] According to H(z)=1-0.9375z -1 Design a first-order high-pass filter and process the endpoint detected speech data through y[n]=x[n]-0.9375×x[n-1] to obtain the pre-emphasized signal, where H(z) is the filter transfer function and z -1 is a delay operator, which indicates the delay of one sampling period of the signal; y[n] is the pre-emphasized signal of the filter, x[n] is the current sampling value of the original speech signal of the filter, x[n-1] is the previous sampling value of the original speech signal of the filter, and 0.9375 is the coefficient;

[0079] The pre-emphasized speech signal is divided into multiple short-time stable signal frames to obtain a framed signal; according to the characteristics of the speech signal, starting from the starting point of the speech signal, the segmentation is performed according to the determined frame length and frame shift, starting from the first sampling point, taking continuous frame length sampling points as the first frame, skipping frame shift sampling points, and taking continuous frame length sampling points from the next position as the second frame, and so on, until the end of the signal;

[0080] According to the framed signal, Windowing is performed to obtain preprocessed speech data, where 0≤n≤N-1, w[n] is the preprocessed speech data, representing the window value of the nth sample point; n is the index of the sample point, starting from 0 to the frame length N-1; and N is the total number of sample points in each frame.

[0081] In the embodiment of the present invention, the transfer function of the given filter is H(z)=1-0.9375z -1, the coefficient of the filter is 0.9375, and the differential equation of the filter is obtained: y[n] = x[n] - 0.9375 × x[n-1]. The speech signal x of the speech segment is initialized according to the differential equation. For each sampling point n from 1 to the signal length, the pre-emphasized signal y[n] of the filter at the current sampling point y[n] is calculated. If there is no previous sampling point for n = 0, x[0] can usually be used directly as y[0]. According to the pre-emphasized signal, the frame length L and frame shift S are determined, such as 256 sample points; frame shift, such as 128 sample points; starting from the starting point of the signal, the sample points x[0] to x[L-1] of the first frame are taken as the first frame, S sample points are skipped, and x[S] to x[S+L-1] are taken as the second frame. The above steps are repeated until the end of the signal to complete the framing. A Hamming window function is applied to each frame signal for windowing processing, and a window function formula is applied to each frame signal for windowing to complete the preprocessing of the speech signal of the speech segment.

[0082] In a preferred embodiment of the present invention, the touch panel module includes a touch panel sensor and a touch control chip; detecting the touch position and outputting the corresponding coordinate value, and using the output coordinate value as the original data of gesture recognition, may include:

[0083] The touch control chip obtains touch detection results by monitoring the capacitance changes on the touch panel sensor;

[0084] According to the touch detection result, the touch position is sampled to obtain an electrical signal, and the change of the electrical signal is mapped to a two-dimensional coordinate to obtain a coordinate value;

[0085] The main control chip performs feature recognition on the received coordinate value sequence to obtain raw data for gesture recognition.

[0086] In an embodiment of the present invention, a touch panel is usually composed of a plurality of capacitive sensors, and these sensors will change in capacitance when touched. A touch control chip periodically measures the capacitance of each capacitive sensor, compares the measured capacitance value with the baseline capacitance value, calculates the capacitance change, and determines whether a touch occurs. Once a touch event is detected, the touch control chip records the number or position of the touch position sensor. The touch panel is usually a two-dimensional grid, and the position of each sensor in the coordinate system is known. According to the grid position of the touch sensor, the touch event is mapped to the two-dimensional coordinate. For example, assuming that the touch panel is an M×N grid, the touch position of the sensor can be represented by the two-dimensional coordinate (x0, y0). If the grid coordinate of the sensor is (i0, j0), the sensor grid coordinate is converted into the screen coordinate, and a coordinate value sequence is obtained from the touch control chip. These coordinate values ​​are the positions of continuous touch events on the touch panel, and feature extraction is performed on the coordinate value sequence, for example, by Calculate the path length of the gesture, where xi and i is the coordinate of the i-th point on the touch path, x i+1 and i+1 is the coordinate of the i+1th point on the touch path; Calculate the path curvature of the gesture, the coordinates of the three points are (x i ,y i ),(x i+1 ,y i+1 ),(x i+2 ,y i+2 ) is the coordinate curvature of i, i+1 and i+2, through Speed ​​calculation: The path speed of the gesture, Δt is the time interval between two adjacent points, v i It is the speed from point i to point i+1. At the same time, statistical features are extracted according to the position distribution of the touch points, such as the path of the gesture. The direction of the gesture is extracted according to the initial and end positions of the touch points. The directions include front, back, left, and right, and the labeled gesture data is obtained. The labeled gesture data is used to train the support vector machine classifier, and the gesture features are customized to have different meanings. For example, the direction of the gesture is forward, which means going straight forward or going to the previous page; the gesture path is circular, which means canceling; the speed of the gesture increases, which means the device is accelerating or the volume is increasing.

[0087] In a preferred embodiment of the present invention, the six-axis sensor module includes a three-axis accelerometer and a three-axis gyroscope, and the motion data includes motion state and direction information, wherein the accelerometer is used to measure the acceleration change of the object in the three axes of X, Y, and Z, and the gyroscope is used to measure the angular velocity change of the object around these three axes; the motion data motion state and direction information of the detection device are complementary to the original data of gesture recognition to obtain gesture data, which may include:

[0088] The three-axis accelerometer can capture the motion status of the device in real time, including movement speed, acceleration and direction changes;

[0089] Get the device's rotation state and direction information through the gyroscope;

[0090] The gesture data is obtained by fusing the six-axis sensor data with the data source of gesture recognition.

[0091] In the embodiment of the present invention, the three-axis accelerometer usually has three sensors, which measure the acceleration in the x, y and z directions respectively. The sensors will generate acceleration data in real time, which are usually output in the form of acceleration values, such as (a x , a y , a z), which represents the acceleration in the x, y, and z directions; the gyroscope measures the rotation rate of the device around three orthogonal axes, and the angular velocity data it outputs can be expressed as (ω x ,ω y ,ω z ), where ω x ,ω y and ω z They are the angular velocities around the x, y, and z axes respectively. By integrating the angular velocities, the rotation angle and direction of the device can be accumulated.

[0092] According to the integration of acceleration data Calculate the instantaneous velocity of the device, where v(t) is the instantaneous velocity at time t, v0 is the initial velocity, and a(t) is the acceleration at time t; Calculate the instantaneous displacement of the device, where s(t) is the displacement at time t and s0 is the initial displacement; The rotation angle of the device is obtained, where θ(t) is the rotation angle at time t, θ0 is the initial angle, and ω(t) is the angular velocity; the data of the accelerometer and gyroscope are fused through the Kalman filter to achieve direction change fusion.

[0093] By calculating the velocity change rate to identify fast gestures, calculating the acceleration change rate to identify acceleration and deceleration gestures, and extracting posture information such as tilt angle and rotation angle from the fused data of the accelerometer and gyroscope, the dynamic and static features of the fused gesture are obtained.

[0094] In a preferred embodiment of the present invention, the gesture data is classified and identified to obtain voice feature data and gesture classification results, which may include:

[0095] Extract gesture features from gesture data, where the gesture features include distance, speed, acceleration, and direction change of the touch point;

[0096] Each gesture feature is converted into fixed-length feature vector data, segmented, and recursively constructed subtrees to obtain the gesture classification result.

[0097] In an embodiment of the present invention, the feature vector data set is divided into a training set and a test set based on the extracted features, such as distance, speed, acceleration, and direction change. Calculate the splitting metric and select the optimal feature for node splitting, where D is the entire dataset, c is the total number of categories in dataset D, and p i is the probability of the i-th class sample in the data set D, va(A) is all possible values ​​of feature A, D v is the data subset when feature A takes the value vv, |D v | is the data subset Dv The number of samples, |D v | is the number of samples in the entire data set D, p v,j is the data subset D v The probability of the jth class sample in ; Starting from the root node, split the data set into subsets according to the selected features, and recursively build subtrees for each subset. Each node of the tree contains a feature and its threshold, which is used to determine the split of the data, calculate the split metric for each feature, select the best feature, divide the data into different subsets according to the threshold of the selected feature, recursively create child nodes for each subset, and repeat the steps. If the preset tree depth is reached or there is insufficient data in the node, stop the recursion. Input the feature vector of the test data into the constructed decision tree, starting from the root node, traverse down along the branches of the tree according to the feature value until reaching the leaf node. The category label on the leaf node is the predicted gesture type.

[0098] In a preferred embodiment of the present invention, the gesture data is classified and identified to obtain voice feature data and gesture classification results, and further includes:

[0099] The preprocessed speech data is processed by MFCC(n)=DCT(log(Me(|FFT(x(t))| 2 ))) is used for feature extraction, where x(t) is the sample of the input speech signal in the time domain, and FFT(x(t)) is to apply the fast Fourier transform (FFT) to each frame signal x(t), converting the signal from the time domain to the frequency domain to obtain the frequency domain signal, |FFT(x(t))| 2 is the square of the amplitude of the frequency domain signal, indicating the energy of each frequency component, Me(|FFT(x(t))| 2 ) is to filter the power spectrum using a Mel filter bank to simulate the auditory perception of the human ear and obtain the Mel frequency power spectrum, log(Me(|FFT(x(t))| 2 )) takes the logarithm of the Mel-frequency power spectrum, and DCT applies discrete cosine transform (DCT) to the logarithmic Mel-frequency power spectrum, converting it from the frequency domain to the cepstrum domain to obtain the Mel-frequency cepstrum coefficients (MFCC).

[0100] In an embodiment of the present invention, after the frame is divided, features are extracted from each frame, and common features include Mel frequency cepstral coefficients (MFCC); a fast Fourier transform (FFT) is applied to each frame signal to convert the signal from the time domain to the frequency domain, and the power spectrum of each frame is calculated. The power spectrum is passed through a Mel filter group to simulate the auditory perception of the human ear, and the logarithm of the power spectrum output by the Mel filter group is taken, and DCT is applied to the logarithmic Mel power spectrum to obtain the Mel frequency cepstral coefficients.

[0101] In a preferred embodiment of the present invention, multiple groups of speech templates and gesture templates are constructed based on speech feature data and gesture classification results to obtain template matching data, which may include:

[0102] According to the speech feature data and the gesture classification results, a speech feature vector and a feature vector after gesture classification are obtained;

[0103] The initial codebook is constructed by randomly selecting feature vectors of the training set and iteratively optimizing to construct a template set;

[0104] According to the template set, the feature vector is mapped to a low-dimensional codebook space and compared with each template in the template set to obtain a comparison result;

[0105] The comparison result is matched with the control instruction to obtain template matching data.

[0106] In the embodiment of the present invention, features are extracted from the speech signal, the speech signal is divided into small frames, and the feature vector of each frame is calculated. The feature vector of each frame can be represented as a d-dimensional vector, where d is the feature dimension, such as MFCC may have 13 or 39 dimensions. At the same time, the feature vector of each gesture is obtained;

[0107] A portion of samples is randomly selected from the feature vector and used as the initial codeword of the codebook. Each codeword is a feature vector. Each feature vector in the training set is assigned to the codeword closest to it. According to the assignment result, the position of each codeword is updated to the average value of the feature vector assigned to it. The assignment and update steps are repeated until the position of the codeword is stable or the set number of iterations is reached. The final set of codewords constitutes the template set of the codebook. By calculating the distance between the feature vector and all codebook templates, for each feature vector, the distance to each template in the template set is calculated, and the minimum distance is found. The feature vector is matched with the position in the codebook to obtain the template matching result. Since the template is matched with the control instruction, the control instruction is generated according to the matching result to realize the generation of the control instruction by voice features or gesture features. For example, if the feature vector is closest to the template "turn left", the control instruction "turn left" is output.

[0108] In a preferred embodiment of the present invention, wirelessly transmitting the template matching data to the receiving end to realize intelligent remote control may include:

[0109] The wireless communication module obtains the processed data from the template matching module, and performs data packaging and modulation processing to achieve wireless transmission;

[0110] The receiving end receives the wireless signal, converts it into a baseband signal, and identifies the specific control instructions after demodulation and analysis;

[0111] According to the control instructions, perform corresponding operations to realize intelligent remote control.

[0112] In the embodiment of the present invention, the wireless communication module obtains data from the template matching module, organizes the data into a format suitable for wireless transmission, usually including header information, data payload and tail check code. The data payload part is encoded, and the encoded digital data is converted into an analog signal, and the signal is sent out through radio waves. The wireless receiving module receives the wireless signal sent from the transmitting end through the antenna, amplifies the received signal, restores the received modulated signal to a baseband signal, converts the baseband signal into a digital signal, decodes the decoded digital signal, restores the original data, extracts the control instruction from the decoded data, performs data parsing to obtain the specific control command, and identifies the control instruction through the parsed data.

[0113] A control method for an intelligent remote controller, the control method comprising the following steps:

[0114] The voice acquisition module captures the voice signal emitted by the user and converts the voice signal into a digital signal;

[0115] Calculating the spectrum entropy of the digital signal using an endpoint detection module to distinguish between speech segments and noise segments, thereby obtaining speech data for endpoint detection;

[0116] Pre-emphasize, frame and window the voice data detected by the endpoint to optimize the signal quality and obtain pre-processed voice data;

[0117] Detecting the user's touch position through the touch panel module and outputting corresponding coordinate values, and using the coordinate values ​​as raw data for gesture recognition;

[0118] The six-axis sensor module is used to detect the motion data, motion state and direction information of the device, which complements the original data of gesture recognition and is processed together to obtain gesture data;

[0119] In the classification module, the pre-processed speech data is subjected to feature extraction, and the gesture data is subjected to classification and recognition, so as to obtain speech feature data and gesture classification results respectively;

[0120] According to the extracted voice feature data and gesture classification results, multiple groups of voice templates and gesture templates are constructed in the template matching module, and the voice signal and gesture data to be recognized are matched with the templates to obtain template matching data;

[0121] The wireless communication module is used to wirelessly transmit the template matching data to the receiving end, and the receiving end performs corresponding control operations according to the received data, thereby realizing the intelligent remote control function.

[0122] In the embodiment of the present invention, through the precise processing of the voice acquisition module and the endpoint detection module, it is possible to effectively distinguish between voice segments and noise segments, optimize the quality of the voice signal, and thus improve the accuracy of voice recognition. The combination of the touchpad and the six-axis sensor allows users to perform more intuitive and flexible operations through touch and gestures. By processing and classifying voice and gesture data in real time, user instructions can be quickly identified, and fast and accurate control responses can be achieved through template matching, thereby improving the efficiency and user experience of intelligent remote control. The wireless communication module reduces the limitations of physical connections, making the remote control operation more flexible. Users can control at a longer distance or without line of sight restrictions, thereby improving the freedom and convenience of use.

[0123] It should be noted that this corresponds to the above method, and all implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0124] The embodiment of the present invention further provides a computing device, comprising: a processor, a memory storing a computer program, wherein when the computer program is executed by the processor, the method described above is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0125] The embodiment of the present invention also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the method described above. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0126] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An intelligent remote controller, characterized in that: include: A voice acquisition module is used to capture the voice signal sent by the user and convert the voice signal into a digital signal; An endpoint detection module, used to distinguish between speech segments and noise segments by calculating the spectrum entropy of digital signals to obtain speech data for endpoint detection; A preprocessing module, used for performing pre-emphasis, framing and windowing processing on the voice data detected by the endpoint to obtain preprocessed voice data; The touch panel module, together with the voice acquisition module, constitutes an interactive multi-modal input, which is used to detect the touch position and output the corresponding coordinate value, and the output coordinate value is used as the raw data for gesture recognition; The six-axis sensor module is used to detect the motion data of the device, which complements the raw data of gesture recognition to obtain gesture data; The classification module is used to extract features from the pre-processed speech data and classify and identify the gesture data to obtain speech feature data and gesture classification results; A template matching module is used to construct multiple groups of voice templates and gesture templates according to the voice feature data and gesture classification results to obtain template matching data; The wireless communication module is used to wirelessly transmit the template matching data to the receiving end to realize intelligent remote control.

2. The intelligent remote controller according to claim 1, characterized in that: The speech segment and the noise segment are distinguished by calculating the spectrum entropy of the digital signal to obtain the speech data for endpoint detection, including: According to the speech signal, a frequency spectrum of the speech signal frame is obtained, and an amplitude spectrum is extracted from the frequency spectrum to obtain an amplitude spectrum; Normalize the amplitude spectrum and calculate to obtain the spectrum entropy; According to the spectral entropy characteristics, the threshold is set by analyzing the pure noise signal, and the spectral entropy of each frame is traversed to distinguish the speech segment and the noise segment to obtain the speech data for endpoint detection. If the spectral entropy of a frame is lower than the set threshold, the frame is considered to belong to the speech segment; otherwise, the frame is considered to belong to the noise segment.

3. The intelligent remote controller according to claim 2, characterized in that: The voice data detected by the endpoint is pre-emphasized, framed and windowed to obtain pre-processed voice data, including: Design a first-order high-pass filter to process the speech data detected by endpoints to obtain a pre-emphasized signal; The pre-emphasized speech signal is divided into multiple short-time stable signal frames to obtain a framed signal; according to the characteristics of the speech signal, starting from the starting point of the speech signal, the segmentation is performed according to the determined frame length and frame shift, starting from the first sampling point, taking continuous frame length sampling points as the first frame, skipping frame shift sampling points, and taking continuous frame length sampling points from the next position as the second frame, and so on, until the end of the signal; According to the framed signal, windowing processing is performed to obtain pre-processed speech data.

4. The intelligent remote controller according to claim 3, characterized in that: The touchpad module includes a touchpad sensor and a touch control chip; detects the touch position and outputs the corresponding coordinate value, and uses the output coordinate value as the raw data for gesture recognition, including: The touch control chip obtains touch detection results by monitoring the capacitance changes on the touch panel sensor; According to the touch detection result, the touch position is sampled to obtain an electrical signal, and the change of the electrical signal is mapped to a two-dimensional coordinate to obtain a coordinate value; The main control chip performs feature recognition on the received coordinate value sequence to obtain raw data for gesture recognition.

5. The intelligent remote controller according to claim 4, characterized in that: The six-axis sensor module contains a three-axis accelerometer and a three-axis gyroscope. The motion data includes motion status and direction information. The accelerometer is used to measure the acceleration changes of the object in the three axes of X, Y, and Z, and the gyroscope is used to measure the angular velocity changes of the object around these three axes.

6. The intelligent remote controller according to claim 5, characterized in that: The motion data motion state and direction information of the detection device complements the original data of gesture recognition to obtain gesture data, including: The three-axis accelerometer can capture the motion status of the device in real time, including movement speed, acceleration and direction changes; Get the device's rotation state and direction information through the gyroscope; The gesture data is obtained by fusing the six-axis sensor data with the data source of gesture recognition.

7. The intelligent remote controller according to claim 6, characterized in that: The gesture data is classified and identified to obtain voice feature data and gesture classification results, including: Extract gesture features from gesture data, where the gesture features include distance, speed, acceleration, and direction change of the touch point; Each gesture feature is converted into fixed-length feature vector data, segmented, and recursively constructed subtrees to obtain the gesture classification result.

8. The intelligent remote controller according to claim 7, characterized in that: The template matching module constructs multiple groups of voice templates and gesture templates based on the voice feature data and gesture classification results to obtain template matching data, including: According to the speech feature data and the gesture classification results, a speech feature vector and a feature vector after gesture classification are obtained; The initial codebook is constructed by randomly selecting feature vectors of the training set and iteratively optimizing to construct a template set; According to the template set, the feature vector is mapped to a low-dimensional codebook space and compared with each template in the template set to obtain a comparison result; The comparison result is matched with the control instruction to obtain template matching data.

9. The intelligent remote controller according to claim 8, characterized in that: Wirelessly transmit the template matching data to the receiving end to achieve intelligent remote control, including: The wireless communication module obtains the processed data from the template matching module, and performs data packaging and modulation processing to achieve wireless transmission; The receiving end receives the wireless signal, converts it into a baseband signal, and identifies the specific control instructions after demodulation and analysis; According to the control instructions, perform corresponding operations to realize intelligent remote control.

10. A control method for an intelligent remote controller according to any one of claims 1 to 9, characterized in that: The control method comprises the following steps: The voice acquisition module captures the voice signal emitted by the user and converts the voice signal into a digital signal; Calculating the spectrum entropy of the digital signal using an endpoint detection module to distinguish between speech segments and noise segments, thereby obtaining speech data for endpoint detection; Pre-emphasize, frame and window the voice data detected by the endpoint to optimize the signal quality and obtain pre-processed voice data; Detecting the user's touch position through the touch panel module and outputting corresponding coordinate values, and using the coordinate values ​​as raw data for gesture recognition; The six-axis sensor module is used to detect the motion data, motion state and direction information of the device, which complements the original data of gesture recognition and is processed together to obtain gesture data; In the classification module, the pre-processed speech data is subjected to feature extraction, and the gesture data is subjected to classification and recognition, so as to obtain speech feature data and gesture classification results respectively; According to the extracted voice feature data and gesture classification results, multiple groups of voice templates and gesture templates are constructed in the template matching module, and the voice signal and gesture data to be recognized are matched with the templates to obtain template matching data; The wireless communication module is used to wirelessly transmit the template matching data to the receiving end, and the receiving end performs corresponding control operations according to the received data, thereby realizing the intelligent remote control function.

Images