A display voice interaction system and method

Abstract

The application relates to the field of display voice interaction, in particular to a display voice interaction system. Voice data in a real-time environment is acquired through a microphone array, the voice data is converted into text data by using an improved RNN-T voice recognition model, a timestamp and a confidence level are marked, and an initial operation instruction is obtained; historical dialogue data is acquired, the initial operation instruction and the historical dialogue data are associated, a context weight is dynamically adjusted through an incremental learning algorithm, a display is taken as a master node, a semantic protocol is established with smart home equipment, the target operation instruction is analyzed, is distributed to associated equipment and is synchronized with a context; and feedback logs in the display and the smart home equipment are analyzed. Settings of the equipment can be automatically adjusted, the display and the equipment are individually and intelligently controlled, and the use experience of a user is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent interaction, and in particular to a display voice interaction system and method. Background Technology

[0002] In current technologies, users have increasingly higher demands for the interactive experience of smart homes. Traditional smart home interaction methods mainly rely on single remote control or simple touch operations, which have many limitations. For example, display voice interaction systems are mostly based on single-turn dialogues or simple command responses, lacking the ability to dynamically integrate complex multi-turn dialogues, contextual relationships, and multimodal inputs (voice + touch / gesture). Although existing technologies improve multi-turn dialogue capabilities through historical conversation data, they have not solved the problem of real-time dynamic adaptation; and relying on Bluetooth and WiFi transmission, they have not achieved semantic coherence for cross-device collaboration. Summary of the Invention

[0003] The purpose of this invention is to solve the above-mentioned problems by designing a display voice interaction system and method.

[0004] To achieve the above objectives, the technical solution of the present invention further includes the following modules in the aforementioned display voice interaction system:

[0005] The voice data acquisition module is used to acquire voice data in the real-time environment through a microphone array, convert the voice data into text data using an improved RNN-T voice recognition model, and mark the timestamp and confidence level to obtain voice commands;

[0006] The multimodal fusion module is used to prioritize the voice commands, display touch operations, and user gestures, with voice commands having the highest weight and touch and gestures as auxiliary commands, to obtain the initial control command.

[0007] The dynamic perception module is used to acquire historical dialogue data, associate the initial control command with the historical dialogue data, and dynamically adjust the context weight through an incremental learning algorithm to obtain the target control command.

[0008] The device collaboration execution module is used to use the display as the master node, establish a semantic protocol with smart home devices, parse the target control commands, distribute them to associated devices, and synchronize the context.

[0009] The execution feedback module is used to analyze feedback logs from displays and smart home devices, record high-frequency user command periods and response preferences, predict potential needs through collaborative filtering algorithms, and interactively display them on the display.

[0010] Furthermore, in the aforementioned display voice interaction system, the voice data acquisition module includes the following sub-modules:

[0011] The data acquisition submodule is used to acquire real-time speech data from the environment through a microphone array and to extract Mel-frequency cepstral coefficient features from the acquired speech signals.

[0012] The feature extraction submodule is used to set the speech data as... ,in , , Indicates the number of microphones in the microphone array. Indicates the total collection time, from The extracted first Frame, number Each MFCC feature is ,in , , Indicates the total number of frames. The dimension representing the MFCC feature;

[0013] The model optimization submodule is used to add a CNN convolutional neural network to the RNN-T speech recognition model to extract local features from the speech data; let the convolutional kernel of the CNN layer be... , bias is The output after the convolution operation is The calculation formula is:

[0014] ;

[0015] in, It is an activation function.

[0016] Furthermore, in the aforementioned display voice interaction system, the voice data acquisition module further includes the following sub-modules:

[0017] The model optimization submodule is used to optimize the output of the CNN convolutional layers. The input is fed into the RNN layer of the RNN-T speech recognition model, and the final output of the encoder is... ,in This indicates that the LSTM in the RNN layer is at the [number]th ... The hidden state at any given moment;

[0018] The data generation submodule is used to generate the first text data using the prediction network in the RNN layer. Let the prediction network at time... The input is Hidden state is The calculation formula is: ;

[0019] The feature fusion submodule is used to fuse the encoder output based on the joint network. and the output of the prediction network By fusing the features, we can obtain the joint characteristics. ,in, Long Short-Term Memory (LSTM) is an abbreviation for Long Short-Term Memory Network, and its calculation formula is: ;

[0020] in, Indicates will and spliced ​​together, Represents the weight matrix. The bias vector is represented by the probability distribution of each word at each time step, which is calculated using the softmax function.

[0021] The instruction generation submodule is used to find the most likely text sequence from the probability distribution using Beam Search to obtain target text data, record the encoder time step corresponding to each word in the target text data, convert the time step into an actual timestamp based on the frame length and frame shift during feature extraction, and calculate the confidence level through the probability product of the decoding path to obtain the speech instruction.

[0022] Furthermore, in the aforementioned display voice interaction system, the multimodal fusion module includes the following sub-modules:

[0023] The operation classification submodule is used to classify the touch operation of the display according to the location, duration and swipe direction characteristics of the touch. It uses computer image recognition algorithms to analyze the user gesture images collected by the image depth sensor, identify the gesture type, and map the touch operation and gesture type to the control intention.

[0024] The priority judgment submodule is used to judge the priority of the voice command, the display touch operation and the user gesture. If the voice command and the control intention conflict within the same time period, it is judged as a conflict situation.

[0025] The instruction acquisition submodule is used to prioritize voice commands in case of conflict. If there are more than three conflicting voice commands, the system will combine historical commands and contextual information to make a judgment. If a voice command conflicts with other input methods, the voice command will be used to obtain the initial control command.

[0026] Furthermore, in the aforementioned display voice interaction system, the dynamic perception module includes the following units:

[0027] The instruction receiving unit is used to, upon receiving an initial control instruction, query relevant historical dialogue data from the database based on the user ID and time range, perform word segmentation on the initial control instruction and the historical dialogue content, split the sentence into individual words, label each word with part-of-speech tagging and entity recognition, and obtain the word segmentation result.

[0028] The similarity calculation unit is used to obtain common keywords in the initial control command and the historical dialogue based on the word segmentation results, and to calculate the semantic similarity between the initial control command and the historical dialogue using cosine similarity.

[0029] The instruction filtering unit is used to filter out historical dialogue data that are highly correlated with the initial control instruction based on a set correlation threshold, and obtain the prediction result.

[0030] The weighted fusion unit is used to calculate the loss between the predicted result and the actual result based on the current context weights and features, update the context weights according to the gradient of the loss function, and perform weighted fusion on the associated historical dialogue data and the initial control command based on the updated context weights to obtain the target control command.

[0031] Furthermore, in the aforementioned display voice interaction system, the device collaborative execution module includes the following units:

[0032] The protocol establishment unit is used to design a layered semantic protocol architecture with the smart home device, with the display as the master node, including at least the application layer, transport layer and physical layer.

[0033] The instruction parsing unit is used to define the data format for the target control specification and the status information fed back by the smart home device to the display, and to parse the target control instruction to obtain the device identification field, action parsing field and parameter extraction field;

[0034] The instruction execution unit is used to acquire smart home devices based on the device identification field, perform corresponding operations on the smart home devices based on the action parsing field and parameter extraction field, and record the execution status of the target control instruction, including instruction content, execution time and execution result.

[0035] Furthermore, in the aforementioned display voice interaction system, the execution feedback module includes the following units:

[0036] The feedback data collection unit is used to collect feedback logs from displays and smart home devices, including at least instruction execution time, instruction content, execution result, and device status;

[0037] Similarity calculation unit, used to assume existence individual users and Different instructions, then the user For instructions Frequency of use This indicates that cosine similarity is used to calculate the similarity between users. and users The similarity between them The calculation formula is: ;

[0038] in, Indicates user For instructions Frequency of use Indicates user With users The sum of the products of frequencies used in all instructions. and Representing users respectively ,user Use the square root of the sum of the squares of the frequencies used for all instructions;

[0039] Predictive calculation unit, used for target users Find the one with the highest similarity A set of neighboring users consists of 1 user group. For unused instructions Predict target users For instructions Potential demand score The calculation formula is: ;

[0040] in, Represents a set of neighboring users For target users The weighted sum of potential demand, with weights based on the similarity between users. ; Represents the set of neighboring users Mid-level users and target users The sum of similarities;

[0041] The demand display unit is used to convert the analysis results and predicted potential demands into natural language text, use TTS speech synthesis technology to convert the natural language text into speech output, and display the analysis results and potential demand predictions in a visual manner on the display screen.

[0042] Furthermore, in the above-described display voice interaction method, the display voice interaction method includes the following steps:

[0043] Voice data in real-time environment is acquired through a microphone array, and the voice data is converted into text data using an improved RNN-T speech recognition model. The text data is then labeled with timestamps and confidence levels to obtain voice commands.

[0044] The voice commands, display touch operations, and user gestures are prioritized, with voice commands having the highest weight and touch and gestures being auxiliary, to obtain the initial control command.

[0045] Acquire historical dialogue data, associate the initial control command with the historical dialogue data, and dynamically adjust the context weights through an incremental learning algorithm to obtain the target control command;

[0046] The display is used as the master node to establish a semantic protocol with smart home devices, parse the target control commands, distribute them to associated devices, and synchronize the context.

[0047] Analyze feedback logs from displays and smart home devices to record high-frequency user command periods and response preferences. Predict potential needs using collaborative filtering algorithms and then interactively display them on the displays.

[0048] Furthermore, in the above-mentioned display voice interaction method, the step of acquiring voice data in the real-time environment through a microphone array includes:

[0049] Real-time speech data is acquired using a microphone array, and the collected speech signals are then subjected to Mel-frequency cepstral coefficient feature extraction.

[0050] Let the voice data be ,in , , Indicates the number of microphones in the microphone array. Indicates the total collection time, from The extracted first Frame, number Each MFCC feature is ,in , , Indicates the total number of frames. The dimension representing the MFCC feature;

[0051] A CNN convolutional neural network is added to the RNN-T speech recognition model to extract local features from the speech data; let the convolutional kernel of the CNN layer be... , bias is The output after the convolution operation is The calculation formula is: ;

[0052] in, It is an activation function.

[0053] Furthermore, in the above-mentioned display voice interaction method, the step of converting the voice data into text data using an improved RNN-T speech recognition model, and marking it with timestamps and confidence levels to obtain voice commands includes:

[0054] The output of the CNN convolutional layer The input is fed into the RNN layer of the RNN-T speech recognition model, and the final output of the encoder is... ,in This indicates that the LSTM in the RNN layer is at the [number]th ... The hidden state at any given moment;

[0055] The first text data is generated using the prediction network in the RNN layer. Let the prediction network at time t be... The input is Hidden state is The calculation formula is: ;

[0056] The encoder output is based on the joint network. and the output of the prediction network By fusing the features, we can obtain the joint characteristics. ,in, Long Short-Term Memory (LSTM) is an abbreviation for Long Short-Term Memory Network, and its calculation formula is: ;

[0057] in, Indicates will and spliced ​​together, Represents the weight matrix. The bias vector is represented by the probability distribution of each word at each time step, which is calculated using the softmax function.

[0058] Beam Search is used to find the most likely text sequence from the probability distribution to obtain the target text data. The encoder time step corresponding to each word in the target text data is recorded. Based on the frame length and frame shift during feature extraction, the time step is converted into an actual timestamp. The confidence level is calculated by multiplying the probabilities of the decoding path to obtain the speech command.

[0059] Its beneficial effects are as follows: 1. More precise control through touch and gestures. Voice commands have the highest weight, ensuring that users can quickly issue commands via voice in most situations, improving operational efficiency; touch and gestures, as auxiliary methods, enrich the interaction and meet diverse user needs. 2. More accurately converting real-time voice data into text data. The addition of timestamps and confidence levels allows the system to more accurately assess the source and accuracy of voice commands. 3. Generating more user-specific control commands based on historical operating habits and current context information. 4. Accurately parsing target control commands and distributing them to associated devices while synchronizing context information. This enables better collaboration between different devices, improving the overall operating efficiency of the display system. 5. Anticipating user needs and providing more personalized services. Attached Figure Description

[0060] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0061] Figure 1 This is a schematic diagram of the first embodiment of a display voice interaction system according to the present invention;

[0062] Figure 2 This is a schematic diagram of a second embodiment of a display voice interaction system according to the present invention;

[0063] Figure 3 This is a schematic diagram of a third embodiment of a display voice interaction system according to the present invention;

[0064] Figure 4 This is a schematic diagram of the first embodiment of a display voice interaction method according to the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0066] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0067] The present invention will now be described in detail with reference to the accompanying drawings, such as... Figure 1 As shown, a display voice interaction system includes the following modules:

[0068] The voice data acquisition module is used to acquire voice data in the real-time environment through a microphone array, convert the voice data into text data using an improved RNN-T voice recognition model, and mark the timestamp and confidence level to obtain voice commands;

[0069] Specifically, this embodiment also includes,

[0070] The data acquisition submodule is used to acquire real-time speech data from the environment through a microphone array and to extract Mel-frequency cepstral coefficient features from the acquired speech signals.

[0071] The feature extraction submodule is used to set the speech data as... ,in , , Indicates the number of microphones in the microphone array. Indicates the total collection time, from The extracted first Frame, number Each MFCC feature is ,in , , Indicates the total number of frames. The dimension representing the MFCC feature;

[0072] The model optimization submodule is used to add a CNN convolutional neural network to the RNN-T speech recognition model to extract local features from the speech data; let the convolutional kernel of the CNN layer be... , bias is The output after the convolution operation is The calculation formula is: ;

[0073] in, It is an activation function.

[0074] The model optimization submodule is used to optimize the output of the CNN convolutional layers. The input is fed into the RNN layer of the RNN-T speech recognition model, and the final output of the encoder is... ,in This indicates that the LSTM in the RNN layer is at the [number]th ... Hidden state at any given moment; data

[0075] The generation submodule is used to generate the first text data using the prediction network in the RNN layer. Let the prediction network at time [time value missing]. The input is Hidden state is The calculation formula is: ;

[0076] The feature fusion submodule is used to fuse the encoder output based on the joint network. and the output of the prediction network By fusing the features, we can obtain the joint characteristics. ,in, Long Short-Term Memory (LSTM) is an abbreviation for Long Short-Term Memory Network, and its calculation formula is: ;

[0077] in, Indicates will and spliced ​​together, Represents the weight matrix. The bias vector is represented by the probability distribution of each word at each time step, which is calculated using the softmax function.

[0078] The instruction generation submodule is used to find the most likely text sequence from the probability distribution using Beam Search to obtain target text data, record the encoder time step corresponding to each word in the target text data, convert the time step into an actual timestamp based on the frame length and frame shift during feature extraction, and calculate the confidence level through the probability product of the decoding path to obtain the speech instruction.

[0079] The multimodal fusion module is used to prioritize the voice commands, display touch operations, and user gestures, with voice commands having the highest weight and touch and gestures as auxiliary commands, to obtain the initial control command.

[0080] Specifically, this embodiment also includes,

[0081] The operation classification submodule is used to classify the touch operation of the display according to the location, duration and swipe direction characteristics of the touch. It uses computer image recognition algorithms to analyze the user gesture images collected by the image depth sensor, identify the gesture type, and map the touch operation and gesture type to the control intention.

[0082] The priority judgment submodule is used to judge the priority of the voice command, the display touch operation and the user gesture. If the voice command and the control intention conflict within the same time period, it is judged as a conflict situation.

[0083] The instruction acquisition submodule is used to prioritize voice commands in case of conflict. If there are more than three conflicting voice commands, the system will combine historical commands and contextual information to make a judgment. If a voice command conflicts with other input methods, the voice command will be used to obtain the initial control command.

[0084] The dynamic perception module is used to acquire historical dialogue data, associate the initial control command with the historical dialogue data, and dynamically adjust the context weight through an incremental learning algorithm to obtain the target control command.

[0085] Specifically, this embodiment also includes an instruction receiving unit, which, after receiving the initial control instruction, queries relevant historical dialogue data from the database according to the user ID and time range, performs word segmentation on the initial control instruction and the historical dialogue content, splits the sentence into individual words, labels each word with part-of-speech and entity recognition, and obtains the word segmentation result;

[0086] The similarity calculation unit is used to obtain common keywords in the initial control command and the historical dialogue based on the word segmentation results, and to calculate the semantic similarity between the initial control command and the historical dialogue using cosine similarity.

[0087] The instruction filtering unit is used to filter out historical dialogue data that are highly correlated with the initial control instruction based on a set correlation threshold, and obtain the prediction result.

[0088] The weighted fusion unit is used to calculate the loss between the predicted result and the actual result based on the current context weights and features, update the context weights according to the gradient of the loss function, and perform weighted fusion on the associated historical dialogue data and the initial control command based on the updated context weights to obtain the target control command.

[0089] The device collaboration execution module is used to use the display as the master node, establish a semantic protocol with smart home devices, parse the target control commands, distribute them to associated devices, and synchronize the context.

[0090] Specifically, this embodiment also includes a protocol establishment unit, used to design a layered semantic protocol architecture with the smart home device, with the display as the master node, including at least the application layer, the transport layer and the physical layer;

[0091] The instruction parsing unit is used to define the data format for the target control specification and the status information fed back by the smart home device to the display, and to parse the target control instruction to obtain the device identification field, action parsing field and parameter extraction field;

[0092] The instruction execution unit is used to acquire smart home devices based on the device identification field, perform corresponding operations on the smart home devices based on the action parsing field and parameter extraction field, and record the execution status of the target control instruction, including instruction content, execution time and execution result.

[0093] The execution feedback module is used to analyze feedback logs from displays and smart home devices, record high-frequency user command periods and response preferences, predict potential needs through collaborative filtering algorithms, and interactively display them on the display.

[0094] Specifically, this embodiment also includes a feedback data collection unit, used to collect feedback logs from the display and smart home devices, including at least instruction execution time, instruction content, execution result and device status;

[0095] Similarity calculation unit, used to assume existence individual users and Different instructions, then the user For instructions Frequency of use This indicates that cosine similarity is used to calculate the similarity between users. and users The similarity between them The calculation formula is: ;

[0096] in, Indicates user For instructions Frequency of use Indicates user With users The sum of the products of frequencies used in all instructions. and Representing users respectively ,user Use the square root of the sum of the squares of the frequencies used for all instructions;

[0097] Predictive calculation unit, used for target users Find the one with the highest similarity A set of neighboring users consists of 1 user group. For unused instructions Predict target users For instructions Potential demand score The calculation formula is: ;

[0098] in, Represents a set of neighboring users For target users The weighted sum of potential demand, with weights based on the similarity between users. ; Represents the set of neighboring users Mid-level users and target users The sum of similarities;

[0099] The demand display unit is used to convert the analysis results and predicted potential demands into natural language text, use TTS speech synthesis technology to convert the natural language text into speech output, and display the analysis results and potential demand predictions in a visual manner on the display screen.

[0100] Its beneficial effects are as follows: 1. More precise control through touch and gestures. Voice commands have the highest weight, ensuring that users can quickly issue commands via voice in most situations, improving operational efficiency; touch and gestures, as auxiliary methods, enrich the interaction and meet diverse user needs. 2. More accurately converting real-time voice data into text data. The addition of timestamps and confidence levels allows the system to more accurately assess the source and accuracy of voice commands. 3. Generating more user-specific control commands based on historical operating habits and current context information. 4. Accurately parsing target control commands and distributing them to associated devices while synchronizing context information. This enables better collaboration between different devices, improving the overall operating efficiency of the smart home system. 5. Anticipating user needs and providing more personalized services.

[0101] In this embodiment, please refer to Figure 2 In a second embodiment of a display voice interaction system according to the present invention, the voice data acquisition module includes the following sub-modules:

[0102] The model optimization submodule is used to optimize the output of the CNN convolutional layers. The input is fed into the RNN layer of the RNN-T speech recognition model, and the final output of the encoder is... ,in This indicates that the LSTM in the RNN layer is at the [number]th ... The hidden state at any given moment;

[0103] The data generation submodule is used to generate the first text data using the prediction network in the RNN layer. Let the prediction network at time... The input is Hidden state is The calculation formula is: ;

[0104] The feature fusion submodule is used to fuse the encoder output based on the joint network. and the output of the prediction network By fusing the features, we can obtain the joint characteristics. ,in, Long Short-Term Memory (LSTM) is an abbreviation for Long Short-Term Memory Network, and its calculation formula is: ;

[0105] in, Indicates will and spliced ​​together, Represents the weight matrix. The bias vector is represented by the probability distribution of each word at each time step, which is calculated using the softmax function.

[0106] The instruction generation submodule is used to find the most likely text sequence from the probability distribution using Beam Search to obtain target text data, record the encoder time step corresponding to each word in the target text data, convert the time step into an actual timestamp based on the frame length and frame shift during feature extraction, and calculate the confidence level through the probability product of the decoding path to obtain the speech instruction.

[0107] Its beneficial effect lies in the fact that the system can more accurately assess the source and accuracy of voice commands. Timestamps can reveal the order of commands and better handle contextual relationships; while confidence levels help the system determine the reliability of commands, allowing for appropriate measures to be taken when confidence levels are low.

[0108] In this embodiment, please refer to Figure 3 A third embodiment of a display voice interaction system according to the present invention includes a dynamic perception module comprising the following sub-units:

[0109] The instruction receiving unit is used to, upon receiving an initial control instruction, query relevant historical dialogue data from the database based on the user ID and time range, perform word segmentation on the initial control instruction and the historical dialogue content, split the sentence into individual words, label each word with part-of-speech tagging and entity recognition, and obtain the word segmentation result.

[0110] The similarity calculation unit is used to obtain common keywords in the initial control command and the historical dialogue based on the word segmentation results, and to calculate the semantic similarity between the initial control command and the historical dialogue using cosine similarity.

[0111] The instruction filtering unit is used to filter out historical dialogue data that are highly correlated with the initial control instruction based on a set correlation threshold, and obtain the prediction result.

[0112] The weighted fusion unit is used to calculate the loss between the predicted result and the actual result based on the current context weights and features, update the context weights according to the gradient of the loss function, and perform weighted fusion on the associated historical dialogue data and the initial control command based on the updated context weights to obtain the target control command.

[0113] Its beneficial effects lie in automatically adjusting device settings based on the user's past operation records of smart home devices at different times, enabling personalized intelligent control of displays and devices. This not only improves the user experience but also better meets the user's needs in different scenarios, making the smart home system more intelligent and user-friendly.

[0114] The above describes a display voice interaction system provided by an embodiment of the present invention. The following describes a display voice interaction method according to an embodiment of the present invention. Please refer to [link / reference]. Figure 4 One embodiment of the display voice interaction method in this invention includes:

[0115] Step 401: Acquire real-time speech data from the environment using a microphone array, convert the speech data into text data using an improved RNN-T speech recognition model, and mark the timestamp and confidence level to obtain speech commands;

[0116] Step 402: Prioritize the voice commands, display touch operations, and user gestures, with voice commands having the highest weight and touch and gestures being auxiliary, to obtain the initial control commands;

[0117] Step 403: Obtain historical dialogue data, associate the initial control command with the historical dialogue data, and dynamically adjust the context weights through an incremental learning algorithm to obtain the target control command.

[0118] Step 404: Use the display as the master node to establish a semantic protocol with smart home devices, parse the target control commands, distribute them to associated devices, and synchronize the context.

[0119] Step 405: Analyze the feedback logs in the display and smart home devices, record the user's high-frequency command periods and response preferences, predict potential needs through collaborative filtering algorithms, and display them interactively on the display.

[0120] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A display voice interaction system, characterized by, The display voice interaction system comprises the following modules: A voice data acquisition module is configured to acquire voice data in a real-time environment through a microphone array, convert the voice data into text data using an improved RNN-T voice recognition model, and mark a timestamp and a confidence level to obtain a voice instruction; A multi-modal fusion module is configured to perform priority judgment on the voice instruction, display touch operation, and user gesture, wherein the voice instruction has the highest weight, and the touch and gesture are auxiliary, to obtain an initial control instruction; An instruction receiving unit is configured to, after receiving the initial control instruction, query relevant historical dialogue data from a database according to a user ID and a time range, perform a word segmentation operation on the initial control instruction and the historical dialogue content, split a sentence into individual words, mark a part of speech and entity recognition for each word, and obtain a word segmentation result; A similarity calculation unit is configured to, according to the word segmentation result, acquire common keywords in the initial control instruction and the historical dialogue, and calculate the semantic similarity between the initial control instruction and the historical dialogue using a cosine similarity; An instruction screening unit is configured to, according to a set correlation threshold, screen historical dialogue data having a high correlation with the initial control instruction to obtain a prediction result; A weighted fusion unit is configured to calculate the loss between the prediction result and an actual result according to a current context weight and a feature, update the context weight according to the gradient of a loss function, and perform weighted fusion on the associated historical dialogue data and the initial control instruction according to the updated context weight to obtain a target control instruction; A device coordination execution module is configured to take the display as a master node, establish a semantic protocol with smart home devices, analyze the target control instruction, distribute the target control instruction to associated devices, and synchronize the context; An execution feedback module is configured to analyze feedback logs in the display and the smart home devices, record a user high-frequency instruction period and a response preference, predict potential demands through a collaborative filtering algorithm, and display an interaction in the display.

2. A display voice interaction system as in claim 1, wherein, The voice data acquisition module comprises the following sub-modules: A data acquisition sub-module is configured to acquire voice data in a real-time environment through a microphone array, and perform Mel frequency cepstral coefficient feature extraction on the acquired voice signal; The feature extraction sub-module is configured to set the voice data as Wherein , , M represents the number of microphones in the microphone array, T represents the total duration of collection, The first frame extracted from the voice data is represented as The first MFCC feature is represented as Wherein , , , M represents the total number of frames, D represents the dimension of the MFCC feature; The model optimization sub-module is configured to add a CNN convolutional neural network in the RNN-T speech recognition model to perform local feature extraction on the speech data. , the bias is , the output after the convolution operation is , and the calculation formula is: ； wherein, is an activation function.

3. A display voice interaction system as in claim 1, wherein, The voice data acquisition module further comprises the following sub-modules: a model optimization sub-module, configured to input the output of the CNN convolutional layer into an RNN layer in an RNN-T speech recognition model, and encode the output of the CNN convolutional layer into a final output of the encoder , wherein , wherein represents a hidden state of an LSTM in the RNN layer at the t-th moment; and represents a hidden state of an LSTM in the RNN layer at the t-th moment. The data generation submodule is configured to generate first text data by using a prediction network in an RNN layer, wherein the prediction network takes as input at time t , and has a hidden state , and the calculation formula is: ； The feature fusion sub-module is configured to fuse the output of the encoder and the output of the prediction network based on a joint network to obtain joint features. wherein, LSTM represents an abbreviation of a long short-term memory network, and a calculation formula is as follows:​​​ ； wherein, represents concatenating and together, represents a weight matrix, represents a bias vector, and a probability distribution of each time step and each vocabulary is calculated by a softmax function; An instruction generation sub-module is configured to use Beam Search to find the most likely text sequence from a probability distribution to obtain target text data, record the encoder time step corresponding to each word in the target text data, convert the time step into an actual timestamp according to the frame length and frame shift during feature extraction, calculate the confidence level through the probability product of the decoding path, and obtain a voice instruction.

4. A display voice interaction system as in claim 1, wherein, The multi-modal fusion module comprises the following sub-modules: An operation classification sub-module is configured to classify display touch operations according to touch position, duration, and sliding direction features, analyze user gesture images collected by an image depth sensor using a computer image recognition algorithm, identify gesture types, and map touch operations and gesture types to control intentions; The priority judgment submodule is configured to judge the priority of the voice instruction, the display touch operation and the user gesture, and determine that a conflict exists if the voice instruction and the operation intention conflict with each other in the same time period; The instruction obtaining submodule is configured to, if the conflict exists, give priority to the voice instruction, and if there are more than three voice instructions that conflict with each other, combine historical instructions and context information to determine the voice instruction, and if the voice instruction conflicts with other input modes, use the voice instruction to obtain an initial operation instruction.

5. A display voice interaction system as in claim 1, wherein, The device cooperative execution module comprises the following units: The protocol establishment unit is configured to use the display as a master node and design a layered semantic protocol architecture with the smart home device, the architecture comprising at least an application layer, a transmission layer and a physical layer; The instruction analysis unit is configured to define a data format of a target operation instruction and state information feedback of the smart home device to the display, analyze the target operation instruction to obtain a device identification field, an action analysis field and a parameter extraction field; The instruction execution unit is configured to obtain the smart home device according to the device identification field, perform a corresponding operation on the smart home device according to the action analysis field and the parameter extraction field, and record execution of the target operation instruction, including instruction content, execution time and execution result.

6. A display voice interaction system as in claim 1, wherein, The execution feedback module comprises the following units: The feedback data collection unit is configured to collect feedback logs in the display and the smart home device, including at least instruction execution time, instruction content, execution result and device state; Similarity calculation unit, used to assume existence individual users and Different instructions, then the user For instructions Frequency of use This indicates that cosine similarity is used to calculate the similarity between users. and users The similarity between them The calculation formula is: ； wherein, represents the user the frequency of use of the instructions, represents the user the sum of the products of the frequency of use of the instructions by the user and respectively represent the user the square root of the sum of the squares of the frequency of use of the instructions by the user by all users.​​ a prediction calculation unit configured to predict a target user finds a user with the highest similarity thereto , denoted as a neighbor user set , the prediction calculation unit predicts the target user scores a potential demand of the target user for the instruction , and a calculation formula is ； wherein, denotes the sum of similarities between the target user and the users in the set of neighbor users to the target user a weighted sum of potential needs, the weights being the similarities between users ; denotes the set of neighbor users the sum of similarities between the target user and the users in the set of neighbor users ; The demand display unit is configured to convert analysis results and predicted potential demand into natural language text, convert the natural language text into voice output using a TTS voice synthesis technology, and display the analysis results and the potential demand prediction in a visual manner on the display.

7. A display voice interaction method, characterized in that, The display voice interaction method comprises the following steps: Obtaining voice data in a real-time environment through a microphone array, converting the voice data into text data using an improved RNN-T voice recognition model, and marking a timestamp and a confidence level to obtain a voice instruction; Judging the priority of the voice instruction, the display touch operation and the user gesture, wherein the voice instruction has the highest weight, and the touch and the gesture are auxiliary, to obtain an initial operation instruction; After receiving the initial operation instruction, querying relevant historical dialogue data from a database according to a user ID and a time range, performing a word segmentation operation on the initial operation instruction and the historical dialogue content to split sentences into individual words, annotating the word segmentation with a part of speech and entity recognition to obtain a word segmentation result; According to the word segmentation result, obtaining common keywords in the initial operation instruction and the historical dialogue, and calculating the semantic similarity between the initial operation instruction and the historical dialogue using a cosine similarity; According to a set correlation threshold, filtering out historical dialogue data with a high correlation degree to the initial operation instruction to obtain a prediction result; According to the current context weight and features, calculating the loss between the prediction result and an actual result, updating the context weight according to the gradient of the loss function, and performing weighted fusion on the correlated historical dialogue data and the initial operation instruction according to the updated context weight to obtain a target operation instruction. The display is taken as a master node, a semantic protocol is established with smart home devices, the target operation instruction is parsed, distributed to associated devices and context is synchronized; Feedback logs in the display and smart home devices are analyzed, user high-frequency instruction time periods and response preferences are recorded, potential demands are predicted through a collaborative filtering algorithm and interactive display is performed in the display.

8. A display voice interaction method according to claim 7, characterized in that, The voice data in the real-time environment is acquired through the microphone array, including: The voice data in the real-time environment is acquired through the microphone array, and mel-frequency cepstral coefficient feature extraction is performed on the collected voice signal; Let the voice data be wherein , , denotes the number of microphones in the microphone array, denotes the total duration of the collection, the i-th frame extracted from the total duration of the collection, the i-th MFCC feature is wherein , , , denotes the total number of frames, denotes the dimension of the MFCC feature; In the RNN-T speech recognition model, a CNN convolutional neural network is added to perform local feature extraction on the speech data; the convolution kernel of the CNN layer is , the bias is , the output after the convolution operation is , and the calculation formula is: ； wherein, is an activation function.

9. The display voice interaction method of claim 7, wherein, The voice data is converted into text data by using an improved RNN-T speech recognition model, and a timestamp and a confidence are marked to obtain a voice instruction, including: The output of the CNN convolutional layers is input into the RNN layer in the RNN-T speech recognition model, and the final output of the encoder is , where represents the hidden state of the LSTM at time step in the RNN layer; and is the output of the RNN layer. The first text data is generated using a prediction network in the RNN layer, where the prediction network takes as input at time , and has a hidden state , and the calculation formula is: ； fusing the output of the encoder and the output of the prediction network based on a joint network to obtain joint features wherein, LSTM stands for long short-term memory network, and the calculation formula is: ； wherein, represents concatenating and together, represents a weight matrix, represents a bias vector, and a probability distribution of each time step and each vocabulary is calculated by a softmax function; A target text data is obtained by using Beam Search to find the most possible text sequence from a probability distribution, an encoder time step corresponding to each vocabulary in the target text data is recorded, the time step is converted into an actual timestamp according to a frame length and a frame shift during feature extraction, and a confidence is calculated through a probability product of a decoding path to obtain a voice instruction.

Images