Multi-modal voice interaction method based on large model, electronic equipment and storage medium

Abstract

The invention provides a multi-mode voice interaction method based on a large model, electronic equipment and a storage medium, and the method comprises the steps: carrying out the voice recognition of the voice data of a first language inputted by a user, and obtaining a first language text; performing language translation on the first language text to translate the first language text into a second language text; performing semantic analysis on the second language text to obtain a semantic analysis result; performing multi-modal knowledge graph construction on the semantic analysis result based on a preset large model to obtain an enhanced semantic understanding result; wherein the multi-modal information related to the second language text is fused in the multi-modal knowledge graph; generating a dialogue strategy of the enhanced semantic understanding result; and performing voice synthesis based on the dialogue strategy to generate reply voice data expressed in the first language. According to the method and the device, multi-language interaction is realized, and meanwhile, various modal information is fused.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing, and in particular to a multimodal speech interaction method, electronic device and storage medium based on a large model. Background Art

[0002] Voice interaction technology has become one of the key technologies in the field of human-computer interaction, aiming to provide users with a more convenient, natural and efficient interaction experience.

[0003] In traditional voice interaction systems, they mainly rely on single-language speech recognition and simple text processing technology to understand the user's intentions and generate responses. However, this approach has many limitations. On the one hand, with the increasing frequency of cross-border communication and multicultural integration, users often need to communicate in different languages ​​in actual use scenarios. Single-language voice interaction systems cannot meet this demand, causing users to encounter language barriers when interacting with devices, greatly affecting the fluency and usability of the interaction.

[0004] On the other hand, existing voice interaction technologies usually process only voice text information, lacking effective integration and utilization of other related modal information. For example, in many practical situations, multimodal information such as images and videos related to voice content can provide rich context and supplementary information for accurately understanding user intentions, but traditional systems find it difficult to integrate these multimodal information into the interaction process, making the interaction process not smart and accurate enough to cope with the complex and changing real-world scenario requirements. Summary of the invention

[0005] The main purpose of the present invention is to provide a multimodal voice interaction method, electronic device and storage medium based on a large model, aiming to overcome the current defects of being unable to interact in multiple languages ​​and unable to integrate multiple modal information.

[0006] To achieve the above object, the present invention provides a multimodal voice interaction method based on a large model, comprising the following steps: Performing speech recognition on the speech data in the first language input by the user to obtain a text in the first language; Performing language translation on the first language text to translate it into a second language text; performing semantic analysis on the second language text to obtain a semantic analysis result; Based on a preset large model, a multimodal knowledge graph is constructed for the semantic analysis result to obtain an enhanced semantic understanding result; wherein the multimodal knowledge graph integrates multimodal information related to the second language text; Generating a dialogue strategy for enhancing the semantic understanding result; Speech synthesis is performed based on the dialogue strategy to generate reply speech data expressed in the first language.

[0007] Furthermore, generating the dialogue strategy for enhancing the semantic understanding result includes: Based on a preset intent classification model, the enhanced semantic understanding result is divided into a plurality of intent categories; wherein the intent classification model is obtained by pre-training; For each intent category, a corresponding dialogue strategy template is matched from a pre-stored dialogue strategy template library; the dialogue strategy template library is pre-built based on multi-language and multi-scenario dialogue data; By using natural language generation technology, the matched dialogue strategy template is filled and adjusted according to the specific content in the enhanced semantic understanding result to generate a personalized dialogue strategy.

[0008] Furthermore, a multimodal knowledge graph is constructed for the semantic analysis results based on a preset large model to obtain enhanced semantic understanding results, including: Encoding the semantic analysis result based on the text encoder in the large model to obtain a text vector; For multimodal information related to the second language text, using a corresponding image encoder and a video encoder to extract and encode features of the image data and the video data, respectively, to obtain an image feature vector and a video feature vector; The image feature vector and the video feature vector are fused into the text vector to construct an initial knowledge graph; wherein the text vector is a core node, and the fused multimodal features are used as peripheral nodes and edges to form an initial knowledge graph; The initial knowledge graph is optimized based on a semantic relationship model to obtain an enhanced semantic understanding result; wherein, the semantic relationship model is based on a multimodal corpus for unsupervised learning, can identify and supplement the semantic relationships in the knowledge graph, and improve the structure and content of the knowledge graph by continuously iteratively adjusting the connection relationship and semantic annotations between nodes.

[0009] Further, the image feature vector and the video feature vector are merged into the text vector, including: Through the cross-modal attention mechanism, the text vector interacts with the image feature vector and the video feature vector to calculate the association weights between different modalities; wherein the cross-modal attention mechanism is based on a multi-head attention architecture and captures the association information between different modalities by calculating multiple attention heads in parallel; According to the calculated association weight, the image feature vector and the video feature vector are fused into the text vector.

[0010] Further, after performing speech synthesis based on the dialogue strategy to generate reply speech data expressed in the first language, the method further includes: Generate a voice data encryption pattern based on the codes of the first language and the second language; generating a communication key based on the semantic analysis result; Based on the voice data encryption mode, the reply voice data is encrypted using the communication key and then fed back to the user's terminal.

[0011] Further, the generating of the voice data encryption mode based on the codes of the first language and the second language includes: Based on the language code mapping table, the codes of the first language and the second language are mapped into a unique first digital identifier and a unique second digital identifier respectively; Calculating a first difference and a second difference between the first digital identifier, the second digital identifier and a preset encryption base value respectively; Encrypting the sum of the first difference and the second difference based on a preset encryption algorithm to obtain an encrypted feature value; The encrypted feature value is binary decomposed, and analyzed in combination with the frequency and duration of the reply voice data, and the encrypted segmentation method and local encryption area are determined through a dynamic programming algorithm.

[0012] Further, generating a communication key based on the semantic analysis result includes: Extracting vocabulary feature vectors, grammatical feature vectors and semantic dependency feature vectors from the semantic analysis results; A graph structure is constructed based on each feature vector, wherein the vocabulary feature vector is used as a node, and the grammatical feature vector and the semantic dependency feature vector are used to determine the weight of the edge; Performing a depth-first traversal on the graph structure, and mapping the node sequence obtained by the traversal into a numerical sequence; Generate a polynomial curve using a numerical sequence, generate a key seed according to the coefficients of the polynomial curve and the control point information; hash the key seed using a hash function to obtain a hash value; The hash value is used as input to generate the communication key through a key stream generation algorithm based on a linear feedback shift register.

[0013] Further, generating a communication key based on the semantic analysis result includes: Encoding the semantic analysis result to obtain encoded characters; Inserting the coded characters into each node in the graph structure one by one in sequence, and assigning values ​​to the edges between adjacent nodes according to the relationship between the characters on the adjacent nodes, to obtain a character graph structure; Find all edges that do not have the same value as the target edges; connect the midpoints of each target edge in sequence to obtain multiple target lines; Searching for nodes in the character graph structure that satisfy a preset relationship with each target link as target nodes; Sequentially combine the characters on the target node as a key seed, and use a hash function to hash the key seed to obtain a hash value; The hash value is used as input to generate the communication key through a key stream generation algorithm based on a linear feedback shift register.

[0014] The present invention also provides a multimodal voice interaction device based on a large model, comprising: A recognition module, used for performing speech recognition on speech data in the first language input by a user to obtain a text in the first language; An analysis module, configured to perform language translation on the first language text into a second language text; and perform semantic analysis on the second language text to obtain a semantic analysis result; A construction module, configured to construct a multimodal knowledge graph for the semantic analysis result based on a preset large model to obtain an enhanced semantic understanding result; wherein the multimodal knowledge graph incorporates multimodal information related to the second language text; A generation module, used to generate a dialogue strategy for enhancing the semantic understanding result; A synthesis module is used to perform speech synthesis based on the dialogue strategy to generate reply speech data expressed in the first language.

[0015] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0016] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0017] The multimodal voice interaction method, electronic device and storage medium based on a large model provided by the present invention include: performing voice recognition on the voice data in the first language input by the user to obtain the first language text; performing language translation on the first language text to translate it into the second language text; performing semantic analysis on the second language text to obtain the semantic analysis result; constructing a multimodal knowledge graph based on the preset large model for the semantic analysis result to obtain an enhanced semantic understanding result; wherein the multimodal knowledge graph integrates the multimodal information related to the second language text; generating a dialogue strategy for the enhanced semantic understanding result; performing speech synthesis based on the dialogue strategy to generate reply voice data expressed in the first language. In the present invention, voice recognition and language translation are performed on the voice data in the first language input by the user to obtain the second language text, and then semantic analysis is performed to obtain the semantic analysis result, and a multimodal knowledge graph is constructed, and finally the reply voice data is generated. It realizes the interaction in multiple languages ​​and integrates multiple modal information at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the steps of a multimodal voice interaction method based on a large model in one embodiment of the present invention; Figure 2 It is a structural block diagram of a multimodal voice interaction device based on a large model in one embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present invention.

[0019] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0021] Reference Figure 1 In one embodiment of the present invention, a multimodal voice interaction method based on a large model is provided, comprising the following steps: Step S1, performing speech recognition on speech data in a first language input by a user to obtain a text in the first language; Step S2, performing language translation on the first language text to translate it into a second language text; performing semantic analysis on the second language text to obtain a semantic analysis result; Step S3, constructing a multimodal knowledge graph for the semantic analysis result based on a preset large model to obtain an enhanced semantic understanding result; wherein the multimodal knowledge graph integrates multimodal information related to the second language text; Step S4, generating a dialogue strategy for enhancing the semantic understanding result; Step S5: Perform speech synthesis based on the dialogue strategy to generate reply speech data expressed in the first language.

[0022] In this embodiment, as described in step S1 above, the voice information spoken by the user in the first language is converted into a corresponding text form so that it can be processed and analyzed more deeply later. It uses speech recognition technology, which is usually based on components such as acoustic models and language models. The acoustic model models and analyzes the acoustic features (such as phoneme features, etc.) in the speech signal to identify the phoneme sequence contained in the speech; the language model, based on the grammar, vocabulary and other rules of the language, combines the identified phoneme sequence to infer the most likely word combination, and finally forms a complete first language text.

[0023] As described in step S2 above, considering that in a cross-language communication scenario, it is necessary to convert the first language text input by the user into the target second language text in order to meet the interaction needs of users with different language backgrounds. This is often done with the help of machine translation technology, such as methods based on statistical machine translation or neural machine translation. Neural machine translation uses deep neural networks, such as the Transformer architecture, to learn a large amount of bilingual parallel corpus, which can automatically capture the mapping relationship between the two languages, and convert the vocabulary, grammatical structure and other information in the first language text into the corresponding expression form in the second language according to the learned pattern, so that the system can communicate and interact effectively with more users who use different languages.

[0024] Semantic analysis is to deeply understand the actual meaning conveyed by the translated second language text and to mine the semantic relationship, theme, intention and other information. This process may use technical means such as lexical analysis, syntactic analysis and semantic role labeling in natural language processing. Lexical analysis will perform operations such as part-of-speech tagging on words in the text; syntactic analysis constructs the grammatical structure tree of the sentence to clarify the relationship between the components; semantic role labeling further determines the semantic role played by each entity in the text, and combines this information to obtain semantic analysis results, such as determining whether a sentence is asking for information, expressing a request or stating a fact. Accurate semantic analysis is the key basis for the subsequent construction of multimodal knowledge graphs and the generation of appropriate dialogue strategies. Only by understanding the semantics of the text can we give a response that fits the user's intention in the interaction.

[0025] As described in step S3 above, the above-mentioned large model, with its powerful parameter scale and learning ability, can deeply mine the semantic analysis results and integrate related multimodal information (such as pictures, videos, audio and other modal contents) to construct a knowledge graph. During the construction process, the large model will first extract features of information of different modalities, such as extracting image features using a convolutional neural network, extracting audio features using an audio processing algorithm, etc., and then find the connection between text semantics and other modal information through a cross-modal association learning mechanism, and structure these associations in the form of a knowledge graph. For example, if the second language text mentions "giant panda", the large model can be associated with multimodal information such as stored pictures of giant pandas and videos introducing the living habits of giant pandas, and construct knowledge graph nodes around the theme of "giant pandas" and the relationship edges between them, thereby enhancing the understanding of text semantics. In this embodiment, the limitation of traditional voice interaction relying only on text information is broken, and the dimension of semantic understanding is enriched by making full use of multimodal information to make the system more accurate and comprehensive in grasping the user's intentions, thereby improving the quality and intelligence of the entire voice interaction.

[0026] As described in step S4 above, appropriate dialogue strategies are formulated based on the semantic understanding results after being enhanced by the multimodal knowledge graph. For example, if the semantic understanding results show that the user is asking about the function of a certain product, dialogue strategies such as detailed introduction of product functions and provision of relevant cases will be generated based on pre-stored rules or effective responses learned in similar scenarios in the past, so as to guide the dialogue to continue smoothly and meet the user's information needs.

[0027] As described in step S5 above, the text content corresponding to the previously generated dialogue strategy is converted back into speech form through speech synthesis technology, and expressed in the first language originally input by the user. Speech synthesis technology models the pronunciation of words, phrases, etc. in the text according to the phonetic rules of the language, and generates natural and fluent speech signals by splicing speech units or based on parameter synthesis. The complete closed loop from user voice input to system voice response is completed, and the interaction results are presented to the user in the form of voice, which conforms to the natural and convenient characteristics of voice interaction, allowing users to intuitively obtain the system's response content and achieve efficient human-computer interaction.

[0028] In the present invention, speech recognition and language translation are performed on the speech data in the first language input by the user to obtain the second language text, and then semantic analysis is performed to obtain the semantic analysis results, and a multimodal knowledge graph is constructed, and finally the reply speech data is generated. It realizes the interaction in multiple languages ​​and integrates multiple modal information at the same time.

[0029] In one embodiment, generating the dialogue strategy for enhancing the semantic understanding result includes: Based on a preset intent classification model, the enhanced semantic understanding result is divided into a plurality of intent categories; wherein the intent classification model is obtained by pre-training; For each intent category, a corresponding dialogue strategy template is matched from a pre-stored dialogue strategy template library; the dialogue strategy template library is pre-built based on multi-language and multi-scenario dialogue data; By using natural language generation technology, the matched dialogue strategy template is filled and adjusted according to the specific content in the enhanced semantic understanding result to generate a personalized dialogue strategy.

[0030] In this embodiment, the above-mentioned intent classification is to extract the core intent of the user from the semantic understanding results after the multimodal knowledge graph is enhanced, so as to accurately match the corresponding dialogue strategy in the future. The intent classification model is usually built based on machine learning or deep learning algorithms, such as support vector machines, convolutional neural networks (CNN) or recurrent neural networks (RNNS) in deep learning and their variants (such as LSTM, GRU, etc.). During the training phase, a large amount of multilingual and multi-scenario text data with intent labels will be collected as training samples. These samples cover various possible user expression intentions, such as asking for information, seeking help, expressing opinions, issuing instructions and other different categories. The model learns how to distinguish different intent categories by learning the mapping relationship between the vocabulary, grammar, semantics and other features of the text in these samples and the corresponding intent labels.

[0031] When faced with new enhanced semantic understanding results, the model can extract key features based on the learned patterns and classify them into corresponding intent categories.

[0032] The construction of the above-mentioned dialogue strategy template library is to reserve a general dialogue response framework for different intentions, different languages ​​and different scenarios in advance. During the construction process, a large amount of real dialogue data is collected, covering expressions in multiple languages ​​and various practical application scenarios (such as shopping, travel, social networking, etc.), and these data are analyzed and sorted to extract common and representative dialogue structures to form dialogue strategy templates. Each template corresponds to a specific intent category. After the intent category to which the enhanced semantic understanding result belongs is determined by the intent classification model, the template that matches it can be searched and matched from this pre-built library. In this embodiment, the efficiency and standardization of dialogue strategy generation are greatly improved. By using the pre-built template library, the tedious process of generating dialogue strategies from scratch each time is avoided, while ensuring that the response conforms to the common effective dialogue mode in structure and logic, laying the foundation for generating high-quality dialogue strategies.

[0033] Natural language generation technology aims to enable computers to generate text content that is fluent and accurate in human language habits. In this step, it will improve the matched dialogue strategy template based on the specific details in the enhanced semantic understanding results. For example, there may be some placeholders in the template that need to be filled according to the specific semantic content.

[0034] If the enhanced semantic understanding result mentions that a specific product is a "certain brand of smartphone", and the matching template contains placeholders for the product brand and type, the natural language generation technology will accurately fill in the corresponding information. At the same time, it will also make appropriate adjustments to the wording and sentence order of the entire template according to the context and semantic relationship to make it more natural and appropriate, and finally generate a personalized dialogue strategy, such as "This certain brand of smartphone has a high-definition screen, a large-capacity battery and other features. It can help you use it for a long time and get a clear visual experience." This enables the dialogue strategy to fit the user's specific expression and actual needs, and realize personalized replies. It avoids the sameness of the reply content, allowing users to feel that the system truly understands their intentions and gives targeted responses, thereby improving the intelligence and user satisfaction of voice interaction, and making the entire interaction process more natural and smooth.

[0035] In one embodiment, a multimodal knowledge graph is constructed for the semantic analysis result based on a preset large model to obtain an enhanced semantic understanding result, including: Encoding the semantic analysis result based on the text encoder in the large model to obtain a text vector; For multimodal information related to the second language text, using a corresponding image encoder and a video encoder to extract and encode features of the image data and the video data, respectively, to obtain an image feature vector and a video feature vector; The image feature vector and the video feature vector are fused into the text vector to construct an initial knowledge graph; wherein the text vector is a core node, and the fused multimodal features are used as peripheral nodes and edges to form an initial knowledge graph; The initial knowledge graph is optimized based on a semantic relationship model to obtain an enhanced semantic understanding result; wherein, the semantic relationship model is based on a multimodal corpus for unsupervised learning, can identify and supplement the semantic relationships in the knowledge graph, and improve the structure and content of the knowledge graph by continuously iteratively adjusting the connection relationship and semantic annotations between nodes.

[0036] In this embodiment, the text encoder plays a key role in vectorizing the text information of the semantic analysis results in the large model. Its principle is usually based on the word embedding technology in deep learning and more complex sequence encoding mechanisms, such as the encoding layer in the Transformer architecture.

[0037] Word embedding technology maps each word in the text to a low-dimensional vector space, so that words with similar semantics are closer in this vector space, so that the semantic features of the words can be captured. For the entire text sequence, the encoder will further consider the order relationship between words and the combination of grammar and semantics, and deeply encode the text through a multi-layer neural network structure, and finally convert the semantic analysis results into a text vector that can represent its overall semantics. The acquisition of text vectors is the basis for subsequent multimodal fusion and the construction of knowledge graphs. After converting text information into vector form, it is convenient for computers to perform numerical processing and analysis, so that it can interact with vector representations of other modalities in the same mathematical space, thereby laying the foundation for the integration of multimodal information. It is the first step to achieve multimodal fusion and directly affects the accuracy and effectiveness of the construction of the entire multimodal knowledge graph.

[0038] Image encoders and video encoders are designed to extract key features from image and video data and convert them into vector representations so that they can be fused with text vectors. Image encoders are often based on the convolutional neural network (CNN) architecture. CNN can automatically learn visual features such as texture, shape, and color in images through components such as convolutional layers and pooling layers. As the number of network layers increases, more abstract and representative high-level features are gradually extracted, and the entire image is finally compressed into an image feature vector.

[0039] The video encoder needs to consider the characteristics of the video data in the time dimension. In addition to using a CNN-like structure to extract features from each frame of the image, it also combines a recurrent neural network (RNN) or its variants (such as LSTM, GRU, etc.) to capture the temporal relationship between frames, thereby extracting the overall characteristics of the video and obtaining a video feature vector.

[0040] By extracting and encoding features from image and video data, we can achieve the conversion of different modal information into a unified vector representation, so that multimodal information can be effectively integrated with text information in subsequent steps. Only by converting each modal information into a vector form that can be compared and calculated with each other can we break down the modal barriers and integrate multimodal information together, providing the necessary conditions for building a comprehensive and rich multimodal knowledge graph, which helps to more fully understand the content related to the second language text.

[0041] The feature vectors of different modalities are fused and organized in a structured form in the form of a knowledge graph. The fusion process usually adopts some specific fusion strategies, such as the method based on the attention mechanism. The attention mechanism can dynamically calculate the association weights between the feature vectors of different modalities, and fuse the key information in the image feature vector and the video feature vector into the text vector based on these weights, highlighting the important associations between the modalities.

[0042] When constructing the initial knowledge graph, the text vector is used as the core node because the text carries the core content of the semantics, and the fused multimodal feature vector is used as the peripheral nodes and edges to represent the multimodal information such as images and videos related to the text and their relationships. The first integration of multimodal information is achieved. By constructing the initial knowledge graph, the originally scattered multimodal information such as text, images, and videos are associated in a structured way, and a knowledge system that can reflect multimodal semantic relationships is initially formed. This provides a basic framework for further optimizing and improving the knowledge graph to obtain enhanced semantic understanding results, and helps to explore the complementary and strengthening role of multimodal information in semantic understanding.

[0043] The purpose of the above-mentioned semantic relationship model is to deeply optimize the initially constructed knowledge graph so that it can more accurately and comprehensively reflect the semantic relationship between multimodal information. The model is based on unsupervised learning of multimodal corpora, which means that it automatically learns the semantic association patterns in a large amount of corpus containing multimodal data such as text, images, and videos, without the need for manual annotation of specific relationship types and other information. By using the semantic relationship model to improve the knowledge graph, more hidden and deep multimodal semantic relationships can be excavated to make up for the deficiencies that may exist in the initial construction process, so that the final knowledge graph can provide richer and more accurate information for semantic understanding, achieve the goal of enhancing semantic understanding, and enable the system to more accurately grasp user intentions and understand the meaning of text based on the multimodal knowledge graph, thereby improving the quality and effect of the entire multimodal voice interaction.

[0044] In one embodiment, fusing the image feature vector and the video feature vector into the text vector includes: Through the cross-modal attention mechanism, the text vector interacts with the image feature vector and the video feature vector to calculate the association weights between different modalities; wherein the cross-modal attention mechanism is based on a multi-head attention architecture and captures the association information between different modalities by calculating multiple attention heads in parallel; According to the calculated association weight, the image feature vector and the video feature vector are fused into the text vector.

[0045] In this embodiment, in multimodal information processing, text, images and videos carry information of different dimensions, and each exists in the form of vectors. In order to effectively integrate these different modal information, a mechanism is needed to measure the degree of correlation between them. The purpose of the cross-modal attention mechanism is to dynamically determine the importance weights of the correlation between the text vector and the image feature vector and the video feature vector, so that the multimodal information can be reasonably fused based on these weights in the future.

[0046] The above multi-head attention architecture is an advanced and efficient design. It captures the correlation information between different modalities by calculating multiple attention heads in parallel. Each attention head can be regarded as an independent "attention perspective" to analyze the correlation between text vectors and other modal vectors from different angles. Taking the multi-head attention mechanism in the common Transformer architecture as an example, it will first perform linear transformations on the input text vector, image feature vector, and video feature vector to obtain the corresponding query, key, and value representations. For each attention head, the similarity between the query and the key is calculated (usually using dot product and other methods) to obtain the attention score, which represents the degree of correlation between different modalities under the attention head. Then, by normalizing the attention score and multiplying and accumulating it with the corresponding value, the feature representation after attention and weighting by the attention head is obtained. Multiple attention heads perform such operations in parallel, each capturing the modal correlation information at different levels and angles. Finally, the results of these different attention heads are spliced ​​or fused to obtain comprehensive and rich inter-modal correlation weight information. For example, for a text vector describing "beach scenery" and the corresponding image feature vector showing the beach scene and video feature vector recording beach activities, different attention heads focus on the relationship between the "waves" mentioned in the text and the wave shape in the image, and the dynamics of the waves in the video, or the relationship between the "beach chairs" in the text and the placement of the beach chairs in the image, and the scene of people using beach chairs in the video, and then determine the corresponding association weights.

[0047] In this embodiment, the degree of association between different modalities in terms of semantics, content, etc. can be accurately calculated, avoiding the problem of simply fusing multimodal information while ignoring the internal connection between them. By mining related information from multiple angles through multiple attention heads, the complex relationship between modalities is more comprehensively reflected, providing a reliable data foundation for subsequent high-quality multimodal fusion. It is a key pre-step to achieve effective fusion of image and video multimodal feature vectors to text vectors, and directly determines whether the fusion effect can give full play to the advantages of multimodal information and improve the accuracy and richness of semantic understanding.

[0048] After obtaining the association weights between the text vector and the image feature vector and the video feature vector, it is necessary to fuse the image and video feature vectors into the text vector based on these weights to truly integrate multimodal information. The specific fusion operation is usually to perform weighted summation of each modal vector according to the weight. By fusing each dimension in this way according to the calculated association weights, the key information in the image and video feature vectors is integrated into the text vector, so that the text vector no longer only contains the semantic information of the text itself, but also incorporates the related image and video multimodal information, forming a new vector representation of comprehensive multimodal features. This step is the key finishing link of the entire multimodal fusion process. By fusing the features of different modalities according to reasonable association weights, the barriers between modalities are successfully broken, multimodal information is brought together, and a unified vector representation that can comprehensively reflect the content related to text, image, and video is constructed. This fused vector serves as the basis for further processing such as constructing a knowledge graph. It plays a vital role in enriching semantic understanding and improving the ability of multimodal voice interaction systems to grasp content. It enables the system to understand and process information related to user input from multiple angles and dimensions, thereby improving the quality and intelligence level of the entire voice interaction.

[0049] In one embodiment, after performing speech synthesis based on the dialogue strategy to generate reply speech data expressed in the first language, the method includes: Generate a voice data encryption pattern based on the codes of the first language and the second language; generating a communication key based on the semantic analysis result; Based on the voice data encryption mode, the reply voice data is encrypted using the communication key and then fed back to the user's terminal.

[0050] In this embodiment, during the multimodal voice interaction process, the generated reply voice data may contain user sensitive information or privacy content. In order to prevent these data from being illegally obtained or tampered with during transmission, they need to be encrypted. The encryption mode is generated based on the codes of the first language and the second language, taking into account the characteristics of different languages ​​(such as grammatical structure, vocabulary characteristics, etc.) and the language conversion involved in voice interaction, and a dedicated encryption method is customized for voice data to improve the pertinence and security of encryption.

[0051] First, the codes of the first language and the second language are usually specific codes or identifiers used to uniquely identify the two languages. These codes are used to determine the encryption mode in combination with pre-set rules or algorithms. For example, different language codes correspond to different encryption algorithm selections, encryption parameter settings, etc. A mapping relationship table can be constructed based on the language codes. After obtaining the codes of the first language and the second language, the corresponding encryption algorithm is found from the table. For example, the code combination corresponds to the use of the AES (Advanced Encryption Standard) algorithm for encryption, and the specific encryption mode details are determined according to the characteristics of the language. For example, for languages ​​with more complex grammatical structures, a segmented encryption mode may be selected to segment the voice data according to semantic units or grammatical structures and then encrypt them separately; for languages ​​with a high vocabulary richness, a local encryption method based on the frequency of specific words is determined to generate an encryption mode suitable for the voice data.

[0052] By combining language codes to generate encryption modes, we can fully consider the differences in the characteristics of different languages ​​in multilingual interactions, make the encryption method more in line with the actual situation of voice data, avoid the use of general encryption methods, effectively improve the security and effectiveness of voice data encryption, provide basic guarantees for the subsequent secure transmission of voice data, and ensure information security during voice interaction.

[0053] The communication key is a key element for encrypting and decrypting voice data. Relying only on fixed, universal keys is prone to security risks and cannot be closely associated with specific voice interaction content. Generating communication keys based on semantic analysis results can make the keys have characteristics related to the current voice interaction semantics, further improving the security of encryption, while also ensuring that the keys are consistent with the specific voice data content. Various valuable semantic information is extracted from the semantic analysis results, such as key subject words, semantic structure features, semantic roles, etc., and then these semantic information are converted into key materials through specific algorithms or models. This makes the encryption operation closely linked to the specific voice interaction semantics. Even if the attacker obtains part of the encrypted voice data, it is difficult to crack the original voice content without understanding the corresponding semantics and the generated key. At the same time, it also avoids the problems of traditional fixed keys being easily cracked and reused, resulting in reduced security. It provides reliable and personalized key protection for the encryption of voice data and enhances the performance of the entire voice interaction system in terms of communication security.

[0054] The first two steps determine the encryption mode and generate the communication key respectively. Then the two are combined to perform the actual encryption operation on the reply voice data, convert the original plaintext voice data into ciphertext, and then send the encrypted voice data to the user terminal. In the encryption process, according to the established encryption mode (such as segmented encryption, local encryption and other specific methods), the communication key is used to encrypt each part of the voice data through the corresponding encryption algorithm (such as AES in symmetric encryption algorithm or asymmetric encryption algorithm, depending on the specific encryption mode selection), and the characteristic information of the voice data (such as the numerical value corresponding to the audio waveform data) is transformed to make it a ciphertext content that cannot be directly recognized and understood, ensuring that even if it is intercepted during the transmission process, it is difficult for a third party to obtain the real information.

[0055] After encryption is completed, the encrypted voice data is sent according to the address and other information of the target terminal through network communication protocols and other means, so that it can be accurately transmitted to the user's terminal device, such as the user's smartphone, smart speaker and other terminals, waiting for the corresponding decryption operation at the receiving end to restore the voice data and realize secure voice interactive communication.

[0056] Through actual encryption operations and correct feedback transmission, the privacy and security of voice data during transmission from the system end to the user terminal are effectively protected, preventing security issues such as information leakage and tampering, allowing users to use multimodal voice interaction services with confidence, improving the reliability of the entire system and user trust.

[0057] In one embodiment, the generating of the voice data encryption mode based on the codes of the first language and the second language includes: Based on the language code mapping table, the codes of the first language and the second language are mapped into a unique first digital identifier and a unique second digital identifier respectively; Calculating a first difference and a second difference between the first digital identifier, the second digital identifier and a preset encryption base value respectively; Encrypting the sum of the first difference and the second difference based on a preset encryption algorithm to obtain an encrypted feature value; The encrypted feature value is binary decomposed, and analyzed in combination with the frequency and duration of the reply voice data, and the encrypted segmentation method and local encryption area are determined through a dynamic programming algorithm.

[0058] In this embodiment, in the scenario where multimodal voice interaction involves multiple languages, in order to generate a suitable voice data encryption mode based on language information, a standardized and unique way to characterize different languages ​​is first required. The language code mapping table is a pre-set correspondence set constructed for this purpose. It corresponds the code of each language (which can be a specific code, symbol, etc.) to a unique digital identifier. The principle of doing this is that numbers are easier to perform mathematical operations and follow established algorithmic logic in subsequent calculations and processing. Through mapping operations, language codes with abstract semantics are converted into digital forms that can be easily processed by computers, laying the foundation for the subsequent generation of encryption modes.

[0059] This mapping method enables different languages ​​to have a unified and standardized digital representation in the subsequent encryption mode generation process, which facilitates various mathematical operations and logical judgments, and avoids the inconvenience caused by directly processing language codes, which are relatively complex and difficult to directly participate in calculations. It is the starting step of the entire encryption mode generation based on language information. Subsequent steps rely on these accurate digital identifiers to ensure the orderliness and operability of the encryption mode generation process.

[0060] The preset encryption base value is a fixed value pre-set by the system. By calculating the difference between the digital identifiers corresponding to the first language and the second language and this encryption base value, a language-related variable factor can be introduced. The principle is that the digital identifiers mapped to different languages ​​are different, and the difference obtained after subtracting the encryption base value is also different. These differences can carry the characteristic information of the language itself, and in the subsequent encryption process, they serve as unique parameters to affect the generation of encryption modes, so that the encryption mode can be associated with a specific language, rather than using a universal, fixed method, which increases the pertinence and diversity of encryption.

[0061] By calculating the difference, the language differences represented by the language codes are cleverly converted into quantifiable numerical differences that can participate in subsequent encryption operations, so that the final generated encryption mode can vary due to different language combinations, better adapting to the personalized needs of voice data encryption in a multi-language environment, avoiding the security vulnerabilities that may exist in a single fixed encryption mode, and enhancing the security and flexibility of encryption.

[0062] After obtaining the first difference and the second difference based on language, they are summed and encrypted using a preset encryption algorithm, the purpose is to further integrate the language-related information and convert it into a confidential encrypted feature value through encryption operation. The encryption algorithm used here can be a common symmetric encryption algorithm (such as AES, etc.) or an asymmetric encryption algorithm (such as RSA, etc.), depending on the overall security design requirements of the system. The encryption process is to use the sum of the difference as input data according to the rules of the selected encryption algorithm, and convert it into an encrypted feature value in ciphertext form through the key in the encryption algorithm (this key is pre-configured by the system for encryption operations at this stage, which is different from the communication key used for subsequent voice data encryption) and a specific encryption transformation operation. The above-mentioned encrypted feature value not only integrates the information related to the first language and the second language (reflected by the difference), but also has confidentiality after encryption processing. It becomes an important intermediate data for further determining the specific encryption method of voice data (such as segmentation method, local encryption area, etc.), and plays a key role in connecting the upper and lower parts of the entire encryption mode generation chain.

[0063] On the one hand, the language-related difference information is encrypted and integrated, so that the language characteristics can be included in the subsequent process in a secure, encrypted form; on the other hand, a unified, encrypted key data foundation is provided for the final determination of the specific details of voice data encryption, ensuring that the process from generating intermediate data based on language to determining the actual encryption method is safe and logically coherent, and ensuring the integrity and security of encryption mode generation.

[0064] The purpose of binary decomposition of the encrypted eigenvalue is to convert it into a binary form that can be more conveniently analyzed and processed by the computer bit by bit, so as to extract information that can be used to guide the specific encryption operation of the voice data. The analysis combined with the frequency and duration information of the reply voice data itself takes into account the characteristics of voice data in terms of acoustic characteristics. Different frequency bands and duration intervals may have different requirements and sensitivities for encryption. The dynamic programming algorithm plays a role here. It is based on the principle of an optimal substructure. By decomposing the problem into multiple sub-problems and recording the optimal solution of each sub-problem, it finally finds the optimal solution to the entire problem. In this scenario, the dynamic programming algorithm is used to comprehensively consider the language-related encryption information contained in the encrypted eigenvalue after binary decomposition and the frequency and duration characteristics of the voice data to determine how to segment the voice data and delineate the local encryption area, so as to achieve the purpose of making full use of language-related information to ensure the pertinence of encryption and combining the characteristics of the voice itself to achieve efficient and reasonable encryption.

[0065] For example, the combination of certain bits of the encrypted eigenvalue after binary decomposition can indicate that the voice data should be segmented according to frequency, with different encryption strengths or encryption algorithm details used for the low-frequency band, medium-frequency band and high-frequency band respectively; at the same time, based on the voice duration and other information in the encrypted eigenvalue, a dynamic programming algorithm is used to calculate which specific time periods of the voice data are more appropriate for local encryption, such as focusing on encryption of time periods containing key semantic information, thereby achieving an encryption mode that is more in line with the actual situation of the voice data and language characteristics.

[0066] The above steps ultimately determine the specific implementation method for voice data encryption. By combining various factors, especially taking into full consideration the language-related information and the acoustic characteristics of the voice data itself, a personalized encryption scheme is generated, which avoids the problem of over-encryption (affecting efficiency) or insufficient encryption (posing security risks) of voice data caused by general encryption methods. The encryption mode can maximize the security and transmission efficiency of voice data in a multi-language and multi-modal voice interaction environment. It is the key finishing link in the entire process of generating voice data encryption mode based on language codes, and is directly related to the actual effect of voice data encryption.

[0067] In one embodiment, generating a communication key based on the semantic analysis result includes: Extracting vocabulary feature vectors, grammatical feature vectors and semantic dependency feature vectors from the semantic analysis results; A graph structure is constructed based on each feature vector, wherein the vocabulary feature vector is used as a node, and the grammatical feature vector and the semantic dependency feature vector are used to determine the weight of the edge; Performing a depth-first traversal on the graph structure, and mapping the node sequence obtained by the traversal into a numerical sequence; Generate a polynomial curve using a numerical sequence, generate a key seed according to the coefficients of the polynomial curve and the control point information; hash the key seed using a hash function to obtain a hash value; The hash value is used as input to generate the communication key through a key stream generation algorithm based on a linear feedback shift register.

[0068] In this embodiment, the semantic analysis result contains rich language information. In order to generate a communication key that is closely related to the specific semantics and has sufficient randomness and security, it is necessary to extract features from different angles. The vocabulary feature vector aims to capture the characteristics of the vocabulary level in the text, such as the word class, word frequency, semantic category and other information of the vocabulary. Through a specific word vector representation method, each word is converted into a vector representation. These vectors are combined to form a vocabulary feature vector, which can reflect the characteristics of the text in terms of vocabulary use. The grammatical feature vector focuses on the grammatical structure of the text, such as the syntactic structure type of the sentence (subject, predicate, object, subject, copula, etc.), the order of arrangement of grammatical components, the complexity of grammatical rules, etc., and grammatical related information is quantified into a vector form through grammatical analysis tools and corresponding encoding methods to reflect the characteristics of the text grammatical level. The semantic dependency feature vector is mainly used to characterize the dependency relationship between different semantic components in the text, such as which words are the executors of the action and which are the recipients. The relationship between each semantic unit is determined by semantic dependency analysis technology and converted into a vector representation, thereby mining the deep semantic structure characteristics of the text.

[0069] The above three feature vectors quantify the results of semantic analysis from the three key dimensions of vocabulary, grammar, and semantics, providing rich and multi-faceted basic materials for the subsequent construction of graph structure and generation of communication keys, so that the finally generated communication keys can fully integrate the language characteristics of the text, avoiding the problems of weak correlation and insufficient randomness that may be caused by generating keys based on only a single dimension. It is an important starting step in the transformation from semantics to computable features in the entire communication key generation process.

[0070] The purpose of constructing the extracted different feature vectors into a graph structure is to present the internal connection between the language elements in the semantic analysis results in an intuitive and easy-to-process way. Data structures such as adjacency matrix or adjacency list in graph theory can be used to realize the construction of the graph. Taking the vocabulary feature vector as a node means that each word has a corresponding node entity in the graph, and the grammatical feature vector and the semantic dependency feature vector are used to determine the weight of the edge. This is because the grammatical rules determine the degree of connection between words in the sentence structure, and the semantic dependency reflects the strength of the association between words at the semantic level. By combining these two feature vectors to set the weight of the edge, it can accurately reflect the association between different words based on grammar and semantics. For example, if two words are grammatically in a direct subject-predicate relationship and have a close semantic dependency (such as "bird" and "sing"), then the weight of the edge between them will be relatively high, otherwise it will be low. The graph structure constructed in this way is like a semantic relationship network, which clearly shows the language logic structure inside the text.

[0071] By constructing a graph structure, the feature vectors of different dimensions extracted previously are effectively integrated into an organic whole, which more intuitively displays the language logical relationship in the semantic analysis results. This provides a clear structural foundation for further mining and utilizing these semantic association information to generate communication keys through subsequent operations such as graph traversal, and is a key step in converting scattered feature vectors into semantic relationship carriers that can be used for key generation.

[0072] Depth-first traversal is a common traversal algorithm in graph theory. It is chosen to traverse the constructed graph structure in order to access the nodes in the graph (i.e., the words corresponding to the vocabulary feature vectors) in a specific order, so as to mine the information of the semantic association order embodied in the graph structure. During the traversal process, the nodes are recorded in the order of access to form a node sequence, which carries the order information of the text semantics according to the logical relationship in the graph. Then this node sequence is mapped to a numerical sequence, which can usually be mapped to a pre-set encoding rule, such as assigning a unique digital code to each word corresponding to the vocabulary feature vector (the encoding value can be determined based on the vocabulary order, word frequency ranking, etc.), and combining the corresponding codes according to the order of the words in the node sequence to obtain a numerical sequence. The purpose of this is to further convert the semantic order information embodied in the graph structure into a numerical form that is convenient for subsequent mathematical processing, in preparation for the generation of keys.

[0073] Through depth-first traversal and mapping operations, semantic features that integrate grammatical and semantic associations are extracted from the graph structure in an orderly manner and converted into numerical sequences, realizing the transformation from semantic logical structure to numerical representation that can be used for mathematical operations. This makes it possible to subsequently use this numerical information to generate keys based on mathematical methods. This is an important intermediate step to further quantify and computate semantic information to serve the generation of communication keys.

[0074] Using numerical sequences to generate polynomial curves is to further explore the hidden rules and features in numerical sequences with the help of the characteristics of mathematical functions. According to the numerical values ​​in the numerical sequence as the sampling points of the polynomial curve, the coefficients of the polynomial are determined by fitting and other mathematical methods. For example, a curve fitting algorithm such as the least squares method is used to find a polynomial function that can pass through these sampling points well. The coefficients of the polynomial contain the key feature information of the numerical sequence. At the same time, some key control points on the polynomial curve (such as extreme points, inflection points, etc.) also carry unique information. The key seed is generated by combining these coefficients and control point information, so that the key seed is closely related to the numerical sequence converted from the semantic analysis results, that is, it is related to the semantics of the text. Then, the key seed is hashed using a hash function. The hash function has the characteristics of one-way and anti-collision. By performing a hash operation on the key seed, it is converted into a fixed-length hash value. This hash value has high randomness and confidentiality, while retaining the association with the original key seed (and then with the semantic analysis results), providing a relatively safe and suitable intermediate value for the final generation of the communication key.

[0075] Linear Feedback Shift Register (LFSR) is a sequence cipher generation mechanism commonly used to generate key streams. It is based on the principle of shift registers. It shifts and feeds back the initial value (i.e., the hash value obtained previously) in the register through feedback logic to continuously generate a new bit sequence, which is the key stream. In practical applications, a sequence of appropriate length and format will be extracted from the key stream according to certain rules (such as grouping) as the final communication key. The principle is that LFSR can use the slight difference in the initial value to generate a key stream with good randomness and a long period. Here, the hash value obtained through the previous multiple steps of semantic-related processing is used as the initial value, so that the generated communication key has both semantic-based association and inherits the security characteristics such as randomness and periodicity brought by LFSR, meeting the security and confidentiality requirements of the key during the communication process.

[0076] By utilizing the LFSR key stream generation algorithm, the hash value obtained through the previous complex processing is converted into the final usable communication key, so that the generated communication key combines the semantic features of the semantic analysis results and good security characteristics. It can effectively encrypt voice data, etc., and ensure communication security in the process of multimodal voice interaction. It is the key finishing link of the entire communication key generation technical solution.

[0077] In one embodiment, generating a communication key based on the semantic analysis result includes: Encoding the semantic analysis result to obtain encoded characters; Inserting the coded characters into each node in the graph structure one by one in sequence, and assigning values ​​to the edges between adjacent nodes according to the relationship between the characters on the adjacent nodes, to obtain a character graph structure; Find all edges that do not have the same value as the target edges; connect the midpoints of each target edge in sequence to obtain multiple target lines; Searching for nodes in the character graph structure that satisfy a preset relationship with each target link as target nodes; Sequentially combine the characters on the target node as a key seed, and use a hash function to hash the key seed to obtain a hash value; The hash value is used as input to generate the communication key through a key stream generation algorithm based on a linear feedback shift register.

[0078] In this embodiment, the semantic analysis result is usually a natural language content presented in the form of text. In order to facilitate subsequent processing in a specific mathematical model such as a graph structure and follow the established algorithm logic, it needs to be converted into a standardized and easy-to-operate encoding form, that is, encoded characters. This encoding process can be implemented based on a variety of methods, such as common character encoding standards (such as ASCII code, Unicode, etc.), or encoding rules designed according to specific application scenarios. Each character, word or semantic unit in the text is converted into a corresponding encoding value according to a certain mapping relationship. The character form corresponding to these encoding values ​​is the encoded character. The advantage of this is that the relatively abstract and complex language information in the semantic analysis results is converted into a discrete symbol form that can be easily recognized and processed by the computer, providing basic materials for subsequent operations such as building a graph structure.

[0079] The purpose of inserting coded characters into the nodes of the graph structure is to organize and present the information in the semantic analysis results in a structured way. The graph structure can well reflect the relationship between elements. Here, the nodes are used to carry the semantic elements represented by the coded characters. The assignment of values ​​to the edges between adjacent nodes based on the relationship between the characters on the adjacent nodes is to further explore and quantify the degree of association between these semantic elements. This relationship can be measured from multiple perspectives, such as the similarity of characters (which can be judged by the numerical difference of character encoding, the grammatical semantic association of characters, etc.), the order of characters in the text, etc. According to these relationships, the corresponding values ​​are assigned to the edges through preset assignment rules (for example, the higher the similarity, the greater the assignment, and the order of adjacent assignments is a specific value, etc.). The character graph structure constructed in this way is like a semantic network that integrates semantic encoding information and semantic element association information. Each node carries the encoded semantic part, and the assignment of edges reflects the closeness or other association characteristics between them.

[0080] In this embodiment, the encoded semantic information is effectively integrated into the graph structure, and the relationship between semantic elements is quantified through the assignment of edges, so that the entire structure not only contains the semantics itself, but also reflects its internal logical connection, providing a clear structural basis for subsequent operations such as finding edges and nodes with specific properties. It is a key link in further mining the semantics that can be used to generate communication key information, allowing subsequent steps to perform targeted processing based on this character graph structure that integrates multiple aspects of information.

[0081] Furthermore, searching for edges that do not have the same value as the target edges is to filter out those edges with unique association characteristics from the character graph structure. These edges represent the unique connection relationship between semantic elements, which is of great value for generating random and unique communication keys. Connecting the midpoints of these target edges to obtain the target connection is to further explore the potential connection between these unique edges, and to construct a new structural clue through the connection. This clue can connect some elements in the graph in a new way based on the unique semantic association represented by the target edge, and it is possible to discover rules or features that are more conducive to generating keys.

[0082] For example, in the constructed character graph structure, if there are several edges with different values ​​such as 3, 4, 5, etc. (and there are no edges with the same value), the edges with different values ​​such as 3 and 4 are selected as target edges, and then the midpoint of each target edge is found (the midpoint position can be determined by geometric methods such as calculating the average value of the coordinates of the nodes at both ends of the edge). These midpoints are connected in order (for example, according to the natural order of the edges in the graph or the order after sorting based on the size of the assigned values), and multiple target lines are obtained. These lines form a new geometric structure in the graph, which contains new information based on unique edge relationships.

[0083] By screening target edges and constructing target connections, unique associated information is extracted from the numerous edges and relationships in the character graph structure. This screened and reorganized information becomes the key basis for the subsequent generation of communication keys with high discrimination and good randomness. It helps to improve the close correlation between communication keys and semantic analysis results, as well as the security and unpredictability of the keys themselves. It is an important step to further explore key information that can be used for key generation based on the character graph structure.

[0084] The above preset relationship is a matching rule between a node and a target line that is pre-set according to the needs of generating communication keys and the semantic association characteristics carried by the character graph structure. By finding nodes that satisfy this preset relationship with each target line, the purpose is to focus on those semantic elements (carried by nodes) that are closely related to the unique lines previously mined. For example, the preset relationship can be set as the vertical distance from the node to the target line is within a certain range (measuring the association from a geometric perspective), or the coded characters represented by the node and the coded characters of the nodes at both ends of the target line have a certain semantic dependency, etc. The target nodes selected according to such rules concentrate the semantic information closely related to the unique semantic associations previously mined through the target edges and target lines, and become the key source for generating key seeds.

[0085] When the target connection has been determined, for each node in the character graph structure, the distance from the node to each target connection is calculated according to the preset distance relationship rules. If the distance from a node to a target connection meets the set range requirements, then this node is determined as the target node. By performing such screening and judgment on all nodes in the entire character graph structure, all target nodes that meet the requirements are found.

[0086] By accurately locating the target node based on the preset relationship, semantic elements closely related to the unique association structure previously mined are extracted from the character graph structure, so that the subsequently generated key seeds can integrate the key semantic information that has been screened and mined in multiple rounds in the character graph structure to the greatest extent, ensuring that the communication key has a deep connection with the semantic analysis results, further improving the key's pertinence and security, and providing core semantic material for generating high-quality communication keys.

[0087] The characters on the target nodes are combined into key seeds in sequence because these target nodes carry key semantic information mined and screened from the semantic analysis results through multiple steps. Combining them forms a character sequence closely related to semantics, which is used as the key seed, so that this seed has the uniqueness and relevance derived from the semantic analysis results. Then the key seed is hashed using a hash function. The characteristics of the hash function, such as unidirectionality and anti-collision, can convert the key seed into a hash value of a fixed length with high randomness and confidentiality. This not only retains the relevance to semantics (because the hash value comes from the key seed generated based on semantics), but also gives it security through the hash function, providing a suitable intermediate value for the subsequent generation of the final communication key.

[0088] The key stream generation algorithm of the linear feedback shift register (LFSR) is based on the principle of the shift register. The initial value (the above hash value) in the register is shifted and fed back through a specific feedback logic to continuously generate a new bit sequence, which is the key stream. In practical applications, a sequence of appropriate length and format is extracted from the key stream according to certain rules (such as grouping, etc.) as the final communication key. The principle is that the LFSR can use the slight difference in the initial value to generate a key stream with good randomness and a long period. Here, the hash value obtained through the previous multiple steps of semantic-related processing is used as the initial value, so that the generated communication key has both semantic-based association and inherits the security characteristics such as randomness and periodicity brought by the LFSR, meeting the security and confidentiality requirements for the key during the communication process.

[0089] By utilizing the LFSR key stream generation algorithm, the hash value obtained through the previous complex processing is converted into the final usable communication key, so that the generated communication key combines the semantic features of the semantic analysis results and good security characteristics. It can effectively encrypt voice data, etc., and ensure communication security in the process of multimodal voice interaction. It is the key finishing link of the entire communication key generation technical solution.

[0090] Reference Figure 2 An embodiment of the present invention further provides a multimodal voice interaction device based on a large model, comprising: A recognition module, used for performing speech recognition on speech data in the first language input by a user to obtain a text in the first language; An analysis module, configured to perform language translation on the first language text into a second language text; and perform semantic analysis on the second language text to obtain a semantic analysis result; A construction module, configured to construct a multimodal knowledge graph for the semantic analysis result based on a preset large model to obtain an enhanced semantic understanding result; wherein the multimodal knowledge graph incorporates multimodal information related to the second language text; A generation module, used to generate a dialogue strategy for enhancing the semantic understanding result; A synthesis module is used to perform speech synthesis based on the dialogue strategy to generate reply speech data expressed in the first language.

[0091] In this embodiment, for the specific implementation of each module in the above-mentioned device embodiment, please refer to the above-mentioned method embodiment, which will not be described in detail here.

[0092] Reference Figure 3 In an embodiment of the present invention, an electronic device is also provided, the internal structure of the electronic device can be as follows Figure 3As shown. The electronic device includes a processor, a memory, a display screen, an input device, a network interface and a database connected via a system bus. Among them, the computer-designed processor is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store the corresponding data in this embodiment. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0093] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the electronic device to which the solution of the present invention is applied.

[0094] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0095] In summary, the multimodal voice interaction method, electronic device and storage medium based on a large model provided in the embodiment of the present invention include: performing voice recognition on the voice data in the first language input by the user to obtain the first language text; performing language translation on the first language text to translate it into the second language text; performing semantic analysis on the second language text to obtain the semantic analysis result; constructing a multimodal knowledge graph based on the preset large model for the semantic analysis result to obtain an enhanced semantic understanding result; wherein the multimodal knowledge graph integrates the multimodal information related to the second language text; generating a dialogue strategy for the enhanced semantic understanding result; performing speech synthesis based on the dialogue strategy to generate reply voice data expressed in the first language. In the present invention, the voice data in the first language input by the user is subjected to voice recognition and language translation to obtain the second language text, and then semantic analysis is performed to obtain the semantic analysis result, and a multimodal knowledge graph is constructed, and finally the reply voice data is generated. It realizes the interaction in multiple languages ​​and integrates multiple modal information at the same time.

[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0097] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0098] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal voice interaction method based on a large model, characterized in that: The following steps are involved: Performing speech recognition on the speech data in the first language input by the user to obtain a text in the first language; performing language translation on the first language text into a second language text; Performing semantic analysis on the second language text to obtain a semantic analysis result; Based on a preset large model, a multimodal knowledge graph is constructed for the semantic analysis result to obtain an enhanced semantic understanding result; wherein the multimodal knowledge graph integrates multimodal information related to the second language text; Generating a dialogue strategy for enhancing the semantic understanding result; Speech synthesis is performed based on the dialogue strategy to generate reply speech data expressed in the first language.

2. The multimodal voice interaction method based on a large model according to claim 1, characterized in that: The dialogue strategy for generating the enhanced semantic understanding result includes: Based on a preset intent classification model, the enhanced semantic understanding result is divided into a plurality of intent categories; wherein the intent classification model is obtained by pre-training; For each intent category, a corresponding dialogue strategy template is matched from a pre-stored dialogue strategy template library; the dialogue strategy template library is pre-built based on multi-language and multi-scenario dialogue data; By using natural language generation technology, the matched dialogue strategy template is filled and adjusted according to the specific content in the enhanced semantic understanding result to generate a personalized dialogue strategy.

3. The multimodal voice interaction method based on a large model according to claim 1, characterized in that: A multimodal knowledge graph is constructed for the semantic analysis results based on a preset large model to obtain enhanced semantic understanding results, including: Encoding the semantic analysis result based on the text encoder in the large model to obtain a text vector; For multimodal information related to the second language text, using a corresponding image encoder and a video encoder to extract and encode features of the image data and the video data, respectively, to obtain an image feature vector and a video feature vector; The image feature vector and the video feature vector are fused into the text vector to construct an initial knowledge graph; wherein the text vector is a core node, and the fused multimodal features are used as peripheral nodes and edges to form an initial knowledge graph; The initial knowledge graph is optimized based on a semantic relationship model to obtain an enhanced semantic understanding result; wherein, the semantic relationship model is based on a multimodal corpus for unsupervised learning, can identify and supplement the semantic relationships in the knowledge graph, and improve the structure and content of the knowledge graph by continuously iteratively adjusting the connection relationship and semantic annotations between nodes.

4. The multimodal voice interaction method based on a large model according to claim 3 is characterized in that: The image feature vector and the video feature vector are merged into the text vector, comprising: Through the cross-modal attention mechanism, the text vector interacts with the image feature vector and the video feature vector to calculate the association weights between different modalities; wherein the cross-modal attention mechanism is based on a multi-head attention architecture and captures the association information between different modalities by calculating multiple attention heads in parallel; According to the calculated association weight, the image feature vector and the video feature vector are fused into the text vector.

5. The multimodal voice interaction method based on a large model according to claim 1, characterized in that: After performing speech synthesis based on the dialogue strategy to generate reply speech data expressed in the first language, the method includes: Generate a voice data encryption pattern based on the codes of the first language and the second language; generating a communication key based on the semantic analysis result; Based on the voice data encryption mode, the reply voice data is encrypted using the communication key and then fed back to the user's terminal.

6. The multimodal voice interaction method based on a large model according to claim 5, characterized in that: The method of generating a voice data encryption mode based on the codes of the first language and the second language includes: Based on the language code mapping table, the codes of the first language and the second language are mapped into a unique first digital identifier and a unique second digital identifier respectively; Calculating a first difference and a second difference between the first digital identifier, the second digital identifier and a preset encryption base value respectively; Encrypting the sum of the first difference and the second difference based on a preset encryption algorithm to obtain an encrypted feature value; The encrypted feature value is binary decomposed, and analyzed in combination with the frequency and duration of the reply voice data, and the encrypted segmentation method and local encryption area are determined through a dynamic programming algorithm.

7. The multimodal voice interaction method based on a large model according to claim 5, characterized in that: Generating a communication key based on the semantic analysis result includes: Extracting vocabulary feature vectors, grammatical feature vectors and semantic dependency feature vectors from the semantic analysis results; A graph structure is constructed based on each feature vector, wherein the vocabulary feature vector is used as a node, and the grammatical feature vector and the semantic dependency feature vector are used to determine the weight of the edge; Performing a depth-first traversal on the graph structure, and mapping the node sequence obtained by the traversal into a numerical sequence; Generate a polynomial curve using a numerical sequence, generate a key seed according to the coefficients of the polynomial curve and the control point information; hash the key seed using a hash function to obtain a hash value; The hash value is used as input to generate the communication key through a key stream generation algorithm based on a linear feedback shift register.

8. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images