Speech adaptive completion system based on multimodal knowledge graph

Abstract

A speech adaptive completion system based on a multimodal knowledge graph includes: a data receiver, a data analyzer and a data reasoner, wherein: the data receiver performs preprocessing according to the received audio and video data and outputs it to the data analyzer; the data analyzer extracts waveform timing features and lip trajectory features from the analysis of speech and images, and obtains a phoneme sequence through multimodal joint representation; the data reasoner performs domain conversation modeling and candidate text prediction based on historical texts, performs text reasoning in combination with phoneme sequences, obtains sentences with semantics, and synthesizes completed speech based on waveform features. The present invention uses a phoneme reasoning model to perform phoneme recognition when the speech modality is missing, and performs domain conversation modeling on the historical text generated by the existing speech based on the semantic relationship between entities in the multimodal knowledge graph, thereby reasoning and generating text with semantics, and synthesizing the speech in combination with the waveform features of the user's speech to form the completed audio.

Description

Technical Field

[0001] The present invention relates to a technology in the field of speech processing, specifically a speech adaptive completion system based on a multimodal knowledge graph for a mobile terminal. Background Art

[0002] Real-time audio and video technology is mostly used in real-time video chat, video conferencing, distance education, smart home, etc. However, in actual use, due to the disorder or loss of data packets during network transmission, the resulting call jitter will cause a significant decrease in call quality. Usually, the receiving end creates audio and video data through a packet loss repair system to fill the audio gaps caused by packet loss or network delay. Audio data completion also faces the following difficulties in the process of mobile audio and video communication: First, audio generation is dominated by deep learning methods, and the opacity of the reasoning process leads to low interpretability of such methods, making it difficult to design or tune for specific scenarios; second, current technologies mainly use single-modal data as the basis for model reasoning, ignoring the ability of mobile terminals to perceive information in multiple modalities, resulting in incomplete system perception of data and information, forming cognitive limitations. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a speech adaptive completion system based on a multimodal knowledge graph. Through the phoneme inference model, phoneme recognition is performed when the speech modality is missing. At the same time, domain conversation modeling is performed on the historical text generated by the existing speech according to the semantic relationship between entities in the multimodal knowledge graph, so as to infer and generate semantically meaningful text, and synthesize the speech in combination with the waveform characteristics of the user's speech to form the completed audio.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a speech adaptive completion system based on a multimodal knowledge graph, comprising: a data receiver, a data analyzer and a data reasoner, wherein: the data receiver performs preprocessing according to received audio and video data and outputs the preprocessing to the data analyzer; the data analyzer extracts waveform timing features and lip trajectory features from speech and images, and obtains a phoneme sequence through multimodal joint representation; the data reasoner performs domain conversation modeling and candidate text prediction according to historical texts, performs text reasoning in combination with the phoneme sequence, obtains a sentence with semantics, and synthesizes the completed speech according to the waveform features.

[0006] The system is further provided with a multimodal data aggregation module, which stores and associates the results of the data receiver and the data analyzer, and provides data support for the data analyzer and the data reasoner.

[0007] The system is further provided with a model management module, which provides model calling and updating for the data receiver, data analyzer and data reasoner.

[0008] The data receiver comprises: a data receiving module, a voice preprocessing module and a video preprocessing module, wherein: the data receiving module receives and analyzes the audio and video data packets of the application, and outputs the voice packets therein to the voice preprocessing module and the video packets to the video preprocessing module respectively; the voice preprocessing module collects and preprocesses the voice packets, takes the low-quality real-time audio with data packet loss as input, performs preliminary processing on the voice modal data through voice data packet detection, voice framing, audio windowing and endpoint detection, obtains the preprocessed waveform and outputs it to the voice analysis module; the video preprocessing module collects and preprocesses the video packets, takes the continuous video images as input, performs preliminary processing on the video modal data through video framing, facial lip shape control point detection, lip area scale normalization and time alignment in sequence, obtains the preprocessed image and outputs it to the image analysis module.

[0009] The data analyzer includes: a speech analysis module, a spatiotemporal image analysis module and a multimodal information fusion module, wherein: the speech analysis module extracts historical text, waveform features and waveform timing features from the preprocessed waveform, and outputs them to the multimodal data aggregation module as speech modal data; the spatiotemporal image analysis module constructs a spatiotemporal graph for the lip control point set of each frame after preprocessing, builds a spatiotemporal graph convolutional neural network, extracts the lip movement features of each frame according to the front and back information of each frame in the spatiotemporal graph, merges them into lip trajectory features, and inputs them into the multimodal data aggregation module as video modal data. Multimodal data aggregation module; the multimodal fusion module realizes feature alignment of waveform timing features and lip trajectory features in a cross-modal interactive manner, trains a cross-modal conversion model, and then uses the hidden state features in the process of mutual conversion between lip trajectory features and waveform timing features as the joint representation between the two modalities. The joint representation information is converted into phoneme information by training the phoneme prediction model to enhance the representation ability of the lip feature modality for phoneme information. For the area where voice data packets are lost, phoneme recognition is performed based on the lip trajectory features and they are spliced ​​into phoneme sequences as the input of the semantic text reasoning module.

[0010] The data reasoner includes: a semantic text reasoning module and a speech completion module, wherein: the semantic text reasoning module identifies the knowledge field involved in the speech packet loss area according to the historical text of the current session, and predicts the candidate text based on the spatiotemporal knowledge graph to prune and optimize the solution space size of text reasoning, and matches the identified phonemes with the text in the solution space, so as to infer and generate the completed text and output it to the speech completion module; the speech completion module synthesizes the missing speech according to the completed text and the collected user speech waveform features, and fills the completed speech into the original speech through speech splicing to form a complete and natural speech segment.

[0011] The activity status of the voice data obtained by the voice data packet detection includes: the voice occurrence area, the voice silence area and the voice data packet loss area, wherein: distinguishing whether the voice is silent is conducive to reducing unnecessary voice recognition and completion, and marking the voice data packet area is for subsequent voice completion of this part of the area. The voice data packet detection performs the first classification of the voice activity, and labels the voice area at this moment as TRUE or NONE according to whether the voice data packet at the current moment is received. The area labeled NONE will be completed by the semantic text reasoning module and the voice completion module, and the area labeled TRUE will further distinguish the areas where the voice appears and disappears by endpoint detection.

[0012] The speech framing described above is not stable due to the time-varying characteristics of the speech signal, and the MFCC feature extraction in the speech analysis module uses Fourier transform, which requires a stable input signal. Therefore, the short-term stability of the speech signal is used to perform framing on the speech signal. The speech framing process adopts an overlapping segmentation strategy, and samples each frame according to a preset frame length and overlapping ratio (frame shift), so that the previous frame smoothly transitions to the next frame and maintains the continuity of the sample.

[0013] The audio windowing adopts a Hamming window, multiplies the speech data in each frame by a window function, highlights the data in the middle, weakens the data information on both sides, effectively solves the problem of spectrum leakage, and thus supports Fourier transform.

[0014] The endpoint detection calculates the short-time energy and the short-time average zero-crossing rate for each frame of data, and classifies the speech occurrence area and the speech silence area in real time through the double threshold comparison method, and labels them as TRUE or FALSE, so as to locate the starting point and the ending point of the effective speech area from the entire speech activity, avoiding the influence of the silent part and the noise part.

[0015] The video framing uses the same sampling frequency as the speech framing to convert the video into an image sequence.

[0016] The facial lip shape control point detection detects the control points of the lips of the character in each frame of the image one by one through an external face recognition engine, including the lip center coordinates, the upper lip middle boundary coordinates, the left corner point coordinates of the mouth, etc.

[0017] The scale normalization of the lip area needs to reduce the influence of the lip size, facial deflection angle, and tilt angle in the image because the subsequent image analysis module only focuses on the relative movement of each lip control point. Therefore, a quadrilateral lip detection frame is fitted according to the left and right corners of the mouth, the lip center coordinates, the upper and lower boundary coordinates of the lips, and the lip center coordinates. The lip control points are rotated and scaled to a uniform size through perspective transformation to achieve scale normalization of the lip area and retain the continuity of the control point movement trajectory.

[0018] The time alignment described above requires that each frame of audio corresponds to a frame of image in order to facilitate cross-modal interaction between speech and video modalities. Since the trajectory of short-time lip control points can be approximately replaced by a simple curve, the set of lip control points corresponding to each frame of audio is fitted by using the Lagrange interpolation method, thereby achieving time alignment from image to audio.

[0019] The lip control point space-time graph is constructed in the following way: utilizing the natural connection relationship between control points, the lip control point sets of all input frames are connected in each frame according to the relationship between the human body's lip control points, and each control point forms a self-loop with itself to construct a spatial graph of each frame, and then the same control points of two adjacent frames are connected to form a temporal edge, representing the lip motion trajectory information between two moments, thereby simultaneously modeling the spatial information and temporal information of the lip control points.

[0020] The lip motion features are extracted in the following way: by constructing a spatiotemporal graph convolutional neural network, for the current frame, the input of the spatial graph convolutional neural network is represented by a 3D matrix (C, T, V), where C represents the feature dimension of the lip control point, the coordinates of the control point are used as features, T represents the current frame and the previous T-1 frame, and V represents the number of lip control points. From a spatial perspective, the graph partitioning strategy is adopted to decompose the graph G of each frame into three subgraphs G1, G2, and G3, which respectively represent the centripetal motion, centrifugal motion, and static motion features of the control points. Each control point in G1 is connected to the neighbor control point that is closer to the lip center than the control point, each control point in G2 is connected to the neighbor control point that is farther away from the lip center than the control point, and each control point in G3 is connected to the control point itself. Therefore, the size of the graph convolution used is (1, V, V) and the number of convolution kernels is 3. The local features of adjacent control points are obtained by weighted averaging. In terms of time, in order to superimpose the temporal features on the spatial features of the current frame, a temporal convolutional neural network is used to fuse the features of the current frame and the previous T-1 frame of each lip control point using a convolution kernel of size (T, 1) to obtain the local features of each control point that change in time. By using spatial and temporal convolution, the lip motion features are extracted, and the output of each frame is (1, V, N 2 ), where N 2 The number of features extracted for each control point is used to concatenate the lip motion features of each frame and output them as (T, V, C2) lip trajectory features.

[0021] The data aggregation refers to: defining domains, text words, phonemes, waveform features, waveform timing features, lip trajectory features and other entity types, attributes and their relationships, taking the input of historical texts, waveform features, waveform timing features of speech modality and lip trajectory features of video modality as different entities, aggregating, storing and associating these entities based on multimodal knowledge graphs, continuously expanding knowledge during the operation of the system, and providing support for the enhancement and verification of text reasoning in subsequent modules. In addition, after the data is compiled, the waveform timing features and lip trajectory features are used as inputs of the multimodal fusion module, the historical text is used as input of the semantic text reasoning module, and the waveform features are used as inputs of the speech completion module.

[0022] The joint representation, i.e., the multimodal joint representation based on Seq2Seq, specifically refers to: the cross-modal interaction is based on the Seq2Seq model, wherein the cross-modal conversion model uses BILSTM as an encoder and a decoder, and is trained by translating from lip trajectory features to waveform timing features and reversely translating from waveform timing features to lip trajectory features to obtain a joint representation of the two modalities.

[0023] The enhanced lip feature modality refers to: using a phoneme inference model with the same structure as the cross-modal conversion model, receiving a joint representation and outputting a temporal phoneme posterior probability matrix y = (y1, y2, ..., yt, ..., yT) of size (T, |A|), where: |A| is the size of the phoneme set A to be identified, and each column of y is (yt1, yt2, ..., yta, ..., ytA), indicating the probability that the tth frame is a certain phoneme a.

[0024] The phonemes to be recognized include: all known phonemes and BLANK, where BLANK is represented by “-”. When the output of LSTM is converted into a phoneme sequence, adjacent phonemes with the same pronunciation are distinguished.

[0025] The phoneme inference model is trained in the following way: CTC is connected as a transcription layer after the BILSTM of the phoneme prediction model decoder, in order to improve the probability p(L|x) that BILSTM outputs the correct result when a given input x is given. Since a phoneme in L is composed of the prediction results of multiple time slices in y, there may be multiple paths π that make up L, that is, B(π) = L, B is the mapping function, then Transcriptional layer CTCs by gradient Adjust the LSTM parameter ω so that for the input sample π∈B -1 (L) makes p(L|x) maximum.

[0026] The phoneme recognition means: for the area where the voice data packet is lost, the lip trajectory features extracted from the video are input into the phoneme inference model, and a phoneme inference vector of size |A| is obtained for each frame, and each value P(a) in the vector, a∈A represents the probability of the frame being each phoneme.

[0027] The domain conversation modeling mentioned above refers to: inferring the knowledge domain in which the semantic context is located based on historical texts, such as the financial industry, travel activities, daily life chats, etc., mainly by defining the discriminability measure key of the domain keywords to infer the knowledge domain involved in the historical text, and then combining the temporal correlation measure EMI (e, w) between text entities to represent the possibility of entity w appearing after entity e within a certain text step length, and splicing EMI (e, w) with a probability greater than a certain threshold into a domain text vector as output, thereby realizing the modeling of domain conversations. First, when training the domain conversation model, an initial text set for each domain is generated based on conversation samples from different domains, and domain entities are associated with text entities in a many-to-many manner in the multimodal knowledge graph. Then, in order to generate recognizable domain keywords, the frequency f of each text word j appearing in the domain text set i is calculated. ij , count the maximum frequency max_f, and calculate the number of times each text word appears in N fields nj , through the formula The degree of discrimination of text word j to domain i is calculated. Compared with the TF-IDF formula, this calculation formula not only takes into account the length of the domain text set and converts the text frequency into frequency, but also ensures that the original part of TF calculation is always non-negative through normalization transformation, thus realizing the discrimination measurement of domain keywords, thereby identifying the knowledge domain involved by searching for keywords in historical texts. Secondly, in each knowledge domain, the mutual information between entities is obtained by calculating the frequency of two text entities appearing successively within a certain text step length. It is used to indicate the relevance of entity2 to entity1 in the text order. When EMI is greater than zero, the greater the value, the greater the probability of occurrence. When EMI is less than zero, the two entities are mutually exclusive, thus defining the correlation measure between text entities in each field. Finally, during the use of the system, new text entities are expanded and associated with the field entities based on the real data of the session. If multiple sessions repeatedly jump between multiple fields, the fields are split and generated based on the unsupervised clustering method.

[0028] The candidate text prediction described above, i.e., candidate text prediction based on the spatiotemporal knowledge graph, specifically refers to: utilizing the temporal characteristics of historical texts to form a spatiotemporal knowledge graph network, inferring the joint probability representation P(w) of the current candidate text, and splicing P(w) with a probability higher than a certain threshold into a candidate text vector, thereby achieving solution space pruning, specifically including: the historical text that has been formed can be regarded as a path that wanders between knowledge graph entities, represented by a combination of several text entity binary vectors (from, to). Due to the real-time nature of audio and video calls, the system cannot fully and accurately predict the future at the current moment. Even if the phoneme recognition is correct, it may not be possible to predict a text entity with truly appropriate semantics. Therefore, multiple alternative paths with smaller weights will be retained on other entity nodes before the entity node at the end of the path for semantic backtracking when reasoning errors occur, thereby forming a path wandering graph G at the current moment t. T , based on the multimodal knowledge graph, the path walk graph at each previous moment is superimposed on the time dimension to form a spatiotemporal knowledge graph G, where space refers to the solution space. In order to improve the reasoning efficiency, based on an assumption that the path walk graph at time T only depends on the path walk graph of the previous s time steps, the spatiotemporal knowledge graph network is trained to optimize and reason about the joint probability distribution of G Where P(G t |G t-s：t-1 ) can be split into Further, Representing all neighbor entity nodes of from_t has two advantages: covering the candidate text space and having the ability to mine frequent patterns. According to the above formula, to establish a text-oriented spatiotemporal graph neural network based on the recurrent neural network RNN, the following formula parameterization steps are required: e from_t is the learnable vector associated with from_t, h t-1 (from_t) is the historical semantic vector of from_t, ω to_t is the classifier parameter. Similarly, P(from_t|G t-s：t-1 )→exp(H t-1 T ·ω from_t ), H t-1 T is the historical semantic vector of the entire walking path. On this basis, and H t =RNN 2 (g(G t ), H t-1 ), that is, the historical semantic vector is recursively updated through RNN, g is an aggregation function with attention mechanism, and the importance weight of neighbor entity nodes to from entity nodes is learned through the attention matrix.

[0029] The matching of the identified phonemes with the text in the solution space refers to: according to the phonemes corresponding to the text, the phoneme inference vector is Cartesian-producted with the domain text vector and the candidate text vector to obtain the text solution space of the intersecting phonemes, and then the probability of each text in the solution space is calculated by the formula P(a)P(w)EMI(e, w), the text with the largest value is used as the completed text for voice completion, the top three texts with the same phonemes are used as candidate texts, and the historical texts are supplemented together to form an alternative path with a smaller weight, which is used as the semantic backtracking that may be generated during the next text reasoning.

[0030] The semantic-based sentence generation is specifically as follows: setting a threshold T and a confidence level α, adding the probabilities of the T text entities at the end of each candidate path, taking the path with the highest probability to concatenate and generate a sentence, and at the same time, shrinking the paths with results less than αT so that the graph only contains paths with results greater than αT, thereby further pruning the path walk graph at the current moment and optimizing the efficiency of continuous reasoning of the system.

[0031] The synthesis of the missing speech refers to: converting the text sequence with timestamps into a phoneme sequence with timestamps, learning the mapping of phonemes to waveform features according to the waveform features of the user's speech through the TTS model, and reversely converting the features into waveforms through the vocoder.

[0032] The speech splicing refers to: splicing the existing speech waveform and the completed speech waveform after stretching and fitting near the connection point, so that the speech transition is smoother and more natural.

[0033] Technical Effects

[0034] Compared with the technical means commonly used in lip feature extraction, that is, extracting image features from lip movement videos and inputting them into a recurrent neural network, the present invention takes the lip control points in the image as the focus of feature extraction, which greatly reduces irrelevant variables in the feature extraction process, and then performs spatiotemporal graph modeling and feature extraction on the spatiotemporal relationship between the lip control points, making the feature extraction process more interpretable.

[0035] Compared with the commonly used technical means in speech completion, namely speech prediction based on historical speech, the present invention largely supplements the information relied on for reasoning through the interaction and complementarity of multimodal features.

[0036] The present invention uses a text pruning reasoning method that combines domain conversation modeling and candidate text prediction based on preset and evolved reference information in the knowledge graph, extracts the historical semantics of the context and the wandering probability of the text path, and combines domain knowledge to make the semantics of the result more appropriate and have higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a method framework diagram of the present invention;

[0038] Figure 2 It is a system structure diagram of an embodiment of the present invention;

[0039] In the figure: the three dotted boxes represent the system input, the system's internal functional modules and the system output respectively; the solid lines and arrows represent the direction of data flow during the interaction of each module; the solid lines without arrows represent the internal model on which the function depends; the dotted lines represent the external model on which the functional module depends. DETAILED DESCRIPTION

[0040] like Figure 1As shown, this embodiment relates to a speech adaptive completion system based on a multimodal knowledge graph, including: a speech preprocessing module, a speech analysis module, a video preprocessing module, a time-space-based image analysis module, a multimodal data aggregation module, a multimodal information fusion module, a semantic text reasoning module and a speech completion module, wherein: the speech preprocessing module collects and preprocesses the speech packets at the receiving end, takes the low-quality real-time audio with data packet loss as input, performs preliminary processing on the speech modality data through speech data packet detection, speech framing, audio windowing and endpoint detection, obtains the preprocessed waveform and outputs it to the speech analysis module; the video preprocessing module collects and preprocesses the video packets at the receiving end The module performs preprocessing and collection. It takes continuous video images as input, performs preliminary processing on the video modality data in sequence through video framing, facial lip shape control point detection, lip area scale normalization and time alignment, obtains the preprocessed image and outputs it to the image analysis module. The speech analysis module extracts historical text, waveform features and waveform timing features from the preprocessed waveform, and outputs them to the multimodal data aggregation module as speech modality data. The spatiotemporal image analysis module constructs a spatiotemporal graph for the lip control point set of each frame after preprocessing, builds a spatiotemporal graph convolutional neural network, extracts the lip motion features of each frame according to the front and back information of each frame in the spatiotemporal graph, and merges them into lip trajectory features as the video modality data. The data is input into the multimodal data aggregation module; the multimodal data aggregation module aggregates, stores and associates the historical text, waveform features, waveform timing features of the speech modality and the lip trajectory features of the video modality to provide support for the fusion and reasoning of subsequent modules; the multimodal fusion module aligns the waveform timing features and the lip trajectory features in a cross-modal interactive manner, trains to obtain a cross-modal conversion model, and then uses the hidden state features in the process of mutual conversion between the lip trajectory features and the waveform timing features as the joint representation between the two modalities, and converts the joint representation information into phoneme information by training the phoneme prediction model, thereby enhancing the representation capability of the lip feature modality for phoneme information. For the voice packet loss area, based on The lip trajectory features are used to identify phonemes and splice them into phoneme sequences as input to the semantic text reasoning module; the semantic text reasoning module identifies the knowledge fields involved in the voice packet loss area based on the historical text of the current session, and predicts the candidate texts based on the spatiotemporal knowledge graph to prune and optimize the solution space size of text reasoning, match the identified phonemes with the text in the solution space, and thus infer and generate the completed text and output it to the speech completion module; the speech completion module synthesizes the missing speech based on the completed text and the collected user speech waveform features, and fills the completed speech into the original speech through speech splicing to form a complete and natural speech segment.

[0041] like Figure 2As shown, the embodiment is divided into a mobile application, a speech adaptive completion system and an infrastructure layer, wherein the speech adaptive completion system is the core content of the entire framework, including a speech preprocessing module, a video preprocessing module, a speech analysis module, a time-space-based video analysis module, a multimodal data aggregation module, a multimodal information fusion module, a semantic text reasoning module, and a speech completion module, which supports the reception, analysis, reasoning and completion of audio and video data through interaction and collaboration between modules.

[0042] The speech preprocessing module comprises: a speech data packet detection unit, a speech framing unit, an audio windowing unit and an endpoint detection unit, wherein: the speech data packet detection unit marks the speech data packet loss area according to whether the speech data packet at the current moment is received, and the speech framing unit uses the short-term stability of the speech signal to perform framing processing on the speech signal. The speech framing process adopts the strategy of overlapping segmentation, samples each frame according to the preset frame length and overlapping ratio (frame shift), so that the previous frame smoothly transitions to the next frame, and maintains the continuity of the sample. The audio windowing unit adopts a Hamming window, multiplies the speech data in each frame by a window function, highlights the data in the middle, and weakens the data information on both sides. The endpoint detection unit calculates the short-term energy and the short-term average zero-crossing rate for each frame of data, and classifies the speech occurrence area and the speech silence area in real time through the double threshold comparison method, so as to locate the starting point and the end point of the effective speech area from the entire speech activity.

[0043] The speech analysis module includes: a speech recognition unit, an MFCC feature extraction unit and a BILSTM-based temporal feature recognition unit, wherein: the speech recognition unit uses an external STT model to obtain text from speech as the context of text reasoning; the MFCC-based feature extraction unit extracts M-dimensional cepstrum feature parameters in the Mel scale frequency domain for each frame of preprocessed speech data input, so as to achieve the purpose of feature extraction and reduce the operation dimension, and splices the cepstrum feature parameters of the current frame and the previous T-1 frame to obtain a waveform feature map of size (1, T, M); however, a pronunciation is usually composed of many frames of speech data, and analyzing the feature vector of a single frame will produce a large error, so it is necessary to combine the temporal context features of the frame; the BILSTM-based temporal feature recognition unit uses a deep bidirectional LSTM network to continue to extract the temporal features of each frame context based on the MFCC waveform feature map, and the output size is (T, N 1 )’s waveform time series feature matrix, N 1 is the number of features in each frame.

[0044] The video preprocessing module includes: a video framing unit, a facial lip shape control point detection unit, a lip area scale normalization unit and a time alignment unit, wherein: the video framing unit converts the video into an image sequence according to the input video data packet using the same sampling frequency as the voice framing; the facial lip shape control point detection unit detects the control points of the lips of the character in each frame of the image one by one through an external face recognition engine, including the lip center coordinates, the upper and middle boundary coordinates of the upper lip, the left corner coordinates of the mouth, etc.; the lip area scale normalization unit fits a quadrilateral lip detection frame according to the left and right corners of the mouth, the lip center coordinates, the upper and lower boundary coordinates of the lips and the lip center coordinates; the lip control points are rotated and scaled to a uniform size through perspective transformation to achieve scale normalization of the lip area and retain the continuity of the control point movement trajectory; the time alignment unit fits a set of lip control points corresponding to each frame of audio by using the Lagrange interpolation method, thereby achieving time alignment of the image to the audio.

[0045] The spatiotemporal image analysis module includes: a lip control point spatiotemporal graph construction unit and a lip motion feature extraction unit, wherein: the lip control point spatiotemporal graph construction unit utilizes the natural connection relationship between control points to connect the lip control point sets of all input frames in each frame according to the relationship between the human body lip control points, and each control point forms a self-loop with itself to construct a spatial graph of each frame, and then connects the same control points of two adjacent frames to form a temporal edge to represent the lip motion trajectory information between two moments, thereby simultaneously modeling the spatial information and temporal information of the lip control points. The lip motion feature extraction unit constructs a spatiotemporal graph convolutional neural network. For the current frame, the input of the spatial graph convolutional neural network is represented by a 3D matrix (C, T, V), wherein C represents the feature dimension of the lip control point, the coordinates of the control point are used as features, T represents the current frame and the previous T-1 frame, and V represents the number of lip control points. From the perspective of space, the graph partitioning strategy is adopted to decompose the graph G of each frame into three sub-graphs G1, G2, and G3, which respectively represent the centripetal motion, centrifugal motion, and static motion features of the control points. Each control point in G1 is connected to the neighboring control points closer to the lip center than the control point, each control point in G2 is connected to the neighboring control points farther away from the lip center than the control point, and each control point in G3 is connected to the control point itself. Therefore, the size of the convolution kernel used in the graph convolution is (1, V, V) and the number of convolution kernels is 3. The local features of adjacent control points are obtained by weighted averaging. In terms of time, in order to superimpose the temporal features on the spatial features of the current frame, a temporal convolutional neural network is used. The convolution kernel of size (T, 1) is used to fuse the features of each lip control point in the current frame and the previous T-1 frame to obtain the local features of each control point that change in time. By using spatial and temporal convolution, the lip motion features are extracted, and the output of each frame is (1, V,), where is the number of features extracted for each control point. The lip motion features of each frame are concatenated and output as (T, V, C2) lip trajectory features.

[0046] The multimodal data aggregation module defines ontology types, attributes and their relationships such as domains, text words, phonemes, waveform features, waveform timing features and lip trajectory features. The inputs of historical texts, waveform features, waveform timing features of speech modality and lip trajectory features of video modality are used as different entities. These entities are aggregated, stored and associated based on the multimodal knowledge graph. During the operation of the system, knowledge is continuously expanded to provide support for the enhancement and verification of text reasoning in subsequent modules. In addition, after the data is compiled, the waveform timing features and lip trajectory features are used as inputs of the multimodal fusion module, the historical text is used as input of the semantic text reasoning module, and the waveform features are used as inputs of the speech completion module.

[0047] The multimodal information fusion module includes: a multimodal joint representation unit based on Seq2Seq, a lip feature modal representation enhancement unit and a phoneme recognition unit based on lip trajectory, wherein: the multimodal joint representation unit based on Seq2Seq uses BILSTM as an encoder and decoder for cross-modal conversion, and is trained by translating from lip trajectory features to waveform timing features and reversely translating from waveform timing features to lip trajectory features to obtain a joint representation of the two modalities. The phoneme inference model of the lip feature modal representation enhancement unit adopts the same structure as the cross-modal conversion model, with the input being the joint representation and the output being a temporal phoneme posterior probability matrix y=(y1, y2, ..., yt, ..., yT) of size (T, |A|), wherein |A| is the size of the phoneme set A to be recognized, and each column of y is (yt1, yt2, ..., yta, ..., ytA), indicating the probability that the tth frame is a certain phoneme a. Among them, the phonemes that need to be recognized include all known phonemes and BLANK, and BLANK is represented by "-". When the output of LSTM is converted into a phoneme sequence, adjacent phonemes with the same pronunciation are distinguished. During the training process, CTC is connected after the BILSTM of the phoneme prediction model decoder as a transcription layer, the purpose is to improve the probability p(L|x) that BILSTM outputs the correct result when a given input x is given. Since a phoneme in L is composed of the prediction results of multiple time slices in y, there may be multiple paths π that make up L, that is, B(π) = L, B is the mapping function, then Transcriptional layer CTCs by gradient Adjust the LSTM parameter ω so that for the input sample π∈B -1 (L) makes p(L|x) maximum. For the speech area where the speech data packet is lost, the phoneme recognition unit based on lip trajectory inputs the lip trajectory features extracted from the video into the phoneme inference model, and obtains the corresponding phoneme inference vector of size |A| for each frame. Each value in the vector represents the probability of each phoneme in the frame.

[0048] The semantic text reasoning module includes: a domain conversation modeling unit, a candidate text prediction unit based on a spatiotemporal knowledge graph, a text reasoning unit and a semantic-based sentence generation unit, wherein: the domain conversation modeling unit identifies the knowledge domain involved according to the historical text of the current conversation, and the candidate text prediction unit based on the spatiotemporal knowledge graph predicts the candidate text to prune and optimize the solution space size of text reasoning. The text reasoning unit matches the identified phonemes with the text in the solution space, and the semantic-based sentence generation unit infers and generates the completed text as the input of the speech completion module.

[0049] The domain conversation modeling unit refers to: inferring the knowledge domain in which the semantic context is located based on the historical text, such as the financial industry, travel activities, daily life chats, etc., mainly by defining the discriminability measure key of the domain keyword to infer the knowledge domain involved in the historical text, and then combining the temporal correlation measure EMI (e, w) between text entities to represent the possibility of entity w appearing after entity e within a certain text step, and splicing EMI (e, w) with a probability greater than a certain threshold into a domain text vector as output, thereby realizing the modeling of domain conversation. First, when training the domain conversation model, an initial text set for each domain is generated based on conversation samples from different domains, and domain entities are associated with text entities in a many-to-many manner in the multimodal knowledge graph. Then, in order to generate recognizable domain keywords, the frequency f of each text word j appearing in the domain text set i is calculated. ij , count the maximum frequency max_f, and calculate the number of times each text word appears in N fields n j , through the formula The degree of discrimination of text word j to domain i is calculated. Compared with the TF-IDF formula, this calculation formula not only takes into account the length of the domain text set and converts the text frequency into frequency, but also ensures that the original part of TF calculation is always non-negative through normalization transformation, thus realizing the discrimination measurement of domain keywords, thereby identifying the knowledge domain involved by searching for keywords in historical texts. Secondly, in each knowledge domain, the mutual information between entities is obtained by calculating the frequency of two text entities appearing successively within a certain text step length. It is used to indicate the relevance of entity2 to entity1 in the text order. When EMI is greater than zero, the greater the value, the greater the probability of occurrence. When EMI is less than zero, the two entities are mutually exclusive, thus defining the correlation measure between text entities in each field. Finally, during the use of the system, new text entities are expanded and associated with the field entities based on the real data of the session. If multiple sessions repeatedly jump between multiple fields, the fields are split and generated based on the unsupervised clustering method.

[0050] The candidate text prediction unit based on the spatiotemporal knowledge graph specifically refers to: using the temporal characteristics of historical texts to form a spatiotemporal knowledge graph network, inferring the joint probability representation P(w) of the current candidate text, and splicing P(w) with a probability higher than a certain threshold into a candidate text vector, thereby achieving solution space pruning, specifically including: the historical text that has been formed can be regarded as a path that wanders between knowledge graph entities, represented by a combination of several text entity binary vectors (from, to). Due to the real-time nature of audio and video calls, the system cannot fully and accurately predict the future at the current moment. Even if the phoneme recognition is correct, it may not be able to predict a text entity with truly appropriate semantics. Therefore, multiple alternative paths with smaller weights will be retained on other entity nodes before the entity node at the end of the path for semantic backtracking when reasoning errors occur, thereby forming a path wandering graph G at the current moment t. T , based on the multimodal knowledge graph, the path walk graph at each previous moment is superimposed on the time dimension to form a spatiotemporal knowledge graph G, where space refers to the solution space. In order to improve the reasoning efficiency, based on an assumption that the path walk graph at time T only depends on the path walk graph of the previous s time steps, the spatiotemporal knowledge graph network is trained to optimize and reason about the joint probability distribution of G Where P(G t |G t-s：t-1 ) can be split into Further, Representing all neighbor entity nodes of from_t has two advantages: covering the candidate text space and having the ability to mine frequent patterns. According to the above formula, to establish a text-oriented spatiotemporal graph neural network based on the recurrent neural network RNN, the following formula parameterization steps are required: e from_t is the learnable vector associated with from_t, h t-1 (from_t) is the historical semantic vector of from_t, ω to_t is the classifier parameter. Similarly, P(from_t|G t-s：t-1 )→exp(H t-1 T ·ω from_t ), H t-1 T is the historical semantic vector of the entire walking path. On this basis, and H t =RNN 2 (g(G t ), H t-1), that is, the historical semantic vector is recursively updated through RNN, g is an aggregation function with attention mechanism, and the importance weight of neighbor entity nodes to from entity nodes is learned through the attention matrix.

[0051] The text reasoning unit: according to the phonemes corresponding to the text, the phoneme reasoning vector is Cartesian-producted with the domain text vector and the candidate text vector to obtain the text solution space of the intersecting phonemes, and then the probability of each text in the solution space is calculated by the formula P(a)P(w)EMI(e, w), the text with the largest value is used as the completed text for voice completion, the top three texts with the same phonemes are used as candidate texts, and the historical texts are supplemented together to form an alternative path with a smaller weight, which is used as the semantic backtracking that may be generated during the next text reasoning.

[0052] The semantic-based sentence generation unit specifically sets a threshold T and a confidence level α, adds the probabilities of the T text entities at the end of each candidate path, takes the path with the highest probability to concatenate and generate a sentence, and at the same time, shrinks the paths with results less than αT so that the graph only contains paths with results greater than αT, thereby further pruning the path walk graph at the current moment and optimizing the efficiency of continuous reasoning of the system.

[0053] The speech completion module comprises: a speech synthesis unit and a speech splicing unit, wherein: the speech synthesis unit converts a text sequence with a timestamp into a phoneme sequence with a timestamp, learns the mapping of phonemes to waveform features according to the user's speech waveform features through a TTS model, and reversely converts the features into waveforms through a vocoder; the speech splicing unit performs stretching and fitting on the existing speech waveform and the completed speech waveform near the connection point and then splices them, so that the speech transition is smoother and more natural.

[0054] Table 1 Comparison of technical characteristics

[0055]

[0056]

[0057] Based on the knowledge graph, this system constructs a spatiotemporal graph with structural significance in image feature extraction, and a spatiotemporal knowledge graph with semantic association in text reasoning, making the reasoning process more explainable. It adopts cross-modal reasoning and uses multimodal knowledge graph as data support to aggregate, associate and complement multimodal information with higher integrity. It models domain conversations and time-series texts, and implements text reasoning with context based on preset and evolving information in the knowledge graph, thereby improving the accuracy of the results.

[0058] Compared with the prior art, the present invention solves the problems of low accuracy of voice completion, low interpretability of the method, and low integrity of information and data in mobile audio and video communication scenarios, and fully utilizes the various modal data of the receiving end for text reasoning and voice completion, thereby improving the intelligence level of the system and providing strong technical support for the repair of voice data packets.

[0059] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A speech adaptive completion system based on multimodal knowledge graph, characterized in that: include: Data receiver, data analyzer and data reasoner, wherein: the data receiver performs preprocessing based on the received audio and video data and outputs the data to the data analyzer; the data analyzer extracts waveform timing features and lip trajectory features from the analysis of speech and images, and obtains a phoneme sequence through multimodal joint representation; the data reasoner performs domain conversation modeling and candidate text prediction based on historical texts, performs text reasoning based on the phoneme sequence, obtains a sentence with semantics, and synthesizes the completed speech based on the waveform features; The data analyzer includes: a speech analysis module, a spatiotemporal image analysis module and a multimodal information fusion module, wherein: the speech analysis module extracts historical text, waveform features and waveform timing features from the preprocessed waveform, and outputs them to the multimodal data aggregation module as speech modal data; the spatiotemporal image analysis module constructs a spatiotemporal graph for the lip control point set of each frame after preprocessing, builds a spatiotemporal graph convolutional neural network, extracts the lip movement features of each frame according to the front and back information of each frame in the spatiotemporal graph, merges them into lip trajectory features, and inputs them into the multimodal data aggregation module as video modal data. Multimodal data aggregation module; the multimodal fusion module realizes feature alignment of waveform timing features and lip trajectory features in a cross-modal interactive manner, trains a cross-modal conversion model, and then uses the hidden state features in the process of mutual conversion between lip trajectory features and waveform timing features as the joint representation between the two modalities. The joint representation information is converted into phoneme information by training the phoneme prediction model to enhance the representation ability of the lip feature modality for phoneme information. For the area where the voice data packet is lost, phoneme recognition is performed based on the lip trajectory features and spliced ​​into a phoneme sequence as the input of the semantic text reasoning module; The lip motion features are extracted in the following way: by constructing a spatiotemporal graph convolutional neural network, for the current frame, the input of the spatial graph convolutional neural network is represented by a 3D matrix (C, T, V), where C represents the feature dimension of the lip control point, the coordinates of the control point are used as features, T represents the current frame and the previous T-1 frame, and V represents the number of lip control points. From a spatial perspective, the graph partitioning strategy is adopted to decompose the graph G of each frame into three sub-graphs G1, G2, and G3, which respectively represent the centripetal motion, centrifugal motion, and stationary motion features of the control points. Each control point in G1 is connected to the neighbor control point that is closer to the lip center than the control point, and each control point in G2 is connected to the neighbor control point that is farther away from the lip center than the control point. Each control point in 3 is connected to the control point itself, so the size of the graph convolution is (1, V, V) and the number of convolution kernels is 3. The local features of adjacent control points are obtained by weighted averaging. In time, in order to superimpose the temporal features on the spatial features of the current frame, a temporal convolutional neural network is used. The convolution kernel of size (T, 1) is used to fuse the features of the current frame and the previous T-1 frame of each lip control point to obtain the local features of each control point that change in time. By using spatial and temporal convolution, the lip motion features are extracted. The output of each frame is (1, V, N2), where N2 is the number of features extracted from each control point. The lip motion features of each frame are spliced ​​and output as (T, V, N2) lip trajectory features; The data aggregation refers to: defining domains, text words, phonemes, waveform features, waveform timing features and lip trajectory feature ontology types, attributes and their relationships, taking the input of historical texts, waveform features, waveform timing features of speech modality and lip trajectory features of video modality as different entities, aggregating, storing and associating these entities based on multimodal knowledge graphs, continuously expanding knowledge during the operation of the system, and providing support for the enhancement and verification of text reasoning in subsequent modules. In addition, after the data is compiled, the waveform timing features and lip trajectory features are used as inputs of the multimodal fusion module, the historical text is used as input of the semantic text reasoning module, and the waveform features are used as inputs of the speech completion module; The joint representation, i.e., the multimodal joint representation based on Seq2Seq, specifically refers to: the cross-modal interaction is based on the Seq2Seq model, wherein the cross-modal conversion model uses BILSTM as an encoder and a decoder, and is trained by translating from lip trajectory features to waveform timing features and reversely translating from waveform timing features to lip trajectory features to obtain a joint representation of the two modalities.

2. The speech adaptive completion system based on multimodal knowledge graph according to claim 1 is characterized in that: The data receiver comprises: a data receiving module, a voice preprocessing module and a video preprocessing module, wherein: the data receiving module receives and analyzes the audio and video data packets of the application, and outputs the voice packets therein to the voice preprocessing module and the video packets to the video preprocessing module respectively; the voice preprocessing module collects and preprocesses the voice packets, takes the low-quality real-time audio with data packet loss as input, performs preliminary processing on the voice modal data through voice data packet detection, voice framing, audio windowing and endpoint detection, obtains the preprocessed waveform and outputs it to the voice analysis module; the video preprocessing module collects and preprocesses the video packets, takes the continuous video images as input, performs preliminary processing on the video modal data through video framing, facial lip shape control point detection, lip area scale normalization and time alignment in sequence, obtains the preprocessed image and outputs it to the image analysis module; The activity state of the voice data obtained by the voice data packet detection includes: a voice occurrence area, a voice silence area and a voice data packet loss area, wherein: distinguishing whether the voice is silent is conducive to reducing unnecessary voice recognition and completion, and marking the voice data packet area is for subsequent voice completion of this part of the area. The voice data packet detection performs a first classification of the voice activity, and labels the voice area at this time as TRUE or NONE according to whether the voice data packet at the current moment is received. The area labeled NONE will be completed by the semantic text reasoning module and the voice completion module, and the area labeled TRUE will further distinguish the areas where the voice appears and disappears by endpoint detection; The speech framing described above is that the overall signal is not stable due to the time-varying characteristics of the speech signal, and the MFCC feature extraction in the speech analysis module uses Fourier transform, which requires a stable input signal. Therefore, the short-term stability of the speech signal is used to perform framing processing on the speech signal. The speech framing process adopts an overlapping segmentation strategy, and samples each frame according to a preset frame length and overlapping ratio, that is, a frame shift, so that the previous frame smoothly transitions to the next frame, and the continuity of the sample is maintained; The video framing uses the same sampling frequency as the voice framing to convert the video into an image sequence; The facial lip shape control point detection detects the control points of the lips of the person in each frame of the image one by one through an external face recognition engine, including the lip center coordinates, the upper lip middle boundary coordinates, and the left corner coordinates of the mouth; The scale normalization of the lip area, because the subsequent image analysis module only focuses on the relative movement of each lip control point, it is necessary to reduce the influence of the lip size, facial deflection angle, and tilt angle in the image. Therefore, a quadrilateral lip detection frame is fitted according to the left and right corners of the mouth, the lip center coordinates, the upper and lower boundary coordinates of the lip, and the lip center coordinates. The lip control points are rotated and scaled to a uniform size through perspective transformation to achieve scale normalization of the lip area and retain the continuity of the control point movement trajectory; The time alignment described above requires that each frame of audio corresponds to a frame of image in order to facilitate cross-modal interaction between speech and video modalities. Since the trajectory of short-time lip control points is approximately replaced by a simple curve, the set of lip control points corresponding to each frame of audio is fitted by using the Lagrange interpolation method, thereby achieving time alignment from image to audio.

3. The speech adaptive completion system based on multimodal knowledge graph according to claim 1 is characterized in that: The data reasoner includes: a semantic text reasoning module and a speech completion module, wherein: the semantic text reasoning module identifies the knowledge field involved in the speech packet loss area according to the historical text of the current session, and predicts the candidate text based on the spatiotemporal knowledge graph to prune and optimize the solution space size of text reasoning, and matches the identified phonemes with the text in the solution space, so as to infer and generate the completed text and output it to the speech completion module; the speech completion module synthesizes the missing speech according to the completed text and the collected user speech waveform features, and fills the completed speech into the original speech through speech splicing to form a complete and natural speech segment.

4. The speech adaptive completion system based on multimodal knowledge graph according to claim 1 is characterized in that: The enhanced lip feature modality refers to: using a phoneme inference model with the same structure as the cross-modal conversion model, receiving a joint representation and outputting a temporal phoneme posterior probability matrix y=(y1,y2,...,yt,...,yT) of size (T,|A|), where: |A| is the size of the phoneme set A to be recognized, and each column of y is (yt1,yt2,...,yta,...,ytA), indicating the probability that the tth frame is a certain phoneme a; The phoneme inference model is trained in the following way: CTC is connected as a transcription layer after the BILSTM of the phoneme prediction model decoder, in order to improve the probability p(L|x) that BILSTM outputs the correct result when a given input x is given. Since a phoneme in L is composed of the prediction results of multiple time slices in y, there are multiple paths π that constitute L, that is, B(π) = L, B is a mapping function, then Transcriptional layer CTCs by gradient Adjust the LSTM parameter ω so that for the input sample π∈B -1 (L) makes p(L|x) maximum; The phoneme recognition means: for the area where the voice data packet is lost, the lip trajectory features extracted from the video are input into the phoneme inference model, and a phoneme inference vector of size |A| is obtained for each frame, and each value P(a), a∈A in the vector represents the probability that the frame is each phoneme.

5. The speech adaptive completion system based on multimodal knowledge graph according to claim 1, characterized in that: The domain conversation modeling refers to: inferring the knowledge domain of the semantic context based on historical texts, inferring the knowledge domain involved in the historical text by defining the discriminability measure key of domain keywords, and then combining the temporal correlation measure EMI(e,w) between text entities to represent the possibility of entity w appearing after entity e within a certain text step, and splicing EMI(e,w) with a probability greater than a certain threshold into a domain text vector as output, thereby realizing the modeling of domain conversations; first, when training the domain conversation model, an initial text set for each domain is generated based on conversation samples from different domains, and domain entities are associated with text entities in a many-to-many manner in a multimodal knowledge graph; then, in order to generate recognizable domain keywords, the frequency f of each text word j appearing in the domain text set i is calculated. ij , count the maximum frequency max_f, and calculate the number of times each text word appears in N fields n j , through the formula Calculate the degree of recognition of text word j for domain i. Compared with the TF-IDF formula, this calculation formula not only takes into account the length of the domain text set and converts the text frequency into frequency, but also ensures that the original part of TF calculation is always non-negative through normalization transformation, thus realizing the recognition measurement of domain keywords, thereby identifying the knowledge domain involved by searching for keywords in historical texts; Secondly, in each knowledge domain, the mutual information between entities is obtained by calculating the frequency of two text entities appearing successively within a certain text step length. It is used to indicate the correlation between entity2 and entity1 in text order. When EMI is greater than zero, the larger the value, the greater the probability of occurrence. When EMI is less than zero, the two entities are mutually exclusive, thereby defining the correlation measure between text entities in each field. Finally, during the use of the system, new text entities are expanded and associated with the field entities based on the real data of the session. If multiple sessions repeatedly jump between multiple fields, the fields are split and generated based on the unsupervised clustering method.

6. The speech adaptive completion system based on multimodal knowledge graph according to claim 1, characterized in that: The candidate text prediction described above, i.e., candidate text prediction based on the spatiotemporal knowledge graph, specifically refers to: utilizing the temporal characteristics of historical texts to form a spatiotemporal knowledge graph network, inferring the joint probability representation P(w) of the current candidate text, and splicing P(w) with a probability higher than a certain threshold into a candidate text vector, thereby realizing the pruning of the solution space, specifically including: regarding the historical text that has been formed as a path that wanders between knowledge graph entities, represented by a combination of several text entity binary vectors (from, to); and due to the real-time nature of audio and video calls, the system cannot fully and accurately predict the future at the current moment. Even if the phoneme recognition is correct, it may not be possible to predict a text entity with truly appropriate semantics. Therefore, multiple alternative paths with smaller weights will be retained on other entity nodes before the entity node at the end of the path for semantic backtracking when reasoning errors occur, thereby forming a path wandering graph G at the current moment t. T , based on the multimodal knowledge graph, the path travel graph at each previous moment is superimposed on the time dimension to form a spatiotemporal knowledge graph G, where space refers to the solution space; in order to improve the reasoning efficiency, based on an assumption: the path travel graph at time T only depends on the path travel graph of the previous s time steps, the spatiotemporal knowledge graph network is trained to optimize and reason about the joint probability distribution of G Where P(G t |G t-s:t-1 ) Further, Representing all neighbor entity nodes of from_t has two advantages: it covers the candidate text space and has the ability to mine frequent patterns; according to the above formula, building a text-oriented spatiotemporal graph neural network based on the recurrent neural network RNN ​​requires the following formula parameterization steps: e from_t is the learnable vector associated with from_t, h t-1 (from_t) is the historical semantic vector of from_t, ω to_t is the classifier parameter. Similarly, P(from_t|G t-s:t-1 )→exp(H t-1 T ·ω from_t ), H t-1 T is the historical semantic vector of the entire walking path; on this basis, and H t =RNN 2 (g(G t ),H t-1 ), that is, the historical semantic vector is recursively updated through RNN, g is an aggregation function with attention mechanism, and the importance weight of neighbor entity nodes to from entity nodes is learned through the attention matrix.

7. The speech adaptive completion system based on multimodal knowledge graph according to claim 3 is characterized in that: The matching of the identified phonemes with the text in the solution space refers to: according to the phonemes corresponding to the text, the phoneme inference vector is Cartesian-producted with the domain text vector and the candidate text vector to obtain the text solution space of the intersecting phonemes, and then the probability of each text in the solution space is calculated by the formula P(a)P(w)EMI(e,w), the text with the largest value is used as the completed text for speech completion, and the top three texts with the same phonemes are used as alternative texts to supplement the historical text together to form an alternative path.

8. The speech adaptive completion system based on multimodal knowledge graph according to claim 1, characterized in that: The semantically-sounding sentence is obtained in the following manner: setting a threshold T and a confidence level α, adding the probabilities of the T text entities at the end of each candidate path, taking the path with the highest probability to concatenate and generate a sentence, and at the same time, shrinking the paths with results less than αT so that the graph only contains paths with results greater than αT, thereby further pruning the path walk graph at the current moment and optimizing the efficiency of continuous reasoning of the system.

9. The speech adaptive completion system based on multimodal knowledge graph according to claim 3 is characterized in that: The synthesizing of the missing speech refers to: converting the text sequence with timestamps into a phoneme sequence with timestamps, learning the mapping of phonemes to waveform features according to the waveform features of the user's speech through the TTS model, and converting the features back into waveforms through the vocoder; The speech splicing refers to: splicing the existing speech waveform and the completed speech waveform after stretching and fitting near the connection point, so that the speech transition is smoother and more natural.

Images