Creation method and system based on speech recognition and text generation model

Abstract

The invention provides a text creation method and system based on speech recognition and a text generation model, and relates to the technical field of natural language processing and artificial intelligence. Carrying out redundant word filtering on the voice data through acoustic feature analysis, and carrying out dialect phoneme correction by using a phoneme mapping table to obtain structured voice data; converting the structured voice data into a text through an encoder-decoder voice recognition model, and extracting a semantic triple and an emotion tag in combination with emotion analysis and dependency syntactic analysis to form structured information; generating a constraint parameter table according to the target body, inputting the structured information and the constraint parameter table into a text generation model, and generating a to-be-processed text; fusing the user preference information with the to-be-processed text to generate a first target text; and evaluating the text quality based on a resonance index threshold, and generating a final target text through iterative optimization. According to the method, the accuracy of voice transcription of the old people is improved, and the effectiveness of text generation in the aspects of form specification and emotion expression is enhanced.

Description

Technical Field

[0001] This disclosure relates to the fields of natural language processing and artificial intelligence, and more specifically, to a text creation method and system based on speech recognition and text generation models. Background Technology

[0002] Currently, the traditional and mainstream methods in speech recognition have shifted from early architectures based on Hidden Markov Models and Gaussian Mixture Models to end-to-end models based on deep learning, such as connectionist temporal classification models. These models have matured in standard Mandarin and clear, uniform speech. However, in speech recognition, these models are not well-suited to non-standard speech patterns commonly found in the elderly, such as heavy dialects, high-frequency redundant words, and emotional pauses. This leads to a high error rate in recognized text and compromised semantic integrity, creating potential for semantic distortion and information loss in the subsequent generation process. Secondly, at the text generation level, while pre-trained models can generate fluent sentences, their training data is mostly derived from general corpora, lacking an internalized understanding and execution ability of strict formal rules such as text meter, tone, parallelism, and imagery. The generated text lacks inherent rhythmic beauty and deep cultural logic, failing to meet the aesthetic and cultural resonance needs of the elderly for traditional text forms. Existing technologies improve the robustness of front-end speech recognition by using signal processing techniques such as volume normalization, and optimize text output format by employing keyword replacement and rhyme checking rules in the back-end. However, these local optimization strategies fail to construct a complete solution encompassing adaptive processing of front-end acoustic features, deep semantic and sentiment analysis in the middle layer, and deep integration of formal rules and content in the back-end. Their drawback lies in the lack of coordination mechanisms between processing modules, making it impossible to systematically solve the problem of converting non-standardized speech input into structured text output.

[0003] Therefore, there is an urgent need to further improve upon existing technologies, taking into account adaptive speech processing, deep semantics, and emotional analysis. By constructing a full-link text generation method and system that integrates a dedicated elderly speech processing mechanism, genre-constrained compilation, and closed-loop correction of resonance indicators, we can achieve stable generation of texts from non-standard speech to texts with strict metrical rules and emotional resonance, thereby more comprehensively enhancing the actual experience and spiritual satisfaction of the elderly in cultural creation and emotional expression through technological means. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this disclosure provides a text creation method and system based on speech recognition and text generation models.

[0005] In a first aspect, embodiments of this disclosure provide a text creation method based on speech recognition and text generation models, including:

[0006] Acquire target user data, which includes: target user's voice data, target genre, and user preference information;

[0007] The voice data of the target user is preprocessed to obtain regularized voice data;

[0008] The regularized speech data is input into the speech recognition model, and the regularized speech data is converted into text through an encoder-decoder structure;

[0009] Based on the LSTM sentiment analysis model, sentiment tags are extracted from the text to construct vectors, and structured information is generated through feature fusion.

[0010] Based on the target genre, a constraint parameter table is generated, and the structured information and the constraint parameter table are fused using a preset text generation model to generate the text to be processed.

[0011] Based on the user preference information, it is fused with the text to be processed to generate the first target text;

[0012] Based on a preset resonance index threshold, the resonance index value corresponding to the first target text is calculated. If the resonance index value is lower than the resonance index threshold, text fine-tuning information is obtained to replace the first target text until the resonance index value reaches the resonance index threshold, and then the second target text is generated.

[0013] In one optional implementation, the acquisition of target user data includes: the target user's voice data, target genre, and user preference information, including:

[0014] Based on the voice data of the target user, the voice data is converted into a digital audio signal using an analog-to-digital converter;

[0015] Based on the digitized audio signal, the spectral features are calculated by Fourier transform and acoustic features are extracted to obtain an acoustic feature sequence, which includes Mel frequency cepstral coefficients and fundamental frequency.

[0016] The acoustic feature sequence is input into a pre-trained redundancy filter. By analyzing the temporal change pattern of the acoustic feature sequence, the audio segment to be processed is identified. The audio segment to be processed includes: the start time point, the end time point, and the audio segment corresponding to the preset pause word.

[0017] In response to the recognition of the audio segment to be processed, the audio segment corresponding to the preset pause word between the start time point and the end time point is cut out;

[0018] The breaks in the cut-off digital audio signal are smoothed to obtain the first normalized speech data;

[0019] Based on the first regularized speech data, dialect phoneme correction is performed using a preset phoneme mapping table to obtain the second regularized speech data.

[0020] In one optional implementation, the step of performing dialect phoneme correction based on the first regularized speech data using a preset phoneme mapping table to obtain the second regularized speech data includes:

[0021] The preset phoneme mapping table is stored in key-value pairs, including dialect phonemes and standard phonemes corresponding to the dialect phonemes;

[0022] The first regularized speech data is localized by an acoustic model to identify dialect phoneme variants. The localization of phoneme boundaries includes the start time point, end time point, and duration of the dialect phoneme variant.

[0023] Based on the preset phoneme mapping table, the standard phoneme corresponding to the dialect phoneme variant is found, and the duration is adjusted according to the duration of the dialect phoneme variant to obtain the duration-adjusted standard phoneme sequence.

[0024] Based on the phoneme boundary localization, the standard phoneme sequence is overlaid with the dialect phoneme variant to obtain the second regularized speech data.

[0025] In one optional implementation, the step of extracting sentiment tags from the text to construct a vector based on the LSTM sentiment analysis model, and generating structured information through feature fusion, includes:

[0026] The second regularized speech data is input into the speech recognition model. The acoustic feature sequence of the second regularized speech data is mapped into the corresponding character sequence output through the encoder-decoder structure to obtain the text sequence after speech recognition.

[0027] Named entity recognition is performed based on the text sequence, and semantic triples are extracted through dependency parsing. The semantic triples include: person, place, and event.

[0028] Simultaneously, the text sequence is input into the LSTM sentiment analysis model. The pre-trained LSTM sentiment analysis model obtains the contextual feature vector of the text sequence and outputs the sentiment label of the text sequence in combination with the sentiment dictionary. The sentiment label includes: positive, negative and neutral.

[0029] The semantic triples and the sentiment tags are stored in string format;

[0030] The sentiment dictionary is a word-sentiment mapping library. By querying the sentiment dictionary, the corresponding sentiment weight vector in the text sequence can be obtained.

[0031] The sentiment weight vector and the context feature vector are fused using the classifier in the LSTM sentiment analysis model to obtain the sentiment label;

[0032] The semantic triples are fused with the sentiment tags to generate structured information.

[0033] In one optional implementation, fusing the semantic triples with the sentiment tags to generate structured information includes:

[0034] Based on the semantic triples, the character field string and the location field string are merged to obtain the image string;

[0035] Using a pre-trained word vector model, the imagery string, event field string, and sentiment tag string are mapped to corresponding imagery vectors, event vectors, and sentiment vectors, respectively.

[0036] By concatenating vectors, the image vector, the event vector, and the emotion vector are fused to obtain a ternary cluster vector, thereby generating the structured information.

[0037] In one optional implementation, the step of generating a constraint parameter table based on the target genre and fusing it with the structured information to generate the text to be processed includes:

[0038] Obtain the target genre and generate the corresponding constraint parameter table, wherein the parameters of the constraint parameter table include at least: number of lines, number of characters per line, rhyme pattern, tonal rules and parallelism requirements;

[0039] The constraint parameter table is compiled, the parameters are converted into corresponding rule control signals, and the rule control signals are integrated and encapsulated to obtain a structured rule graph.

[0040] The ternary cluster vector and the structured rule graph are input into a Transformer-based text generation model;

[0041] In the decoding and generation process of the text generation model, the content semantic generation conditions are provided based on the ternary cluster vector, and the metrical form control conditions are provided based on the structured rule graph;

[0042] The semantic generation conditions of the content are fused with the metrical form control conditions through an attention mechanism to generate the text to be processed.

[0043] In one optional implementation, the step of fusing the content semantic generation conditions with the metrical form control conditions through an attention mechanism to generate the text to be processed includes:

[0044] Obtain user preference information, which includes at least the set of words selected by the user and rhyme information;

[0045] By calculating language similarity, the user preference information is used to replace the corresponding words in the text to be processed to generate the first target text;

[0046] Based on the first target text, evaluation and correction are performed using a preset resonance index threshold.

[0047] In one optional implementation, the evaluation and correction based on the first target text using a preset resonance index threshold includes:

[0048] Calculate the resonance index value of the first target text and compare the resonance index value with a preset resonance index threshold. The resonance index includes: theme coverage, rhythm compliance, emotional consistency and readability.

[0049] Based on the first target text and the triple cluster vector, the matching degree between the semantic vector of the first target text and the triple cluster vector is calculated by a keyword matching algorithm to obtain the topic coverage rate. The threshold of the topic coverage rate is set as a first threshold.

[0050] The rhyme detection algorithm is used to calculate the proportion of the first target text that conforms to the preset rhyme compliance, and the rhyme compliance rate is obtained. The threshold of the rhyme compliance rate is set as a second threshold.

[0051] Based on the LSTM sentiment analysis model, the matching degree between the first target text and the sentiment tag is calculated to obtain the sentiment consistency. The threshold of the sentiment consistency is set as a third threshold.

[0052] The readability score of the first target text is calculated based on the prosodic compliance rate, and the threshold of the readability score is set as a fourth threshold.

[0053] If the resonance index value is lower than the resonance index threshold, the first target text is replaced to generate the second target text.

[0054] In an optional implementation, the step of replacing the first target text to generate the second target text when the resonance index value is lower than the resonance index threshold includes:

[0055] Calculate the resonance index value of the first target text, and compare the resonance index value with the corresponding resonance index threshold.

[0056] Identify resonance index items that do not reach the resonance index threshold, and mark the text segments in the first target text to obtain the text segments to be processed;

[0057] The text fine-tuning information is obtained, and the text fragment to be processed is replaced by semantic similarity calculation to obtain the target text to be evaluated;

[0058] By performing evaluation and replacement on the target text to be evaluated, and when the resonance index value reaches the corresponding resonance index threshold, the second target text is output.

[0059] Secondly, embodiments of this disclosure also provide a text creation system based on speech recognition and text generation models, including:

[0060] The data acquisition module is used to acquire target user data, which includes: target user voice data, target genre, and user preference information;

[0061] The preprocessing module is used to preprocess the voice data of the target user to obtain regularized voice data;

[0062] The conversion module is used to input the regularized speech data into the speech recognition model and convert the regularized speech data into text through an encoder-decoder structure;

[0063] The semantic understanding module is used to extract the sentiment tags of the text based on the LSTM sentiment analysis model, construct vectors, and generate structured information through feature fusion.

[0064] The fusion module is used to generate a constraint parameter table based on the target genre, and to fuse the structured information with the constraint parameter table using a preset text generation model to generate the text to be processed.

[0065] The generation module is used to fuse the user preference information with the text to be processed to generate a first target text;

[0066] The evaluation and correction module is used to calculate the resonance index value corresponding to the first target text based on a preset resonance index threshold. If the resonance index value is lower than the resonance index threshold, text fine-tuning information is obtained until the resonance index value reaches the resonance index threshold, and then the second target text is generated.

[0067] Compared with existing technologies, this invention improves the accuracy and robustness of speech recognition for the elderly by constructing a preprocessing system that includes redundant word filtering and dialect phoneme correction, achieving stable conversion from non-standard speech to regular text. By integrating semantic analysis and emotion modeling techniques, it effectively enhances the ability to extract personal experiences and emotional elements from text. By compiling textual formal constraints into a structured rule graph and integrating it with semantic content, it improves the compliance of generated text with formal norms such as meter and rhyme. By establishing an evaluation and correction mechanism based on multi-dimensional resonance indicators, it optimizes the overall performance of text in terms of thematic coverage, emotional consistency, rhythmic compliance, and readability. At the same time, through localized privacy protection design, it ensures user data security, enabling the generated text works to more accurately convey the emotional core of the elderly, significantly improving the expressive satisfaction and cultural participation of the elderly population. Attached Figure Description

[0068] Figure 1 A flowchart illustrating the text creation method based on speech recognition and text generation models provided in this disclosure embodiment;

[0069] Figure 2 A flowchart illustrating the generation of a phoneme mapping table based on speech recognition, as provided in this embodiment of the disclosure;

[0070] Figure 3 A flowchart illustrating text generation based on semantic triples and sentiment vectors provided in this embodiment of the disclosure;

[0071] Figure 4 This is a schematic diagram of a text creation system based on speech recognition and text generation models provided in an embodiment of this disclosure. Detailed Implementation

[0072] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0073] In this embodiment, the technical solution of the present invention is essentially an integrated natural language processing and artificial intelligence algorithm system. Through specific acoustic feature extraction algorithms, parameterized training and inference processes of deep learning models, and mathematical operations of vector fusion and attention mechanisms, it achieves automated conversion from non-standard speech input to structured text output.

[0074] Specifically, the technical solution of this invention addresses the shortcomings of existing technologies in speech recognition, such as insufficient robustness to elderly dialects and redundant words, and the weak internalization and execution capability of text generation models for formal constraints. Through the compilation and encapsulation of a pre-defined phoneme mapping table and constraint parameter table, and the quantitative calculation of multi-dimensional resonance indicators, this invention provides a repeatable and quantifiable technical path, ensuring the accuracy and consistency of the output results, rather than relying on subjective intellectual activity or abstract rules.

[0075] In this application, the text creation methods and systems include poetry creation methods and systems.

[0076] See Figure 1 The diagram shows a flowchart of a text creation method based on speech recognition and text generation models provided in this disclosure. The method includes steps S101 to S107, wherein:

[0077] S101: Obtain target user data, the target user data including: target user's voice data, target genre and user preference information;

[0078] S102: Preprocess the voice data of the target user to obtain regularized voice data;

[0079] S103: Input the regularized speech data into the speech recognition model, and convert the regularized speech data into text through an encoder-decoder structure;

[0080] S104: Based on the LSTM sentiment analysis model, extract the sentiment tags of the text to construct vectors, and generate structured information through feature fusion;

[0081] S105: Based on the target genre, generate a constraint parameter table, and use a preset text generation model to fuse the structured information with the constraint parameter table to generate the text to be processed;

[0082] S106: Based on the user preference information, merge it with the text to be processed to generate the first target text;

[0083] S107: Based on a preset resonance index threshold, calculate the resonance index value corresponding to the first target text. If the resonance index value is lower than the resonance index threshold, obtain text fine-tuning information to replace the first target text until the resonance index value reaches the resonance index threshold, and generate the second target text.

[0084] In practice, the user's original voice data is acquired and converted into a digital audio signal using an analog-to-digital converter.

[0085] The raw voice data is obtained through the audio acquisition device of the user terminal, which records the analog signal of the continuous waveform of the sound in the time domain. The user terminal includes: a mobile phone, headphones, and a computer microphone.

[0086] In a specific implementation, based on the digitized audio signal, the spectral characteristics are calculated by Fourier transform and acoustic features are extracted to obtain an acoustic feature sequence, which includes Mel frequency cepstral coefficients and fundamental frequency;

[0087] The Mel frequency cepstral coefficients smooth and compress the spectrum by simulating human hearing, extract the formant structure representing the shape of the vocal tract, and describe the emitted phonemes and words through the temporal combination of values, and reflect the speaker's timbre characteristics.

[0088] The fundamental frequency is used to quantify the pitch of speech by calculating the frequency of vocal cord vibration;

[0089] The acoustic feature sequence is input into a pre-trained redundancy filter. By analyzing the temporal change pattern of the acoustic feature sequence, the audio segment to be processed is identified. The audio segment to be processed includes: a start time point, an end time point, and an audio segment corresponding to a preset pause word. Redundancy words include: “um”, “ah”, “that”, and “then”.

[0090] As an optional implementation, the pre-trained redundant word filter can adopt a combined structure model of a one-dimensional convolutional neural network and a gated recurrent unit. The input of the redundant word filter includes the Mel frequency cepstral coefficients and the fundamental frequency, and the output is the probability value of each time frame belonging to a redundant word.

[0091] For example, based on the original speech data, the audio waveform "[mute][um][mute][that][mute][me][ah][mute][today I'm going][mute][park][inside][mute][walk][mute]" is obtained. The system calculates the probability of redundant words for these time frame segments in turn through the redundant word filter, and uses a sliding window of a predetermined length to judge the average probability. If the average probability within the window exceeds a set threshold (such as 0.7), the start time point and end time point corresponding to the window are marked as audio segments to be processed.

[0092] For example, the corresponding audio segments “[mute][um]”, “[mute][that]”, and “[ah][mute]” exceed the threshold after calculation and are determined to be the audio segments to be processed. Valid content such as “I” and “park” are excluded by the filter due to their low probability of being redundant. At the same time, for redundant content with repetitive semantics in “inside”, the redundant word filter further uses post-processing logic to determine and mark it as the audio segments to be processed.

[0093] In response to the recognition of the audio segment to be processed, the audio segment corresponding to the preset pause word between the start time point and the end time point is cut out;

[0094] For example, the cut-off digital audio signal is [I], [today], [go], [park], [take a walk]. The breakpoints of the cut-off digital audio signal are smoothed to obtain the first regularized speech data.

[0095] As an optional implementation, the smoothing process includes:

[0096] By using overlapping fading processing, a signal segment is selected before the start time point and a signal segment is selected after the end time point. The two signal segments are cross-faded using a linear function to achieve a smooth energy transition.

[0097] By applying a window function, a smoothing window function is used at the breakpoint to weight the signal at the breakpoint, causing its amplitude to decay smoothly.

[0098] By using waveform similarity-based splicing processing, points with waveform phase alignment are found at the breakpoints and spliced ​​together.

[0099] Based on the first regularized speech data, dialect phoneme correction is performed using a preset phoneme mapping table to obtain the second regularized speech data.

[0100] As an optional implementation, the preset phoneme mapping table is stored in key-value pairs, including dialect phonemes and standard phonemes corresponding to the dialect phonemes;

[0101] The preset phoneme mapping table serves as a structured correspondence database, storing the dialect phonemes and the standard phonemes in key-value pairs.

[0102] The “key” is used to store the corrected dialect phoneme, and the “value” is used to store the standard phoneme corresponding to the dialect phoneme.

[0103] refer to Figure 2 As shown, the generation of the phoneme mapping table includes:

[0104] S201: Establish an original corpus by collecting speech data from dialect areas and their corresponding standard pronunciation annotations;

[0105] S202: Standardized dialect audio data is obtained by performing format unification, noise reduction and enhancement, volume normalization and silent segment removal on the original corpus through automated audio processing;

[0106] S203: Use a speech recognition system to convert the standardized dialect audio data into a phoneme sequence and generate a corresponding standard Mandarin phoneme sequence;

[0107] S204: Align the dialect phonemes and the standard phonemes on the time axis through forced alignment technology, establish the corresponding relationship of phonemes, and form a phoneme mapping table.

[0108] Exemplarily, taking Sichuan dialect as an example, by collecting Sichuan dialect audio-visual and text data in the public network, an original corpus is established, and the original corpus includes: dialect dubbing, local news, short videos, and audiobooks; and a Sichuan dialect phoneme mapping table is formed.

[0109] Exemplarily, based on the first regularized speech data "I went for a walk in the park today", the standard phonemes are " ", and the Sichuan dialect phonemes are "[ng][jin][tian][i][gong][yan][san][bu]";

[0110] Based on the comparison between the standard phonemes and the Sichuan dialect phonemes, it is obtained that in the character "我", the Sichuan dialect initial consonant "[ng]" corresponds to the zero initial consonant or the initial consonant "w" in standard Mandarin; in the character "去", when the initial consonants are "j, q, x", the Sichuan dialect final uses the front tooth rhyme "i" to replace the rounded front rhyme "ü" in standard Mandarin; in the character "园", the Sichuan dialect pronounces the final "[üan]" as "[yan]".

[0111] By comparing with the phoneme mapping table, mark "ng", "i", "yan" as dialect phoneme variants, and perform phoneme boundary localization through an acoustic model. The phoneme boundary localization includes: the start time point, end time point and duration length of the dialect phoneme variant.

[0112] As an optional implementation manner, the acoustic model is based on a deep neural network and consists of an input layer, a feature extraction layer and an output layer. The input layer receives the first regularized speech data. The feature extraction layer includes three layers of bidirectional long short-term memory networks. The output layer is a fully connected layer, and combines a softmax activation function to output the probability distribution of each time frame on all preset phoneme categories.

[0113] The preset phoneme categories are a set of phonemes defined in the acoustic model training stage, including all standard phonemes and dialect phoneme variants.

[0114] By using a temporal classification decoding algorithm to process the output probability distribution, the temporal classification decoding algorithm traverses each time frame and selects the phoneme label with the highest probability;

[0115] The time frame is an analysis unit obtained by segmenting the audio signal into frames, and the phoneme labels come from a predefined set of phonemes during the acoustic model training phase.

[0116] As an optional implementation, adjacent identical phoneme tags are merged and blank tags representing silence are deleted to generate a phoneme sequence with time boundary information, while recording the start frame index and end frame index of each phoneme segment.

[0117] Based on the audio sampling rate and frame shift parameters, the frame index is converted into a millisecond timestamp;

[0118] The starting time point is obtained by multiplying the starting frame index by the frame shift parameter and then dividing by the audio sampling rate.

[0119] The end time point is obtained by multiplying the end frame index by the frame shift parameter and then dividing by the audio sampling rate.

[0120] The duration is obtained by subtracting the start time from the end time.

[0121] The audio sampling rate is the number of samples collected per second during analog-to-digital conversion, and the frame shift parameter is the time interval between adjacent analysis frames.

[0122] Based on the start and end times of the dialect phoneme variant, the corresponding audio segment is cut out from the first normalized speech data, and the adjusted standard phoneme waveform is embedded at that position.

[0123] Based on the phoneme boundary localization, the standard phoneme sequence is overlaid with the dialect phoneme variant to obtain the second regularized speech data after phoneme correction.

[0124] For example, the start and end times of a specified phoneme (e.g., "ng") in the audio waveform data are calculated based on an acoustic model, and the range of sample points corresponding to it in the audio waveform data is determined according to a preset audio sampling rate.

[0125] By performing array slicing operations, the original waveform data within the range of the located sample points is removed from the first regularized speech data to form a blank audio segment;

[0126] According to the preset phoneme mapping table, the original waveform of the standard phoneme corresponding to the dialect phoneme variant is found, and the duration is adjusted by digital signal processing algorithm so that its duration is completely consistent with the removed dialect phoneme variant.

[0127] In practice, the audio duration can be stretched or compressed by superimposing WSOLA waveforms to obtain the standard phoneme sequence "[wo][jin][tian][qv][gong][üan][san][bu]", which serves as the second normalized speech data.

[0128] In a specific implementation, the second regularized speech data is input into a speech recognition model. The acoustic feature sequence of the second regularized speech data is mapped to the corresponding character sequence through an encoder-decoder structure, and the resulting text sequence after speech recognition is obtained.

[0129] See Figure 3 The flowchart shown is a process for text generation based on semantic triples and sentiment vectors provided in this embodiment of the disclosure, including:

[0130] For example, the second regularized speech data, "I went for a walk in the park today," is input via voice. After acoustic feature extraction, a Mel-spectrum feature sequence is obtained. The encoder converts the feature sequence into an acoustic feature representation. The decoder generates the corresponding text sequence step by step based on an attention mechanism and finally outputs the text sequence "I went for a walk in the park today."

[0131] Named entity recognition is performed based on the text sequence, and semantic triples are extracted through dependency parsing. The semantic triples include: person, place, and event.

[0132] The named entity recognition uses a deep learning-based sequence labeling model to assign entity category labels to each word in the text sequence and extract entity information based on the output entity label sequence.

[0133] The dependency parsing uses a neural network-based dependency parser to construct a grammatical structure tree of the statement. It traverses the grammatical structure tree to identify the core predicate verb, determines the person field based on the subject-predicate relationship, determines the event field based on the verb-object relationship, and determines the location field based on the adverbial of place, forming semantic triples.

[0134] As an optional implementation, named entity recognition is performed based on the text sequence, using a sequence labeling model based on BERT-BiLSTM-CRF;

[0135] The sequence labeling model adopts the BIO labeling system, where B represents the start of an entity, I represents the interior of an entity, and O represents a non-entity. The specific labeling system includes B-PER for the start of a person entity, I-PER for the interior of a person entity, B-LOC for the start of a location entity, I-LOC for the interior of a location entity, B-ORG for the start of an organization entity, I-ORG for the interior of an organization entity, and O for a non-entity.

[0136] For example, taking the text sequence "Today I went for a walk in the park" as an example, after word segmentation, the resulting word sequence contains "I", "today", "go", "park", and "walk".

[0137] The named entity recognition process converts each word into a context-sensitive vector representation through a BERT encoding layer.

[0138] The vector representation of "park" contains semantic information from the context of "go" and "take a walk";

[0139] The sequence is processed in two directions, from front to back and from back to front, using a BiLSTM layer; the feature vector of each position is output; finally, the globally optimal label sequence is output using the CRF layer with the transition matrix obtained from training; the label sequence is "O", "O", "O", "B-LOC", "O".

[0140] Among them, "I" is marked with O to indicate a non-entity; "Today" is marked with O to indicate a non-entity; "Go" is marked with O to indicate a non-entity; "Park" is marked with B-LOC to indicate the start of a location entity; "Walk" is marked with O to indicate a non-entity; "Park" corresponding to the B-LOC label is identified as a location entity.

[0141] Dependency parsing is performed on the text sequence “Today I went for a walk in the park” to build a dependency tree, identify the core predicate verb “go”, find the subject “I” through the subject-predicate relation, find the object “park walk” through the verb-object relation, and finally generate the semantic triple {person: I, location: park, event: walk}.

[0142] In practice, the text sequence is simultaneously input into the LSTM sentiment analysis model. The pre-trained LSTM sentiment analysis model obtains the contextual feature vector of the text sequence and outputs the sentiment label of the text sequence in combination with the sentiment dictionary. The sentiment label includes: positive, negative and neutral.

[0143] The sentiment dictionary is a word-sentiment mapping library. By querying the sentiment dictionary, the corresponding sentiment weight vector in the text sequence can be obtained.

[0144] For example, a query can be performed using the BosonNLP sentiment dictionary, which is stored in key-value pairs, where the key is a word and the value is the corresponding sentiment weight vector.

[0145] In practice, the emotional labels include three categories: positive, negative, and neutral, which correspond to positive, negative, and neutral emotional tendencies, respectively.

[0146] The specific weight values ​​of the sentiment labels can be flexibly adjusted by the user according to the actual application scenario and target needs. For example, positive labels correspond to positive weights, negative labels correspond to negative weights, and neutral labels correspond to zero or near-zero weights, so as to be appropriately integrated with the context feature vector in the classifier of the sentiment analysis model.

[0147] For example, the input text sequence is segmented to obtain word vectors for [I, today, go, park, walk], and each word in the word vectors is traversed to match the query in the sentiment dictionary;

[0148] For words that match successfully, the sentiment weight vector is extracted; words that do not match successfully are assigned a neutral label weight by default.

[0149] The sentiment weight vector and the context feature vector are fused using the classifier in the LSTM model to obtain the sentiment label.

[0150] For example, the classifier uses a multilayer perceptron architecture to dimensionally align the sentiment weight vector with the context feature vector through a fully connected layer, performs weighted fusion through a feature gating mechanism, and outputs the sentiment label through a softmax function.

[0151] Based on the semantic triples, the character field string and the location field string are merged to obtain the image string;

[0152] Using a pre-trained word vector model, the imagery string, event field string, and sentiment tag string are mapped to corresponding imagery vectors, event vectors, and sentiment vectors, respectively.

[0153] By concatenating vectors, the image vector, the event vector, and the emotion vector are fused to generate a ternary cluster vector, thus generating the structured information.

[0154] For example, the ternary cluster vector can be obtained (imagery: I am in the park, event: taking a walk, emotion: pleasant and calm).

[0155] Based on the user-defined target genre, a constraint parameter table is generated, and the structured information and the constraint parameter table are fused using a preset text generation model to generate the text to be processed.

[0156] The target genre includes poetry.

[0157] The poetic genres mentioned include at least: five-character regulated verse, seven-character quatrain, and free verse;

[0158] By querying the poetry genre, a corresponding constraint parameter table is generated, where the parameters of the constraint parameter table include: the number of lines, the number of characters per line, the rhyming pattern, the tonal pattern rules, and the antithesis requirements.

[0159] As an optional implementation, the number of lines of the seven-character quatrain is constrained to 4 lines, the number of characters per line is constrained to 7 characters, the rhyming pattern is constrained to the last characters of the 2nd and 4th lines rhyming with level tones, the tonal pattern rules are constrained to the tonal pattern format of the seven-character quatrain, the first line is "Ping Ping Ze Ze Ping Ping Ze", the second line is "Ze Ze Ping Ping Ze Ze Ping", and the antithesis requirements are not constrained.

[0160] Based on the constraint parameter table, compilation is performed, the parameters are converted into corresponding rule control signals, and the rule control signals are integrated and encapsulated to obtain a structured rule graph;

[0161] The ternary cluster vector and the structured rule graph are input into a text generation model based on Transformer;

[0162] In the decoding and generation process of the text generation model, content semantic generation conditions are provided based on the ternary cluster vector, and metrical form control conditions are provided based on the structured rule graph;

[0163] The content semantic generation conditions and the metrical form control conditions are fused through an attention mechanism to generate a text to be processed.

[0164] In a specific implementation, the text generation model generates a text to be processed based on the event vector and the image vector in the ternary cluster vector under the tonal pattern constraints of the structured rule graph;

[0165] The content semantic generation conditions extract the "taking a walk" event and the "park" scene, the metrical form control conditions limit the tonal pattern format to "Ping Ping Ze Ze Ping Ping Ze", and after being fused through the attention mechanism, a text to be processed that meets the metrical requirements is output, such as strolling leisurely, wandering, fragrant garden, forest garden;

[0166] If the structured rule graph requires the second line to rhyme, the text generation model generates the text to be processed, such as jian, ran, huan, strengthening the positive weight in the emotional weight vector;

[0167] Based on the user preference information, it is fused with the text to be processed to generate a first target text (such as Strolling leisurely in the fragrant garden at sunset, the wind caresses the willow lines and I am at ease. Suddenly seeing the fallen flowers flowing with the water, only then do I know that tranquility is true joy);

[0168] Based on the first target text, evaluation and correction are performed through a preset resonance index threshold.

[0169] As an optional implementation, the resonance index value of the first target text is calculated and compared with a preset resonance index threshold. The resonance index includes: theme coverage, rhythm compliance, emotional consistency and readability.

[0170] The topic coverage rate is the degree of matching between the generated first target text and the original topic.

[0171] The prosodic compliance rate measures the proportion of the first target text that conforms to traditional prosodic norms. It mainly detects three dimensions: whether the number of characters in each sentence meets the requirement of seven characters, whether the last character of even-numbered sentences rhymes with a level tone, and whether the tones of words and phrases follow established rules. The prosodic compliance rate index is obtained by calculating the percentage of qualified positions out of all detected positions.

[0172] The emotional consistency is used to evaluate the consistency between the emotional expression of the first target text and the original emotional features. Based on the probability distribution output by the sentiment analysis model, it detects whether the emotional tone of the words in the text matches the expected tone.

[0173] The readability is a comprehensive indicator for evaluating the reading experience of a text, examining factors such as sentence fluency, syllable harmony, and semantic comprehension difficulty.

[0174] Based on the first target text and the triple cluster vector, the number of keywords in the first target text that match the triple cluster vector is divided by the total number of keywords contained in the triple cluster vector using a keyword matching algorithm. The resulting quotient is then multiplied by 100% to obtain the topic coverage rate. The threshold of the topic coverage rate is set as a first threshold.

[0175] The rhyme detection algorithm divides the number of lines of poetry in the first target text that meet the preset rhyme requirements by the total number of sentences in the first target text, and then multiplies the resulting quotient by 100% to obtain the rhyme compliance rate. The threshold of the rhyme compliance rate is set as the second threshold.

[0176] Based on the LSTM sentiment analysis model, the matching probability between the first target text and the preset sentiment tag is calculated by the LSTM sentiment analysis model. The matching probability value is multiplied by 100% to obtain the sentiment consistency. The threshold of the sentiment consistency is set as the third threshold.

[0177] Based on the rhythm compliance rate, a weighted calculation is performed by combining the tonal coordination and poetic fluency analysis of the first target text. The tonal coordination is obtained by evaluating the compliance of the distribution of Chinese characters' tones in the poem, and the poetic fluency is obtained by analyzing the semantic coherence and rhythm between the poem lines. The evaluation indicators are weighted and summed according to preset weights to obtain the readability score of the first target text. The threshold of the readability score is set as the fourth threshold.

[0178] As an alternative implementation, the first threshold, the second threshold, the third threshold, and the fourth threshold can be set respectively according to the generation goals of different target genres and the evaluation requirements of the system, so as to reflect the passing standards of theme coverage rate, rhyme compliance rate, emotion consistency, and recitability. The initial setting of each threshold can be determined based on the statistical analysis of historical text corpora and the evaluation results of domain experts, and can be dynamically adjusted according to user feedback or model performance during the operation of the system.

[0179] When any resonance index in the evaluation result fails to reach the corresponding threshold, a replacement operation is performed on the first target text by obtaining the text fine-tuning information of the user to generate a second target text.

[0180] For example, in the case where the theme coverage rate is lower than the first threshold, theme-related imagery can be replaced based on the imagery list of the ternary cluster vector; in the case where the rhyme compliance rate is lower than the second threshold, the rhyming pattern of the poem can be adjusted according to the constraint parameter table; in the case where the emotion consistency is lower than the third threshold, emotion words that do not match the target emotion can be replaced; in the case where the recitability score is lower than the fourth threshold, the sentence structure can be merged and reconstructed. By iteratively performing evaluation and correction until all resonance indexes meet the corresponding threshold conditions, the second target text is output.

[0181] In the specific implementation, parsing is performed based on the ternary cluster vector to obtain a keyword set (such as "I, park, take a walk, happy, quiet"); word segmentation matching is performed on the first target text (such as matching "fragrant garden" as "park" and "stroll leisurely" as "take a walk"), and the number of matched keywords and the total number of keywords are identified. In the case where keywords are not matched (such as "I" not being matched), the calculated theme coverage rate will decrease.

[0182] Based on the rhyme ending characteristics of each sentence in the first target text (such as the rhyme endings of the characters "jian", "ran", "huan", etc. are all of the "an" category), it is judged that the rhyme ending characteristics meet the rhyming requirements;

[0183] In the specific implementation, the LSTM emotion analysis model identifies emotion expression words in the text (such as "joyfully", "pure joy" corresponding to the positive and happy emotion dimension, "quiet" corresponding to the quiet dimension, and "fallen flowers flowing with the water" having a negative meaning), and calculates the relative strength contribution values of various emotion dimensions in the text;

[0184] Through multi-dimensional fusion calculation using an emotion weight vector, an initial calculated value of emotion consistency is obtained, and a probability calibration function, such as the Sigmoid function, is used to map the original score of emotion consistency to the range of 0 to 1 to achieve quantitative emotion evaluation.

[0185] Regarding readability, the importance of three dimensions—rhythmic compliance, tonal harmony, and poetic fluency—was determined based on the text corpus, and weights were assigned to each to form a comprehensive evaluation index system. Rhythmic compliance evaluates whether the verses conform to traditional rhyming rules and tonal norms; tonal harmony assesses whether the tonal variations in the verses are harmonious and appropriate; and poetic fluency focuses on the semantic coherence and naturalness of the entire poem. A readability score is derived by combining the evaluations of these three dimensions.

[0186] For example, if the theme coverage is insufficient in the first target text, text fine-tuning information is obtained (e.g., changing "leisurely stroll" to "I tread"), and the theme integrity is improved by adding the subject "I"; if the emotional consistency is insufficient, user text fine-tuning information is obtained (e.g., replacing "fallen flowers" with "fish shadow"), and the original negative image of "fish shadow" with positive connotations is used to replace it, ensuring that the emotional expression of the text is consistent with the expected tone. Through evaluation and iterative correction, until the resonance index value reaches the resonance index threshold, the second target text is generated.

[0187] Based on the user-provided voice data and the second target text, a user corpus summary is created through a local summary generation process, and the user corpus summary is stored in a local database in encrypted form.

[0188] The system responds to user commands input through an interactive interface, including undo commands, delete commands, upload commands, and complete commands.

[0189] In response to the revocation command, the data state before the command is restored through a data rollback operation;

[0190] In response to the deletion command, stored historical user data and personalized settings are removed via a data clearing operation;

[0191] In response to the upload command, the de-identified structured information vector is uploaded to the cloud server via a secure transmission protocol;

[0192] In response to the completion command, all temporary data is cleared through a data cleanup operation, retaining only the final confirmed target text.

[0193] Reference Figure 4 As shown in the embodiments of this disclosure, a text creation system based on speech recognition and text generation models is provided. The system includes: a data acquisition module 10, a preprocessing module 20, a conversion module 30, a semantic understanding module 40, a fusion module 50, a generation module 60, and an evaluation and correction module 70, wherein:

[0194] Data acquisition module 10: used to acquire target user data, the target user data including: target user's voice data, target genre and user preference information;

[0195] Preprocessing module 20: used to preprocess the voice data of the target user to obtain regularized voice data;

[0196] Conversion module 30: used to input the regularized speech data into the speech recognition model, and convert the regularized speech data into text through an encoder-decoder structure;

[0197] Semantic understanding module 40: used to extract sentiment tags from the text based on the LSTM sentiment analysis model, construct vectors, and generate structured information through feature fusion;

[0198] Fusion module 50: Used to generate a constraint parameter table based on the target genre, and to fuse the structured information with the constraint parameter table using a preset text generation model to generate the text to be processed;

[0199] Generation module 60: used to generate a first target text by fusing the user preference information with the text to be processed;

[0200] Evaluation and correction module 70: is used to calculate the resonance index value corresponding to the first target text based on a preset resonance index threshold, and to obtain text fine-tuning information when the resonance index value is lower than the resonance index threshold, until the resonance index value reaches the resonance index threshold, and then generate the second target text.

[0201] Those skilled in the art will understand that, in the methods described above in the specific embodiments, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic. It should be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0202] In the description of this specification, the terms "exemplary," "for example," "specifically," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

Claims

1. A text creation method based on speech recognition and text generation models, characterized in that, The method includes: Acquire target user data, which includes: target user's voice data, target genre, and user preference information; The voice data of the target user is preprocessed to obtain regularized voice data; The regularized speech data is input into the speech recognition model, and the regularized speech data is converted into text through an encoder-decoder structure; Based on the LSTM sentiment analysis model, sentiment tags are extracted from the text to construct vectors, and structured information is generated through feature fusion. Based on the target genre, a constraint parameter table is generated, and the structured information and the constraint parameter table are fused using a preset text generation model to generate the text to be processed. Based on the user preference information, it is fused with the text to be processed to generate the first target text; Based on a preset resonance index threshold, the resonance index value corresponding to the first target text is calculated. If the resonance index value is lower than the resonance index threshold, text fine-tuning information is obtained to replace the first target text until the resonance index value reaches the resonance index threshold, and then the second target text is generated.

2. The text creation method based on speech recognition and text generation model according to claim 1, characterized in that, The acquisition of target user data includes: the target user's voice data, target genre, and user preference information, including: Based on the voice data of the target user, the voice data is converted into a digital audio signal using an analog-to-digital converter; Based on the digitized audio signal, the spectral features are calculated by Fourier transform and acoustic features are extracted to obtain an acoustic feature sequence, which includes Mel frequency cepstral coefficients and fundamental frequency. The acoustic feature sequence is input into a pre-trained redundancy filter. By analyzing the temporal change pattern of the acoustic feature sequence, the audio segment to be processed is identified. The audio segment to be processed includes: the start time point, the end time point, and the audio segment corresponding to the preset pause word. In response to the recognition of the audio segment to be processed, the audio segment corresponding to the preset pause word between the start time point and the end time point is cut out; The breaks in the cut-off digital audio signal are smoothed to obtain the first normalized speech data; Based on the first regularized speech data, dialect phoneme correction is performed using a preset phoneme mapping table to obtain the second regularized speech data.

3. The text creation method based on speech recognition and text generation model according to claim 2, characterized in that, The step of obtaining second regularized speech data by performing dialect phoneme correction based on the first regularized speech data using a preset phoneme mapping table includes: The preset phoneme mapping table is stored in key-value pairs, including dialect phonemes and standard phonemes corresponding to the dialect phonemes; The first regularized speech data is localized by an acoustic model to identify dialect phoneme variants. The localization of phoneme boundaries includes the start time point, end time point, and duration of the dialect phoneme variant. Based on the preset phoneme mapping table, the standard phoneme corresponding to the dialect phoneme variant is found, and the duration is adjusted according to the duration of the dialect phoneme variant to obtain the duration-adjusted standard phoneme sequence. Based on the phoneme boundary localization, the standard phoneme sequence is overlaid with the dialect phoneme variant to obtain the second regularized speech data.

4. The text creation method based on speech recognition and text generation model according to claim 3, characterized in that, The LSTM-based sentiment analysis model extracts sentiment tags from the text to construct vectors, and generates structured information through feature fusion, including: The second regularized speech data is input into the speech recognition model. The acoustic feature sequence of the second regularized speech data is mapped into the corresponding character sequence output through the encoder-decoder structure to obtain the text sequence after speech recognition. Named entity recognition is performed based on the text sequence, and semantic triples are extracted through dependency parsing. The semantic triples include: person, place, and event. Simultaneously, the text sequence is input into the LSTM sentiment analysis model. The pre-trained LSTM sentiment analysis model obtains the contextual feature vector of the text sequence and outputs the sentiment label of the text sequence in combination with the sentiment dictionary. The sentiment label includes: positive, negative and neutral. The semantic triples and the sentiment tags are stored in string format; The sentiment dictionary is a word-sentiment mapping library. By querying the sentiment dictionary, the corresponding sentiment weight vector in the text sequence can be obtained. The sentiment weight vector and the context feature vector are fused using the classifier in the LSTM model to obtain the sentiment label; The semantic triples are fused with the sentiment tags to generate structured information.

5. The text creation method based on speech recognition and text generation model according to claim 4, characterized in that, The step of fusing the semantic triples with the sentiment tags to generate structured information includes: Based on the semantic triples, the character field string and the location field string are merged to obtain the image string; Using a pre-trained word vector model, the imagery string, event field string, and sentiment tag string are mapped to corresponding imagery vectors, event vectors, and sentiment vectors, respectively. By concatenating vectors, the image vector, the event vector, and the emotion vector are fused to obtain a ternary cluster vector, thereby generating the structured information.

6. The text creation method based on speech recognition and text generation model according to claim 5, characterized in that, The process of generating a constraint parameter table based on the target genre and fusing it with the structured information to generate the text to be processed includes: Obtain the target genre and generate the corresponding constraint parameter table, wherein the parameters of the constraint parameter table include at least: number of lines, number of characters per line, rhyme pattern, tonal rules and parallelism requirements; The constraint parameter table is compiled, the parameters are converted into corresponding rule control signals, and the rule control signals are integrated and encapsulated to obtain a structured rule graph. The ternary cluster vector and the structured rule graph are input into a Transformer-based text generation model; In the decoding and generation process of the text generation model, the content semantic generation conditions are provided based on the ternary cluster vector, and the metrical form control conditions are provided based on the structured rule graph; The semantic generation conditions of the content are fused with the metrical form control conditions through an attention mechanism to generate the text to be processed.

7. The text creation method based on speech recognition and text generation model according to claim 6, characterized in that, The process of fusing the content semantic generation conditions with the metrical form control conditions through an attention mechanism to generate the text to be processed includes: Obtain user preference information, which includes at least the set of words selected by the user and rhyme information; By calculating language similarity, the user preference information is used to replace the corresponding words in the text to be processed to generate the first target text; Based on the first target text, evaluation and correction are performed using a preset resonance index threshold.

8. The text creation method based on speech recognition and text generation model according to claim 7, characterized in that, The evaluation and correction based on the first target text, using a preset resonance index threshold, includes: Calculate the resonance index value of the first target text and compare the resonance index value with a preset resonance index threshold. The resonance index includes: theme coverage, rhythm compliance, emotional consistency and readability. Based on the first target text and the triple cluster vector, the matching degree between the semantic vector of the first target text and the triple cluster vector is calculated by a keyword matching algorithm to obtain the topic coverage rate. The threshold of the topic coverage rate is set as a first threshold. The rhyme detection algorithm is used to calculate the proportion of the first target text that conforms to the preset rhyme compliance, and the rhyme compliance rate is obtained. The threshold of the rhyme compliance rate is set as a second threshold. Based on the LSTM sentiment analysis model, the matching degree between the first target text and the sentiment tag is calculated to obtain the sentiment consistency. The threshold of the sentiment consistency is set as a third threshold. The readability score of the first target text is calculated based on the prosodic compliance rate, and the threshold of the readability score is set as a fourth threshold. If the resonance index value is lower than the resonance index threshold, the first target text is replaced to generate the second target text.

9. The text creation method based on speech recognition and text generation model according to claim 8, characterized in that, The step of replacing the first target text to generate the second target text when the resonance index value is lower than the resonance index threshold includes: Calculate the resonance index value of the first target text, and compare the resonance index value with the corresponding resonance index threshold. Identify resonance index items that do not reach the resonance index threshold, and mark the text segments in the first target text to obtain the text segments to be processed; The text fine-tuning information is obtained, and the text fragment to be processed is replaced by semantic similarity calculation to obtain the target text to be evaluated; By performing evaluation and replacement on the target text to be evaluated, and when the resonance index value reaches the corresponding resonance index threshold, the second target text is output.

10. A text creation system based on speech recognition and text generation models, used to implement the text creation method based on speech recognition and text generation models as described in any one of claims 1-9, characterized in that, The system includes: The data acquisition module is used to acquire target user data, which includes: target user voice data, target genre, and user preference information; The preprocessing module is used to preprocess the voice data of the target user to obtain regularized voice data; The conversion module is used to input the regularized speech data into the speech recognition model and convert the regularized speech data into text through an encoder-decoder structure; The semantic understanding module is used to extract the sentiment tags of the text based on the LSTM sentiment analysis model, construct vectors, and generate structured information through feature fusion. The fusion module is used to generate a constraint parameter table based on the target genre, and to fuse the structured information with the constraint parameter table using a preset text generation model to generate the text to be processed. The generation module is used to fuse the user preference information with the text to be processed to generate a first target text; The evaluation and correction module is used to calculate the resonance index value corresponding to the first target text based on a preset resonance index threshold. If the resonance index value is lower than the resonance index threshold, text fine-tuning information is obtained until the resonance index value reaches the resonance index threshold, and then the second target text is generated.

Images