Audio and video synthesis method and device based on text content quality detection and medium

Abstract

Embodiments of the present disclosure disclose an audio and video synthesis method and device based on text content quality detection and a medium. A specific implementation of the method includes: receiving writing requirement information sent by a user end; generating initial auxiliary writing content information based on the writing requirement information; performing scene entity extraction processing on the initial auxiliary writing content information; performing additional content injection and style adaptation processing on the initial auxiliary writing content information; performing text content quality detection processing on the auxiliary writing content information to obtain a text content quality score; in response to determining that the text content quality score is greater than a preset score value, generating an audio or a video corresponding to the auxiliary writing content information and containing additional content based on the writing requirement information and the auxiliary writing content information; and sending the auxiliary writing content information and the audio or the video to the user end for playing the audio or the video at the user end. The implementation reduces waste of network transmission resources.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to an audio-visual synthesis method, apparatus, and medium based on text content quality detection. Background Technology

[0002] With the rapid development of artificial intelligence technology, AI-assisted writing tools have been widely applied in scenarios such as novel writing, copywriting, and content generation. These tools use natural language processing technology to understand users' writing needs (such as theme, style, and scenario), automatically generating text content that meets the requirements. Audio and video synthesis based on text content quality detection is a technology that generates supplementary writing content based on user writing needs and pushes additional content (e.g., advertising copy, emotional copy). Currently, the common methods for generating supplementary writing content based on user writing needs and pushing additional content are: after generating the supplementary writing content, additional content is pushed through pop-ups, inserts, etc., or by relying on keyword matching to directly insert the matched additional content copy into the generated text.

[0003] However, when using the above method to generate supplementary writing content based on user writing needs and push additional content, the following technical problems often arise: Because the supplementary content in pop-ups and inserts is semantically unrelated to the written text, it easily distracts users. Users need to frequently click the close button to continue watching or listening, resulting in a poor user experience. Furthermore, the directly inserted supplementary content relies solely on keyword matching, failing to match supplementary content with the contextual characteristics and entity elements of the written text. It also lacks style adaptation and quality checks on the text after insertion, leading to semantic logical gaps in the generated supplementary writing content and inconsistent text style. This results in poor quality supplementary writing content, requiring users to regenerate multiple times to obtain usable content. Each generation necessitates resending the generated supplementary writing content, wasting network transmission resources.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose audio and video synthesis and push methods, apparatuses, electronic devices, and computer-readable media based on text content quality detection to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide an audio-visual synthesis method based on text content quality detection. The method includes: receiving writing requirement information sent by a user terminal; generating initial auxiliary writing content information based on the writing requirement information; performing scene entity extraction processing on the initial auxiliary writing content information to obtain scene entity element information; retrieving at least one additional content information matching the scene entity element information from a preset additional content information library; performing additional content injection and style adaptation processing on the initial auxiliary writing content information based on the at least one additional content information to obtain auxiliary writing content information; performing text content quality detection processing on the auxiliary writing content information to obtain a text content quality score; in response to determining that the text content quality score is greater than a preset score value, generating audio or video corresponding to the auxiliary writing content information and containing additional content based on the writing requirement information and the auxiliary writing content information; and sending the auxiliary writing content information, the audio, or the video to the user terminal for playback on the user terminal.

[0008] Secondly, some embodiments of this disclosure provide an audio-visual synthesis apparatus based on text content quality detection. The apparatus includes: a receiving unit configured to receive writing requirement information sent by a user terminal; a first generation unit configured to generate initial auxiliary writing content information based on the writing requirement information; a scene entity extraction unit configured to perform scene entity extraction processing on the initial auxiliary writing content information to obtain scene entity element information; a retrieval unit configured to retrieve at least one additional content information matching the scene entity element information from a preset additional content information database; and an additional content injection unit configured to inject additional content based on the at least one additional content information. The initial auxiliary writing content information undergoes additional content injection and style adaptation processing to obtain auxiliary writing content information; a text content quality detection unit is configured to perform text content quality detection processing on the auxiliary writing content information to obtain a text content quality score; a second generation unit is configured to, in response to determining that the text content quality score is greater than a preset score value, generate audio or video corresponding to the auxiliary writing content information and containing additional content based on the writing requirement information and the auxiliary writing content information; and a sending unit is configured to send the auxiliary writing content information, the audio or video to the user terminal for playback on the user terminal.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above-described embodiments of this disclosure have the following beneficial effects: the audio-visual synthesis method based on text content quality detection in some embodiments of this disclosure improves the user experience and reduces the waste of network transmission resources. Specifically, the reasons for poor user experience and waste of network transmission resources are as follows: because the additional content in pop-ups and inserts is semantically unrelated to the written text, it easily interferes with the user. The user needs to click the close button frequently to continue watching or listening, resulting in a poor user experience. Furthermore, the directly inserted additional content relies solely on keyword matching, without matching and retrieving additional content based on the scene characteristics and entity elements of the written text, and without style adaptation and quality detection of the text after inserting additional content. This leads to logical gaps in the generated auxiliary writing content information in terms of semantics, and the text style is inconsistent, resulting in poor quality of the generated auxiliary writing content. Users need to regenerate multiple times to obtain usable content, and each generation requires resending the generated auxiliary writing content, resulting in waste of network transmission resources. Based on this, the audio-visual synthesis method based on text content quality detection in some embodiments of this disclosure first receives writing requirement information sent by the user terminal. Based on the above writing requirement information, initial auxiliary writing content information is generated. This allows us to obtain initial auxiliary writing content information that meets the user's writing needs. Next, scene entity extraction processing is performed on the initial auxiliary writing content information to obtain scene entity element information. Thus, scene and entity elements are extracted from the writing text, i.e., the initial auxiliary writing content information, to obtain scene entity element information. Then, at least one piece of supplementary content information matching the aforementioned scene entity element information is retrieved from a pre-defined supplementary content information database. This allows us to retrieve supplementary content related to the current writing scenario, reducing semantic bias caused by keyword dependence and making supplementary content a natural extension of the content rather than a distractor. Then, based on the aforementioned at least one piece of supplementary content information, supplementary content injection and style adaptation processing are performed on the initial auxiliary writing content information to obtain auxiliary writing content information. This allows the expression of the supplementary content text to be converted to a form consistent with the style of the original text (such as sentence structure, vocabulary, and rhetoric alignment), eliminating inconsistencies in style and maintaining the logical coherence of the text. Finally, text content quality detection processing is performed on the aforementioned auxiliary writing content information to obtain a text content quality score. This allows for text content quality detection processing of the auxiliary writing content information after the insertion of supplementary content. Next, in response to determining that the text content quality score is greater than a preset score, based on the writing requirement information and the auxiliary writing content information, an audio or video recording corresponding to the auxiliary writing content information and including additional content is generated. Finally, the auxiliary writing content information, the audio, or the video is sent to the user terminal for playback on the user terminal.Therefore, when the quality of the supplementary writing content is high, the audio or video converted from the writing text with the added supplementary content, along with the supplementary writing content information, can be sent to the user's end. Users can obtain usable content all at once without having to regenerate multiple times; each generation only requires sending one high-quality result, thus reducing the waste of network transmission resources caused by multiple regenerations. Furthermore, because it uses scene entity element information extracted from scene entities to retrieve supplementary content, integrates the supplementary content into the writing text, and generates audio and video, it replaces traditional pop-ups, inserts, and other isolated supplementary content display formats, reducing the interference of pop-ups and inserts that are semantically irrelevant to the writing text content and improving the user experience. Moreover, style adaptation processing and text content quality detection are performed during the insertion of supplementary content. When the quality of the supplementary writing content is high, the supplementary writing content information is sent to the user's end, allowing users to obtain usable content all at once without having to regenerate multiple times; each generation only requires sending one high-quality result, thus reducing the waste of network transmission resources caused by multiple regenerations. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the audio-visual synthesis method based on text content quality detection according to the present disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of the audio and video synthesis apparatus based on text content quality detection according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flowchart 100 is shown, illustrating some embodiments of the audio-visual synthesis method based on text content quality detection according to this disclosure. This audio-visual synthesis method based on text content quality detection includes the following steps: Step 101: Receive writing request information sent by the user.

[0021] In some embodiments, the execution entity (e.g., a computing device) of the audio-visual synthesis method based on text content quality detection can receive writing requirement information sent by a user terminal. Here, the user terminal refers to the terminal device used by the user to input writing requirements, including but not limited to smartphones, tablets, and personal computers. The writing requirement information refers to a set of parameters submitted by the user through the user terminal via text input, voice input, or gesture selection, used to guide the generation of text content. The writing requirement information includes writing theme, scene element description information, and text style and effect presentation identifiers. The effect presentation identifiers include plot visualization identifiers or voice broadcast identifiers. The writing theme refers to the core event or topic of the text content that the user expects to generate, such as "friends' gathering," "bedtime story," or "product introduction." The scene element description information refers to the user's description of the time, place, character relationships, environmental details, etc., of the text occurrence, used to define the specific scene of text generation. The text style refers to the user's expected text language expression characteristics, including sentence structure, vocabulary preferences, and emotional tone, such as "colloquial style," "formal written style," or "childlike style." The aforementioned effect rendering identifiers refer to the final output format specified by the user, including either a narrative visualization identifier or a voice narration identifier. Specifically, the narrative visualization identifier requests that the generated text be composited into a video format. The voice narration identifier requests that the generated text be composited into an audio format. For example: A user enters the writing topic "Winnie the Pooh and Honey" into an AI writing application on their smartphone, describes the scene elements as "forest, bear, honey jar," selects "warm and childlike" as the effect rendering identifier, and then clicks the submit button.

[0022] Step 102: Generate initial auxiliary writing content information based on writing needs information.

[0023] In some embodiments, the aforementioned execution entity may generate initial auxiliary writing content information based on the aforementioned writing requirement information.

[0024] In some optional implementations of certain embodiments, the aforementioned execution entity may generate initial auxiliary writing content information based on the aforementioned writing requirement information through the following steps: The first step is to structurally decompose the above writing requirements information to obtain requirement analysis information. This requirement analysis information includes the writing theme, scene element description information, and text style. In practice, the executing entity calls a string parsing function or a regular expression matching algorithm to extract the values ​​of the writing theme field as the writing theme, the scene element description information field as the scene element description information, and the text style field as the text style. Then, the executing entity can determine the extracted writing theme, scene element description information, and text style as the requirement analysis information. The writing theme can be the core event or topic of the text content the user expects to generate; for example, the writing theme could be "Winnie the Pooh collecting honey." The scene element description information can be a textual description of the time, place, relationships between characters, environmental details, etc., of the event; for example, the scene element description information could be "in the forest, a bear, a honey jar." The text style can be the text language expression characteristics the user expects, including sentence structure, vocabulary preferences, and emotional tone; for example, the text style could be "warm and childlike."

[0025] The second step involves inputting the aforementioned requirements analysis information into a pre-defined writing model to generate text content that conforms to the aforementioned writing theme, contains scene elements represented by the aforementioned scene element description information, and has a language style consistent with the aforementioned text style. This text content serves as initial auxiliary writing content information. The pre-defined writing model can be a generative pre-trained model (e.g., the GPT series models) used to generate natural language text from the requirements analysis information. In practice, the executing entity concatenates the writing theme, scene element description information, and text style from the requirements analysis information into a complete prompt word sequence. The format of this prompt word sequence is: "Generate text according to the following requirements: theme is [writing theme], scene is [scene element description information], style is [text style]". Then, the executing entity passes this prompt word sequence as input to the pre-defined writing model to obtain text content that conforms to the aforementioned writing theme, contains scene elements represented by the aforementioned scene element description information, and has a language style consistent with the aforementioned text style. This initial auxiliary writing content information refers to the original text generated by the pre-defined writing model without any additional information injected. The original text mentioned above can be a writing text that satisfies the above writing theme, contains the scene elements mentioned in the above scene element description information (such as "forest", "little bear", "honey jar"), and whose overall language style (such as short sentences, colloquialism, positive emotion) is consistent with the above text style.

[0026] Step 103: Perform scene entity extraction processing on the above initial auxiliary writing content information to obtain scene entity element information.

[0027] In some embodiments, the aforementioned execution entity may perform scene entity extraction processing on the aforementioned initial auxiliary writing content information to obtain scene entity element information.

[0028] In some optional implementations of certain embodiments, the aforementioned execution entity may perform scene entity extraction processing on the initial auxiliary writing content information through the following steps to obtain scene entity element information: The first step involves semantic parsing the initial writing assistance content to obtain a word sequence and a part-of-speech tagging (POS) information set. Each word in the word sequence corresponds to a POS tag in the POS information set. In practice, the executing entity can use Chinese word segmentation technology to segment the initial writing assistance content to obtain a word sequence. Then, the executing entity can input the word sequence into a POS tagging model (e.g., a sequence tagging model) to predict a POS tag for each word, resulting in a POS tag information set. Each POS tag in the POS tag information set can be a POS tag. For example, for the initial writing assistance content "Winnie the Pooh collected a jar of honey in the forest," after semantic parsing, the word sequence is {"Winnie the Pooh," "in," "forest," "collected," "a jar," "honey"}, and the corresponding POS tag information set is {noun, preposition, noun, verb, quantifier phrase, noun}.

[0029] The second step involves extracting a keyword set from the aforementioned word sequence based on the part-of-speech tagging information set. This keyword set includes scene keywords, character keywords, item keywords, and action keywords. The scene keywords include at least one scene keyword; the character keywords include at least one character keyword; the item keywords include at least one character keyword; and the action keywords include at least one action keyword. Scene keywords describe elements of the scene such as location, environment, and time, such as "in the forest," "restaurant," and "evening." Character keywords describe participants in the text, including role names, pronouns, or titles, such as "Winnie the Pooh" and "it." Item keywords describe concrete or abstract nouns of objects appearing in the text, such as "honey" and "jar." Action keywords describe verbs or verb phrases describing the actions or behaviors of characters or objects in the text, such as "picking" and "picked." In practice, the executing entity first filters out words with parts of speech of noun, noun phrase, verb, or verb phrase based on the part-of-speech tagging information set, forming a candidate word set. Then, the aforementioned execution entity uses a pre-built scene entity dictionary (e.g., a dictionary containing common scene terms such as "forest," "restaurant," and "park") or a deep learning-based keyword extraction model (e.g., the TextRank algorithm or a sequence labeling model based on a bidirectional encoder representation model) to identify and label words belonging to the scene, person, item, and action categories from the candidate word set. The execution entity can then store the identified scene category words in scene keyword information, person category words in person keyword information, item category words in item keyword information, and action category words in action keyword information.

[0030] The third step involves performing sentiment analysis and contextual analysis on the aforementioned scenario keyword information to obtain scenario atmosphere element information. This scenario atmosphere element information includes scenario type identifiers and scenario atmosphere identifiers. The scenario type identifier is a code or label used to uniquely identify a scenario category (e.g., "outdoor forest scenario," "indoor party scenario," "business meeting scenario"). The scenario atmosphere identifier is a label used to describe the emotional tone conveyed by the scenario (e.g., "relaxed and cheerful," "warm and harmonious," "tense and serious"). In practice, the executing entity can obtain scenario atmosphere element information by performing sentiment analysis and contextual analysis on the aforementioned scenario keyword information as follows: First, for each scenario keyword, which includes at least one scenario keyword, the scenario keyword can be combined with its adjacent words in the aforementioned word sequence (e.g., the first two words and the last two words) to form a short text fragment. Then, VADER sentiment analysis technology is used to perform sentiment analysis on the short text fragment to obtain a sentiment polarity value (e.g., between -1 and +1). Afterward, the executing entity can determine the average of the obtained sentiment polarity values ​​as the scenario atmosphere sentiment value. If the emotional value of the scene atmosphere (e.g., 0.7) is greater than or equal to a preset positive threshold, the scene atmosphere is identified as "relaxed and cheerful" or "warm and harmonious". If the emotional polarity value is less than or equal to a preset negative threshold (e.g., -0.7), the scene atmosphere is identified as "tense and serious" or "sad". If the emotional polarity value is greater than the negative threshold but less than the positive threshold, the scene atmosphere is identified as "neutral" or "calm". The aforementioned execution entity can input scene keyword information, including at least one scene keyword, into a pre-trained scene classification model to obtain a scene type identifier. The scene classification model can be a zero-shot classification model with at least one scene keyword as input and a scene type identifier as output. For example, the scene classification model can input at least one scene keyword as {"in the forest", "restaurant", "evening"}, and output a scene type identifier such as "outdoor dining party scene". Optionally, for one or more scene keywords, the aforementioned execution entity can sequentially search for the scene type corresponding to each keyword in a preset scene type mapping dictionary. The frequency of each scene type is counted, and the identifier of the scene type with the highest frequency is selected as the scene type identifier. If multiple occurrences are tied, the selection can be based on a preset priority (e.g., outdoor > indoor > catering), or "General Scene" can be output. If no scene type is matched for any keyword, the default "General Scene" identifier is output. The preset scene type mapping dictionary can be a preset lookup table containing various mapping relationships. Each mapping relationship consists of a scene type identifier and a set of corresponding scene keywords.

[0031] The fourth step involves generating entity element information based on the aforementioned character keyword information, item keyword information, and action keyword information. In practice, the executing entity can identify each keyword from the aforementioned scene keyword information (including at least one scene keyword), character keyword information (including at least one character keyword), and item keyword information (including at least one character keyword) as an entity element. Each identified entity element is then designated as entity element information.

[0032] The fifth step is to determine the above-mentioned scene atmosphere element information and the above-mentioned entity element information as scene entity element information.

[0033] Step 104: Retrieve at least one piece of additional content information that matches the scene entity element information from the preset additional content information database.

[0034] In some embodiments, the executing entity may retrieve at least one additional content information that matches the scenario entity element information from a preset additional content information database. Each of the at least one additional content information may be text (e.g., advertising copy, emotional copy, etc.).

[0035] In some optional implementations of certain embodiments, the aforementioned execution entity may retrieve at least one piece of additional content information that matches the aforementioned scene entity element information from a preset additional content information database through the following steps: The first step is to generate a first search condition based on the scene type identifier and scene atmosphere identifier in the aforementioned scene entity element information. In practice, the executing entity combines the scene type identifier and scene atmosphere identifier into a structured query condition as the first search condition. The first search condition can be a scene type tag pre-labeled in the additional content that is consistent with the scene type identifier, and a scene atmosphere tag pre-labeled in the additional content that is consistent with the scene atmosphere identifier. For example, if the scene type identifier is "outdoor natural scene" and the scene atmosphere identifier is "relaxed and pleasant", then the first search condition can be expressed as: (scene_type == "outdoor natural scene") AND (scene_atmosphere == "relaxed and pleasant").

[0036] The second step involves generating a second search condition based on the entity element information in the aforementioned scenario. In practice, the executing entity can determine the second search condition as the condition that the intersection of the keyword tag set of the supplementary content and the entity element information set is not empty. Specifically, the second search condition can be a match between the pre-labeled keyword tags of the supplementary content and at least one entity element in the aforementioned entity element information.

[0037] The third step involves performing a joint search on the aforementioned first and second search criteria within the pre-defined supplementary content information database to obtain at least one supplementary content piece of information. This pre-defined supplementary content information database refers to a pre-constructed database storing multiple supplementary content pieces of information and their associated tags. In practice, the executing entity first selects multiple supplementary content pieces of information matching the scene type and atmosphere based on the first search criteria as an initial supplementary content information set. Then, at least one initial supplementary content piece of information from the initial supplementary content information set that satisfies the second search criteria is determined as at least one supplementary content piece of information.

[0038] Step 105: Based on at least one additional content information, perform additional content injection and style adaptation processing on the initial auxiliary writing content information to obtain auxiliary writing content information.

[0039] In some embodiments, the executing entity may, based on at least one additional content information, inject additional content and perform style adaptation processing on the initial auxiliary writing content information to obtain auxiliary writing content information. The aforementioned scene entity element information includes scene atmosphere element information and the aforementioned entity element information, wherein the aforementioned entity element information includes various entity elements.

[0040] In addressing the technical problems in the background by employing the aforementioned solutions, the application scenario—AI-assisted writing with additional content injection and style adaptation—often presents the following technical challenges: Relying solely on keyword matching to directly insert additional content without quantitative analysis of the original text's stylistic features (such as sentence complexity, rhetorical preferences, and vocabulary habits) or assessing the style match between the additional content and the original text, leads to semantic and stylistical conflicts between the inserted content and the original text. This poor user experience results in frequent regeneration requests, each requiring audio / video encoder re-encoding and network retransmission, wasting computer resources. To address the specific requirements of this application scenario, the AI-assisted writing system must output high-quality, stylistically consistent supplementary writing content in a single pass, ensuring the quality of the generated content and preventing users from repeatedly requesting regeneration due to awkward additional content. Faced with these technical challenges, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the aforementioned execution entity may perform additional content injection and style adaptation processing on the initial auxiliary writing content information based on at least one additional content information to obtain auxiliary writing content information: The first step is to segment the initial writing assistance information into sentences to obtain a sentence sequence. In practice, the executing entity calls regular expressions or sentence segmentation functions in natural language processing tools to divide the initial writing assistance information into multiple sentences, which are then arranged sequentially to obtain a sentence sequence. For example, if the initial writing assistance information is "Winnie the Pooh picked a jar of honey in the forest. He jumped up happily.", the resulting sentence sequence after segmentation is {"Winnie the Pooh picked a jar of honey in the forest.", "He jumped up happily."}.

[0041] The second step involves performing dependency parsing on the sentence sequence to obtain a set of syntax trees. In practice, the execution entity calls the dependency parsing interface of an open-source natural language processing tool (such as LTP Natural Language Processing Tool) to process each sentence in the sentence sequence sequentially and output a set of syntax trees. Each syntax tree in the set can be a tree-like data structure representing the grammatical structure of a sentence, and these syntax trees can be dependency syntax trees.

[0042] The third step involves generating sentence structure complexity feature information based on the aforementioned set of syntactic trees. In practice, for each syntactic tree in the set, the execution entity can sum the depths of each node in the syntactic tree (with the root node having a depth of 0) and divide the sum by the total number of nodes to obtain the average syntactic depth of each node. Then, the execution entity can determine the mean of the average syntactic depths of all nodes as the sentence structure complexity feature information.

[0043] The fourth step involves performing rhetorical style detection on the initial auxiliary writing content information to obtain rhetorical type labels. These rhetorical type labels refer to identifiers for one or more rhetorical categories, such as "personification" or "parallelism." The rhetorical type label can be an enumerated value or a list of labels. The Chinese rhetorical device recognition technology performs rhetorical style detection on the initial auxiliary writing content information to obtain rhetorical type labels.

[0044] The fifth step involves identifying lexical preferences in the initial auxiliary writing content information to obtain preferred lexical feature information. In practice, the executing entity first segments the initial auxiliary writing content information into words, then counts the frequency of each word, and selects the top predetermined number of segments as high-frequency words, arranged in descending order of frequency, to obtain a high-frequency word sequence. Simultaneously, it calculates the proportion of colloquial vocabulary based on a colloquial dictionary (containing modal particles such as "la," "oh," and "ya," and commonly used colloquial words), and the proportion of formal vocabulary based on a formal dictionary (containing words like "therefore," "however," etc.). The proportions of colloquial and formal vocabulary are combined into a two-dimensional vector as the first feature vector. Next, word embedding processing is performed on the high-frequency word sequence (for example, using a One-Hot encoding model), resulting in a vector representing the high-frequency word sequence as the second feature vector. Finally, the first and second feature vectors are concatenated, and the resulting vector is determined as the preferred lexical feature information.

[0045] Step 6: Based on the sentence structure complexity features, rhetoric type labels, and preferred vocabulary features mentioned above, generate the original text style feature vector. In practice, the sentence structure complexity features can be converted into a numerical vector as the complexity vector. Next, one-hot encoding is used to convert the rhetoric type labels into numerical vectors as the rhetoric type vector. Then, the complexity vector, rhetoric type vector, and the vectors represented by the preferred vocabulary features are concatenated, and the concatenated vector is determined as the original text style feature vector.

[0046] Step 7: Perform the following processing based on each of the at least one piece of additional content information mentioned above: The first sub-step involves extracting text style features from the aforementioned supplementary content information to obtain a supplementary content style feature vector. It should be noted that the method used to extract text style features from the supplementary content information to obtain the supplementary content style feature vector is the same as the operations in steps one through six above (i.e., performing operations similar to those in steps one through six sequentially, such as sentence segmentation, dependency parsing, syntactic complexity calculation, rhetoric detection, and lexical preference recognition).

[0047] The second sub-step involves determining the target similarity between the style feature vector of the additional content and the style feature vector of the original text. This similarity can be cosine similarity.

[0048] The third sub-step involves determining that the target similarity is greater than or equal to a preset similarity threshold, and then identifying the additional content information as the additional content information to be injected. The preset similarity threshold can be a pre-defined value (e.g., 0.7).

[0049] The fourth sub-step involves identifying sentences containing at least one of the aforementioned entity elements from the initial auxiliary writing content information as initial sentences to be inserted. Here, an initial sentence to be inserted refers to a sentence in the sentence sequence that contains at least one of the aforementioned entity elements. In practice, the executing entity iterates through each sentence in the sentence sequence, checking whether one of the aforementioned entity elements appears in the sentence. If it does, the sentence is added to the initial set of sentences to be inserted.

[0050] The fifth sub-step involves determining the semantic similarity between each initial sentence to be inserted and the aforementioned additional content information to be injected, and identifying the initial sentence to be inserted with the highest semantic similarity as the target sentence to be inserted. In practice, the execution entity can use a pre-trained sentence embedding model (such as Sentence-BERT) or a word vector-based average pooling method to convert each initial sentence to be inserted and the aforementioned additional content information to be injected into sentence vectors, and then calculate the cosine similarity. The initial sentence to be inserted with the highest similarity is then selected as the target sentence to be inserted.

[0051] The sixth sub-step involves performing style adaptation processing on the aforementioned target sentence to be inserted, resulting in style-aligned supplementary content information. In practice, the executing entity can rewrite the text representing the supplementary content information to be injected using a pre-trained style transfer model (e.g., a text style conversion model) or rule-based methods (e.g., template filling, thesaurus replacement) based on the original text's style feature vector, obtaining the rewritten text as style-aligned supplementary content information. For example, if the original text has a colloquial and childlike style, the style-aligned supplementary content information rewritten from "honey, all-natural, with a rich taste" could be "Try a bite of this honey, it's all-natural, sweet, and delicious!"

[0052] The seventh sub-step involves inserting the style-aligned supplementary content information into the initial auxiliary writing content information after the target sentence to be inserted, thereby updating the initial auxiliary writing content information.

[0053] The above technical solution, combined with steps 106 to 108 and related content, serves as an inventive point of this disclosure, solving the technical problem of "waste of computer resources." Factors leading to this waste often include: relying solely on keyword matching to directly insert additional content without quantitative analysis of the original text's style characteristics (such as sentence complexity, rhetorical preferences, and vocabulary habits), and without judging the matching degree between the style of the additional content and the original text, resulting in semantic and stylistic conflicts between the inserted content and the original text. Users frequently trigger regeneration operations due to poor experience, and each regeneration requires re-encoding of the audio / video encoder and retransmission over the network, causing a waste of computer resources. Solving these factors can reduce the waste of computer resources. To achieve this, firstly, the initial auxiliary writing content information is segmented into sentences to obtain a sentence sequence; then, dependency parsing is performed on the sentence sequence to obtain a set of syntax trees. This allows the acquisition of the grammatical structure information of each sentence. Based on the set of syntax trees, sentence structure complexity feature information is generated. This allows for the quantification of text complexity at the sentence level (e.g., average syntactic depth, clause density), distinguishing between colloquial short sentences and formal long sentences. Then, rhetorical style detection is performed on the initial auxiliary writing content information to obtain rhetorical type labels; lexical preference identification is also performed on the initial auxiliary writing content information to obtain preferred lexical feature information. This allows for the identification of rhetorical devices and word usage habits in the text. Based on the sentence structure complexity feature information, the rhetorical type labels, and the preferred lexical feature information, a style feature vector of the original text is generated. This integrates the style features of the original text across the three dimensions of syntax, rhetoric, and vocabulary into a computable numerical vector. Next, text style feature extraction is performed on each of the at least one additional content information to obtain an additional content style feature vector; the similarity between the additional content style feature vector and the original text style feature vector is determined as the target similarity; in response to the determination that the target similarity is greater than or equal to a preset similarity threshold, the additional content information is identified as the additional content information to be injected. Therefore, supplementary content with a style closely resembling the original text is selected from the candidate supplementary content, reducing style conflicts at the source. Next, sentences containing at least one of the various entity elements from the initial supplementary writing content are identified as initial sentences to be inserted. The semantic similarity between each initial sentence to be inserted and the supplementary content to be injected is determined, and the sentence with the highest semantic similarity is selected as the target sentence to be inserted. This accurately locates the appropriate insertion position of the supplementary content in the original text, ensuring semantic coherence between the supplementary content and adjacent sentences.Then, based on the target sentence to be inserted, the supplementary content information to be injected is style-adapted to obtain style-aligned supplementary content information. This style-aligned supplementary content information is then inserted after the target sentence in the initial auxiliary writing content information to update the initial auxiliary writing content information. The updated initial auxiliary writing content information is then determined as the auxiliary writing content information. This achieves a deep semantic and stylistic integration between the supplementary content and the original text, ensuring stylistic consistency of the writing content information after the supplementary content is inserted. This results in semantically logically coherent and stylistically unified auxiliary writing content information. Combined with steps 106 to 108, semantically logically coherent and stylistically unified auxiliary writing content information can be pushed to the user, reducing the need for users to repeatedly request regeneration due to awkward supplementary content. This reduces the number of re-encoding operations for audio / video encoders, the amount of redundant network traffic, and the waste of computing resources.

[0054] Step 8: Confirm the updated initial auxiliary writing content information as the auxiliary writing content information.

[0055] Step 106: Perform text content quality detection on the auxiliary writing information to obtain a text content quality score.

[0056] In some embodiments, the aforementioned executing entity may perform text content quality detection processing on the aforementioned auxiliary writing content information to obtain a text content quality score.

[0057] In some optional implementations of certain embodiments, the aforementioned execution entity may perform text content quality detection processing on the aforementioned auxiliary writing content information through the following steps to obtain a text content quality score: The first step involves detecting spelling and grammatical errors in the aforementioned supplementary writing content to obtain a language standardization score. Here, "spelling errors" refers to misused Chinese characters in the text due to similar pronunciation or form. "Grammatical errors" refers to sentences that do not conform to grammatical rules or lack semantic logic. The language standardization score is a numerical value used to quantify the correctness of the text in terms of spelling and grammar. In practice, the implementing entity can use a Chinese spelling detection model (e.g., the ChineseErrorCorrector large model) to perform error detection processing on the supplementary writing content, obtaining the error probability of each character. Then, the ratio of the number of characters with error probabilities greater than a preset probability value to the total number of characters included in the supplementary writing content can be determined as the first value. Next, the implementing entity can determine the total number of sentences included in the supplementary writing content as the total number of sentences. For each sentence in the supplementary writing content, the sentence can be input into a pre-trained grammatical error detection model to obtain a result indicating whether the sentence is grammatically incorrect. Next, the number of detection results indicating a sentence is grammatically incorrect is defined as the number of grammatically incorrect sentences. Then, the ratio of the number of grammatically incorrect sentences to the total number of sentences can be defined as the second value. Finally, the average of the first and second values ​​can be defined as the language standardization score. The aforementioned grammatically incorrect sentence detection model can be a sequence classifier or a text classification model that takes sentences as input and outputs whether a sentence is grammatically incorrect. As an example, at least one sample sentence and the detection result corresponding to each of the at least one sample sentence can be obtained first. Then, each of the at least one sample sentence is used as input, and the detection result corresponding to each of the at least one sample sentence is used as the expected output to train a ground feature extraction model.

[0058] The second step involves performing semantic coherence detection on the aforementioned auxiliary writing content information to obtain a coherence detection score. In practice, the executing entity can segment the auxiliary writing content information into sentences to obtain a sequence of writing content sentences. Then, the executing entity can use a sentence embedding model to convert each writing content sentence in the sequence into a sentence vector, resulting in a sentence vector sequence. These sentence vectors can represent the writing content sentences. Next, for every two adjacent sentence vectors in the sentence vector sequence, the executing entity can determine the cosine similarity between the two adjacent sentence vectors as the similarity score. Finally, the executing entity can determine the coherence detection score by taking the arithmetic mean of the determined similarities.

[0059] The third step involves performing an overall style consistency check on the aforementioned writing aids to obtain an overall style consistency score. This score is a numerical value used to quantify text style stability, ranging from 0 to 1, where 1 indicates complete style consistency and 0 indicates extreme style inconsistency. In practice, the implementing entity divides the writing aids into multiple text segments according to a fixed-length time window (e.g., every 100 characters) or according to natural paragraphs (if paragraph markers exist), resulting in a text segment sequence. Then, the BERT model is used to extract semantic style features from each text segment in the sequence, yielding a semantic feature vector. A dependency parsing tool (e.g., LTP or Stanford Parser) is used to extract syntactic structure features from each segment, yielding a syntactic feature vector. The semantic and syntactic feature vectors are then concatenated and fused to obtain a style feature vector corresponding to each text segment. Next, the arithmetic mean of the style feature vectors of all text segments is calculated to obtain the average feature vector. Finally, the cosine distance between the feature vector of each segment and the average feature vector is calculated. Next, the mean of the obtained cosine distances is determined as the average cosine distance. Finally, the aforementioned execution entity can determine the sum of the preset value (such as 1) and the average cosine distance as the denominator, and then use the preset value as the numerator, determining the ratio of the numerator to the denominator as the overall style consistency detection score.

[0060] The fourth step involves generating a text content quality score based on the aforementioned language standardization score, coherence detection score, and overall style consistency detection score. In practice, the implementing entity can set the weight of the language standardization score to a first preset weight (e.g., 0.4), the coherence detection score to a second preset weight (e.g., 0.3), and the overall style consistency detection score to a third preset weight (e.g., 0.3). Then, each score (language standardization score, coherence detection score, and overall style consistency detection score) can be multiplied by its corresponding weight to obtain a weighted score. Finally, the sum of these weighted scores can be used to determine the text content quality score.

[0061] Step 107: In response to determining that the text content quality score is greater than the preset score value, based on the writing requirement information and the auxiliary writing content information, generate audio or video that corresponds to the auxiliary writing content information and includes additional content.

[0062] In some embodiments, the execution entity may, in response to determining that the text content quality score is greater than a preset score value, generate audio or video that corresponds to the auxiliary writing content information and includes additional content, based on the writing requirement information and the auxiliary writing content information.

[0063] In some optional implementations of certain embodiments, the aforementioned executing entity may generate audio or video that corresponds to the aforementioned auxiliary writing content information and includes additional content, based on the aforementioned writing requirement information and the aforementioned auxiliary writing content information, through the following steps: The first step, in response to determining that the effect presentation identifiers included in the above writing requirements information are plot visualization identifiers, is to perform the following video compositing steps: The first sub-step involves performing script semantic parsing on the aforementioned auxiliary writing content information to obtain multi-shot narrative storyboard text information. In practice, the executing entity can call the application programming interface of a publicly available large language model (such as GPT-4, Claude, or Tongyi Qianwen) to perform script semantic parsing on the aforementioned auxiliary writing content information according to a preset prompt word template, thereby obtaining multi-shot narrative storyboard text information. The aforementioned multi-shot narrative storyboard text information is a structured data object or text description that describes the scene description, character actions, dialogue content, camera movement, and estimated duration of each shot in the video. The prompt template above could be: "Please break down the following story text into multiple storyboards. Each storyboard includes: scene description, character actions, dialogue, camera movement (such as push, pull, pan, tilt), and duration (seconds)." For example, the supplementary writing information could be: "Winnie the Pooh collected a jar of honey in the forest. He happily took a sip." The multi-shot narrative storyboard text information could be: "Storyboard 1: Scene: Forest, Action: Winnie the Pooh bends down to collect honey, Shot: Medium shot, Duration 3 seconds"; Storyboard 2: "Scene: Forest, Action: Winnie the Pooh holds up the jar to taste the honey, expression happy, Shot: Close-up, Duration 2 seconds." The second sub-step involves generating video generation prompts based on the aforementioned multi-shot narrative storyboard text information and scene entity element information. The executing entity merges the description of each shot in the multi-shot narrative storyboard text information with the style constraints (e.g., scene type, atmosphere identifier) ​​and entity elements (e.g., character name, item name) in the scene entity element information to generate detailed prompts for each shot. The specific fusion method is as follows: The executing entity first extracts the scene type identifier (e.g., "outdoor natural scene") and the scene atmosphere identifier (e.g., "relaxed and cheerful") from the scene entity element information, converting them into words describing the visual style (e.g., converting "relaxed and cheerful" into "bright colors, happy character expression, light movements"). Simultaneously, it extracts the character name (e.g., "Winnie the Pooh") from the entity element list in the scene entity element information, retaining its original name to ensure the video generation model can generate consistent character images. Then, for each shot in the multi-shot narrative storyboard text information, the executing entity generates prompts for that shot using template filling. The template above can be designed as follows: "[Scene Type], [Character] is [Action Description], [Scene Atmosphere] visual style, [Camera Movement] shot, duration [Duration] seconds. Here, [Scene Type] is obtained from the scene type identifier above; [Character] is obtained from the character name in the entity element list; [Action Description] is obtained from the action description field of this shot; [Scene Atmosphere] is converted from the scene atmosphere identifier above; [Camera Movement] is obtained from the camera movement field of this shot; [Duration] is obtained from the duration field of this shot. If the shot description includes dialogue, "The character said: '[Dialogue Content]'" can be added to the end of the prompt. The executing entity concatenates all the prompts for each shot in shot order using separators (such as line breaks or semicolons) into a complete string as the video generation prompt.

[0064] The third sub-step involves using the aforementioned video generation prompts to call a pre-trained end-to-end video generation model to generate a sequence of video frames. This end-to-end video generation model can be any video generation model (e.g., OpenAI's Sora model, the Tongyi Wanxiang video generation model, etc.).

[0065] The fourth sub-step involves performing audio-visual synchronization alignment processing on the aforementioned auxiliary writing content, resulting in a video that corresponds to the auxiliary writing content information and includes additional content. In practice, the executing entity can convert the auxiliary writing content information into an audio file using text-to-speech (TTS) technology. Next, the executing entity uses the open-source FFmpeg tool to merge the aforementioned video frame sequence into a single video file without audio, and then merges the audio file into the video file, resulting in a video that corresponds to the aforementioned auxiliary writing content information and includes additional content.

[0066] In the second step, in response to determining that the effect presentation identifier included in the writing demand information is a voice broadcast identifier, the following audio synthesis step is executed: Sub-step 1: perform text normalization processing on the auxiliary writing content information to generate a pronunciation annotation sequence. In practice, the executing subject may adopt a preset general text normalization tool (such as the open-source cn2an and dateparser libraries, or a self-written regular expression matching rule) to convert non-standard written forms such as numbers, symbols, and abbreviations in the auxiliary writing content information into colloquial expression forms that conform to pronunciation rules. For example, "2024年" is converted to "er ling er si nian", and "99%" is converted to "bai fen zhi jiu shi jiu". The converted auxiliary writing content information is determined as a normalized text. The preset speech synthesis engine is called to input the normalized text as a parameter, and the engine will automatically convert the normalized text into a pronunciation annotation sequence internally. For example, the normalized text may be: "xiao xiong wei ni cai le yi guan feng mi", and the pronunciation annotation sequence may be {xiao3, xiong2, wei2, ni2, cai3, le5, yi1, guan4, feng1, mi4}. The preset speech synthesis engine refers to a software module that converts text into speech, which internally includes a text front-end processing module (converting text into a pronunciation annotation sequence) and an acoustic model / vocoder (converting the pronunciation annotation sequence into waveform audio). The preset speech synthesis engine may be a TTS speech synthesis system.

[0067] Sub-step 2: input the pronunciation annotation sequence into a preset speech synthesis engine, to obtain an audio corresponding to the auxiliary writing content information and containing additional content. In practice, the executing subject may input the pronunciation annotation sequence into the acoustic model (for example, the Tacotron2 model) / vocoder (converting the pronunciation annotation sequence into waveform audio) of the preset speech synthesis engine, to obtain an audio corresponding to the auxiliary writing content information and containing additional content. The audio may be an audio that reads all the text content in the auxiliary writing content information.

[0068] In response to determining that the text content quality score is less than or equal to a preset score value, the executing subject may further determine the initial auxiliary writing content information as the auxiliary writing content information.

[0069] Step 108: transmit the auxiliary writing content information, the audio or the video to a user terminal, so as to play the audio or the video at the user terminal.

[0070] In some embodiments, the executing subject may transmit the auxiliary writing content information, the audio or the video to the user terminal, so as to play the audio or the video at the user terminal.

[0071] In addressing the technical problems mentioned above by employing the aforementioned technical solutions, the real-time transmission of AI-assisted writing-generated audio or video in unreliable network environments often presents the following technical challenges for the application scenario: Due to dynamic fluctuations in network bandwidth and the unknown buffer status at the user end, using a fixed bitrate for audio and video transmission can lead to data loss and playback stuttering when bandwidth decreases, forcing users to refresh or re-request. Conversely, a fixed low bitrate when bandwidth is ample wastes available bandwidth and prolongs transmission time. Furthermore, neglecting to consider the receiver buffer occupancy rate can result in buffer overflow (data loss) or starvation (playback interruption), both of which trigger user intervention and retransmission, leading to wasted computer resources due to repeated encoding by the audio / video encoder and repeated transmission across the network link. Therefore, the following requirements are necessary for this application scenario: the AI-assisted writing system must transmit the generated audio or video containing additional content to the user end in real-time and smoothly, dynamically adjusting the transmission bitrate to adapt to changes in network bandwidth and the receiver buffer status, ensuring smooth playback without stuttering or overflow while maximizing network bandwidth savings.

[0072] In some optional implementations of certain embodiments, the executing entity may send the aforementioned auxiliary writing content information, the aforementioned audio, or the aforementioned video to the aforementioned user terminal through the following steps, so as to play the aforementioned audio or the aforementioned video on the aforementioned user terminal: The first step is to send the aforementioned writing assistance information to the aforementioned user terminal. In practice, the executing entity sends the aforementioned writing assistance information in text form (e.g., plain text) to the aforementioned user terminal via the Hypertext Transfer Protocol (HTTP) or WebSocket protocol for the user to preview.

[0073] The second step is to perform network bandwidth probing on the aforementioned user terminals to obtain the current available bandwidth value. In practice, the executing entity can use a bandwidth estimation algorithm to perform network bandwidth probing on the aforementioned user terminals to obtain the current available bandwidth value. Here, the current available bandwidth value refers to the amount of data that can be transmitted per unit time (in megabits per second).

[0074] The third step is to determine the initial transmission rate based on the currently available bandwidth value. In practice, the executing entity multiplies the currently available bandwidth value by a safety factor (e.g., 0.8) to obtain the initial transmission rate to avoid network congestion. For example, if the currently available bandwidth is 5 megabits per second, the initial transmission rate is set to 4 megabits per second.

[0075] The fourth step involves transcoding the audio or video file according to the initial transmission bitrate to obtain a media stream. Transcoding refers to converting the encoding format, resolution, bitrate, and other parameters of the original audio or video file into a new encoded stream that matches the initial transmission bitrate. The purpose is to reduce the size of the media data to accommodate network bandwidth and avoid transmission stuttering. The media stream refers to a continuous audio and video data stream suitable for network segmented transmission, generated after transcoding, such as Transport Stream (TS) or segmented MP4 format. In practice, the execution entity calls the open-source FFmpeg audio and video processing tool, using the generated audio or video as input, setting the output video bitrate to equal the initial transmission bitrate, and setting the output audio bitrate to a preset fixed value (e.g., 128 kilobits per second) and specifying the output container format as Transport Stream (TS) or segmented MP4. The execution entity then calls FFmpeg to perform the transcoding operation using these parameters, converting the original file (video) into a media stream with a bitrate matching the network bandwidth.

[0076] The fifth step involves dividing the aforementioned media stream into a sequence of media segment information according to a preset fixed time interval. Each media segment information has a corresponding timestamp and sequence number. The preset fixed time interval refers to the duration of media covered by each segment, such as 2 seconds or 4 seconds. The media segment information sequence is a list arranged chronologically, containing media data blocks and their metadata (timestamp, sequence number). Each media segment information refers to an independent data unit obtained by cutting a continuous media stream (such as audio or video) at preset fixed time intervals (e.g., every 2 seconds). Each independent data unit includes a media data block and its metadata (timestamp, sequence number). The media data block can be binary audio or video data. For example, a 30-second video, divided into 2-second segments, generates 15 media segment information: media segment information 0 corresponds to seconds 0-2, media segment information 1 corresponds to seconds 2-4, and so on.

[0077] Step 6: Group the above media segment information sequence to obtain a media segment information group sequence. In practice, the executing entity can sequentially divide every preset number (e.g., 4) consecutive media segment information into groups to obtain a media segment information group sequence (it should be noted that when the number of undivided segments in the last group is less than the preset number, the undivided segments are grouped together). For example, the media segment information sequence can be {media segment information 1, media segment information 2, media segment information 3, media segment information 4, media segment information 5}. If the preset number is 4, then the media segment information group sequence can be {{media segment information 1, media segment information 2, media segment information 3, media segment information 4}, {media segment information 5}}. Step 7: For each media segment information group in the above media segment information group sequence, perform the following steps: The first sub-step involves performing forward error correction encoding on the aforementioned media fragment information group to obtain a redundant fragment information group. In practice, the execution entity can use Reed-Solomon encoding or XOR encoding to perform forward error correction encoding on the aforementioned media fragment information group to obtain a redundant fragment information group. The aforementioned redundant fragment information group contains a set of check data blocks used to verify the media fragment information group. For example, the media fragment information group can be {media fragment information 1, media fragment information 2, media fragment information 3, media fragment information 4}. Reed-Solomon encoding is performed on the media fragment information group, setting the number of redundant data blocks to 2. After encoding, two check data blocks are generated, denoted as check data block 1 as redundant fragment information and check data block 2 as redundant fragment information, respectively. Then, the redundant fragment information group is {check data block 1, check data block 2}.

[0078] The second sub-step involves interleaving the media fragment information in the media fragment information group with the redundant fragment information in the redundant fragment information group to obtain a media redundant interleaved fragment information group as the transmission block. In practice, the aforementioned execution entity generates a list, first adding the first media fragment information, then the second media fragment information, then the first check data block, then the third media fragment information, then the fourth media fragment information, and finally the second check data block, forming the media redundant interleaved fragment information group.

[0079] The eighth step is to sort the obtained transmission blocks to obtain a transmission block sequence. In practice, the above-mentioned execution entity can sort the transmission blocks corresponding to each media fragment information group according to the order of the media fragment information groups in the media fragment information group sequence to obtain a transmission block sequence.

[0080] Step 9: Based on the initial transmission rate and transmission block sequence, perform the following transmission process: Sub-step one involves sending the first transmission block in the transmission block sequence to the aforementioned user terminal at the current transmission bitrate, and deleting the first transmission block from the transmission block sequence to update the transmission block sequence.

[0081] Sub-step two involves monitoring the receive buffer occupancy rate of the aforementioned user terminal. The receive buffer occupancy rate refers to the ratio of the duration of cached but unplayed media data in the user terminal's receive buffer to the preset total buffer capacity. For example, if the preset total buffer capacity is 10 seconds, and the current cached but unplayed data duration is 3 seconds, then the receive buffer occupancy rate is 30%. The aforementioned cached but unplayed media data duration refers to the total playback duration of media segment information (i.e., information representing the original media segments) received by the user terminal but not yet retrieved by the player. In practice, the aforementioned user terminal periodically (e.g., every 200 milliseconds) sends a feedback message to the execution entity, reporting the current cached duration or percentage of the buffer.

[0082] Sub-step three involves, in response to determining that the aforementioned receive buffer occupancy rate is less than or equal to a preset lower threshold, performing a positive incremental update on the initial transmit bitrate, and then re-executing the aforementioned transmit process based on the updated initial transmit bitrate and the updated transmit block sequence. Here, the aforementioned preset lower threshold refers to the lower limit of the buffer occupancy rate (e.g., 20% or 2 seconds). The aforementioned positive incremental update process refers to increasing the current transmit bitrate by a step size (e.g., increasing by 10% or a fixed value of 0.5 megabits per second).

[0083] Sub-step four involves, in response to determining that the aforementioned receive buffer occupancy rate is greater than or equal to a preset upper limit threshold, performing a negative incremental update on the initial transmit bitrate, and then re-executing the aforementioned transmit process based on the updated initial transmit bitrate and the updated transmit block sequence. Here, the aforementioned preset upper limit threshold refers to the upper limit of the buffer occupancy rate (e.g., 80% or 8 seconds). The aforementioned negative incremental update process refers to reducing the current transmit bitrate by one step (e.g., reducing it by 10% or a fixed value of 0.5 megabits per second).

[0084] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "playback stuttering, data loss, and resulting waste of computer resources caused by dynamic fluctuations in network bandwidth and unknown receiving buffer status." Factors leading to wasted computer resources often include: due to dynamic fluctuations in network bandwidth and unknown receiving buffer status at the user end, when transmitting audio and video at a fixed bitrate, a decrease in bandwidth can lead to data packet loss and playback stuttering, forcing the user to refresh or re-request. When bandwidth is sufficient, a fixed low bitrate wastes available bandwidth and prolongs transmission time. Simultaneously, if the receiving buffer occupancy rate is not considered, buffer overflow (data loss) or starvation (playback interruption) may occur, both of which trigger user intervention and retransmission, resulting in wasted computer resources due to repeated encoding by the audio / video encoder and repeated transmission by the network link. Solving these factors can achieve the effect of adapting to network fluctuations, avoiding stuttering and overflow, and reducing the waste of computing resources caused by invalid retransmissions and repeated encoding. To achieve this effect, firstly, the aforementioned auxiliary writing content information is sent to the aforementioned user end. Thus, the user can first obtain a preview of the text content. Then, network bandwidth is probed at the aforementioned user end to obtain the current available bandwidth value. Based on the currently available bandwidth, an initial transmission bitrate is determined. This ensures that the initial transmission bitrate matches the current network conditions, avoiding congestion due to excessively high bitrates or wasted bandwidth due to excessively low bitrates. According to the initial transmission bitrate, the audio or video is transcoded to obtain a media stream. This converts the original audio / video file into a media stream with a bitrate adapted to the network bandwidth. The media stream is then divided into a media segment information sequence at preset fixed time intervals, with each segment bearing a timestamp and sequence number. This discretizes the continuous media stream into independently transmittable and verifiable units, facilitating packet loss recovery and adaptive transmission. The media segment information sequence is grouped to obtain a media segment information group sequence. For each media segment information group in the media segment information group sequence, the following steps are performed: forward error correction coding is performed on the media segment information group to obtain a redundant segment information group; the original media segments and redundant segments are interleaved to obtain a transmission block. Therefore, the receiving end can recover a small number of lost fragments without retransmission, reducing round-trip delays and retransmission traffic caused by packet loss, and saving network bandwidth and server processing resources. Then, the obtained transmission blocks are sorted to obtain a transmission block sequence. Next, the initial transmission code rate is determined as the current transmission code rate. Then, the first transmission block in the transmission block sequence is sent to the aforementioned user end at the current transmission code rate, and the already sent transmission blocks are removed from the sequence. This achieves sequential, on-demand block transmission. Afterwards, the receive buffer occupancy rate of the aforementioned user end is monitored. In response to determining that the receive buffer occupancy rate is less than or equal to a preset lower threshold, the current transmission code rate is updated positively (increasing the code rate), and the transmission process is executed again based on the updated code rate and the remaining transmission block sequence.Next, in response to the determination that the aforementioned receive buffer occupancy rate is greater than or equal to a preset upper limit threshold, the current transmit bitrate is updated negatively (the bitrate is reduced), and transmission continues. This dynamically adjusts the transmit rate in real time: accelerating filling when the buffer is nearing depletion to prevent playback stuttering; and slowing down when the buffer is nearing overflow to avoid data loss. This closed-loop control reduces the number of times the user needs to refresh, retry, or re-request due to stuttering or overflow, thereby reducing additional encoding and transmission processing caused by user intervention and minimizing waste of computer resources. Furthermore, the use of bandwidth detection to preset the initial bitrate, forward error correction to reduce retransmissions, and receive buffer feedback to drive dynamic bitrate adjustment allows audio and video transmission to adapt to network fluctuations and buffer status, ensuring smooth and stable playback. Users do not need to repeatedly request regeneration or retransmission due to stuttering or overflow, further reducing waste of computer resources.

[0085] The above-described embodiments of this disclosure have the following beneficial effects: the audio-visual synthesis method based on text content quality detection in some embodiments of this disclosure improves the user experience and reduces the waste of network transmission resources. Specifically, the reasons for poor user experience and waste of network transmission resources are as follows: because the additional content in pop-ups and inserts is semantically unrelated to the written text, it easily interferes with the user. The user needs to click the close button frequently to continue watching or listening, resulting in a poor user experience. Furthermore, the directly inserted additional content relies solely on keyword matching, without matching and retrieving additional content based on the scene characteristics and entity elements of the written text, and without style adaptation and quality detection of the text after inserting additional content. This leads to logical gaps in the generated auxiliary writing content information in terms of semantics, and the text style is inconsistent, resulting in poor quality of the generated auxiliary writing content. Users need to regenerate multiple times to obtain usable content, and each generation requires resending the generated auxiliary writing content, resulting in waste of network transmission resources. Based on this, the audio-visual synthesis method based on text content quality detection in some embodiments of this disclosure first receives writing requirement information sent by the user terminal. Based on the above writing requirement information, initial auxiliary writing content information is generated. This allows us to obtain initial auxiliary writing content information that meets the user's writing needs. Next, scene entity extraction processing is performed on the initial auxiliary writing content information to obtain scene entity element information. Thus, scene and entity elements are extracted from the writing text, i.e., the initial auxiliary writing content information, to obtain scene entity element information. Then, at least one piece of supplementary content information matching the aforementioned scene entity element information is retrieved from a pre-defined supplementary content information database. This allows us to retrieve supplementary content related to the current writing scenario, reducing semantic bias caused by keyword dependence and making supplementary content a natural extension of the content rather than a distractor. Then, based on the aforementioned at least one piece of supplementary content information, supplementary content injection and style adaptation processing are performed on the initial auxiliary writing content information to obtain auxiliary writing content information. This allows the expression of the supplementary content text to be converted to a form consistent with the style of the original text (such as sentence structure, vocabulary, and rhetoric alignment), eliminating inconsistencies in style and maintaining the logical coherence of the text. Finally, text content quality detection processing is performed on the aforementioned auxiliary writing content information to obtain a text content quality score. This allows for text content quality detection processing of the auxiliary writing content information after the insertion of supplementary content. Next, in response to determining that the text content quality score is greater than a preset score, based on the writing requirement information and the auxiliary writing content information, an audio or video recording corresponding to the auxiliary writing content information and including additional content is generated. Finally, the auxiliary writing content information, the audio, or the video is sent to the user terminal for playback on the user terminal.Therefore, when the quality of the supplementary writing content is high, the audio or video converted from the writing text with the added supplementary content, along with the supplementary writing content information, can be sent to the user's end. Users can obtain usable content all at once without having to regenerate multiple times; each generation only requires sending one high-quality result, thus reducing the waste of network transmission resources caused by multiple regenerations. Furthermore, because it uses scene entity element information extracted from scene entities to retrieve supplementary content, integrates the supplementary content into the writing text, and generates audio and video, it replaces traditional pop-ups, inserts, and other isolated supplementary content display formats, reducing the interference of pop-ups and inserts that are semantically irrelevant to the writing text content and improving the user experience. Moreover, style adaptation processing and text content quality detection are performed during the insertion of supplementary content. When the quality of the supplementary writing content is high, the supplementary writing content information is sent to the user's end, allowing users to obtain usable content all at once without having to regenerate multiple times; each generation only requires sending one high-quality result, thus reducing the waste of network transmission resources caused by multiple regenerations.

[0086] Further reference Figure 2 As an implementation of the methods shown in the figures, this disclosure provides some embodiments of an audio-visual synthesis apparatus based on text content quality detection. These apparatus embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0087] like Figure 2As shown, an audio-visual synthesis apparatus 200 based on text content quality detection in some embodiments includes: a receiving unit 201, a first generation unit 202, a scene entity extraction unit 203, a retrieval unit 204, an additional content injection unit 205, a text content quality detection unit 206, a second generation unit 207, and a sending unit 208. The receiving unit 201 is configured to receive writing requirement information sent by a user terminal; the first generation unit 202 is configured to generate initial auxiliary writing content information based on the writing requirement information; the scene entity extraction unit 203 is configured to perform scene entity extraction processing on the initial auxiliary writing content information to obtain scene entity element information; the retrieval unit 204 is configured to retrieve at least one additional content information matching the scene entity element information from a preset additional content information database; and the additional content injection unit 205 is configured to inject additional content into the initial auxiliary writing content information based on the at least one additional content information. The system performs style adaptation processing to obtain auxiliary writing content information; the text content quality detection unit 206 is configured to perform text content quality detection processing on the auxiliary writing content information to obtain a text content quality score; the second generation unit 207 is configured to, in response to determining that the text content quality score is greater than a preset score value, generate audio or video corresponding to the auxiliary writing content information and containing additional content based on the writing requirement information and the auxiliary writing content information; the sending unit 208 is configured to send the auxiliary writing content information, the audio or the video to the user terminal to play the audio or the video on the user terminal.

[0088] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the method described above correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0089] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0090] like Figure 3As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0091] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0092] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0093] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0094] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0095] A computer-readable medium may be included in an electronic device or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: receive writing requirement information sent by a user terminal; generate initial auxiliary writing content information based on the writing requirement information; perform scene entity extraction processing on the initial auxiliary writing content information to obtain scene entity element information; retrieve at least one supplementary content information matching the scene entity element information from a preset supplementary content information database; perform supplementary content injection and style adaptation processing on the initial auxiliary writing content information based on the at least one supplementary content information to obtain auxiliary writing content information; perform text content quality detection processing on the auxiliary writing content information to obtain a text content quality score; in response to determining that the text content quality score is greater than a preset score value, generate audio or video corresponding to the auxiliary writing content information and containing supplementary content based on the writing requirement information and the auxiliary writing content information; and send the auxiliary writing content information, the audio, or the video to the user terminal for playback on the user terminal.

[0096] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0098] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a receiving unit, a first generating unit, a scene entity extraction unit, a retrieval unit, an additional content injection unit, a text content quality detection unit, a second generating unit, and a sending unit. The names of these units do not necessarily limit the specific unit; for example, the receiving unit may also be described as "a unit that generates initial auxiliary writing content information based on the aforementioned writing requirement information."

[0099] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0100] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of technical features, but should also cover other technical solutions formed by arbitrary combinations of technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An audio-visual synthesis method based on text content quality detection, comprising: Receive writing request information sent by the user client; Based on the writing requirements information, initial auxiliary writing content information is generated; The initial auxiliary writing content information is processed by scene entity extraction to obtain scene entity element information; Retrieve at least one piece of additional content information that matches the scene entity element information from a preset additional content information database; Based on the at least one additional content information, additional content injection and style adaptation processing are performed on the initial auxiliary writing content information to obtain auxiliary writing content information; The text content quality of the auxiliary writing information is processed by text content quality detection to obtain a text content quality score; In response to determining that the text content quality score is greater than a preset score value, based on the writing requirement information and the auxiliary writing content information, an audio or video corresponding to the auxiliary writing content information and containing additional content is generated; The auxiliary writing content information, the audio, or the video is sent to the user terminal so that the audio or video can be played on the user terminal.

2. The method according to claim 1, wherein, The process of generating initial auxiliary writing content information based on the writing requirements information includes: The writing requirement information is decomposed into a structure to obtain requirement analysis information, which includes writing topic, scene element description information and text style. The required analysis information is input into a preset writing model to generate text content that conforms to the writing theme, contains scene elements represented by the scene element description information, and has a language style consistent with the text style as initial auxiliary writing content information.

3. The method according to claim 1, wherein, The process of extracting scene entities from the initial assisted writing content information to obtain scene entity element information includes: The initial auxiliary writing content information is subjected to semantic parsing to obtain a word sequence and a part-of-speech tagging information set, wherein each word in the word sequence corresponds to a part-of-speech tagging information set; Based on the part-of-speech tagging information set, a keyword set is extracted from the word sequence. The keyword information set includes scene keyword information, character keyword information, item keyword information, and action keyword information. The scene keyword information includes at least one scene keyword, the character keyword information includes at least one character keyword, the item keyword information includes at least one character keyword, and the action keyword information includes at least one action keyword. The scene keyword information is processed by sentiment and context analysis to obtain scene atmosphere element information, wherein the scene atmosphere element information includes scene type identifier and scene atmosphere identifier; Based on the character keyword information, the item keyword information, and the action keyword information, entity element information is generated; The scene atmosphere element information and the entity element information are determined as scene entity element information.

4. The method according to claim 3, wherein, The step of retrieving at least one piece of supplementary content information from a preset supplementary content information database that matches the scene entity element information includes: Based on the scene type identifier and scene atmosphere identifier in the scene entity element information, a first search condition is generated; Based on the entity element information in the scene entity element information, a second search condition is generated; Based on the first search criteria and the second search criteria, a joint search process is performed in the preset supplementary content information database to obtain at least one supplementary content information.

5. The method according to claim 1, wherein, The text content quality detection process performed on the auxiliary writing content information to obtain a text content quality score includes: The auxiliary writing content information is checked for spelling and grammatical errors to obtain a language standardization score; The semantic coherence between sentences of the auxiliary writing content information is detected to obtain a coherence detection score; The overall style consistency of the auxiliary writing content information is checked to obtain an overall style consistency score. A text content quality score is generated based on the language standardization score, the coherence detection score, and the overall style consistency detection score.

6. The method according to claim 1, wherein, The writing requirement information includes writing theme, scene element description information, and text style and effect presentation identifiers. The effect presentation identifiers include plot visualization identifiers or voice narration identifiers. The process of generating audio or video corresponding to the auxiliary writing content information and containing additional content, based on the writing requirement information and the auxiliary writing content information, includes: In response to determining that the effect presentation identifier included in the writing requirement information is a narrative visualization identifier, the following video compositing steps are performed: The script semantic analysis process is performed on the auxiliary writing content information to obtain multi-shot narrative storyboard text information; Based on the multi-camera narrative storyboard text information and the scene entity element information, video generation prompts are generated; Based on the video generation prompts, a pre-trained end-to-end video generation model is invoked to generate a video frame sequence; Based on the auxiliary writing content, the video frame sequence is subjected to audio-visual synchronization alignment processing to obtain a video that corresponds to the auxiliary writing content information and includes additional content. In response to determining that the effect presentation identifier included in the writing requirement information is a voice broadcast identifier, the following audio synthesis steps are performed: The text regularization process is applied to the auxiliary writing content information to generate a pronunciation annotation sequence; The pronunciation annotation sequence is input into a preset speech synthesis engine to obtain audio that corresponds to the writing assistance content information and includes additional content.

7. An audio-visual synthesis device based on text content quality detection, comprising: The receiving unit is configured to receive writing request information sent by the user client; The first generation unit is configured to generate initial auxiliary writing content information based on the writing requirement information; The scene entity extraction unit is configured to perform scene entity extraction processing on the initial auxiliary writing content information to obtain scene entity element information; The retrieval unit is configured to retrieve at least one piece of additional content information that matches the scene entity element information from a preset additional content information database; The additional content injection unit is configured to perform additional content injection and style adaptation processing on the initial auxiliary writing content information based on the at least one additional content information to obtain auxiliary writing content information. The text content quality detection unit is configured to perform text content quality detection processing on the auxiliary writing content information to obtain a text content quality score. The second generation unit is configured to, in response to determining that the text content quality score is greater than a preset score value, generate audio or video that corresponds to the auxiliary writing content information and includes additional content, based on the writing requirement information and the auxiliary writing content information. The sending unit is configured to send the auxiliary writing content information, the audio, or the video to the user terminal to play the audio or the video on the user terminal.

8. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images