Method and device for generating audio and video material

Abstract

The present application provides a method and device for generating audio and video materials, wherein the method for generating audio and video materials comprises: receiving a request for generating audio and video materials for a phoneme sequence, wherein the request for generating audio and video materials comprises a role identifier; determining a target phoneme information set based on the role identifier, wherein the target phoneme information set comprises phoneme information and video materials corresponding to the phoneme information; obtaining a phoneme information sequence in the target phoneme information set according to the phoneme sequence; determining the video material corresponding to each phoneme information in the phoneme information sequence, and generating the target audio and video material corresponding to the phoneme sequence.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for generating audio and video materials. The present application also relates to an apparatus for generating audio and video materials, a computing device, and a computer-readable storage medium. Background Art

[0002] With the continuous development of video processing technology, users' demand for video processing is also increasing. Currently, in order to enhance the user's video viewing experience, the original video can be re-created to generate audio and video that meets the needs. For example, the repeated content in the original video can be deleted, and the wonderful clips in the original video can be edited.

[0003] At present, the editing operation of audio and video is limited to the processing of the original video content and the audio corresponding to the original video content, and the amount of audio and video materials is relatively small. In addition, when the video does not contain the original sound required for user creation, it is necessary to manually generate audio and video materials based on the original video.

[0004] However, the above-mentioned method of manually generating audio and video materials has low material efficiency, which affects the user's creative efficiency and creative experience. Summary of the invention

[0005] In view of this, the embodiment of the present application provides a method for generating audio and video materials. The present application also relates to an apparatus for generating audio and video materials, a computing device, and a computer-readable storage medium to solve the technical problem of low efficiency in generating audio and video materials in the prior art.

[0006] According to a first aspect of an embodiment of the present application, a method for generating audio and video material is provided, comprising:

[0007] Receiving an audio / video material generation request for a phoneme sequence, wherein the audio / video material generation request includes a role identifier;

[0008] Determining a target phoneme information set based on the role identifier, wherein the target phoneme information set includes phoneme information and video material corresponding to the phoneme information;

[0009] Acquire a phoneme information sequence in the target phoneme information set according to the phoneme sequence;

[0010] Determine the video material corresponding to each phoneme information in the phoneme information sequence, and generate the target audio and video material corresponding to the phoneme sequence.

[0011] According to a second aspect of an embodiment of the present application, there is provided a device for generating audio and video material, including:

[0012] A receiving module is configured to receive an audio and video material generation request for a phoneme sequence, wherein the audio and video material generation request includes a role identifier;

[0013] A determination module is configured to determine a target phoneme information set based on the role identifier, wherein the target phoneme information set includes phoneme information and video material corresponding to the phoneme information;

[0014] An acquisition module, configured to acquire a phoneme information sequence in the target phoneme information set according to the phoneme sequence;

[0015] The generation module is configured to determine the video material corresponding to each phoneme information in the phoneme information sequence, and generate the target audio and video material corresponding to the phoneme sequence.

[0016] According to a third aspect of an embodiment of the present application, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the method for generating audio and video materials when executing the computer instructions.

[0017] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer instructions, and when the computer instructions are executed by a processor, the steps of the method for generating audio and video materials are implemented.

[0018] The method for generating audio and video materials provided in the present application receives a request for generating audio and video materials for a phoneme sequence, wherein the request for generating audio and video materials includes a role identifier; determines a target phoneme information set based on the role identifier, wherein the target phoneme information set includes phoneme information and video materials corresponding to the phoneme information; obtains a phoneme information sequence in the target phoneme information set according to the phoneme sequence; determines the video material corresponding to each phoneme information in the phoneme information sequence, and generates the target audio and video material corresponding to the phoneme sequence.

[0019] An embodiment of the present application realizes obtaining a corresponding phoneme information sequence according to a phoneme sequence, and then generating a target audio and video material according to the phoneme information sequence and video material, thereby improving the efficiency of generating audio and video materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a method for generating audio and video materials provided in one embodiment of the present application;

[0021] Figure 2 is a processing flow chart of a method for generating audio and video materials applied to a phoneme sequence z provided in an embodiment of the present application;

[0022] Figure 3It is a schematic diagram of a method for generating audio and video materials provided by an embodiment of the present application;

[0023] Figure 4 It is a structural schematic diagram of a device for generating audio and video materials provided in one embodiment of the present application;

[0024] Figure 5 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0025] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0026] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to any or all possible combinations of one or more associated listed items.

[0027] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] First, the terms involved in one or more embodiments of the present application are explained.

[0029] PCM (Pulse Code Modulation) format: Pulse code modulation format is an audio format.

[0030] Overlap & fade in and out: In order to make the spelling product sound like a rapid and continuous single pronunciation, it is necessary to allow some audio overlap between the initials and finals when they are spliced. At the same time, there should be a fade-out sound effect at the end of the initials and a fade-in sound effect at the beginning of the finals, making the synthesis effect more natural.

[0031] Spelling: In video creation, when you need to use a pronunciation made by a certain character that is not in the original sound material, you will look for the consonants and vowels in the original sound material to synthesize the required pronunciation, and then splice and adjust them to obtain the corresponding material.

[0032] Currently, when creating a video, it is necessary to confirm all the required pronunciations according to the conceived copy and compare them with the original audio of the material. If there is no corresponding original sound for the pronunciation required for the creation, it is necessary to use the spelling ability to achieve it; the pronunciation that needs to be spelled is decomposed into initials and finals, and the original audio containing the required initials and finals is found in the original audio and cut out; in the audio processing software and other software, the original initials and finals required for spelling are dragged into the audio track, and the initials and finals are cut out respectively. The initials need to retain more finals, and then the connection between the initials and finals is overlapped to a certain extent, and the volume of the overlapping part is faded in and out by curve. By constantly adjusting the audio, the required audio is obtained, and the corresponding pictures and subtitles are added.

[0033] However, under the above-mentioned creative method, the creator needs to count the pronunciations of all the word materials in the original audio in advance, and then compare the required pronunciations with the original audio to confirm which pronunciations need to be spelled. This process is time-consuming and laborious. In addition, the spelling process requires the use of professional audio processing software, and repeated adjustments are required to obtain high-quality spelling audio. The threshold is high and time-consuming. Ultimately, only a few professional creators can spell out and use the required pronunciations with better effects, resulting in creativity being limited by the original audio and limited creative freedom.

[0034] The method for generating audio and video materials of the present application expands the existing audio material library and improves the efficiency of generating audio and video materials; thereby enabling users to create videos without being restricted to the original video, thereby improving the user creation experience.

[0035] In the present application, a method for generating audio and video materials is provided. The present application also relates to an apparatus for generating audio and video materials, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0036] Figure 1 A flowchart of a method for generating audio and video materials according to an embodiment of the present application is shown, which specifically includes the following steps:

[0037] Step 102: receiving an audio / video material generation request for a phoneme sequence, wherein the audio / video material generation request includes a role identifier.

[0038] Among them, the audio and video material generation request refers to a request to generate audio and video corresponding to the phoneme sequence, and the audio and video include audio and video content corresponding to the phoneme sequence; phoneme refers to the smallest speech unit divided according to language attributes. The phoneme in this embodiment refers to the initial consonant in Pinyin, such as "b, p, m", etc., or the final vowel, such as "ai, ei, ao, ou", etc.; the phoneme sequence refers to a sequence composed of phonemes, and at least one corresponding text can be determined according to the phoneme sequence. For example, the phoneme sequence is "wo" composed of the initial consonant "w" and the final vowel "o", and the corresponding text can be determined according to the phoneme sequence "wo", such as "I", "hold", etc. For another example, the phoneme sequence is "guli", and the corresponding text can be determined as "encouragement", "grain", etc. according to the phoneme sequence.

[0039] Specifically, the audio and video material generation request includes a role identifier; the role identifier refers to a field that can uniquely represent the target role. For example, if a video includes role A and role B, role identifier "1" can represent role A, and role identifier "2" can represent role B. The role identifier in the audio and video material generation request can facilitate subsequent determination of the data corresponding to the role identifier for generating audio and video materials.

[0040] For example, an audio / video material generation request Q for a phoneme sequence "gongji" is received, wherein the audio / video material generation request Q includes a role identifier "x".

[0041] By receiving a request for generating audio and video material including a role identifier for a phoneme sequence, it is convenient to subsequently determine data corresponding to the target role identifier for generating the audio and video material corresponding to the phoneme sequence.

[0042] In practical applications, the phoneme sequence can be obtained according to the target text, that is, the target text can be recognized to obtain the phoneme sequence; and a request for generating audio and video materials for the target text can also be received, and the target text can be recognized to obtain the phoneme sequence.

[0043] Specifically, before receiving the request for generating audio and video material for the phoneme sequence, the following steps may also be included:

[0044] Get the target text;

[0045] The target text is recognized to obtain a phoneme sequence corresponding to the target text.

[0046] Among them, the target text refers to the text that can obtain a phoneme sequence through recognition; in practical applications, the target text can be determined according to business requirements. For example, if the text "today" is a common text, then "today" can be used as the target text; after determining the target text, recognize the target text to obtain the phoneme sequence corresponding to the target text. For example, recognizing the target text "turn right" obtains the corresponding phoneme sequence "xiang you zhuan".

[0047] In addition, it is also possible to recognize the phoneme sequence in the target audio, the technician splices the phonemes to obtain the phoneme sequence, etc. The present application does not make specific limitations on the source of the phoneme sequence.

[0048] Furthermore, in the case where the target text consists of multiple characters, each character included in the target text can be split, and the phoneme corresponding to each character can be determined to form the phoneme sequence corresponding to the target text.

[0049] Specifically, the method for recognizing the phoneme sequence of the target text may include:

[0050] Split the target text to obtain the target characters in the target text;

[0051] Recognize the phoneme subsequence corresponding to each target character, and form the phoneme sequence corresponding to the target text by each phoneme subsequence.

[0052] Among them, the target character refers to the character included in the target text. For example, if the target text is "turn left", the target characters recognized in the target text are "xiang", "you", and "zhuan" respectively; the phoneme subsequence refers to the phoneme sequence corresponding to the target character. For example, it is determined that the target characters are "xiang" and "you" respectively. The phoneme subsequence corresponding to the recognized target character "xiang" is "xiang", and the phoneme subsequence corresponding to the recognized target character "you" is "you"; after determining the phoneme subsequence corresponding to each target character, splice each phoneme subsequence to obtain the phoneme sequence corresponding to the target text.

[0053] In practical applications, the way to recognize the phoneme sequence can be: input the target text into a pre-trained phoneme sequence recognition model, and obtain the phoneme sequence output by the phoneme sequence recognition model according to the target text; or, recognize each target character in the target text, then determine the phoneme subsequence corresponding to each target text, and splice each phoneme subsequence to obtain the phoneme sequence.

[0054] For example, determine the target text "yesterday"; recognize the target characters "zuo" and "tian" contained in the target text; recognize the target character "zuo" to obtain the corresponding phoneme subsequence "zuo", recognize the target character "tian", and obtain the corresponding phoneme subsequence "tian"; for the phoneme subsequences corresponding to the target characters, concatenate the phoneme subsequences according to the order of the target characters in the target text to obtain the phoneme sequence "zuo tian" corresponding to the target text.

[0055] By obtaining the corresponding phoneme sequence according to the target text, or directly obtaining a preset phoneme sequence, the channels for obtaining the phoneme sequence are enriched and the richness of the phoneme sequence is improved.

[0056] Step 104: determining a target phoneme information set based on the role identifier, wherein the target phoneme information set includes phoneme information and video material corresponding to the phoneme information.

[0057] Among them, the target phoneme information set refers to a set corresponding to the role identification, consisting of phoneme information and video materials corresponding to the phoneme information; in actual applications, different role identifications correspond to different phoneme information sets, such as the phoneme information set a1 corresponding to the role identification "1"; the target phoneme information corresponding to the target role is included in the target phoneme set, that is, the target phoneme information set corresponding to the target role can be determined according to the role identification; the phoneme information refers to the time interval corresponding to the phoneme in the audio and video, for example, the phoneme G is the final ang, and the time interval corresponding to the final ang in the audio and video is determined to be from the 3rd second to the 4th second, then the phoneme information of the phoneme G is determined to be from the 3rd second to the 4th second; the video material corresponding to the phoneme information refers to the video clip captured in the audio and video according to the phoneme information.

[0058] Specifically, after determining the role identifier in the audio and video material generation request, a target phoneme information set corresponding to the role identifier is determined in at least one created phoneme information set based on the role identifier; the target phoneme information set includes the phoneme information of the target role corresponding to the role identifier and the video clip corresponding to the phoneme information.

[0059] For example, based on the character identifier "3", the phoneme information set a3 corresponding to the character identifier "3" is determined in the phoneme information set a1, the phoneme information set a2 and the phoneme information set a3; the phoneme information set a3 includes the phoneme information of the target character "Zhang San" corresponding to the character identifier "3" and the video clip corresponding to the phoneme information.

[0060] By determining the target phoneme information set corresponding to the role identifier, audio and video materials can be subsequently generated based on the phoneme information in the target phoneme information set and the video clip corresponding to the phoneme information.

[0061] Before determining the corresponding target phoneme information set based on the role identifier, it is necessary to create phoneme information sets corresponding to different roles, so that the corresponding phoneme information set can be queried based on the role identifier subsequently.

[0062] Specifically, before determining the target phoneme information set based on the role identifier, it may further include:

[0063] Determine the audio-visual content to be processed, and identify the role phoneme information of the target role in the audio-visual content to be processed;

[0064] Based on the role phoneme information, intercept video materials in the audio-visual content to be processed;

[0065] Generate the role phoneme information set corresponding to the target role according to the role phoneme information and the video materials.

[0066] Among them, the audio-visual content to be processed refers to an audio-visual file containing audio content and video content corresponding to the audio content; the target role refers to the role included in the audio-visual content to be processed; multiple roles may be included in the audio-visual content to be processed. Identify the audio segments corresponding to the phonemes of each role in the audio-visual content to be processed during playback, and generate the role phoneme information corresponding to the role based on the start time point and end time point of the audio segment. For example, in the TV drama video L, there is audio corresponding to role A and audio corresponding to role B. After identifying the audio segment corresponding to role A, determine the start time point "3 minutes and 25 seconds" and end time point "3 minutes and 54 seconds" of the audio segment, and generate the role phoneme information corresponding to role A based on the start time point and end time point.

[0067] For another example, identify the audio segment corresponding to the initial consonant "zh" of the character "张" read by role S in the audio-visual content, and obtain the phoneme information corresponding to the initial consonant "zh".

[0068] After identifying the role phoneme information, the role phoneme information can also be reviewed to ensure its accuracy; after determining the role phoneme information, intercept the video materials corresponding to the role phoneme information in the audio-visual content to be processed. For example, intercept the video segment from the 4th second to the 8th second in the audio-visual content H to be processed according to the role phoneme information "from the 4th second to the 8th second" as the video materials corresponding to the role phoneme information; generate the role phoneme information set corresponding to the target role according to the determined role phoneme information and the video materials corresponding to the role phoneme information.

[0069] Furthermore, the method for identifying the role phoneme information of the target role in the audio-visual content to be processed may include:

[0070] Determine the preset phoneme recognition rule;

[0071] Identify the initial consonant phonemes and final phonemes of the target character in the audio of the audio and video to be processed according to the preset phoneme recognition rule to form a phoneme set;

[0072] Determine the time interval information corresponding to each phoneme in the phoneme set in the audio, and generate character phoneme information corresponding to each phoneme.

[0073] Among them, the preset phoneme recognition rules refer to the rules for identifying the phonemes corresponding to the target character in the audio and video to be processed; the audio of the audio and video to be processed refers to the audio content in the audio and video to be processed; the initial consonant phoneme refers to the consonant before the final phoneme, which can form a complete phoneme sequence together with the final phoneme; the final phoneme refers to the phoneme that can form a complete phoneme sequence together with the initial consonant phoneme.

[0074] For example, the initial consonant phonemes and final phonemes corresponding to the target character are identified in the audio and video to be processed through the preset phoneme recognition rule R.

[0075] The purpose of this application to identify the initial consonant phonemes and final consonant phonemes in the audio corresponding to the audio and video is to obtain a phoneme sequence not included in the audio and video to be processed by combining and splicing different initial consonant phonemes with different final consonant phonemes, and then subsequently obtain the audio and video material corresponding to the phoneme sequence.

[0076] A phoneme set is generated from the collected initial consonant phonemes and final consonant phonemes corresponding to the audio and video to be processed; the time interval information corresponding to each phoneme in the phoneme set in the audio of the audio and video to be processed is determined, for example, the time interval information of phoneme K in the audio is determined to be "the 3rd second to the 5th second", specifically, the time interval information corresponding to each initial consonant phoneme and the time interval information corresponding to each final consonant phoneme in the phoneme set are determined; the time interval information corresponding to each phoneme is used as the phoneme information corresponding to the phoneme.

[0077] Step 106: Acquire a phoneme information sequence in the target phoneme information set according to the phoneme sequence.

[0078] Among them, the phoneme information sequence refers to a sequence generated by the phoneme information corresponding to the phoneme sequence; the phoneme information corresponding to the phonemes in the phoneme sequence is obtained in the target phoneme set according to the phoneme sequence, and the phoneme information sequence corresponding to the phoneme sequence is composed according to the phoneme information corresponding to each phoneme in the determined phoneme sequence.

[0079] For example, determine the character phoneme information set corresponding to the phoneme sequence "hao" and the character identification "3"; determine the phoneme information m1 "1.5 seconds to 1.6 seconds" corresponding to the phoneme "h" and the phoneme information m2 "1.6 seconds to 1.8 seconds" corresponding to the phoneme "ao" in the character phoneme information set according to the phoneme sequence; generate a phoneme information sequence corresponding to the phoneme sequence "hao" from the phoneme information m1 and the phoneme information m2.

[0080] The phoneme information sequence is determined in the target phoneme information set according to the phoneme sequence, so that audio and video materials corresponding to the phoneme sequence can be subsequently generated based on the phoneme information sequence.

[0081] Step 108: Determine the video material corresponding to each phoneme information in the phoneme information sequence, and generate the target audio and video material corresponding to the phoneme sequence.

[0082] After determining the phoneme information sequence in the target phoneme information set, determine the video material corresponding to the phonemes in the phoneme information sequence in the target phoneme information set according to the phoneme information sequence; generate the target audio and video material based on the phoneme information and the video material corresponding to the phoneme information.

[0083] Specifically, the method of determining the video material corresponding to each phoneme information in the phoneme information sequence and generating the target audio and video material corresponding to the phoneme sequence may include:

[0084] Based on a preset interception rule, intercept an audio segment of an initial format corresponding to each phoneme information in the audio corresponding to the phoneme information sequence;

[0085] splicing each audio clip into a target audio material according to a preset splicing rule;

[0086] A target video material is generated based on the video material corresponding to each phoneme information, and the target audio material and the target video material are aligned to obtain the target audio and video material.

[0087] Among them, the preset interception rule refers to the rule for intercepting audio clips in the audio; the initial format refers to the format corresponding to the audio clip; for example, according to the interception rule, an audio clip in the PCM format is intercepted in the audio; by intercepting the audio clip in the initial format that is convenient for audio processing, the audio clips can be spliced ​​later; the preset splicing rule refers to the rule for splicing audio clips, for example, the preset splicing rule is the rule for overlapping and splicing initial consonant audio clips and final consonant audio clips; in actual applications, since the audio effect obtained by simply splicing audio clips is not good, the preset splicing rules can be used to splice the audio clips in audio splicing methods such as overlapping and fading in and out to obtain the target audio material; the target audio material refers to the audio material used to generate the target audio and video material; the target video material refers to the video material used to generate the target audio and video material.

[0088] For example, according to the preset interception rule, an audio segment in PCM format corresponding to each phoneme in the phoneme sequence G is intercepted from the audio corresponding to the phoneme sequence G; each audio segment is spliced ​​according to the preset splicing rule to obtain the target audio material; the video material corresponding to each phoneme in the phoneme sequence G is spliced ​​into the target video material; the target video material and the target audio material are aligned to obtain the target audio and video material.

[0089] In practical applications, the method of aligning the target audio material and the target video material to obtain the target audio and video material may include:

[0090] Determine audio duration information corresponding to the target audio material, and video duration information corresponding to the target video material;

[0091] The audio duration information is adjusted to be consistent with the video duration information, and a target audio and video material is generated based on the adjusted target video material and target audio material.

[0092] The audio duration information refers to the time length required to play the target audio material; the video duration information refers to the video playback information corresponding to the target video material.

[0093] Specifically, since the audio duration information corresponding to the target audio material will be added or increased in the process of generating the target audio material, in order to make the audio and video durations of the generated target audio and video material consistent, it is necessary to adjust the audio duration and / or video duration; that is, adjust the video duration information and the audio duration information to be consistent, and then splice the target audio material and the target video material to obtain the target audio and video material.

[0094] It should be noted that, in actual applications, the target audio and video material can be generated based on the target audio material and part of the video material in the target video material. For example, the target audio and video material can be generated only based on the video material corresponding to the final phonemes. At this time, the audio duration and the video duration may be inconsistent, so the audio material or the video material also needs to be processed by speed change, etc. to ensure the consistency between the audio duration information and the video duration information.

[0095] After generating the target audio and video material corresponding to the phoneme sequence, the method may further include:

[0096] The target audio and video material and the phoneme sequence are stored in an audio and video material library in correspondence.

[0097] By storing the target audio and video material and the phoneme sequence in correspondence in the audio and video material library, the user can search or browse the audio and video material in the audio and video material library later.

[0098] Furthermore, after the target audio and video material and the phoneme sequence are stored in the audio and video material library in correspondence, the following steps may also be included:

[0099] Receiving a video generation request for a target character, wherein the video generation request includes a phoneme sequence to be processed;

[0100] Acquiring the audio and video material to be processed from the audio and video material library based on the phoneme sequence to be processed;

[0101] Generate a target video according to the audio and video materials to be processed.

[0102] Among them, the video generation request refers to a request to generate a video based on audio and video materials; the video generation request can be a video generation request for a target character, for example, receiving a video generation request for a target character "Zhang San", that is, generating a video generation request with the target character "Zhang San" as the audio material.

[0103] The video generation request carries a phoneme sequence to be processed, which is used to obtain audio and video materials corresponding to the phoneme sequence to be processed in the audio and video material library; the audio and video materials to be processed refer to the audio and video materials corresponding to the phoneme sequence to be processed; and the target video is generated based on the video materials to be processed.

[0104] In practical applications, the video generation request may include video processing information; the specific method of generating the target audio and video according to the target audio and video material may include:

[0105] The target audio and video material is processed based on the video processing information to obtain the target audio and video.

[0106] The video processing information refers to information used to process the target audio and video material. For example, the video processing information is to adjust the target audio and video material to double speed, or to add a black and white filter effect to the target audio and video material.

[0107] For example, the video processing information is determined to adjust the target audio and video material to 0.5 times the speed; after the audio and video material H is determined, the audio and video material H is adjusted to 0.5 times the speed according to the video processing information.

[0108] In actual applications, users can query corresponding audio and video materials in the audio and video material library by entering text or phoneme sequences; when the content entered by the user in the search box is text, after the corresponding audio and video material is searched, the default subtitles of the audio and video material are consistent with the text entered by the user; then the video processing information can include information for adjusting the default subtitles.

[0109] Specifically, when the target audio and video material includes default subtitles, the target audio and video material is processed based on the video processing information to obtain the target video, and the method may include:

[0110] Determining the default subtitles to be processed in the target audio and video material;

[0111] The to-be-processed default subtitles are adjusted based on the video processing information to obtain a target video containing target subtitles.

[0112] The default subtitles to be processed refer to the subtitles that need to be adjusted in the target audio and video material; the subtitle adjustment information contained in the video processing information is determined, and the default subtitles to be processed are adjusted according to the subtitle adjustment information to obtain the target subtitles, that is, to obtain the target video containing the target subtitles.

[0113] For example, it is determined that the default subtitle to be processed in the target audio and video material is "Today's weather is sunny"; and the subtitle adjustment information is determined to be "delete today and sunny, and add cloudy". Then, the default subtitle to be processed is adjusted according to the subtitle adjustment information to obtain a target video containing the target subtitle "The weather is cloudy".

[0114] The method for generating audio and video materials of the present application receives an audio and video material generation request for a phoneme sequence, wherein the audio and video material generation request includes a role identifier; determines a target phoneme information set based on the role identifier, wherein the target phoneme information set includes phoneme information and video materials corresponding to the phoneme information; obtains a phoneme information sequence in the target phoneme information set according to the phoneme sequence; determines the video material corresponding to each phoneme information in the phoneme information sequence, and generates the target audio and video material corresponding to the phoneme sequence. By obtaining the corresponding phoneme information sequence according to the phoneme sequence, and then generating the target audio and video material according to the phoneme information sequence and the video material, the efficiency of generating audio and video materials is improved.

[0115] The following combination Figure 2 Taking the application of the method for generating audio and video materials provided by the present application in the phoneme sequence z as an example, the method for generating audio and video materials is further described. Figure 2 A processing flow chart of a method for generating audio and video materials applied to a phoneme sequence z provided in an embodiment of the present application is shown, which specifically includes the following steps:

[0116] Step 202: Obtain a target text Q, and obtain a phoneme sequence z based on the target text Q.

[0117] Specifically, the target text Q is the text "goodbye", and the target character "Zhang San" does not have the audio of "goodbye" in any video; recognize the target text Q and obtain the phoneme sequence "zaijian".

[0118] Step 204: receiving an audio / video material generation request for the phoneme sequence z, wherein the audio / video material generation request includes a role identifier "1".

[0119] Specifically, the role "Zhang San" can be determined according to the role identifier "1"; and a request for generating audio and video materials for the phoneme sequence z is received so as to subsequently generate audio and video materials corresponding to the role "Zhang San".

[0120] Step 206: Determine a target phoneme information set based on the role identifier "1", wherein the target phoneme information set includes the phoneme information and the video material corresponding to the phoneme information.

[0121] Specifically, before determining the target phoneme information set based on the role identifier "1", it is necessary to generate the target phoneme information set. The specific generation process may include: determining the audio and video to be processed, which contains the video content corresponding to the role "Zhang San" in the audio and video to be processed; identifying the initial consonant phonemes and final phonemes of the target role "Zhang San" in the audio of the audio and video to be processed according to preset phoneme recognition rules to form a phoneme set; determining the time interval information corresponding to each phoneme in the phoneme set in the audio of the video, and generating the role phoneme information corresponding to each phoneme; intercepting video materials in the audio and video to be processed based on the role phoneme information; generating the target phoneme information set corresponding to the target role "Zhang San" based on the acquired role phoneme information and video materials.

[0122] Step 208: Obtain a phoneme information sequence in the target phoneme information set according to the phoneme sequence z.

[0123] Specifically, the phonemes included in the phoneme sequence z are compared with the phonemes in the target phoneme information set respectively, the phoneme information corresponding to each phoneme in each phoneme sequence z is determined, and the determined phoneme information constitutes a phoneme information sequence.

[0124] Step 210: extracting an audio segment corresponding to each phoneme information in the phoneme information sequence from the audio corresponding to the phoneme information sequence.

[0125] Specifically, an audio segment corresponding to each phoneme in the audio is intercepted according to the phoneme information, that is, the time interval of the audio; in order to facilitate the subsequent splicing processing of the audio segments, the audio segments in the PCM format can be intercepted.

[0126] Step 212: splice each audio segment into a target audio material according to a preset splicing rule.

[0127] Specifically, each audio clip is overlapped and spliced ​​to obtain the target audio material.

[0128] Step 214: Generate a target video material based on the video materials corresponding to each phoneme information, and align the target audio material and the target video material to obtain a target audio-visual material.

[0129] Specifically, splice the video materials corresponding to each phoneme information in the target phoneme information set into a target video material; adjust the durations of the target video material and the target audio material to be the same, and form a target audio-visual material based on the adjusted target video material and target audio material.

[0130] Step 216: Correspondingly store the phoneme sequence and the target audio-visual material in the audio-visual material library.

[0131] For the method for generating an audio-visual material of the present application, an audio-visual material generation request for a phoneme sequence is received, wherein the audio-visual material generation request includes a role identifier; a target phoneme information set is determined based on the role identifier, wherein the target phoneme information set includes phoneme information and video materials corresponding to the phoneme information; a phoneme information sequence is obtained from the target phoneme information set according to the phoneme sequence; video materials corresponding to each phoneme information in the phoneme information sequence are determined, and a target audio-visual material corresponding to the phoneme sequence is generated. By obtaining a corresponding phoneme information sequence according to the phoneme sequence, and then generating a target audio-visual material according to the phoneme information sequence and the video materials, the efficiency of generating the audio-visual material is improved.

[0132] The following combines Figure 3 to further illustrate the method for generating an audio-visual material of the present application, wherein Figure 3 shows a schematic diagram of the method for generating an audio-visual material provided by an embodiment of the present application.

[0133] Figure 3 It is shown that the server may include a front-end module, a server-side module, and a multimedia module.

[0134] After receiving a spelling request for a target role, the front-end module initiates a spelling task; based on the spelling request, selects an initial consonant sentence in the original sound video of the target role, and determines the initial consonant time period corresponding to each initial consonant sentence in the initial consonant sentence; selects a medial and final sentence in the original sound video of the target role, and determines the medial and final time period corresponding to each medial and final sentence; confirms the product duration, that is, confirms the video duration of the medial and final part of the medial and final character, for example, determines the duration of the medial and final "ang" of the character "Wang" in the original sound video corresponding to the video in the original sound video; sends the determined time information to the server-side module for further processing.

[0135] The server-side module integrates the data of the determined time period and the source of the materials; specifically, determines the initial consonant source sentence materials and the initial consonant source sentence material selection time period, the final consonant source sentence materials and the final consonant source sentence material selection time period, the picture source sentence materials and the picture source sentence material selection time period; based on the material source and the corresponding selection time period, it is convenient to subsequently collect video and audio to generate audio and video materials.

[0136] The multimedia module includes a synthesis SDK for synthesizing audio and video materials; the synthesis SDK extracts audio clips in PCM format from the initial consonant material according to the selected time period based on the data sorted by the server module, and extracts audio clips in PCM format from the final consonant material according to the selected time period; the initial consonant audio clips and the corresponding final consonant audio clips are synthesized by fading in and out, and the audio duration information of the synthesized audio is adjusted, such as by speed change and other operations, so that it is consistent with the picture duration information corresponding to the picture, to obtain a product PCM; the adjusted audio is used as audio material; the picture clip is extracted from the picture material according to the selected time period as the video material, that is, the product picture; the audio material and the picture material are spliced ​​to obtain audio and video material in MP4 format, that is, the product MP4; the spliced ​​audio and video material can be further adjusted until it meets the preset material requirements and is stored in the material library; the generated audio and video material can be returned to the server module and the front-end module.

[0137] Corresponding to the above method embodiment, the present application also provides an embodiment of a device for generating audio and video materials. Figure 4 A schematic structural diagram of a device for generating audio and video materials provided in one embodiment of the present application is shown.

[0138] like Figure 4 As shown, the device comprises:

[0139] The receiving module 402 is configured to receive an audio / video material generation request for a phoneme sequence, wherein the audio / video material generation request includes a role identifier;

[0140] The determination module 404 is configured to determine a target phoneme information set based on the role identifier, wherein the target phoneme information set includes phoneme information and video material corresponding to the phoneme information;

[0141] An acquisition module 406 is configured to acquire a phoneme information sequence in the target phoneme information set according to the phoneme sequence;

[0142] The generation module 408 is configured to determine the video material corresponding to each phoneme information in the phoneme information sequence, and generate the target audio and video material corresponding to the phoneme sequence.

[0143] Optionally, the device further includes a generating submodule configured to:

[0144] Determine the audio and video to be processed, and identify the character phoneme information of the target character in the audio and video to be processed;

[0145] Based on the character phoneme information, intercept video material in the audio and video to be processed;

[0146] A character phoneme information set corresponding to the target character is generated according to the character phoneme information and the video material.

[0147] Optionally, the generating submodule is further configured to:

[0148] Determining preset phoneme recognition rules;

[0149] Identify the initial consonant phonemes and final phonemes of the target character in the audio of the audio and video to be processed according to the preset phoneme recognition rule to form a phoneme set;

[0150] Determine the time interval information corresponding to each phoneme in the phoneme set in the audio, and generate character phoneme information corresponding to each phoneme.

[0151] Optionally, the device further includes an acquisition submodule configured to:

[0152] Get the target text;

[0153] The target text is recognized to obtain a phoneme sequence corresponding to the target text.

[0154] Optionally, the acquisition submodule is further configured to:

[0155] Splitting the target text to obtain target characters in the target text;

[0156] The phoneme subsequence corresponding to each target text is identified, and the phoneme sequence corresponding to the target text is composed of each phoneme subsequence.

[0157] Optionally, the device further includes a storage module configured to:

[0158] The target audio and video material and the phoneme sequence are stored in an audio and video material library in correspondence.

[0159] Optionally, the device further includes a target video generation module configured to:

[0160] Receiving a video generation request for a target character, wherein the video generation request includes a phoneme sequence to be processed;

[0161] Acquiring the audio and video material to be processed from the audio and video material library based on the phoneme sequence to be processed;

[0162] Generate a target video according to the audio and video materials to be processed.

[0163] Optionally, the video generation request includes video processing information; and the target video generation module is further configured to:

[0164] The audio and video material to be processed is processed based on the video processing information to obtain a target video.

[0165] Optionally, the audio and video material to be processed includes default subtitles; and the target video generation module is further configured to:

[0166] Determining the default subtitles to be processed in the audio and video material to be processed;

[0167] The to-be-processed default subtitles are adjusted based on the video processing information to obtain a target video containing target subtitles.

[0168] Optionally, the generating module 408 is further configured to:

[0169] Based on a preset interception rule, intercept an audio segment of an initial format corresponding to each phoneme information in the audio corresponding to the phoneme information sequence;

[0170] splicing each audio clip into a target audio material according to a preset splicing rule;

[0171] A target video material is generated based on the video material corresponding to each phoneme information, and the target audio material and the target video material are aligned to obtain the target audio and video material.

[0172] Optionally, the generating module 408 is further configured to:

[0173] Determine audio duration information corresponding to the target audio material, and video duration information corresponding to the target video material;

[0174] The audio duration information is adjusted to be consistent with the video duration information, and a target audio and video material is generated based on the adjusted target video material and target audio material.

[0175] The device for generating audio and video materials of the present application comprises a receiving module configured to receive an audio and video material generation request for a phoneme sequence, wherein the audio and video material generation request includes a role identifier; a determining module configured to determine a target phoneme information set based on the role identifier, wherein the target phoneme information set includes phoneme information and video materials corresponding to the phoneme information; an acquiring module configured to acquire a phoneme information sequence in the target phoneme information set according to the phoneme sequence; and a generating module configured to determine the video materials corresponding to each phoneme information in the phoneme information sequence, and generate the target audio and video material corresponding to the phoneme sequence. By acquiring the corresponding phoneme information sequence and then generating the target audio and video material according to the phoneme information sequence and the video material, the effect of improving the efficiency of generating audio and video materials is achieved.

[0176] The above is a schematic scheme of a device for generating audio and video materials of this embodiment. It should be noted that the technical scheme of the device for generating audio and video materials and the technical scheme of the method for generating audio and video materials described above are of the same concept, and the details not described in detail in the technical scheme of the device for generating audio and video materials can be found in the description of the technical scheme of the method for generating audio and video materials described above.

[0177] Figure 5 The block diagram of a computing device 500 provided according to an embodiment of the present application is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0178] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0179] In one embodiment of the present application, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0180] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.

[0181] Among them, when the processor 520 executes the computer instructions, the steps of the method for generating audio and video materials are implemented.

[0182] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned method for generating audio and video materials belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned method for generating audio and video materials.

[0183] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the method for generating audio and video materials as described above.

[0184] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned method for generating audio and video materials belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned method for generating audio and video materials.

[0185] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0186] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0187] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0188] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0189] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A method for generating audio and video materials, characterized in that: include: Receiving an audio / video material generation request for a phoneme sequence, wherein the audio / video material generation request includes a role identifier; Determining a target phoneme information set based on the role identifier, wherein the target phoneme information set includes phoneme information and video material corresponding to the phoneme information; Acquire a phoneme information sequence in the target phoneme information set according to the phoneme sequence; Determine the video material corresponding to each phoneme information in the phoneme information sequence, and generate the target audio and video material corresponding to the phoneme sequence; Among them, the target phoneme information set refers to a set corresponding to the role identifier, consisting of phoneme information and video materials corresponding to the phoneme information; the phoneme sequence refers to a sequence composed of phonemes; the phoneme information refers to the time interval corresponding to the phoneme in the audio and video; the phoneme information sequence refers to a sequence generated by the phoneme information corresponding to the phoneme sequence; The step of determining the video material corresponding to each phoneme information in the phoneme information sequence and generating the target audio and video material corresponding to the phoneme sequence includes: Based on a preset interception rule, intercept an audio segment of an initial format corresponding to each phoneme information in the audio corresponding to the phoneme information sequence; splicing each audio clip into a target audio material according to a preset splicing rule; A target video material is generated based on the video material corresponding to each phoneme information, and the target audio material and the target video material are aligned to obtain the target audio and video material.

2. The method according to claim 1, characterized in that Before determining the target phoneme information set based on the role identifier, the method further includes: Determine the audio and video to be processed, and identify the character phoneme information of the target character in the audio and video to be processed; Based on the character phoneme information, intercept video material in the audio and video to be processed; A character phoneme information set corresponding to the target character is generated according to the character phoneme information and the video material.

3. The method according to claim 2, characterized in that Identifying the character phoneme information of the target character in the audio and video to be processed includes: Determining preset phoneme recognition rules; Identify the initial consonant phonemes and final phonemes of the target character in the audio of the audio and video to be processed according to the preset phoneme recognition rule to form a phoneme set; Determine the time interval information corresponding to each phoneme in the phoneme set in the audio, and generate character phoneme information corresponding to each phoneme.

4. The method according to claim 1, characterized in that Before receiving a request for generating audio and video materials for a phoneme sequence, the method further includes: Get the target text; The target text is recognized to obtain a phoneme sequence corresponding to the target text.

5. The method according to claim 4, characterized in that Recognizing the phoneme sequence of the target text, including: Splitting the target text to obtain target characters in the target text; The phoneme subsequence corresponding to each target text is identified, and the phoneme sequence corresponding to the target text is composed of each phoneme subsequence.

6. The method according to claim 1, characterized in that After generating the target audio and video material corresponding to the phoneme sequence, the method further includes: The target audio and video material and the phoneme sequence are stored in an audio and video material library in correspondence.

7. The method according to claim 6, characterized in that After the target audio and video material and the phoneme sequence are stored in the audio and video material library in correspondence, the method further includes: Receiving a video generation request for a target character, wherein the video generation request includes a phoneme sequence to be processed; Acquiring the audio and video material to be processed from the audio and video material library based on the phoneme sequence to be processed; Generate a target video according to the audio and video materials to be processed.

8. The method according to claim 7, characterized in that The video generation request includes video processing information; Generating a target video according to the audio and video material to be processed includes: The audio and video material to be processed is processed based on the video processing information to obtain a target video.

9. The method according to claim 8, characterized in that The audio and video material to be processed includes default subtitles; Processing the audio and video material to be processed based on the video processing information to obtain a target video includes: Determining the default subtitles to be processed in the audio and video material to be processed; The to-be-processed default subtitles are adjusted based on the video processing information to obtain a target video containing target subtitles.

10. The method according to claim 1, characterized in that Aligning the target audio material and the target video material to obtain the target audio and video material includes: Determine audio duration information corresponding to the target audio material, and video duration information corresponding to the target video material; The audio duration information is adjusted to be consistent with the video duration information, and a target audio and video material is generated based on the adjusted target video material and target audio material.

11. A device for generating audio and video materials, characterized in that: include: A receiving module is configured to receive an audio and video material generation request for a phoneme sequence, wherein the audio and video material generation request includes a role identifier; A determination module is configured to determine a target phoneme information set based on the role identifier, wherein the target phoneme information set includes phoneme information and video material corresponding to the phoneme information; An acquisition module, configured to acquire a phoneme information sequence in the target phoneme information set according to the phoneme sequence; A generating module is configured to determine the video material corresponding to each phoneme information in the phoneme information sequence, and generate a target audio and video material corresponding to the phoneme sequence; Among them, the target phoneme information set refers to a set corresponding to the role identifier, consisting of phoneme information and video materials corresponding to the phoneme information; the phoneme sequence refers to a sequence composed of phonemes; the phoneme information refers to the time interval corresponding to the phoneme in the audio and video; the phoneme information sequence refers to a sequence generated by the phoneme information corresponding to the phoneme sequence; The step of determining the video material corresponding to each phoneme information in the phoneme information sequence and generating the target audio and video material corresponding to the phoneme sequence includes: Based on a preset interception rule, intercept an audio segment of an initial format corresponding to each phoneme information in the audio corresponding to the phoneme information sequence; splicing each audio clip into a target audio material according to a preset splicing rule; A target video material is generated based on the video material corresponding to each phoneme information, and the target audio material and the target video material are aligned to obtain the target audio and video material.

12. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer instructions, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented.

14. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented.

Images