A radio and television program production method and system based on artificial intelligence

Abstract

The present invention relates to the field of audio processing technology, and in particular to a method and system for producing radio and television programs based on artificial intelligence. The method obtains an audio segment; based on the voiceprint embedding vector corresponding to the human voice at each moment in the audio segment, the audio segment is divided into a single-sound sub-segment and a multi-sound sub-segment; based on the intensity of the human voice at each moment in the single-sound sub-segment corresponding to each character in the multi-sound sub-segment, the enhanced audio of the human voice of each character in the multi-sound sub-segment is obtained, and then each sentence corresponding to each character is obtained; based on the structure and narration of each sentence corresponding to each character, the display subtitles of the television program are obtained. The present invention enhances the audio of each character in the multi-sound sub-segment, thereby improving the accuracy of identifying the speech content of each character, and then accurately obtaining each sentence corresponding to each character, and then analyzing each sentence, effectively improving the accuracy and completeness of the subtitle display.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to a radio and television program production method and system based on artificial intelligence. Background Art

[0002] Radio and television program production is a systematic process encompassing four phases: pre-production planning, production execution, post-production, and broadcast feedback. After recording according to the script, a large amount of raw footage is generated. Post-production involves editing this material to suit the program's characteristics, preserving the effective material and playback logic, and synchronizing the audio and video. Whether it's a variety show, news, or interview program, it often includes a significant amount of speech. Subtitles must be edited based on the speech to ensure a complete and engaging video. Therefore, ensuring accurate subtitle display in television programs is crucial to enhance the audience's viewing experience.

[0003] With the rise of artificial intelligence, computers can analyze and process audio and identify the speech content of each person in the audio. However, in reality, variety shows or interviews often have multiple people speaking at the same time, different people have different speaking habits, and speech is omitted. As a result, the subtitles displayed in TV programs by identifying audio are inaccurate and incomplete, affecting the audience's accurate understanding of the speech content of the people in the TV program. Summary of the Invention

[0004] In order to solve the technical problem of inaccurate and incomplete subtitles displayed in television programs due to multiple people speaking simultaneously, different speaking habits among different people, and omissions of speech in variety shows or interviews, the purpose of the present invention is to provide a radio and television program production method and system based on artificial intelligence. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for producing a radio and television program based on artificial intelligence, the method comprising the following steps:

[0006] Get each audio segment in the audio file;

[0007] Based on the voiceprint embedding vector corresponding to the human voice at each moment in each audio segment, the audio segment is divided into single-sound sub-segments and multi-sound sub-segments;

[0008] Based on the intensities of the human voice at each frequency at each moment in the single sound subsegment corresponding to each character in each multi-sound subsegment, obtaining reference intensities of the human voice at each frequency at each moment in each multi-sound subsegment, thereby obtaining enhanced audio of the human voice of each character in each multi-sound subsegment;

[0009] Each single sound sub-segment and the enhanced audio are identified to obtain each sentence corresponding to each character; and display subtitles of the television program are obtained based on the structure and narration of each sentence corresponding to each character.

[0010] Furthermore, the method for obtaining the single sound sub-segment and the multi-sound sub-segment is:

[0011] Get the voiceprint embedding vector corresponding to each person appearing in the audio file, and use them as the reference voiceprint embedding vector;

[0012] For any moment in any audio segment, obtain the cosine similarity between the voiceprint embedding vector corresponding to the human voice at that moment and each reference voiceprint embedding vector, and use it as the degree of character matching corresponding to the human voice at that moment;

[0013] When the character matching degree is greater than the character matching degree threshold, the character corresponding to the reference voiceprint embedding vector is used as the matching character of the human voice at that moment;

[0014] When the number of matching characters is greater than 1, this moment is a multi-voice overlapping moment;

[0015] When the number of matching characters is equal to 1, the moment is a single voice moment;

[0016] The local audio segment formed by the continuous overlapping moments of multiple voices in the audio segment is used as a multi-voice sub-segment;

[0017] A local audio segment formed by continuous single voice moments in the audio segment is used as a single voice sub-segment.

[0018] Furthermore, the method for obtaining the character matching degree threshold is:

[0019] Obtain the cosine similarity of any two reference voiceprint embedding vectors as the first similarity value;

[0020] The largest first similarity value is used as the character matching degree threshold.

[0021] Furthermore, the reference intensity is obtained by:

[0022] For the a-th polyphonic sub-segment, all local polyphonic sub-segments corresponding to the continuous human voices of the same character in the a-th polyphonic sub-segment are used as reference sound segments; wherein the character corresponding to each moment in the same reference sound segment is the same;

[0023] For the t-th moment in the i-th reference sound segment of the a-th multi-sound sub-segment, the intensity of the human voice at the k-th frequency at the t-th moment is used as the target intensity;

[0024] For the p-th character in the i-th reference sound segment, the intensity of the human voice at the k-th frequency at each moment in all the single sound sub-segments corresponding to the p-th character is used as the reference intensity;

[0025] taking the difference between the target intensity and each of the reference intensities as a first difference;

[0026] The result of negative correlation and normalization of the mean of the first difference is used as the intensity adjustment weight of the voice of the p-th character at the k-th frequency at the t-th moment;

[0027] The product of the target intensity and the intensity adjustment weight is used as the reference intensity corresponding to the voice of the p-th character at the k-th frequency at the t-th moment in the i-th reference sound segment of the a-th multi-sound sub-segment.

[0028] Furthermore, the method for obtaining display subtitles of a television program is:

[0029] For any character in the audio file, the degree of component deficiency of each sentence of the character is obtained based on the difference in balance factor and the difference in the number of leaf nodes of the syntax tree corresponding to each sentence of the character and each other sentence of the character;

[0030] When the degree of component deficiency is greater than a preset component deficiency threshold, the corresponding sentence is a component deficiency sentence of the character; wherein a sentence is a complete sentence;

[0031] For any component constituting a sentence and any component-missing sentence, the probability of the component-missing sentence missing the component is obtained based on the lack of the component in the sentence and the proportion of sentences containing the component in all sentences of the character;

[0032] When the probability is greater than a preset probability threshold, the content corresponding to the component in the adjacent sentence before the sentence where the component is missing is used as the content of the component in the sentence where the component is missing, and is displayed in brackets in the subtitles;

[0033] For any sentence, the ratio of the corresponding duration of the sentence to the number of words in the sentence is used as the reference time interval between any two adjacent words in the sentence;

[0034] When the time interval between any two adjacent words in the sentence is greater than a preset multiple of the reference time interval, the sentence is displayed separately in the subtitles.

[0035] Furthermore, the method for obtaining the degree of deficiency of the component is:

[0036] For any sentence of the character, obtain the balance factor difference between the syntactic tree corresponding to the sentence and each other sentence of the character, and use them as the second difference;

[0037] Obtain the difference in the number of leaf nodes between the syntactic tree corresponding to the sentence and each other sentence of the character, and use them as the third difference;

[0038] The cumulative value of the second difference and the cumulative value of the third difference are added together and normalized, and the result is used as the degree of component deficiency of the sentence.

[0039] Furthermore, the method for obtaining the degree of possibility is:

[0040] The result of negative correlation and normalization of the number of times the component appears in the sentence lacking the component is used as the reference lack degree of the component in the sentence lacking the component;

[0041] The ratio of the number of sentences containing the component in all sentences of the character to the total number of sentences of the character is used as the reference weight of the component in the corresponding sentences of the character;

[0042] The product of the reference weight and the reference lack degree is used as the probability degree of the lack of the component in the component-lacking sentence.

[0043] Furthermore, the method for obtaining the audio segment is:

[0044] The audio in the audio file with continuous moments of sound greater than the preset decibel level is divided into audio segments.

[0045] Furthermore, the artificial intelligence-based radio and television program production method further includes:

[0046] Each moment in each audio segment is consistent with the moment corresponding to each video frame.

[0047] In the second aspect, another embodiment of the present invention provides a radio and television program production system based on artificial intelligence, the system comprising: a memory, a processor, and a computer program stored in the memory and running on the processor, when the processor executes the computer program, it implements the steps of any one of the above methods.

[0048] The present invention has the following beneficial effects:

[0049] The present invention first divides the audio segment into single-sound sub-segments and multi-sound sub-segments based on the voiceprint embedding vector corresponding to the human voice at each moment in each audio segment, thereby improving the efficiency of audio processing; then, based on the intensity of the human voice at each moment in the single-sound sub-segment corresponding to each character in each multi-sound sub-segment, the reference intensity of the human voice of each character at each moment in each multi-sound sub-segment is obtained, accurately reflecting the sound intensity corresponding to the voice of each character at each moment in the multi-sound sub-segment, effectively reducing the interference between the voices of different characters in the multi-sound sub-segment, and improving the accuracy of obtaining the content spoken by each character in the multi-sound sub-segment; then, the enhanced audio of the human voice of each character in each multi-sound sub-segment is obtained, so that each sentence corresponding to each character in the audio file can be accurately identified; in order to ensure that the subtitles of the TV program are displayed more accurately and completely, the structure and narration of each sentence corresponding to each character are analyzed, so as to avoid the situation where the subtitles are incomplete and the sound and picture are out of sync due to missing sentence components or sentence pauses, so that the display subtitles of the TV program can be accurately obtained, effectively improving the audience's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A schematic flow chart of a method for producing radio and television programs based on artificial intelligence provided by one embodiment of the present invention;

[0052] Figure 2 A flow chart of a method for obtaining subtitles for a television program provided by one embodiment of the present invention;

[0053] Figure 3 A structural diagram of a radio and television program production system based on artificial intelligence provided by one embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a computer device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0055] To further illustrate the technical means and effects of the present invention to achieve the intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effects of an artificial intelligence-based radio and television program production method and system proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable form.

[0056] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0057] The following describes in detail a specific solution of a radio and television program production method and system based on artificial intelligence provided by the present invention with reference to the accompanying drawings.

[0058] Example 1:

[0059] This invention proposes a method for producing radio and television programs based on artificial intelligence. Figure 1 , which shows a schematic flow chart of a radio and television program production method based on artificial intelligence provided by one embodiment of the present invention, the method comprising the following steps:

[0060] Step S1: Acquire each audio segment in the audio file.

[0061] Specifically, this embodiment takes any TV program as an example for analysis. It should be noted that all subsequent TV programs are this TV program. In the post-production process, after the editor completes the editing of the original material, the video of the TV program is filled with subtitles, and then the audio file corresponding to the TV program is obtained. It is known that the minimum sound that the human ear can hear is 20 decibels, and thus this embodiment sets the preset decibel to 20 decibels. The implementer can set the size of the preset decibel according to the actual situation, and it is not limited here. The audio at the continuous moments when the sound in the audio file is greater than the preset decibel is divided into audio segments. Among them, each audio segment contains the voice of the character.

[0062] It should be noted that, in order to ensure the consistency of the audio and video time axes, this embodiment keeps each moment in each audio segment consistent with the moment corresponding to each video frame.

[0063] Step S2: Based on the voiceprint embedding vector corresponding to the human voice at each moment in each audio segment, the audio segment is divided into single-sound sub-segments and multi-sound sub-segments.

[0064] Specifically, it is known that the speech between characters in an audio file is overlapping and carried out sequentially, and there are situations where multiple people speak at the same time. When using existing technologies to identify the content of what each person said when multiple people speak at the same time, it is easy to identify inaccurately because the voices of different characters interfere with each other. It is very easy to accurately identify audio segments in which only one person speaks using existing technologies. Therefore, for local audio segments in which multiple people speak at the same time, it is necessary to enhance the voices of each character to ensure that the subtitles of subsequent TV programs are more accurate. Therefore, this embodiment first determines whether the voice at each moment in each audio segment is a single character or multiple characters based on the voiceprint embedding vector corresponding to the voice at each moment in each audio segment, and then divides the audio segment into single-sound sub-segments and multi-sound sub-segments. Among them, a single-sound sub-segment is an audio segment corresponding to the voice of only one character, and a multi-sound sub-segment is an audio segment corresponding to multiple characters speaking at the same time.

[0065] Preferably, in one possible implementation of this embodiment, the method for obtaining single-sound sub-segments and multiple-sound sub-segments is as follows: It is known that the voiceprint characteristics of each character are different, and then this embodiment first manually obtains the exclusive audio segments corresponding to a preset number of sentences spoken by each character appearing in the TV program. This embodiment sets the preset number to 10, that is, each character speaks 10 complete sentences alone. The implementer can set the size of the preset number according to actual conditions, and there is no limit here; then a pre-trained voiceprint encoder is used to extract the voiceprint embedding vector of the exclusive audio segment of each character appearing in the audio file, and all of them are used as reference voiceprint embedding vectors. At this point, the reference voiceprint embedding vector corresponding to each character is obtained, accurately representing the voiceprint characteristics corresponding to each character. Among them, using a pre-trained voiceprint encoder to extract voiceprint embedding vectors is a well-known technology and will not be described in detail;

[0066] For any moment in any audio segment, the pre-trained voiceprint encoder is also used to extract the voiceprint embedding vector corresponding to the human voice at that moment, and then the cosine similarity between the voiceprint embedding vector corresponding to the human voice at that moment and each reference voiceprint embedding vector is obtained, and all of them are used as the degree of matching of the person corresponding to the human voice at that moment; wherein, the greater the degree of character matching, the more likely the person corresponding to the corresponding reference voiceprint embedding vector is the person corresponding to the human voice at that moment, therefore, when the degree of character matching is greater than the character matching degree threshold, the person corresponding to the corresponding reference voiceprint embedding vector is used as the matching person of the human voice at that moment. wherein, the method for obtaining the character matching degree threshold is: considering that there is a certain similarity in the voiceprint features between different people, this embodiment obtains the cosine similarity of any two reference voiceprint embedding vectors as the first similarity value; then the largest first similarity value is used as the character matching degree threshold;

[0067] When the number of matching characters is greater than 1, the moment is a multi-voice overlapping moment; when the number of matching characters is equal to 1, the moment is a single-voice moment; then the local audio segment formed by the continuous multi-voice overlapping moments in the audio segment is regarded as a multi-voice sub-segment; the local audio segment formed by the continuous single-voice moments in the audio segment is regarded as a single-voice sub-segment.

[0068] At this point, the single-sound sub-segments and multi-sound sub-segments in each audio segment are accurately obtained.

[0069] Step S3: Based on the intensity of the human voice at each frequency at each moment in the single sound sub-segment corresponding to each character in each multi-sound sub-segment, obtain the reference intensity of the human voice of each character at each moment in each multi-sound sub-segment, and then obtain the enhanced audio of the human voice of each character in each multi-sound sub-segment.

[0070] Specifically, for a single-sound sub-segment, speech recognition can be performed directly. However, for a multi-sound sub-segment, when the speech content of each character contained in the multi-sound sub-segment is to be identified, the voices of other characters are relatively noise, and the voices of other characters should be weakened, retaining the spectrum of the desired character, thereby obtaining the enhanced audio of the desired character. When the difference between the intensity corresponding to the human voice at a certain moment in a multi-sound sub-segment at each frequency and the intensity corresponding to the human voice at each frequency in the single-sound sub-segment corresponding to a certain character at that moment in the multi-sound sub-segment is greater, it means that the voice of the character at that moment in the multi-sound sub-segment is more interfered with by the voices of other characters, and the intensity corresponding to the human voice at that moment at each frequency needs to be adjusted more, so as to accurately obtain the intensity corresponding to the human voice of the character at that moment at each frequency, and then accurately obtain the speech content of the character at that moment. Therefore, this embodiment obtains the reference intensity of the human voice of each character at each moment in each multi-sound subsegment at each frequency based on the intensity of the human voice at each frequency in the single-sound subsegment corresponding to each character in each multi-sound subsegment, and then obtains the enhanced audio of the human voice of each character in each multi-sound subsegment.

[0071] Preferably, in one achievable method of this embodiment, the method for obtaining the reference intensity is: taking into account that not all the multiple characters in the multi-sound sub-segment keep speaking continuously, in order to more accurately enhance the audio of each character in the multi-sound sub-segment, and then accurately obtain the speech content of each character in each multi-sound sub-segment, it is necessary to further divide each multi-sound sub-segment, and treat the audio segments corresponding to the continuous speaking of the same character in each multi-sound sub-segment as new independent audio segments for subsequent analysis. This embodiment takes the ath multi-sound sub-segment as an example for analysis. For the ath multi-sound sub-segment, the local multi-sound sub-segments corresponding to the continuous human voice corresponding to the same character in the ath multi-sound sub-segment are all used as reference sound segments; wherein, the characters corresponding to each moment in the same reference sound segment are the same; for the tth moment in the i-th reference sound segment of the ath multi-sound sub-segment, the intensity corresponding to the human voice at the t-th moment at the k-th frequency is used as the target intensity; for the p-th character in the i-th reference sound segment, the intensity corresponding to the human voice at each moment in all single sound sub-segments corresponding to the p-th character at the k-th frequency is used as the reference intensity; it should be noted that if there is no corresponding single sound sub-segment for the p-th character, the intensity corresponding to the human voice at each moment in the exclusive audio segment corresponding to the p-th character at the k-th frequency is used as the reference intensity;

[0072] The absolute value of the difference between the target intensity and each reference intensity is taken as the first difference; the larger the first difference, the greater the interference degree of the voice of the p-th character at the k-th frequency by the voices of other characters at the t-th moment, and the denoising degree needs to be increased. Therefore, in this embodiment, the result of negative correlation and normalization of the mean of the first difference is taken as the intensity adjustment weight of the voice of the p-th character at the k-th frequency at the t-th moment; the product of the target intensity and the intensity adjustment weight is then taken as the reference intensity corresponding to the voice of the p-th character at the k-th frequency at the t-th moment in the i-th reference sound segment of the a-th multi-sound sub-segment. It should be noted that in this embodiment, the mean of the first difference is negatively correlated and normalized by exp(-x), where exp is an exponential function with a natural constant as the base, and x represents the mean of the first difference.

[0073] At this point, the reference intensity corresponding to the human voice of each character at each moment in the i-th reference sound segment at each frequency is obtained, and then the enhanced audio corresponding to the human voice of each character in the i-th reference sound segment is obtained; according to the method of obtaining the enhanced audio corresponding to the human voice of each character in the i-th reference sound segment, the enhanced audio corresponding to the human voice of each character in each reference sound segment in the a-th multi-sound sub-segment is obtained, and the enhanced audio corresponding to the human voice of each character in the a-th multi-sound sub-segment is determined, and then the enhanced audio corresponding to the human voice of each character in each multi-sound sub-segment is obtained.

[0074] Step S4: Identify each single sound sub-segment and the enhanced audio to obtain each sentence corresponding to each character; obtain display subtitles of the TV program based on the structure and narration of each sentence corresponding to each character.

[0075] Specifically, existing technologies are used to identify each single sound segment and enhanced audio, accurately capturing each sentence corresponding to each character. Subtitles are known to display the speaker's speech simultaneously. Due to the varying speaking habits of different characters, some characters may use inverted sentences, resulting in inaccurate punctuation of their speech in the subtitles. Furthermore, some characters may omit their spoken content, resulting in incomplete subtitle content, seriously impacting the display quality of the subtitles. For example, taking the sentences "I like this program very much" and "Me too," the former sentence has the object placed before the subject, and the correct word order should be "I like this program very much." The latter sentence omits "I like this program very much," resulting in an incomplete sentence for "Me too." To make subtitles more intelligent and accessible, sentences with missing components should be supplemented. Inverted sentences can be analyzed to determine their completeness and whether they contain missing components by analyzing their structure. Therefore, this embodiment analyzes the structure of each character's sentences to capture their speaking habits, facilitating more accurate and reasonable acquisition of subtitles corresponding to each character's audio.

[0076] On the other hand, in the process of a character speaking a certain sentence, in order to create a sense of suspense, there will be a pause in the narration. If the subtitles are fully presented during the pause, it will lead to spoilers. Therefore, not every sentence of every character needs to be fully presented. For sentences that create a sense of suspense, it is necessary to first show the part of the sentence before the speaker pauses, and show the complete sentence after the pause rather than the part after that, so as to ensure the complete effect of the subtitles and the effect of the TV program.

[0077] Therefore, this embodiment accurately obtains the display subtitles of the television program according to the structure and narration of each sentence corresponding to each character.

[0078] Preferably, in one possible implementation of this embodiment, the method for obtaining the display subtitles of a television program is described in Figure 2 , which presents a flow chart of a method for obtaining display subtitles of a television program provided by this embodiment, the method comprising the following steps:

[0079] Step S201: For any character in the audio file, the degree of component deficiency of each sentence of the character is obtained based on the balance factor difference and the leaf node number difference of the syntax tree corresponding to each sentence of the character and each other sentence of the character.

[0080] For the sake of clarity, this embodiment uses any character in an audio file as an example for analysis. First, each sentence of the character is converted into text using the speech recognition model Whisper. Then, the text corresponding to each sentence is analyzed using the jsSyntaxTree software to draw a syntax tree for each sentence. The use of the speech recognition model Whisper to convert each sentence into text and the use of the jsSyntaxTree software to draw a syntax tree are both well-known technologies and will not be described in detail. The balance factors of the syntax trees corresponding to each sentence of the character should be similar, because the speaking habits of the character will not change quickly. Considering that some of the words spoken by the character may have some components omitted, there will be obvious differences between the syntax trees corresponding to the omitted sentences and all the sentences spoken by the character. Therefore, this embodiment obtains the degree of component omission of each sentence of the character based on the difference in balance factors and the difference in the number of leaf nodes between the syntax trees corresponding to each sentence of the character and all other sentences of the character. It should be noted that if the number of sentences spoken by the character in the audio file is less than a first preset number, all sentences in the character's exclusive audio segment are defaulted to the words spoken by the character in the audio file, where each sentence in the exclusive audio segment is a complete sentence. In this embodiment, the first preset number is set to 10. The implementer can set the first preset number to a larger value based on actual circumstances, and this is not limited here. The greater the degree of missing components, the more likely the corresponding sentence is to lack sentence components.

[0081] Preferably, in one possible implementation of this embodiment, the degree of component missing is determined by: for any sentence of the character, obtaining the absolute value of the difference between the balance factor of the syntactic tree corresponding to the sentence and each of the other sentences of the character, each as a second difference; the larger the second difference, the more abnormal the sentence and the more likely it is to be incomplete; wherein, the method for obtaining the balance factor of the syntactic tree is well known in the art and will not be described in detail; further obtaining the absolute value of the difference between the number of leaf nodes in the syntactic tree corresponding to the sentence and each of the other sentences of the character, each as a third difference; the larger the third difference, the greater the likelihood that the sentence is abnormal, indirectly indicating that the sentence is more likely to be incomplete; therefore, in this embodiment, the cumulative value of the second difference is added to the cumulative value of the third difference and the result of normalization is used as the degree of component missing of the sentence. It should be noted that in this embodiment, the cumulative value of the second difference is normalized by the norm normalization function. It should be noted that among all the sentences uttered by the character, sentences with missing components must account for a small proportion.

[0082] At this point, the degree of missing components in each sentence of each character is obtained.

[0083] Step S202: When the component missing degree is greater than a preset component missing degree threshold, the corresponding sentence is a component missing sentence of the character; wherein a sentence is a complete sentence.

[0084] The greater the degree of component omission, the more likely the corresponding sentence is to contain a component. Therefore, this embodiment sets the preset component omission threshold at 0.7. Implementers can adjust the preset component omission threshold based on actual circumstances, and this is not limited here. When the component omission degree exceeds the preset component omission threshold, the corresponding sentence is considered a component-omission sentence for that character; a sentence is considered a complete sentence.

[0085] At this point, the components of each character are accurately obtained.

[0086] Step S203: For any component constituting a sentence and any component-missing sentence, obtain the possibility that the component-missing sentence lacks the component based on the lack of the component and the proportion of sentences containing the component in all sentences of the character.

[0087] In order to determine the missing component of each component-deficient sentence for the character, and then for any component that constitutes a sentence and any component-deficient sentence for the character, the more sentences containing the component in the sentences corresponding to the character, the more likely the component is to be present in every sentence of the character. When the component is missing in the component-deficient sentence, the more likely the component needs to be supplemented in the component-deficient sentence. Therefore, this embodiment obtains the probability that the component-deficient sentence lacks the component based on the situation that the component is missing in the component-deficient sentence and the proportion of sentences containing the component in all sentences of the character. The greater the probability, the more likely the component-deficient sentence is to lack the content corresponding to the component.

[0088] Preferably, in one possible implementation of this embodiment, the method for obtaining the degree of possibility is: negatively correlate and normalize the number of times the component appears in the sentence where the component is missing, and use this as the reference degree of absence of the component in the sentence where the component is missing; wherein the number of times the component appears in the sentence where the component is missing is always 0 or 1, and therefore, the greater the reference degree of absence, the less likely the component is to exist in the sentence where the component is missing. It should be noted that in this embodiment, exp(-X) is negatively correlated and normalized with the number of times the component appears in the sentence where the component is missing, wherein exp is an exponential function with a natural constant as the base, and X represents the number of times the component appears in the sentence where the component is missing. In order to analyze whether the content of the component needs to be supplemented when the component is missing in the sentence where the component is missing, the ratio of the number of sentences containing the component in all sentences of the character to the number of all sentences of the character is obtained as the reference weight of the component in the sentence corresponding to the character; wherein, the greater the reference weight, the more important the component is in the sentence of the character, and if the component is missing in the sentence where the component is missing, the more the content of the component needs to be supplemented. In order to accurately obtain the possibility that the content of the component-deficient sentence lacks the component and needs to be supplemented, the product of the reference weight and the reference lack degree is used as the possibility degree that the component-deficient sentence lacks the component.

[0089] Step S204: When the probability is greater than a preset probability threshold, the content corresponding to the component in the previous adjacent sentence of the sentence where the component is missing is used as the content of the component in the sentence where the component is missing, and is supplemented and displayed in brackets in the subtitles.

[0090] It is known that the greater the degree of possibility, the more the component in the sentence where the component is missing needs to be supplemented, and thus this embodiment sets the preset degree of possibility threshold to 0.6. The implementer can set the size of the preset degree of possibility threshold according to the actual situation, and it is not limited here. When the degree of possibility is greater than the preset degree of possibility threshold, the component is the missing component of the sentence where the component is missing and the content of the component in the sentence where the component is missing needs to be supplemented; it is known that the conversations between the characters on the timeline are closely connected, and thus this embodiment uses the content corresponding to the component in the adjacent sentence before the sentence where the component is missing as the content of the component in the sentence where the component is missing, and displays it in brackets in the subtitles. For example, the subtitles are displayed as "I also (like this show)". It should be noted that if the number of adjacent sentences before the sentence where the component is missing is not unique, the content corresponding to the component in the sentence with the smallest degree of component missing will be used as the content corresponding to the component in the sentence where the component is missing.

[0091] At this point, the components of each character are accurately obtained and the content that needs to be supplemented in the missing components in the missing sentence is missing.

[0092] Step S205: For any sentence, the ratio of the corresponding duration of the sentence to the number of words contained in the sentence is used as the reference time interval between any two adjacent words in the sentence; when the time interval between any two adjacent words in the sentence is greater than a preset multiple of the reference time interval, the sentence is displayed separately in the subtitles.

[0093] After obtaining all complete sentences for each character, the subtitles need to be adjusted based on the speaker's pauses. For any sentence, the ratio of the sentence's duration to the number of words in the sentence serves as the reference time interval between any two adjacent words in the sentence. When a sentence requires suspense, there will be pauses during the narration. Therefore, when the time interval between any two adjacent words in the sentence is greater than a preset multiple of the reference time interval, for the sake of subtitle effectiveness, the subtitles for that sentence should be filled in separately. Specifically, the half of the sentence before the pause is first filled in the video at the corresponding time. For completeness, the second half of the sentence should be filled in with the entire sentence, not just the content of the second half after the pause. For example, taking "My favorite program is XXX," the half of the sentence before the pause would appear in the subtitles as "My favorite program is," and the second half would appear as "My favorite program is XXX." In this embodiment, the preset multiple is set to 2. Implementers can adjust the preset multiple based on actual circumstances, and this is not limited here.

[0094] At this point, the subtitles corresponding to each sentence of each character are accurately obtained, and then all subtitle sentences are filled into the preset position of the corresponding video through the editing software. Usually, the preset position is set to the center position below the video, which is not limited here, and the subtitle production is accurately completed, which effectively improves the accuracy and completeness of the subtitle display, and finally accurately obtains the finished TV program.

[0095] In summary, this embodiment obtains an audio segment; based on the voiceprint embedding vector corresponding to the human voice at each moment in the audio segment, the audio segment is divided into a single-sound sub-segment and a multi-sound sub-segment; based on the intensity of the human voice at each moment in the single-sound sub-segment corresponding to each character in the multi-sound sub-segment, the enhanced audio of the human voice of each character in the multi-sound sub-segment is obtained, and then each sentence corresponding to each character is obtained; based on the structure and narration of each sentence corresponding to each character, the display subtitles of the TV program are obtained. The present invention enhances the audio of each character in the multi-sound sub-segment, improves the accuracy of identifying the speech content of each character, and then accurately obtains each sentence corresponding to each character, and then analyzes each sentence, effectively improving the accuracy and completeness of the subtitle display.

[0096] Example 2:

[0097] The present invention also proposes a radio and television program production system based on artificial intelligence, see Figure 3 , which shows a structural diagram of a radio and television program production system based on artificial intelligence provided by an embodiment of the present invention. The system includes: an audio segment acquisition module 10, an audio division module 20, an enhanced audio acquisition module 30 and a display subtitle acquisition module 40.

[0098] The audio segment acquisition module 10 is used to acquire each audio segment in the audio file.

[0099] The audio segmentation module 20 is configured to divide the audio segment into single-sound sub-segments and multi-sound sub-segments based on the voiceprint embedding vector corresponding to the human voice at each moment in each audio segment.

[0100] The enhanced audio acquisition module 30 is used to obtain the reference intensity of the human voice of each character at each moment in each multi-sound subsegment at each frequency based on the intensity of the human voice at each frequency in the single sound subsegment corresponding to each character in each multi-sound subsegment, and then obtain the enhanced audio of the human voice of each character in each multi-sound subsegment.

[0101] The display subtitle acquisition module 40 is used to identify each single sound sub-segment and the enhanced audio to obtain each sentence corresponding to each character; and obtain the display subtitles of the TV program based on the structure and narration of each sentence corresponding to each character.

[0102] It should be noted that the system provided in the above embodiment is merely an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the above embodiment provides an artificial intelligence-based radio and television program production system and an artificial intelligence-based radio and television program production method embodiment, which are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0103] Example 3:

[0104] The present invention also proposes an artificial intelligence-based radio and television program production device, comprising a memory and a processor, wherein the memory stores executable program code, and the processor is configured to call and execute the executable program code to perform an artificial intelligence-based radio and television program production method provided in an embodiment of the present application. The device can be a chip, component, or module, wherein the chip may include a connected processor and memory; wherein the memory is configured to store instructions, and when the processor calls and executes the instructions, the chip can execute the artificial intelligence-based radio and television program production method provided in the above embodiment.

[0105] In addition, the present application also protects a computer device, see Figure 4 The computer device includes a memory 401, a processor 402, and a computer program 403 stored in the memory 401 and running on the processor 402, wherein when the processor 402 executes the computer program 403, the computer device can execute any one of the artificial intelligence-based radio and television program production methods introduced above.

[0106] Example 4:

[0107] This embodiment also provides a computer-readable storage medium, which stores computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned related method steps to implement an artificial intelligence-based radio and television program production method provided in the above embodiment.

[0108] Example 5:

[0109] This embodiment also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the artificial intelligence-based radio and television program production method provided in the above embodiment.

[0110] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0111] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A method for producing radio and television programs based on artificial intelligence, characterized in that: The method comprises the following steps: Get each audio segment in the audio file; Based on the voiceprint embedding vector corresponding to the human voice at each moment in each audio segment, the audio segment is divided into single-sound sub-segments and multi-sound sub-segments; Based on the intensities of the human voice at each frequency at each moment in the single sound subsegment corresponding to each character in each multi-sound subsegment, obtaining reference intensities of the human voice at each frequency at each moment in each multi-sound subsegment, thereby obtaining enhanced audio of the human voice of each character in each multi-sound subsegment; Identifying each single sound sub-segment and the enhanced audio to obtain each sentence corresponding to each character; obtaining display subtitles of the television program based on the structure and narration of each sentence corresponding to each character; The method for obtaining the display subtitles of a television program is as follows: For any character in the audio file, the degree of component deficiency of each sentence of the character is obtained based on the difference in balance factor and the difference in the number of leaf nodes of the syntax tree corresponding to each sentence of the character and each other sentence of the character; When the degree of component deficiency is greater than a preset component deficiency threshold, the corresponding sentence is a component deficiency sentence of the character; wherein a sentence is a complete sentence; For any component constituting a sentence and any component-missing sentence, the probability of the component-missing sentence missing the component is obtained based on the lack of the component in the sentence and the proportion of sentences containing the component in all sentences of the character; When the probability is greater than a preset probability threshold, the content corresponding to the component in the adjacent sentence before the sentence where the component is missing is used as the content of the component in the sentence where the component is missing, and is displayed in brackets in the subtitles; For any sentence, the ratio of the corresponding duration of the sentence to the number of words in the sentence is used as the reference time interval between any two adjacent words in the sentence; When the time interval between any two adjacent words in the sentence is greater than a preset multiple of the reference time interval, the sentence is displayed separately in the subtitles.

2. The method for producing a radio and television program based on artificial intelligence according to claim 1, wherein: The method for obtaining the single sound sub-segment and the multi-sound sub-segment is: Get the voiceprint embedding vector corresponding to each person appearing in the audio file, and use them as the reference voiceprint embedding vector; For any moment in any audio segment, obtain the cosine similarity between the voiceprint embedding vector corresponding to the human voice at that moment and each reference voiceprint embedding vector, and use it as the degree of character matching corresponding to the human voice at that moment; When the character matching degree is greater than the character matching degree threshold, the character corresponding to the reference voiceprint embedding vector is used as the matching character of the human voice at that moment; When the number of matching characters is greater than 1, this moment is a multi-voice overlapping moment; When the number of matching characters is equal to 1, the moment is a single voice moment; The local audio segment formed by the continuous overlapping moments of multiple voices in the audio segment is used as a multi-voice sub-segment; A local audio segment formed by continuous single voice moments in the audio segment is used as a single voice sub-segment.

3. The method for producing a radio and television program based on artificial intelligence according to claim 2, wherein: The method for obtaining the character matching degree threshold is as follows: Obtain the cosine similarity of any two reference voiceprint embedding vectors as the first similarity value; The largest first similarity value is used as the character matching degree threshold.

4. The method for producing a radio and television program based on artificial intelligence according to claim 1, wherein: The method for obtaining the reference intensity is: For the a-th polyphonic sub-segment, all local polyphonic sub-segments corresponding to the continuous human voices of the same character in the a-th polyphonic sub-segment are used as reference sound segments; wherein the character corresponding to each moment in the same reference sound segment is the same; For the t-th moment in the i-th reference sound segment of the a-th multi-sound sub-segment, the intensity of the human voice at the k-th frequency at the t-th moment is used as the target intensity; For the p-th character in the i-th reference sound segment, the intensity of the human voice at the k-th frequency at each moment in all the single sound sub-segments corresponding to the p-th character is used as the reference intensity; taking the difference between the target intensity and each of the reference intensities as a first difference; Negatively correlate and normalize the mean of the first differences, and use this as the intensity adjustment weight for the voice of the p-th character at the k-th frequency at the t-th moment. The negative correlation and normalization are achieved using exp(-x), where exp is an exponential function with a natural constant as its base, and x represents the mean of the first differences. The product of the target intensity and the intensity adjustment weight is used as the reference intensity corresponding to the voice of the p-th character at the k-th frequency at the t-th moment in the i-th reference sound segment of the a-th multi-sound sub-segment.

5. The method for producing a radio and television program based on artificial intelligence according to claim 1, wherein: The method for obtaining the degree of deficiency of the component is as follows: For any sentence of the character, obtain the balance factor difference between the syntactic tree corresponding to the sentence and each other sentence of the character, and use them as the second difference; Obtain the difference in the number of leaf nodes between the syntactic tree corresponding to the sentence and each other sentence of the character, and use them as the third difference; The cumulative value of the second difference and the cumulative value of the third difference are added together and normalized, and the result is used as the degree of component deficiency of the sentence.

6. The method for producing a radio and television program based on artificial intelligence according to claim 1, wherein: The method for obtaining the degree of possibility is: The result of negative correlation and normalization of the number of times the component appears in the sentence lacking the component is used as the reference lack degree of the component in the sentence lacking the component; The ratio of the number of sentences containing the component in all sentences of the character to the total number of sentences of the character is used as the reference weight of the component in the corresponding sentences of the character; The product of the reference weight and the reference lack degree is used as the probability degree of the lack of the component in the component-lacking sentence.

7. The method for producing a radio and television program based on artificial intelligence according to claim 1, wherein: The method for obtaining the audio segment is: The audio in the audio file with continuous moments of sound greater than the preset decibel level is divided into audio segments.

8. The method for producing a radio and television program based on artificial intelligence according to claim 1, wherein: The artificial intelligence-based radio and television program production method further includes: Each moment in each audio segment is consistent with the moment corresponding to each video frame.

9. A radio and television program production system based on artificial intelligence, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When executing the computer program, the processor implements the steps of the artificial intelligence-based radio and television program production method described in any one of claims 1 to 8.

Images