Sound effect video generation method and device, equipment and storage medium

Abstract

The invention relates to a sound effect video generation method and apparatus, a device and a storage medium. The method comprises the steps of obtaining a target video to which a sound effect is to be added and a target script corresponding to the target video; according to a preset script segmentation rule, segmenting the target script to obtain at least one script slice; according to each script slice and a trained sentiment analysis model, determining a sentiment label of each script slice; according to the emotion label of each script slice and a trained sound effect generation model, determining a sound effect fragment corresponding to each script slice; and adding the sound effect fragment corresponding to each script slice into the target video to obtain the target video with the added sound effect. The sound effect obtained by the method is more fit with the target video, and the playing effect of the target video with the sound effect is better.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for generating audio and video effects. Background Art

[0002] With the development of technology, sound effects play an important role in conveying emotions and creating atmosphere in current film and television production and viewing experience. Therefore, it becomes particularly important to add sound effects to videos.

[0003] In the prior art, when a technician needs to add sound effects to a video clip, the technician can find the sound effect to be used in the sound effects library based on his or her understanding of the video clip, add the sound effect to the audio part of the video clip, and obtain a video with the sound effect added.

[0004] Since the sound effect is found in the sound effect library and is not obtained based on the video clip, the sound effect obtained in this way may not be suitable for the video clip, resulting in poor playback effect of the video with the sound effect added. Summary of the invention

[0005] The present application provides a method, device, equipment and storage medium for generating audio and video effects. The audio effects obtained by the method are more closely aligned with the target video, and the target video with the audio effects added has a better playback effect.

[0006] In a first aspect, the present application provides a method for generating an audio effect video, the method comprising:

[0007] Obtain a target video to which sound effects are to be added and a target script corresponding to the target video;

[0008] According to a preset script segmentation rule, the target script is segmented to obtain at least one script slice;

[0009] Determine the sentiment label of each script slice based on each script slice and the trained sentiment analysis model;

[0010] Determine the sound effect segment corresponding to each script slice based on the emotional label of each script slice and the trained sound effect generation model;

[0011] The sound effect clip corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

[0012] Optionally, adding the sound effect clip corresponding to each script slice to the target video to obtain the target video with the sound effect added includes:

[0013] Obtain the time period corresponding to each script slice in the target video;

[0014] For each script slice, according to the time period corresponding to the script slice in the target video, a corresponding sound effect segment is added to the target video;

[0015] When the sound effect clips corresponding to all script slices are added to the target video, the target video with the sound effects added is obtained.

[0016] Optionally, the script segmentation rule is:

[0017] Based on the trained element labeling model, labeling the script elements in the target script to obtain the labeling positions in the target script;

[0018] Determining the script elements in the target script according to the marked positions in the target script;

[0019] The target script is cut according to the script elements in the target script to obtain script slices in the target script.

[0020] Optionally, the method comprises:

[0021] For each script slice, according to the marked position in the script slice, the script element included in the script slice is determined, the content of the script element is analyzed to obtain an analysis result, and based on the analysis result, the sound effect corresponding to the script element is adjusted to obtain a sound effect adjustment segment corresponding to the script slice;

[0022] The step of adding the sound effect clip corresponding to each script slice to the target video to obtain the target video with the sound effect added includes:

[0023] The sound effect adjustment segment corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

[0024] Optionally, after obtaining the target video to which the sound effect has been added, the method further includes:

[0025] When playing a video, obtain the current playback position, user behavior data, and facial data;

[0026] According to the current playback position, determining a current emotion label corresponding to the current playback position;

[0027] Determining a target emotion tag of the user according to the behavior data and the facial data;

[0028] According to the current emotion tag and the target emotion tag, it is determined whether to regenerate the sound effect segment at the current playback position.

[0029] Optionally, determining whether to regenerate the sound effect segment at the current playback position according to the current emotion tag and the target emotion tag includes:

[0030] Determine whether the current emotion label is consistent with the target emotion label;

[0031] When there is inconsistency, it is determined to regenerate the sound effect segment at the current playback position according to the target emotion tag.

[0032] Optionally, determining the sentiment label of each script slice according to each script slice and the trained sentiment analysis model includes:

[0033] For each script slice, a word segmenter is used to perform word segmentation on the script slice to obtain a word segmentation result, and the word segmentation result is input into a trained sentiment analysis model to obtain a sentiment label for the script slice.

[0034] In a second aspect, the present application provides a device for generating audio and video effects, the device comprising:

[0035] An acquisition unit, used for acquiring a target video to be added with sound effects and a target script corresponding to the target video;

[0036] A segmentation unit, configured to segment the target script according to a preset script segmentation rule to obtain at least one script slice;

[0037] A first determination unit is used to determine the sentiment label of each script slice according to each script slice and the trained sentiment analysis model;

[0038] A second determination unit is used to determine the sound effect segment corresponding to each script slice according to the emotion label of each script slice and the trained sound effect generation model;

[0039] The adding unit is used to add the sound effect clip corresponding to each script slice to the target video to obtain the target video with the sound effect added.

[0040] Optionally, add units for:

[0041] Obtain the time period corresponding to each script slice in the target video;

[0042] For each script slice, according to the time period corresponding to the script slice in the target video, a corresponding sound effect segment is added to the target video;

[0043] When the sound effect clips corresponding to all script slices are added to the target video, the target video with the sound effects added is obtained.

[0044] Optionally, the device further includes a rule determination unit, wherein the rule determination unit is configured to:

[0045] Based on the trained element labeling model, labeling the script elements in the target script to obtain the labeling positions in the target script;

[0046] Determining the script elements in the target script according to the marked positions in the target script;

[0047] The target script is cut according to the script elements in the target script to obtain script slices in the target script.

[0048] Optionally, the device comprises an adjustment unit, wherein the adjustment unit is used to:

[0049] For each script slice, according to the marked position in the script slice, the script element included in the script slice is determined, the content of the script element is analyzed to obtain an analysis result, and based on the analysis result, the sound effect corresponding to the script element is adjusted to obtain a sound effect adjustment segment corresponding to the script slice;

[0050] The adding unit is used for:

[0051] The sound effect adjustment segment corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

[0052] Optionally, the device further comprises a regeneration unit, wherein the regeneration unit is configured to:

[0053] When playing a video, obtain the current playback position, user behavior data, and facial data;

[0054] According to the current playback position, determining a current emotion label corresponding to the current playback position;

[0055] Determining a target emotion tag of the user according to the behavior data and the facial data;

[0056] According to the current emotion tag and the target emotion tag, it is determined whether to regenerate the sound effect segment at the current playback position.

[0057] Optionally, the regeneration unit is used to:

[0058] Determine whether the current emotion label is consistent with the target emotion label;

[0059] When there is inconsistency, it is determined to regenerate the sound effect segment at the current playback position according to the target emotion tag.

[0060] Optionally, the first determining unit is used to:

[0061] For each script slice, a word segmenter is used to perform word segmentation on the script slice to obtain a word segmentation result, and the word segmentation result is input into a trained sentiment analysis model to obtain a sentiment label for the script slice.

[0062] In a third aspect, the present application provides an audio and video generation device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to:

[0063] Obtain a target video to which sound effects are to be added and a target script corresponding to the target video;

[0064] According to a preset script segmentation rule, the target script is segmented to obtain at least one script slice;

[0065] Determine the sentiment label of each script slice based on each script slice and the trained sentiment analysis model;

[0066] Determine the sound effect segment corresponding to each script slice based on the emotional label of each script slice and the trained sound effect generation model;

[0067] The sound effect clip corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

[0068] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the above-mentioned audio and video generation method.

[0069] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: In the embodiment of the present application, the target video to be added with sound effects and its corresponding target script are obtained, and the target script is segmented according to the script segmentation rules to obtain at least one script slice, and the emotion label of each script slice is obtained according to each script slice and the trained emotion analysis model, and the sound effect segment corresponding to each script slice is obtained according to the emotion label of each script slice and the trained sound effect generation model, and the sound effect segment corresponding to each script slice is added to the target video to obtain the target video with the sound effects added. In the above process, the sound effects added to the target video are generated according to the emotion label of the script slice, and the script reflects the content of the video. In essence, the sound effects added to the target video are generated according to the emotion of the video content, so that the sound effects obtained are more in line with the target video, and the target video with the sound effects added has a better playback effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0073] Figure 1 A flowchart of a method for generating audio and video effects provided in an embodiment of the present application;

[0074] Figure 2 A flowchart of a script segmentation method provided in an embodiment of the present application;

[0075] Figure 3 A flowchart of a sound effect adjustment method provided in an embodiment of the present application;

[0076] Figure 4 A flowchart of another sound effect adjustment method provided in an embodiment of the present application;

[0077] Figure 5 A flowchart of another method for adding sound effects provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of a flow chart of a device for generating audio and video effects provided in an embodiment of the present application;

[0079] Figure 7 A schematic diagram of an audio and video generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0081] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0082] With the development of science and technology, in the current film and television production and viewing experience, sound effects play an important role in conveying emotions and creating atmosphere. Therefore, adding sound effects to videos has become particularly important. In the prior art, when a technician needs to add sound effects to a certain video clip, the technician can find the sound effect to be used in the sound effect library based on his or her understanding of the video clip, add the sound effect to the audio part of the video clip, and obtain a video with the sound effect added. Since the sound effect is found in the sound effect library and is not obtained based on the video clip, the sound effect obtained in this way may not be suitable for the video clip, resulting in poor playback effect of the video with the sound effect added.

[0083] In an embodiment of the present application, a target video to be added with sound effects and its corresponding target script are obtained, and the target script is segmented according to the script segmentation rules to obtain at least one script slice, and the emotion label of each script slice is obtained according to each script slice and the trained emotion analysis model, and the emotion label of each script slice is input into the trained sound effect generation model to obtain the sound effect segment corresponding to each script slice, and the target video to which the sound effects have been added is determined according to the target video and the sound effect segment corresponding to each script slice. In the above process, the sound effects added to the target video are generated according to the emotion labels of the script slices, and the script reflects the content of the video. In essence, the sound effects added to the target video are generated according to the emotions of the video content, so that the sound effects obtained are more closely matched to the target video, and the target video with the sound effects added has a better playback effect.

[0084] In summary, the embodiment of the present application provides a method for generating a sound effect video, wherein the sound effect obtained by the method is more closely matched with the target video, and the target video with the sound effect added has a better playback effect, such as Figure 1 As shown, the specific steps include:

[0085] Step 101, obtain the target video to which the sound effect is to be added and the target script corresponding to the target video.

[0086] Among them, the target video is the video to be added with sound effects, which is shot based on the target script.

[0087] In this step, the execution device may be a server storing the target video and the target script, or a terminal storing the target video and the target script. In the present application, the execution device may automatically detect the stored video, determine whether the video needs to be added with sound effects, and trigger the method described in the present application when the video needs to be added with sound effects. The method described in the present application may also be triggered when an instruction to add sound effects to the video is received. The method described in the present application may also be automatically triggered in other scenarios, which will not be described one by one here.

[0088] In this step, when the execution device needs to add sound effects to the target video, the execution device needs to first obtain the target video and the target script corresponding to the target video.

[0089] Step 102, segment the target script according to a preset script segmentation rule to obtain at least one script slice.

[0090] Among them, when the script slice is one, the script slice is the entire content of the target script, that is, the script slice is the target script. When the script slice is multiple, the script slice is part of the content in the target script.

[0091] In this step, the preset script segmentation rules can be set for the technician and stored in the execution device. When the execution device needs to segment the target script, the execution device can obtain the pre-stored script segmentation rules and segment the target script according to the script segmentation rules to obtain at least one script slice.

[0092] Step 103, determining the sentiment label of each script slice according to each script slice and the trained sentiment analysis model.

[0093] Among them, the emotion label indicates the emotion of the script slice content. For example, when the emotion label is relaxed, the emotion label indicates that the content of the script slice is relaxed. The emotion analysis model is a natural language model, which is used to analyze the emotion of the script slice and obtain the emotion label of the script slice. The training method of the emotion analysis model is similar to the training method of the prior art machine learning model, and will not be repeated here.

[0094] In this step, for each script slice, the execution device can directly input the script slice into the trained sentiment analysis model to obtain the sentiment label of the script slice. Further, in order to output more accurately, the execution device can also pre-process each script slice in advance so that the sentiment analysis module can more accurately analyze each script slice. After pre-processing the script slice, the processed script slice is input into the trained sentiment analysis model to obtain the sentiment label of the script slice. The pre-processing can be word segmentation processing of the script slice.

[0095] Step 104, determining the sound effect segment corresponding to each script slice according to the emotion label of each script slice and the trained sound effect generation model.

[0096] The sound effect generation model is a machine learning model used to generate sound effect segments based on emotion tags. The input of the sound effect generation model is the emotion tag, and the output is the sound effect segment. The training method of the sound effect generation model is similar to the training method in the prior art, and will not be described here one by one.

[0097] In one embodiment, the execution device can input the emotion label of each script slice into a trained sound effect generation model to obtain the sound effect segment corresponding to each script slice.

[0098] It should be noted that the duration of the sound effect segment generated based on the above method may not match the duration of the video segment corresponding to the script slice in the target video. Therefore, in order to solve the above problem, the duration of the sound effect segment can be adjusted after it is obtained.

[0099] In another implementation, since the script slice corresponds to the video segment of the target video, in order to ensure that the duration of the sound effect segment meets the requirements, the sound effect segment can be generated based on the required duration. Specifically, for each script slice, the execution device can determine the video segment corresponding to the script slice in the target video, input the duration of the video segment and the emotional label of the script slice into the trained sound effect generation model, and obtain the sound effect segment corresponding to the script slice.

[0100] Step 105, adding the sound effect clip corresponding to each script slice to the target video to obtain the target video with the sound effect added.

[0101] In this step, for each script slice, the execution device adds the sound effect clip corresponding to the script slice to the position corresponding to the target video according to the position of the script slice. After the sound effect clip corresponding to each script slice is added to the target video, the target video with the sound effect added is obtained.

[0102] In an embodiment of the present application, a target video to be added with sound effects and its corresponding target script are obtained, and the target script is segmented according to the script segmentation rules to obtain at least one script slice, and the emotion label of each script slice is obtained according to each script slice and the trained emotion analysis model, and the sound effect segment corresponding to each script slice is obtained according to the emotion label of each script slice and the trained sound effect generation model, and the sound effect segment corresponding to each script slice is added to the target video to obtain the target video with the sound effects added. In the above process, the sound effects added to the target video are generated according to the emotion label of the script slice, and the script reflects the content of the video. In essence, the sound effects added to the target video are generated according to the emotion of the video content, so that the sound effects obtained are more closely matched to the target video, and the target video with the sound effects added has a better playback effect.

[0103] In the embodiment of the present application, since the script includes elements such as scenes, character dialogues, locations, time, and plot development, these elements can reflect part of the plot content, and the emotions of the content belonging to the same plot are often the same or similar, therefore, the content belonging to the same plot can be regarded as a script slice. Therefore, the embodiment of the present application provides a script segmentation method, which can segment the target script to obtain multiple script slices, such as Figure 2 As shown, including:

[0104] Step 201 , based on the trained element labeling model, label the script elements in the target script to obtain the labeling positions in the target script.

[0105] The script elements include scenes, character dialogues, locations, time, plot development, etc. The element labeling model is a machine learning model used to label the target script. That is, the input of the element labeling model is the target script, and the output is the target script with the label position.

[0106] Furthermore, the training process of the element labeling model is as follows: first obtain multiple sample scripts, then the technicians can label the sample scripts to obtain labeled sample scripts and use them as benchmark data. Then, randomly select a sample script from the multiple sample scripts, and input the sample script into the element labeling model to obtain predicted data, which is also a labeled sample script. Compare the benchmark data with the predicted data to adjust the element labeling model. Determine whether the adjusted element labeling model meets the preset conditions. If it does not meet the conditions, use other sample scripts to continue to adjust the adjusted element labeling model until the adjusted element labeling model meets the preset conditions to obtain a trained element labeling model.

[0107] In this step, the target script is input into the trained element labeling model to obtain the target script with the marked points, and in the marked target script, the marking position in the target script is obtained.

[0108] Step 202, determining the script elements in the target script according to the marked positions in the target script.

[0109] The marked position is used to determine the script element. For example, the marked position includes a marked starting point and a marked ending point. The marked starting point is used to indicate the beginning of the script element, and the marked ending point is used to indicate the end of the script element. In this way, the script element can be determined by the marked position. Of course, the marked position can also only have the beginning or end of the script element, and the content between the two marked positions is regarded as a script element.

[0110] Step 203, cutting the target script according to the script elements in the target script to obtain script slices in the target script.

[0111] Since this step requires segmenting script elements belonging to the same plot into the same script slice, the execution device can use a pre-trained natural language model to identify the content of each script element and divide script elements with similar content into the same script slice.

[0112] In order to obtain a sound effect that is more consistent with the video, after obtaining the sound effect segment of each script slice, the embodiment of the present application can also adjust the obtained sound effect segment to make the adjusted sound effect segment more consistent with the video, so that the user watching the video has an immersive feeling. Therefore, the embodiment of the present application provides a sound effect adjustment method, the method is as follows Figure 3 As shown, specifically including:

[0113] Step 301, for each script slice, determine the script elements included in the script slice according to the marked position in the script slice.

[0114] During the sound effect adjustment stage, the script slice also carries the marked position, so that the execution device can determine the script elements included in the script slice based on the marked position in the script slice.

[0115] Step 302, analyze the content of the script element to obtain the analysis result of the script element.

[0116] In this step, the pre-trained natural language model is used to analyze the content of the script elements and determine the analysis results.

[0117] For example, when the script element is a scene, the analysis result is a description of the scene; when the script element is a character dialogue, the analysis result is the emotion and tone in the dialogue.

[0118] Step 303, based on the analysis result of the script element, adjust the sound effect corresponding to the script element.

[0119] In this step, according to the analysis results, the sound effects corresponding to the script elements are slightly adjusted, such as introducing new sound effects and modifying the original sound effects, thereby obtaining the sound effect adjustment segment corresponding to the script element. After adjusting the sound effects corresponding to all the script elements included in the script slice, the sound effect adjustment segment corresponding to the script slice is obtained.

[0120] For example, when the script element is a scene, select background sound effects or environmental sound effects that match the description of the scene. For example, when the script element is a scene, and the analysis result shows that the scene is inside a spaceship, you can add sound effects that simulate the space environment, such as low mechanical sounds or electric current sounds. When the script element is a character dialogue, and the analysis result shows that the dialogue shows surprise or tension, you can add enhanced sound effects, such as rapid music or alarm sounds.

[0121] Step 304, when the sound effect adjustment segment corresponding to each script slice is obtained, the sound effect adjustment segment corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

[0122] In this step, after obtaining the sound effect adjustment segment corresponding to each script slice, the sound effect adjustment segment corresponding to each script slice can be added to the target video to obtain the target video with the sound effect added.

[0123] In the embodiment of the present application, the sound effects in the target video are adjusted according to the reaction of the user watching the video, so that the adjusted sound effects are more closely matched with the video, and the video playback effect is further improved. Therefore, the embodiment of the present application provides a sound effect adjustment method, which can adjust the sound effects in the video. Figure 4 As shown, specifically including:

[0124] Step 401, when playing a video, obtain the current playing position, user's behavior data and facial data.

[0125] The execution device may be a terminal for playing the video, such as a mobile phone, a computer, a notebook, etc. When playing the video, the execution device may periodically collect the user's behavior data and facial data. The collection method may be to capture an image of the user watching the video, and then collect the behavior data and facial data from the image.

[0126] In this step, when playing the video, the execution device can obtain the current playback position, the user's behavior data and facial data at intervals. Alternatively, the technician can pre-stamp the position to be detected, and when playing the video, the execution device detects the timestamp in real time, and when the timestamp is detected, the current playback position, the user's behavior data and facial data are obtained.

[0127] Furthermore, since the user behavior data and facial data at a certain point in time cannot accurately express the user's emotions, the execution device can periodically collect the user's behavior data and facial data and store them. When these data need to be obtained, they can be directly obtained from the storage space.

[0128] Step 402: Determine the current emotion tag corresponding to the current playback position according to the current playback position.

[0129] In one embodiment, before executing this step, the execution device can determine the time period corresponding to each script slice based on the content corresponding to each script slice in the target video, so that the emotion label corresponding to each time period can be determined based on the time period corresponding to each script slice and the emotion label corresponding to each script slice. Afterwards, the execution device can store the emotion label corresponding to each time period. When it is necessary to execute this step, the execution device can determine the time period to which the current playback position belongs based on the current playback position, and determine the emotion label corresponding to the time period as the label corresponding to the current position.

[0130] In another embodiment, after the execution device obtains the current playback position, it determines the sound effect segment corresponding to the current playback position, and based on the sound effect segment, determines the emotion label of the script slice corresponding to the sound effect segment, and determines the emotion label as the current emotion label corresponding to the current playback position.

[0131] Step 403, determining the target emotion tag of the user based on the behavior data and the facial data.

[0132] In one embodiment, the execution device stores the user behaviors and facial expressions corresponding to each emotion tag. When step 403 needs to be executed, the user behaviors are identified based on the behavior data, and the user's facial expressions are identified based on the facial data. When the user behaviors and facial expressions of the user are obtained, the execution device can determine the user's current emotion tag based on the user behaviors and facial expressions corresponding to each emotion tag stored in advance, as well as the identified user behaviors and facial expressions.

[0133] It should be noted that if there are multiple groups of behavioral data and facial data obtained, the emotion tags corresponding to each group can be determined based on these data, and the emotion tag with the largest number of occurrences can be determined as the current emotion tag.

[0134] Step 404, determining whether to regenerate the sound effect segment at the current playback position based on the current emotion tag and the target emotion tag.

[0135] In this step, it is determined whether to regenerate the sound effect segment at the current playback position based on whether the current emotion label is consistent with the target emotion label.

[0136] In the embodiment of the present application, when the current emotion label and the target emotion label are consistent, it means that the currently played sound effect brings an immersive viewing experience to the user, and there is no need to regenerate the currently played sound effect. When the two are inconsistent, it means that the currently played sound effect cannot bring an immersive viewing experience to the user, and the currently played sound effect needs to be regenerated. Therefore, the embodiment of the present application provides an emotion label detection method, which is a further limitation of step 404, and the specific steps are:

[0137] Determine whether the current emotion tag is consistent with the target emotion tag. If they are inconsistent, determine to regenerate the sound effect segment at the current playback position based on the target emotion tag. In this step, the current emotion tag is compared with the target emotion tag to determine whether the current emotion tag is consistent with the target emotion tag. If they are inconsistent, execute step 103 based on the target emotion tag.

[0138] In the embodiment of the present application, a method for adding sound effects is also provided, which is a further limitation of step 104. The specific steps are as follows: Figure 5 As shown, including:

[0139] Step 501, obtaining the time period corresponding to each script slice in the target video.

[0140] In this step, for each script slice, the starting content and the ending content of the script slice can be aligned with the content in the target video respectively to obtain the starting time point and the ending time point. Then, the time period corresponding to the script slice in the target video can be determined based on the starting time point and the ending time point.

[0141] In the above process, the technician aligns the script slice content with the target video, and can also use alignment technology to align the two.

[0142] Step 502, for each script slice, add the corresponding sound effect clip to the target video according to the time period corresponding to the script slice in the target video.

[0143] In this step, according to the time period corresponding to each script slice in the target video, the sound effect clip corresponding to each script slice is added to the position corresponding to the time period, so as to add the sound effect clip corresponding to each script slice to the target video.

[0144] Step 503, when the sound effect clips corresponding to all script slices are added to the target video, the target video with the sound effects added is obtained.

[0145] In this step, the execution device determines whether the sound effect clips corresponding to all the script slices are obtained. If yes, the target video with the sound effect added is obtained.

[0146] In the embodiment of the present application, a script slicing processing method is also provided, which is a further limitation of step 104, and the specific steps include:

[0147] For each script slice, a word segmenter is used to segment the script slice to obtain a word segmentation result, which is then input into a trained sentiment analysis model to obtain a sentiment label for the script slice.

[0148] The word segmenter is used to divide a sentence into multiple words. The word segmenter used in this step may be a machine learning model or may not be a machine learning model, which is not limited here, as long as it can achieve word segmentation processing. The word segmentation result includes at least one word.

[0149] In this step, in order to make the sentiment analysis model better identify the script slice, a word segmenter can be used to divide the content in the script slice into word segmentation results including at least one word. After that, the word segmentation result is input into the trained sentiment analysis model to obtain the sentiment label of the script slice.

[0150] In the embodiment of the present application, a device for generating audio and video effects is also provided, which corresponds to the method embodiment, as shown in the following example: Figure 6 As shown, including:

[0151] An acquisition unit 601 is used to acquire a target video to be added with sound effects and a target script corresponding to the target video;

[0152] A segmentation unit 602 is used to segment the target script according to a preset script segmentation rule to obtain at least one script slice;

[0153] A first determining unit 603 is used to determine the sentiment label of each script slice according to each script slice and the trained sentiment analysis model;

[0154] The second determining unit 604 is used to determine the sound effect segment corresponding to each script slice according to the emotion label of each script slice and the trained sound effect generation model;

[0155] The adding unit 605 is used to add the sound effect segment corresponding to each script slice to the target video to obtain the target video with the sound effect added.

[0156] Optionally, a unit 605 is added for:

[0157] Obtain the time period corresponding to each script slice in the target video;

[0158] For each script slice, according to the time period corresponding to the script slice in the target video, a corresponding sound effect segment is added to the target video;

[0159] When the sound effect clips corresponding to all script slices are added to the target video, the target video with the sound effects added is obtained.

[0160] Optionally, the device further includes a rule determination unit 606, wherein the rule determination unit 606 is configured to:

[0161] Based on the trained element labeling model, labeling the script elements in the target script to obtain the labeling positions in the target script;

[0162] Determining the script elements in the target script according to the marked positions in the target script;

[0163] The target script is cut according to the script elements in the target script to obtain script slices in the target script.

[0164] Optionally, the device includes an adjustment unit 607, and the adjustment unit 607 is used to:

[0165] For each script slice, according to the marked position in the script slice, the script element included in the script slice is determined, the content of the script element is analyzed to obtain an analysis result, and based on the analysis result, the sound effect corresponding to the script element is adjusted to obtain a sound effect adjustment segment corresponding to the script slice;

[0166] The adding unit 605 is used for:

[0167] The sound effect adjustment segment corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

[0168] Optionally, the apparatus further includes a regeneration unit 608, wherein the regeneration unit 608 is configured to:

[0169] When playing a video, obtain the current playback position, user behavior data, and facial data;

[0170] According to the current playback position, determining a current emotion label corresponding to the current playback position;

[0171] Determining a target emotion tag of the user according to the behavior data and the facial data;

[0172] According to the current emotion tag and the target emotion tag, it is determined whether to regenerate the sound effect segment at the current playback position.

[0173] Optionally, the regeneration unit 608 is used to:

[0174] Determine whether the current emotion label is consistent with the target emotion label;

[0175] When there is inconsistency, it is determined to regenerate the sound effect segment at the current playback position according to the target emotion tag.

[0176] Optionally, the first determining unit 603 is configured to:

[0177] For each script slice, a word segmenter is used to perform word segmentation on the script slice to obtain a word segmentation result, and the word segmentation result is input into a trained sentiment analysis model to obtain a sentiment label for the script slice.

[0178] like Figure 7 As shown, the embodiment of the present application provides an audio and video generation device, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0179] Memory 703, used for storing computer programs;

[0180] In one embodiment of the present application, the processor 701 is used to execute the program stored in the memory 703 to implement the audio and video generation method provided by any of the above method embodiments, including:

[0181] Obtain a target video to which sound effects are to be added and a target script corresponding to the target video;

[0182] According to a preset script segmentation rule, the target script is segmented to obtain at least one script slice;

[0183] Determine the sentiment label of each script slice based on each script slice and the trained sentiment analysis model;

[0184] Determine the sound effect segment corresponding to each script slice based on the emotional label of each script slice and the trained sound effect generation model;

[0185] The sound effect clip corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

[0186] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the audio and video generation method provided in any of the aforementioned method embodiments are implemented.

[0187] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0189] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0190] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating audio and video effects, characterized in that: The method comprises: Obtain a target video to which sound effects are to be added and a target script corresponding to the target video; According to a preset script segmentation rule, the target script is segmented to obtain at least one script slice; Determine the sentiment label of each script slice based on each script slice and the trained sentiment analysis model; Determine the sound effect segment corresponding to each script slice based on the emotional label of each script slice and the trained sound effect generation model; The sound effect clip corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

2. The method according to claim 1, characterized in that: The step of adding the sound effect clip corresponding to each script slice to the target video to obtain the target video with the sound effect added includes: Obtain the time period corresponding to each script slice in the target video; For each script slice, according to the time period corresponding to the script slice in the target video, a corresponding sound effect segment is added to the target video; When the sound effect clips corresponding to all script slices are added to the target video, the target video with the sound effects added is obtained.

3. The method according to claim 1, characterized in that: The script segmentation rules are as follows: Based on the trained element labeling model, labeling the script elements in the target script to obtain the labeling positions in the target script; Determining the script elements in the target script according to the marked positions in the target script; The target script is cut according to the script elements in the target script to obtain script slices in the target script.

4. The method according to claim 3, characterized in that: The method comprises: For each script slice, according to the marked position in the script slice, the script element included in the script slice is determined, the content of the script element is analyzed to obtain an analysis result, and based on the analysis result, the sound effect corresponding to the script element is adjusted to obtain a sound effect adjustment segment corresponding to the script slice; The step of adding the sound effect clip corresponding to each script slice to the target video to obtain the target video with the sound effect added includes: The sound effect adjustment segment corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

5. The method according to claim 1, characterized in that: After obtaining the target video with added sound effects, the method further includes: When playing a video, obtain the current playback position, user behavior data, and facial data; According to the current playback position, determining a current emotion label corresponding to the current playback position; Determining a target emotion tag of the user according to the behavior data and the facial data; According to the current emotion tag and the target emotion tag, it is determined whether to regenerate the sound effect segment at the current playback position.

6. The method according to claim 5, characterized in that: The step of determining whether to regenerate the sound effect segment at the current playback position according to the current emotion tag and the target emotion tag includes: Determine whether the current emotion label is consistent with the target emotion label; When there is inconsistency, it is determined to regenerate the sound effect segment at the current playback position according to the target emotion tag.

7. The method according to claim 1, characterized in that: Determining the sentiment label of each script slice according to each script slice and the trained sentiment analysis model includes: For each script slice, a word segmenter is used to perform word segmentation on the script slice to obtain a word segmentation result, and the word segmentation result is input into a trained sentiment analysis model to obtain a sentiment label for the script slice.

8. A device for generating audio and video effects, characterized in that: The device comprises: An acquisition unit, used for acquiring a target video to be added with sound effects and a target script corresponding to the target video; A segmentation unit, configured to segment the target script according to a preset script segmentation rule to obtain at least one script slice; A first determination unit is used to determine the sentiment label of each script slice according to each script slice and the trained sentiment analysis model; A second determination unit is used to determine the sound effect segment corresponding to each script slice according to the emotion label of each script slice and the trained sound effect generation model; The adding unit is used to add the sound effect clip corresponding to each script slice to the target video to obtain the target video with the sound effect added.

9. A device for generating audio and video effects, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to: Obtain a target video to which sound effects are to be added and a target script corresponding to the target video; According to a preset script segmentation rule, the target script is segmented to obtain at least one script slice; Determine the sentiment label of each script slice based on each script slice and the trained sentiment analysis model; Determine the sound effect segment corresponding to each script slice based on the emotional label of each script slice and the trained sound effect generation model; The sound effect clip corresponding to each script slice is added to the target video to obtain the target video with the sound effect added.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating audio and video effects as described in any one of claims 1 to 7 is implemented.

Images