Audio data processing method, device, equipment, and storage medium
The method and apparatus allow users to preview and select sound effects at desired positions in audio data, enhancing user experience by ensuring selected sound effects are preferred, thus addressing the limitations of existing voice changer technologies.
Patent Information
- Application Number
- JP2023513340
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-08-26
- Filing Date
- 2021-08-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-08-26
AI Technical Summary
Existing methods for adding voice changer sound effects to videos do not meet user needs for previewing and selecting desired sound effects.
A method and apparatus for processing audio data that allows users to select a playback position and preview a sound effect, adding it to an audio clip, and receive a command to add a second sound effect with specified length, enabling user selection of preferred sound effects.
Enables users to select and add sound effects they like, simplifying operations and improving user experience by allowing preview and reducing the need for repetitive additions.
Smart Images

Figure 0007581490000001 
Figure 0007581490000002 
Figure 0007581490000003
Abstract
Description
[Technical field]
[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This disclosure claims priority to a Chinese patent application filed with the China National Intellectual Property Office on August 26, 2020, bearing application number 202010873112.3, and entitled "Audio data processing method, device, apparatus, and storage medium," the entire contents of which are incorporated herein by reference.
[0002] [Technical field] TECHNICAL FIELD The embodiments of the present disclosure relate to the field of audio data processing technology, and in particular to an audio data processing method, device, apparatus, and storage medium. [Background technology]
[0003] The video application provided by the related technology has a voice changer function, which allows users to add their favorite voice changer sound effects to videos. However, the method of adding voice changer sound effects to videos in the related technology cannot meet users' needs. Summary of the Invention [Problem to be solved by the invention]
[0004] To solve the above technical problems, or at least a part thereof, embodiments of the present disclosure provide a method, an apparatus, a device, and a storage medium for processing audio data. [Means for solving the problem]
[0005] A first aspect of an embodiment of the present disclosure provides a method for processing audio data, the method including the steps of: acquiring a first playback position in first audio data and a user preview command for a first sound effect; adding the first sound effect to a first audio clip in the first audio data to generate and play sound effect preview data, the first audio clip being an audio clip in the first audio data having a start position at the first playback position; and, if a first addition command for a second sound effect is received from the user, adding a second sound effect to a second audio clip in the first audio data to acquire second audio data, the first addition command including information of a first addition length of the second sound effect in the first audio data, the second audio clip being an audio clip in the first audio data having a start position at the first playback position and having a length of the first addition length.
[0006] A second aspect of an embodiment of the present disclosure provides an apparatus for processing audio data, the apparatus comprising: a first acquisition module for acquiring a first playback position in the first audio data and a listening command for a first sound effect by a user; a first sound effect adding module that adds a first sound effect to a first audio clip in first audio data to generate and play back sound effect preview data, the first audio clip being an audio clip in the first audio data whose start position is a first playback position; a first receiving module for receiving a first addition command for a second sound effect from a user, the first addition command including information of a first addition length of the second sound effect in the first audio data; The present invention includes a second sound effect adding module that adds a second sound effect to a second audio clip in the first audio data to obtain second audio data, the second audio clip being an audio clip whose start position is the first playback position and whose length is the first addition length.
[0007] A third aspect of an embodiment of the present disclosure provides a terminal device, the terminal device including a memory and a processor, the memory storing a computer program, and the computer program being capable of realizing the method of the first aspect above when executed by the processor.
[0008] A fourth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored therein, the computer program being capable of implementing the method of the first aspect above when executed by a processor. Effect of the Invention
[0009] Compared with the prior art, the technical solutions provided by the embodiments of the present disclosure have the following advantages: In an embodiment of the present disclosure, a first playback position in a first audio data and a preview command for a first sound effect from a user are obtained, and the first sound effect is added to a first audio clip in the first audio data to obtain sound effect preview data for preview. Then, when the sound effect preview data is played and a first addition command for a second sound effect from a user is received, the second sound effect is added to a second audio clip in the first audio data, the second sound effect being started at the first playback position, based on information on the first addition length included in the first addition command, to obtain second audio data. According to the solution of the embodiment of the present disclosure, a user can select an arbitrary position in audio data to preview a sound effect, and add a sound effect that the user likes to an audio clip of the audio data based on the preview result, so that compared with the case where the sound effect addition effect cannot be previewed, the user can select a sound effect that the user likes through preview and add the sound effect that the user likes to the audio data, and thus the sound effect that the user finally selects and adds to the audio data can be guaranteed to be the sound effect that the user likes, and the user does not need to repeatedly add the added sound effect again due to dissatisfaction with the added sound effect, simplifying the user's operation and improving the user experience. In addition, according to the solution of the embodiment of the present disclosure, a user can add a specific sound effect to a specific audio clip in the audio data, and can add multiple sound effects correspondingly to multiple audio clips of the audio data, enriching the effect of adding sound effects, improving the interest of adding sound effects, and enhancing the user experience. [Brief description of the drawings]
[0010] The drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification serve to explain the principles of the present disclosure. In order to more clearly explain the technical solutions of the embodiments or prior art of the present disclosure, the following briefly introduces drawings that need to be used in the description of the embodiments or prior art, and other drawings can also be obtained based on these drawings, on the premise that it is obvious to those skilled in the art that they do not make efforts worthy of inventive step. [Figure 1] 1 is a flowchart of a method for processing audio data according to an embodiment of the present disclosure. [Diagram 2] AB is a schematic diagram of an operation method of an operation interface according to an embodiment of the present disclosure. [Figure 3A] 1 is a schematic diagram of a method for adding sound effects according to an embodiment of the present disclosure. [Figure 3B] 1 is a schematic diagram of another sound effect adding method according to an embodiment of the present disclosure. [Figure 4] 5A-5B are schematic diagrams of another sound effect adding method according to an embodiment of the present disclosure. [Diagram 5] 5A-5B are schematic diagrams of another sound effect adding method according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram of an audio smoothing method according to an embodiment of the present disclosure. [Figure 7] 11 is a flowchart of another audio data processing method according to an embodiment of the present disclosure. [Figure 8] 1 is a schematic diagram of a method for adding sound effects according to an embodiment of the present disclosure. [Figure 9] 1 is a schematic diagram illustrating the configuration of an audio data processing device according to an embodiment of the present disclosure. [Figure 10] FIG. 2 is a schematic diagram illustrating the configuration of a terminal device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] In order to make the above objectives, features and advantages of the present disclosure more clearly comprehensible, the solutions of the present disclosure are further described below, whereby the embodiments and features in the embodiments of the present disclosure may be combined with each other if they do not conflict.
[0012] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure; however, the present disclosure may be embodied in other ways than those described herein, and it is apparent that the embodiments in the specification are merely some of the embodiments of the present disclosure, rather than all of the embodiments.
[0013] FIG. 1 is a flowchart of an audio data processing method according to an embodiment of the present disclosure, which is executed by a terminal device. Exemplarily, the terminal device may be understood as a device having audio processing capability and video playback capability, such as a mobile phone, a tablet computer, a laptop, a desktop computer, a smart TV, etc. In some embodiments, the terminal device according to the present embodiment is equipped with an audio collecting device (e.g., may be, but is not limited to, a microphone), the terminal device collects audio data by the audio collecting device, and adds one or more audio effects to the audio data by the method of the present embodiment, the audio effect according to the present embodiment being an audio form that is superimposed on the audio data to generate a specific audio expression effect (e.g., may be, but is not limited to, a vibration effect, an electronic music effect, a character effect) in the original audio data. In some other embodiments, the terminal device according to the present embodiment may be equipped with a shooting device and an audio collecting device at the same time, the shooting device captures a video image, and at the same time, the audio collecting device collects audio data to generate a video with audio, and the terminal device uses the method of the present embodiment to add one or more audio effects to the audio data of the video with audio. As shown in FIG. 1, the method provided by this embodiment includes the following steps. Step 101: A first playback position in first audio data and a user's preview command for a first sound effect are obtained.
[0014] In this embodiment, the name of the "first audio data" does not have any other meaning, but is merely for easily distinguishing between multiple audio data related to this embodiment. The first audio data according to this embodiment may be understood as audio data of a video to be edited in a video editing interface, or audio data recorded in another manner. The first audio data according to this embodiment may be audio data to which one or more sound effects are added, or may be audio data to which no sound effects are added. If the first audio data is specifically audio data to which one sound effect is added, the sound effect may be added to all audio data of the first audio data, or may be added to an audio clip of the first audio data. If the first audio data is specifically audio data to which multiple sound effects are added, the multiple sound effects may be understood to be added to different audio clips of the audio data, for example, one sound effect is added to one audio clip.
[0015] The first playback position in this embodiment may be understood as a position selected by the user from the first audio data, or as a default position in the first audio data if the user does not select a position, such as the start position or middle position of the audio data, but is not limited to these positions.
[0016] In this embodiment, the operation of obtaining the first playback position in the first audio data and the user's listening command for the first sound effect is triggered by at least one of a predetermined operation by the user on the operation interface and a predetermined voice command.
[0017] Illustratively, FIGS. 2A-2B are schematic diagrams of an operation method of an operation interface according to an embodiment of the present disclosure. As shown in FIGS. 2A-2B, the operation interface includes a display area 21, a first operation area 22, and a second operation area 23. The first operation area 22 includes a playback progress control, and a playback position of the first audio data can be selected by moving the position of the focus in the playback progress control. The second operation area 23 includes at least one sound effect control, and each sound effect control corresponds to one sound effect. If the first audio data is specifically audio data of a video, the interface shown in FIGS. 2A and 2B is specifically a video editing interface. In this case, the display area 21 is for displaying a video image, and the playback progress control in the first operation area 22 is for adjusting the playback position of the video, and in this case, based on the corresponding relationship between the video playback position and the first audio data, the playback position of the video is mapped to the first audio data to obtain the playback position of the first audio data (i.e., the first playback position in this embodiment). If the first audio data is simply audio data collected by an audio collection device and has no corresponding video image, the interface of Figures 2A and 2B is specifically an audio editing interface, in which case the display area 21 may be for displaying the audio curve of the first audio data, and the focus position on the playback progress control indicates the playback position of the first audio data.
[0018] Referring to FIG. 2A, in the interface shown in FIG. 2A, a user can perform a click operation at any position on the playback progress axis (i.e., the straight line on which the focus of FIG. 2A is located). For example, in one exemplary embodiment, the interface shown in FIG. 2A displays presentation information to the user to suggest the user to select a start position of a sound effect, and the presentation information can help the user quickly select a position on the playback progress axis. For example, when the user clicks on a position on the playback progress axis, the focus of the playback progress control moves to the position clicked by the user. In this case, the first playback position according to this embodiment can be obtained by mapping the position of the focus on the playback progress axis to the first audio data. Or, in another embodiment, the user drags to move the focus on the playback progress axis based on the presentation information, and when the movement of the focus stops, the first playback position according to this embodiment can be obtained by mapping the position of the focus on the playback progress axis to the first audio data.
[0019] 2B, after the user selects the first playback position, the user performs a predetermined operation on a certain sound effect control in the second operation area 23 to trigger a preview command, and the predetermined operation may be, for example, one of a single click on a certain sound effect control, a continuous multiple click on a certain sound effect control, and a slide of a certain sound effect control in a certain direction, but is not limited to these predetermined operations. For example, when the user performs a predetermined operation on the sound effect control 2 in FIG. 2B, a preview command of the sound effect corresponding to the sound effect control 2 is triggered, and in this case, the sound effect of the sound effect control 2 is the first sound effect in this embodiment, and if the user is not satisfied with the sound effect preview effect of the sound effect control 2, the user may subsequently perform a predetermined operation on another sound effect control in the second operation area 23 to trigger and preview the sound effect of the other sound effect control. In some embodiments, the display interface of FIG. 2B may further display a prompt message on how to preview sound effects, so as to instruct the user on how to quickly preview sound effects and reduce the user's difficulty in using the interface.
[0020] Of course, the above is not the only limitation to this embodiment, but merely an exemplary description based on FIGS. 2A and 2B.
[0021] Step 102: A first sound effect is added to a first audio clip in the first audio data, and sound effect preview data is generated and played back, the first audio clip being an audio clip in the first audio data whose starting position is a first playback position.
[0022] Illustratively, the first sound effect adding method according to this embodiment includes the following: In one method, the first sound effect is added to all audio data after the first playback position in the first audio data. For example, FIG. 3A is a schematic diagram of a method for adding sound effects according to an embodiment of the present disclosure. As shown in FIG. 3A, the first audio data is specifically audio data having a length of m seconds, and the first playback position selected by the user is specifically the fth second position of the audio data. In this way, in one exemplary method for adding sound effects, the first sound effect can be added to an audio clip from fth second to mth second.
[0023] In another method, the first sound effect is added to an audio clip of the first audio data, starting from a first playback position and having a predetermined time length (for example, 1 second, but not limited thereto). For example, FIG. 3B is a schematic diagram of another method of adding sound effects according to an embodiment of the present disclosure, and as shown in FIG. 3B, the first audio data is illustratively audio data having a length of G seconds, and the first playback position selected by the user is illustratively the Kth second position of the audio data. In this way, in one exemplary method of adding sound effects, the first sound effect can be added to the audio data within a predetermined time length after the Kth second of the first audio data.
[0024] In addition, although this embodiment provides the above two first sound effect adding methods, the first sound effect adding method of the embodiment does not only have the above two, and in fact, the first sound effect adding method is not limited to one or several fixed adding methods, and can be set according to needs.
[0025] With reference to the interface shown in FIG. 2A or FIG. 2B, when playing the sound effect preview data, the focus on the playback progress control moves synchronously on the playback progress axis. In one example, after the playback of the sound effect preview data is completed, the position of the focus on the playback progress axis is restored to the start position of the sound effect selected by the user. That is, after the playback of the sound effect preview data is completed, the playback position of the first audio data of this embodiment is restored to the first playback position of this embodiment. Of course, this is just one implementation of this embodiment, not all embodiments, and in fact, in other implementations, the focus position may not be restored to the initial position selected by the user after the playback of the sound effect preview data is completed.
[0026] In this embodiment, after the playback of the sound effect preview data is completed, the playback position of the first audio data is restored to the first playback position, making it easy for the user to preview multiple sound effects, and when the user needs to preview a sound effect other than the current sound effect, the user does not need to reselect the start position of the sound effect, simplifying the user's operation and improving the user experience. Also, when the user likes the current sound effect, the user does not need to search again for the start position of the sound effect at the time of preview, simplifying the user's operation and improving the accuracy of adding sound effects.
[0027] Step 103: Receive a first addition command for the second sound effect by the user, where the first addition command includes information of a first addition length of the second sound effect in the first audio data.
[0028] Exemplarily, the first additional command in this embodiment is triggered by a predetermined interface operation or a voice command, and the predetermined interface operation may be, for example, a long press operation on the sound effect control, a slide operation in a predetermined direction, or a successive click operation on the sound effect control, but is not limited to these operations.
[0029] Referring to the display interface shown in FIG. 2A or FIG. 2B, in some embodiments, the display interface of the terminal device can also display prompting information to prompt the user how to add sound effects to the first audio data, and the prompting message can enable the user to quickly add sound effects to the first audio data.
[0030] In this embodiment, presentation information instructing the user how to add sound effects is displayed on the display interface, so that the user can quickly learn how to add sound effects and smoothly add sound effects to audio data, thereby improving the smoothness and experience of user operations.
[0031] Exemplarily, the first additional length information in this embodiment includes information on a cutoff position of a sound effect, or information on a duration of a sound effect.
[0032] Step 104: Add the second sound effect to a second audio clip in the first audio data to obtain second audio data, where the second audio clip is an audio clip whose start position is the first playback position and whose length is the first added length.
[0033] The second sound effect according to the present embodiment may be the first sound effect previewed by the user, or may be an unpreviewed sound effect selected by the user from the operation interface.
[0034] In this embodiment, the second sound effect adding method includes at least some of the following: In one addition method, after obtaining a first playback position in the first audio data, the user presses and holds the sound effect control to add a second sound effect corresponding to the sound effect control to the first audio data from the first playback position, and when the user lifts his / her finger, the sound effect addition is stopped and an audio clip of the first addition length (i.e., the second audio clip in this embodiment) is obtained. For example, Figures 4A-4B are schematic diagrams of another sound effect addition method according to an embodiment of the present disclosure, and in Figures 4A-4B, the playback position f is the sound effect start addition position selected by the user (i.e., the first playback position). As shown in FIG. 4A, when a user presses and holds the sound effect control n, if it is detected that the pressing time by the user is longer than a first predetermined time (for example, it may be 0.5 seconds, but is not limited thereto), the sound effect (i.e., the second sound effect) corresponding to the sound effect control n is added to the first audio data from the playback position f, in this case, the effect with the added sound effect is played from the playback position f, and the focus in the playback progress control moves along the playback progress axis according to the arrow direction in FIG. 4A as the audio data is played. As shown in FIG. 4B, when the focus movement moves to the position u, when the user lifts his finger, the addition of the sound effect is stopped at the position u, and the audio corresponding to the positions f to u in the first audio data is the second audio clip according to this embodiment.
[0035] In another adding method, after obtaining a first playback position in the first audio data, the user triggers the adding operation of the sound effect by sliding a sound effect control in a predetermined direction. When the user slides a certain sound effect control in a predetermined direction, the sound effect corresponding to the sound effect control (i.e., the second sound effect) is added to the first audio data from the first playback position, and when the user slides the sound effect control again in the predetermined direction or in the opposite direction to the predetermined direction, the addition of the sound effect is stopped, and the end position of the sound effect in the first audio data is temporarily called the third playback position. In this case, the audio clip between the first playback position and the third playback position in the first audio data is the second audio clip in this embodiment. For example, FIG. 5A-FIG. 5B are schematic diagrams of another sound effect adding method according to an embodiment of the present disclosure, and in FIG. 5A-FIG. 5B, the playback position f is the start addition position (i.e., the first playback position) of the sound effect selected by the user. As shown in Fig. 5A, when the user slides the sound effect control n upward, the sound effect (i.e., the second sound effect) corresponding to the sound effect control n is added to the first audio data from the playback position f, in this case, the effect with the added sound effect is played from the playback position f, and the focus on the playback progress control moves along the playback progress axis according to the arrow direction in Fig. 5A as the first audio data is played. As shown in Fig. 5B, when the focus moves to the position u, when the user slides the sound effect control n downward, the addition of the sound effect stops at the position u, the position u is the sound effect end position, and the audio clip corresponding to the position f to the position u in the first audio data is the second audio clip according to this embodiment.
[0036] In yet another adding method, after obtaining a first playback position in the first audio data, the user triggers the adding operation of the sound effect by continuously clicking on the sound effect control, in which case the sound effect corresponding to the sound effect control (i.e., the second sound effect) is added to the first audio data from the first playback position, and the user stops adding the sound effect by performing a continuous click operation on the sound effect control again, so that the audio clip from the first playback position in the first audio data to the stop of adding the sound effect is the second audio clip in this embodiment.
[0037] Of course, although only three methods of adding sound effects are provided here, these three methods are not the only methods in this embodiment, and in fact, the method of adding sound effects in this embodiment is not limited to one or several methods, but may be set according to needs.
[0038] It should be noted that, in the above method of the present embodiment, one sound effect is described to be added to one audio clip of audio data, but as can be understood by those skilled in the art, multiple sound effects may be added to one audio data according to the above method. For example, after obtaining second audio data according to the above method, the present embodiment may continuously use the above method to obtain a second playback position in the second audio data and a second addition command for a third sound effect by a user, and add a third sound effect to a third audio clip in the second audio data according to the second addition command to obtain third audio data, in this way, the obtained third audio data is audio data including two sound effects, and the third audio clip may be understood as an audio clip whose starting position is the second playback position in the second audio data and whose length is the second addition length.
[0039] Furthermore, considering that two different sound effects are connected, there is a possibility that a sonic boom will occur at the connection point of the sound effects. To solve this problem, the present embodiment further provides a smoothing processing method, which performs a fade-out process on the earlier one of the two sound effects and a fade-in process on the later one, thereby avoiding the occurrence of a sonic boom and enabling a smooth transition at the connection point of the two sound effects. For example, FIG. 6 is a schematic diagram of an audio smoothing method according to an embodiment of the present disclosure, and the width between the two horizontal lines in FIG. 6 indicates the volume. As shown in FIG. 6, sound effect 61 in FIG. 6 is a second sound effect according to the present embodiment, sound effect 62 is a third sound effect according to the present embodiment, and the end position 63 of the second sound effect is the start position of the third sound effect. In this case, when smoothing is performed on the connection point between the second sound effect and the third sound effect, it is necessary to perform a fade-out process on the second sound effect so that the volume of the second sound effect is reduced at a position 64 that is a predetermined distance away from the end position 63 and reduced to 0 at the end position 63. In this case, since the volume of the audio data does not increase or decrease, it is necessary to perform a fade-in process on the third sound effect from position 64 so that the volume of the third sound effect gradually increases from position 64 and increases to a set value at the end position 63. In this way, by performing a cross fade-out process on two consecutive sound effects, the connection point of the two sound effects is smoothly transitioned, the occurrence of a sonic boom is avoided, and the user experience is improved. Of course, the smoothing method provided in FIG. 6 is not the only limitation of this embodiment, but is merely an illustrative explanation, and in fact, related art has already provided a wide variety of audio smoothing methods, any one of which may be used in this embodiment according to needs.
[0040] In this embodiment, a first playback position in the first audio data and a user preview command for a first sound effect are obtained, and the first sound effect is added to a first audio clip in the first audio data to obtain sound effect preview data for preview, the sound effect preview data is played, and when a first addition command for a second sound effect is received from the user, the second sound effect is added to a second audio clip in the first audio data, the second sound effect having the first playback position as a starting position, based on information on the first addition length included in the first addition command, to obtain second audio data. According to the solution of the embodiment, a user can select an arbitrary position in audio data to preview the sound effect, and add a favorite sound effect to an audio clip of the audio data according to the preview result, so that compared with the case where the sound effect adding effect cannot be previewed, the user can select a favorite sound effect by previewing, and add the favorite sound effect to the audio data, and thus, the sound effect finally selected by the user to be added to the audio data can be guaranteed to be the favorite sound effect of the user, and the user does not need to repeatedly add the added sound effect again due to dissatisfaction with the added sound effect, simplifying the user's operation and improving the user experience. In addition, according to the solution of the embodiment, a user can add a specific sound effect to a specific audio clip in the audio data, and add multiple sound effects to multiple audio clips of the audio data correspondingly, improving the interest of adding sound effects and enhancing the user experience.
[0041] FIG. 7 is a flowchart of another audio data processing method according to an embodiment of the present disclosure. As shown in FIG. 7, the above first sound effect adding method includes the following steps: Step 701: Recognize a target voice from a first audio clip based on a pre-defined voice recognition model. Step 702: A first sound effect is added to a target sound in a first audio clip to generate preview data of the first sound effect in the first audio clip.
[0042] Exemplarily, the target sound in this embodiment includes at least one of a human voice, an animal voice, a vehicle voice, an instrument voice, a background sound, and a foreground sound.
[0043] Exemplarily, the speech recognition model according to the present embodiment may be understood as a model that is pre-trained by a model training method and can recognize and extract one or more of the above target voices, which may be at least one of, but is not limited to, a support vector machine model, a deep neural network model, a logistic regression model, etc.
[0044] Referring to the method of FIG. 3A and FIG. 3B, after obtaining a first audio clip starting from the first playback position based on the first playback position, the embodiment provides an editing interface to the user, the interface includes at least one voice option. The user can select one or more of the voice options as target voices, and then a voice recognition model for recognizing the target voices is called, and the target voices are recognized from the first audio clip by the model, and the positions of the target voices are obtained. Furthermore, after obtaining the positions of the target voices, one executable form adds a first sound effect to the positions of the target voices to generate preview data, and another executable form first separates the audio data of the target voice from the positions of the target voice, adds the first sound effect to the audio data of the target voice to obtain voice change data, and then adds the voice change data to the positions of the target voices in the first audio clip to obtain preview data. For example, Fig. 8 is a schematic diagram of a sound effect adding method according to an embodiment of the present disclosure, in which the horizontal lines in Fig. 8 indicate audio data, and as shown in Fig. 8, the positions where the target voice is recognized based on the voice recognition model are q and w. When performing the sound effect adding operation, the sound effect can be directly added to the q and w positions of the audio data, or the target voice data is separated from the q and w positions, and then the sound effect is added to the target voice at the q position and the target voice at the w position, respectively, to obtain the voice change data at the q position and the voice change data at the w position, and then the voice change data at the q position is added to the q position of the audio data, and the voice change data at the w position is added to the w position of the audio data to obtain the preview data. Of course, this is not the only limitation on the sound effect adding method of this embodiment, but is merely an illustrative description.
[0045] It should be noted that in some embodiments, the sound effect adding method of this embodiment may be applied to the second sound effect adding process of the above embodiment, i.e., the method of this embodiment can add a second sound effect to the target audio of the above second audio clip, and the specific adding method can be referred to the method of this embodiment, and will not be described in detail here.
[0046] In this embodiment, a target voice in an audio clip is recognized by a preset voice recognition model, and sound effects are added to the target voice of the audio clip, so that the sound effect addition effect is more interesting and improves the user experience.
[0047] 9 is a schematic diagram of a configuration of an audio data processing device according to an embodiment of the present disclosure, and the processing device may be understood as the above terminal device or a part of a functional module in the above terminal device. As shown in FIG. 9, the processing device 90 includes: a first acquisition module 91 for acquiring a first playback position in the first audio data and a listening command for a first sound effect by a user; a first sound effect adding module 92 that adds the first sound effect to a first audio clip in the first audio data to generate and play back sound effect preview data, the first audio clip being an audio clip in the first audio data whose start position is the first playback position; a first receiving module 93 for receiving a first addition command for a second sound effect from a user, the first addition command including information on a first addition length of the second sound effect in the first audio data; and a second sound effect adding module 94 that adds the second sound effect to a second audio clip in the first audio data to obtain second audio data, the second audio clip being an audio clip whose start position is the first playback position and whose length is the first added length.
[0048] In one embodiment, the first audio data is audio data of a video to be edited in a video editing interface.
[0049] In one embodiment, the first acquisition module 91 is a first display unit for displaying the video editing interface, the video editing interface including a video playback progress control and a sound effect control for the first sound effect; an acquisition unit for acquiring the first playback position selected by a user via the playback progress control and the preview command triggered by the user via the sound effect control.
[0050] In one embodiment, the processing unit 90 includes: The audio playback device may further include a restoration module for restoring a playback position of the first audio data to the first playback position after the playback of the sound effect preview data is completed.
[0051] In one embodiment, the first sound effect adding module 92 includes: a recognition unit for recognizing a target speech from the first audio clip based on a predefined speech recognition model; and a sound effect adding unit that adds the first sound effect to the target sound in the first audio clip to generate preview data of the first sound effect in the first audio clip.
[0052] In one embodiment, the processing unit 90 includes: a display module for displaying at least one audio option; The system may further include a determination module for determining a voice selected by the user from the at least one voice option as a target voice.
[0053] In one embodiment, the target voice is: The sound includes at least one of human voices, animal voices, vehicle voices, instrument voices, background sounds, and foreground sounds.
[0054] In one embodiment, the second sound effect addition module 94 adds the second sound effect to the target voice in the second audio clip.
[0055] In one embodiment, the processing unit 90 includes: a second acquisition module for acquiring a second playback position in the second audio data and a second addition command for a third sound effect by the user, the second addition command including information on a second addition length of the third sound effect in the second audio data; The method may further include a third sound effect adding module that adds the third sound effect to a third audio clip in the second audio data and obtains the third audio data, the third audio clip being an audio clip in the second audio data having the second playback position as a starting position and having a length that is the second addition length.
[0056] In one embodiment, the processing unit 90 includes: The audio data processing device may further include a smoothing processing module that performs a fade-out process on the second audio effect and a fade-in process on the third audio effect when the end position of the second audio effect and the second playback position are two consecutive playback positions in the third audio data.
[0057] The apparatus provided by this embodiment can implement the method of any one of the embodiments of Figures 1 to 8 above, and the implementation manner and beneficial effects are similar, so no further description is given here.
[0058] An embodiment of the present disclosure further provides a terminal device, the terminal device including a processor and a memory, a computer program stored in the memory, and the computer program, when executed by the processor, can realize the method of any one of the embodiments of Figures 1 to 8 above.
[0059] Illustratively, FIG. 10 is a schematic diagram of a terminal device according to an embodiment of the present disclosure. Hereinafter, a schematic diagram of a terminal device 1000 according to an embodiment of the present disclosure will be described with specific reference to FIG. 10. The terminal device 1000 according to the embodiment of the present disclosure includes, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable media players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The terminal device in FIG. 10 is merely an example, and should not impose any limitations on the functions and scope of use of the embodiment of the present disclosure.
[0060] As shown in Fig. 10, the terminal device 1000 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1001 that performs various appropriate operations and processes based on programs stored in a read only memory (ROM) 1002 or programs loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 further stores various programs and data necessary for the operation of the terminal device 1000. The processing unit 1001, the ROM 1002 and the RAM 1003 are connected to each other by a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0061] Typically, input devices 1006 including, for example, a touch screen, a touch panel, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyro, etc., output devices 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc., storage devices 1008 including, for example, a magnetic tape, a hard disk, etc., and communication devices 1009 may be connected to the I / O interface 1005. The communication devices 1009 enable the terminal device 1000 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 13 shows the terminal device 1000 having various devices, it is not required to implement or include all the devices shown. Alternatively, more or fewer devices may be implemented or included.
[0062] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowcharts may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the method shown in the flowchart. In such an embodiment, the computer program is downloaded and installed from a network via a communication device 1009, or installed from a storage device 1008, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, it performs the above-mentioned functions defined in the method of the embodiment of the present disclosure.
[0063] It should be noted that the above computer readable medium of the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the above. The computer readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection having one or more leads, a portable computer magnetic disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact magnetic disk read only memory (CD-ROM), an optical memory, a magnetic memory, or any suitable combination of the above. In the present disclosure, the computer readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, the computer readable signal medium includes a data signal propagated in baseband or as part of a carrier, the data signal carrying computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may be any computer-readable medium, except for a computer-readable storage medium, which may transmit, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium may be transmitted by any suitable medium, including, but not limited to, wire, optical cable, RF (radio frequency), or the like, or any suitable combination of the above.
[0064] In some embodiments, clients and servers communicate using any network protocol now known or later developed, such as HyperText Transfer Protocol (HTTP), and may be connected to each other via any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet, and an end-to-end network (e.g., an adhoc end-to-end network), and any network now known or later developed.
[0065] The computer-readable medium may be included in the terminal device, or may be present separately from the terminal device without being disposed in the terminal device.
[0066] The computer-readable medium carries one or more programs, which, when executed by the terminal device, cause the terminal device to perform the following steps: acquiring a first playback position in the first audio data and a user's preview command for a first sound effect; adding the first sound effect to a first audio clip in the first audio data to generate and play back sound effect preview data, where the first audio clip is an audio clip in the first audio data whose starting position is the first playback position; and, if a first addition command for a second sound effect is received from the user, adding a second sound effect to a second audio clip in the first audio data to acquire second audio data, where the first addition command includes information of a first addition length of the second sound effect in the first audio data, and the second audio clip is an audio clip in the first audio data whose starting position is the first playback position and whose length is the first addition length.
[0067] Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and further including, but not limited to, conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. When a remote computer is involved, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., connected via the Internet using an Internet Service Provider).
[0068] The flowcharts and block diagrams in the drawings illustrate possible system architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or part of code, which includes one or more executable instructions for implementing a certain logical function. It should be noted that in some implementations, the functions noted in the blocks may occur in a different order from the order shown in the figures. For example, two blocks shown in succession may actually be executed essentially in parallel, or in some cases in the reverse order, depending on the functions involved. Also, herein, each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts may be implemented in a dedicated system by hardware for executing a certain function or operation, or in a combination of dedicated hardware and computer instructions.
[0069] The relevant units described in the embodiments of the present disclosure may be implemented in the form of software or hardware. In some cases, the names of the units do not constitute limitations on the units themselves.
[0070] As used herein, the functions described above may be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used include, but are not limited to, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0071] In the specification of the present disclosure, a machine-readable medium may be a tangible medium that contains or stores a program used by or in combination with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection by one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact magnetic disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0072] An embodiment of the present disclosure further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the method of any one of the embodiments of Figures 1 to 8 above can be realized, and since the implementation manner and beneficial effects thereof are similar, they will not be described in detail here.
[0073] In this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or sequence between these entities or operations. In addition, the terms "comprise" and "include" or any other variations thereof are intended to include a non-exclusive inclusion, whereby a process, method, article, or device that includes a set of elements not only includes those elements, but also includes other elements not expressly listed, or further includes the inherent elements of such process, method, article, or equipment. Unless further limited, an element defined by the phrase "includes" does not exclude that the process, method, article, or equipment that includes the element includes other identical elements.
[0074] The above are merely specific embodiments of the present disclosure, enabling those skilled in the art to understand or realize the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to these embodiments described herein, but is accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. 1. A method for processing audio data, comprising the steps of: obtaining a first playback position in the first audio data and a user listening command for at least one first sound effect; a step of adding the at least one first sound effect to a first audio clip in the first audio data to generate at least one sound effect preview data, and playing and previewing the at least one sound effect preview data, the first audio clip being an audio clip in the first audio data whose start position is the first playback position; selecting one first sound effect as a second sound effect from the at least one first sound effect sampled; receiving a first addition command for the second sound effect from a user, the first addition command including information of a first addition length of the second sound effect in the first audio data; and adding the second sound effect to a second audio clip in the first audio data to obtain second audio data, the second audio clip being an audio clip having a starting position at the first playback position and a length equal to the first added length.
2. 2. The method of claim 1, wherein the first audio data is audio data of a video being edited in a video editing interface.
3. The step of acquiring a first playback position in the first audio data and a user's preview command for a first sound effect includes: displaying the video editing interface including a video playback progress control and a sound effect control for the first sound effect; and obtaining the first playback position selected by a user via the playback progress control and the preview command triggered by the user via the sound effects control.
4. 4. The method according to claim 1, further comprising the step of restoring the playback position of the first audio data to the first playback position after the playback of the sound effect preview data is completed.
5. the step of adding the first sound effect to a first audio clip in the first audio data to generate sound effect preview data includes: recognizing a target voice from the first audio clip based on a predefined voice recognition model and determining a location of the target voice; 2. The method of claim 1, further comprising: adding the first sound effect to the position of the target sound in the first audio clip to generate preview data of the first sound effect in the first audio clip.
6. Prior to recognizing a target speech from the first audio clip based on the predefined speech recognition model, the method further comprises: displaying at least one audio option; 6. The method of claim 5, further comprising determining a voice selected by the user from the at least one voice option as a target voice.
7. The step of adding the second sound effect to a second audio clip in the first audio data comprises:
7. The method of claim 5 or 6, further comprising the step of adding the second sound effect to the target voice in the second audio clip.
8. After adding the second sound effect to a second audio clip in the first audio data to obtain second audio data, the method further comprises: obtaining a second playback position in the second audio data and a second addition command for a third sound effect by the user, the second addition command including information on a second addition length of the third sound effect in the second audio data; 2. The method of claim 1, further comprising the step of adding the third sound effect to a third audio clip in the second audio data to obtain third audio data, the third audio clip being an audio clip in the second audio data that has a starting position at the second playback position and a length that is the second added length.
9. 9. The method of claim 8, further comprising the step of performing a fade-out process on the second sound effect and a fade-in process on the third sound effect when the end position of the second sound effect and the second playback position are two consecutive playback positions in the third audio data.
10. 1. An apparatus for processing audio data, comprising: a first acquisition module for acquiring a first playback position in the first audio data and a listening command for at least one first sound effect by a user; a first sound effect adding module for adding the at least one first sound effect to a first audio clip in the first audio data to generate at least one sound effect preview data, and for playing and previewing the at least one sound effect preview data, the first audio clip being an audio clip whose start position is the first playback position in the first audio data; a first receiving module for selecting one first sound effect as a second sound effect from the at least one first sound effect sampled, and receiving a first addition command for the second sound effect from a user, the first addition command including information on a first addition length of the second sound effect in the first audio data; and a second sound effect adding module that adds the second sound effect to a second audio clip in the first audio data to obtain second audio data, the second audio clip being an audio clip whose start position is the first playback position and whose length is the first added length.
11. A terminal device comprising a memory and a processor, the memory storing a computer program, the computer program implementing the method according to any one of claims 1 to 9 when executed by the processor.
12. 10. A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Data processing method, apparatus, terminal, and storage medium
CN109346111A
Synchronous reproducing device, music piece automatic remix system, synchronous reproducing method and synchronous reproducing program
JP2009204907A
Audio processing device, audio processing method, audio processing program and audio processing integrated circuit
JP2014095806A
Information processing system and information processing apparatus
JP2019205151A
