Audio editing method, system and device and storage medium

Abstract

The invention provides an audio editing method, system and device and a storage medium, and relates to the technical field of computers, in particular to the technical fields of data processing, large language models, audio editing and the like. The method comprises the following steps that: middleware inputs audio to be edited and an editing instruction into an audio editing model; the audio editing model determines a corresponding editing suggestion based on the to-be-edited audio and the editing instruction, and feeds back the editing suggestion to the middleware; the middleware determines a corresponding editing material according to the editing suggestion, constructs a cue word according to the editing suggestion and the editing material, and inputs the cue word and the editing material into an audio editing model; and the audio editing model edits the to-be-edited audio based on the cue word and the editing material to obtain the edited audio. According to the invention, the audio editing efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as data processing, large language models, and audio editing. Background Art

[0002] With the widespread use of smart terminals, more and more users are recording oral broadcasts, that is, users dictate a piece of content, and smart terminals record and save the oral content. In oral broadcast recording, there are often slips of the tongue, missing content, or the voice style does not match the audio content. In this case, re-recording is required, resulting in a lot of time wasted. How to realize automatic editing or modification of audio content is a technical problem that needs to be solved. Summary of the invention

[0003] The present disclosure provides an audio editing method, system, device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided an audio editing method, which is applied to an audio editing system including middleware and an audio editing model, and the method includes:

[0005] The middleware inputs the audio to be edited and the editing instructions into the audio editing model;

[0006] The audio editing model determines corresponding editing suggestions based on the audio to be edited and the editing instructions, and feeds the editing suggestions back to the middleware;

[0007] The middleware determines the corresponding editing material according to the editing suggestion, constructs prompt words according to the editing suggestion and the editing material, and inputs the prompt words and the editing material into the audio editing model;

[0008] The audio editing model edits the audio to be edited based on the prompt words and the editing material to obtain the edited audio.

[0009] An audio editing system includes middleware and an audio editing model; wherein:

[0010] The middleware is used to input the audio to be edited and the editing instructions into the audio editing model, receive the editing suggestions returned by the audio editing model; determine the corresponding editing materials according to the editing suggestions, construct prompt words according to the editing suggestions and the editing materials, and input the prompt words and the editing materials into the audio editing model;

[0011] The audio editing model is used to determine corresponding editing suggestions based on the audio to be edited and the editing instructions, and to feed back the editing suggestions to the middleware; it is also used to receive prompt words and editing materials, and edit the audio to be edited based on the prompt words and the editing materials to obtain the edited audio.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] This disclosure achieves a deep integration of user editing intent and audio processing technology through the close collaboration of middleware and audio editing model. This method not only improves the efficiency and accuracy of audio editing work, but also creates a convenient and intelligent audio creation and editing environment for users, making the audio editing process smoother and more intuitive.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 is a flowchart of an implementation of an audio editing method 100 according to an embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of a content delivery method in an audio editing method proposed in an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of the structure of an audio editing system 300 according to an embodiment of the present disclosure;

[0024] Figure 4 A schematic block diagram of an example electronic device 400 that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of types. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0027] With the widespread use of smart terminals, more and more users are recording oral broadcasts. Users dictate a piece of content, and smart terminals record and save the oral content. In oral broadcast recording, there are often slips of the tongue, missing content, or the voice style does not match the audio content. In this case, re-recording is required, resulting in a lot of time wasted. If there is an audio editor that can help automatically correct audio problems or edit audio content, it will greatly improve efficiency. At the same time, the editor can also be used as a tool for audio background noise removal and background music generation.

[0028] The existing audio editing systems mainly use a single expert model to implement a single task; this type of expert model can realize audio editing in a single field and cannot cover a wider field; for other fields outside its field of expertise, the audio editing effect of the expert model still has defects, and it is easy to edit the audio incorrectly, causing the original audio to lose texture or content.

[0029] In order to solve the above problems, the embodiments of the present disclosure propose an audio editing method and system. Among them, the audio editing method and system proposed in the embodiments of the present disclosure can be applied to electronic devices such as mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, game consoles, e-book readers, multimedia playback devices, wearable devices, etc. The electronic device receives the audio to be edited and the editing instructions input by the user, and automatically edits the audio to be edited according to the editing instructions. Alternatively, the audio editing method and system proposed in the embodiments of the present disclosure can also be applied to a cloud server, receiving the audio to be edited and the editing instructions sent by the user through the terminal device, automatically editing the audio to be edited according to the editing instructions, and returning the edited audio to the terminal device.

[0030] Figure 1 is a flowchart of an implementation of an audio editing method 100 according to an embodiment of the present disclosure, comprising:

[0031] S110, the middleware inputs the audio to be edited and the editing instruction into the audio editing model;

[0032] S120, the audio editing model determines corresponding editing suggestions based on the audio to be edited and the editing instruction, and feeds back the editing suggestions to the middleware;

[0033] S130, the middleware determines the corresponding editing material according to the editing suggestion, constructs prompt words according to the editing suggestion and the editing material, and inputs the prompt words and the editing material into the audio editing model;

[0034] S140: The audio editing model edits the audio to be edited based on the prompt words and the editing material to obtain the edited audio.

[0035] In the audio editing method proposed in the embodiment of the present application, the deep integration of user editing intention and audio processing technology is achieved through the close cooperation of middleware and audio editing model. This method not only improves the efficiency and accuracy of audio editing, but also brings a more convenient and intelligent audio editing experience to users.

[0036] In some implementations, the audio editing model identifies the audio to be edited to determine the text corresponding to the audio to be edited; the audio editing model determines the corresponding editing suggestions based on the audio text and the audio editing instructions.

[0037] The audio editing model receives an audio file to be edited, which may contain multiple sound elements, such as the voice of the main speaker, the voices of other people speaking, background music, and environmental noise. In order to accurately extract the voice of the main speaker, the audio editing model will pre-process the audio to be edited, including using filters to filter out background noise, adjusting the overall volume level of the audio by gain, and using the Voice Activity Detection (VAD) method to identify the human voice in the audio.

[0038] The core of the VAD method is to deeply analyze the subtle features in the audio to accurately distinguish between the human voice segment and the inactive segment that does not contain the human voice (such as long periods of silence, environmental noise or other non-speech type background sound, etc.). When building an audio editing model, a human voice detection module can be included in the audio editing model; this human voice detection module is based on the VAD method and can automatically identify the human voice part of the audio, so that the audio editing model can focus more on the processing and use of the human voice.

[0039] To achieve this goal, the human voice detection module can use machine learning or deep learning technology. The human voice detection module can be obtained by pre-training. During the training process, the human voice detection module, as a part of the audio editing model, can be trained as a whole together with other parts of the audio editing model; or, the human voice detection module can be trained, and after the training of the human voice detection module is completed, the audio editing model is trained again. During the training process, the parameters of the human voice detection module are frozen, and only the parameters of other parts of the audio editing model except the human voice detection module are adjusted.

[0040] In the process of training the human voice detection module, a diverse audio data set can be prepared first, which should contain rich human voice samples and various types of non-human voice interference (such as natural environment noise, mechanical equipment sound, background music, etc.). Subsequently, through feature extraction technology, multi-dimensional feature vectors that can characterize human voice features, such as time domain features and frequency domain features, are extracted from each audio sample. These feature vectors are used as input to train the human voice detection module so that the module can effectively distinguish human voice samples from non-human voice samples. After distinguishing the human voice samples from non-human voice samples in the audio to be edited, the audio editing model performs channel separation on the human voice samples. Specifically, the audio editing model has a strong analytical ability and can deeply analyze the characteristics of the human voice samples in the audio to be edited based on the pre-saved voice information of the main speaker as a reference. Through this refined analysis process, the model can accurately extract the unique voice of the main speaker from the complex audio environment and achieve efficient extraction and separation of the sound.

[0041] Frame the voice of the main speaker, and convert each frame of the audio waveform into audio features using Linear Prediction Cepstral Coefficients (LPCC) and Mel-Frequency Cepstral Coefficients (MFCC), and input them into the audio editing model.

[0042] Using the acoustic module and language module in the audio editing model, each frame of audio information can be converted into text.

[0043] The acoustic module is used to process the audio features of each frame of audio, and output the phoneme information in the audio, that is, the smallest speech unit, which carries the pronunciation information of the speech. For example, in Chinese pinyin, "ma", in this syllable, "m" and "a" are two different pronunciation actions, so these are two different phonemes. The acoustic module can be obtained through model training. Before training the acoustic module, a large amount of audio data needs to be collected, and these audio data cover different speakers, speech rates, intonations, etc. Extract the features of the audio data, and correspond and label the audio features with phonemes, and input the labeled results into the model for training.

[0044] Based on the factor information recognized by the acoustic module, retrieve the corresponding characters or words in the character library. For Chinese, retrieve the Chinese characters corresponding to the pinyin; for English, retrieve the words corresponding to the phonetic symbols.

[0045] The language module is used to evaluate the association probability between each character or word and the phoneme to determine the finally recognized audio text. The language module can be obtained through model training. Before training the language module, a large amount of text data needs to be collected, and these text data cover news reports, literary works, social media posts, etc. Preprocess the text data, including word segmentation, duplicate removal, etc., label the content in the processed text data with the corresponding phonemes, and input the labeled results into the model for training.

[0046] The audio editing model determines the corresponding editing suggestions according to the recognized audio text and the editing instructions. For example, the user records an audio about a historical event, and the content of the audio text after recognition is "September 2, 1945, is the day when the second zhang ended", and the editing instruction proposed by the user is "Help me correct the typos in this sentence". The audio editing model outputs the editing suggestion according to the audio text and the editing instruction as "The typo in this sentence is 'zhang', which can be modified to 'zhan'".

[0047] Or, a user records an audio of a self-introduction, and the recognized audio text content is "Hello everyone, I am XX", and the user's editing instruction is "Help me make this sentence smooth", and the audio editing model outputs the editing suggestion "This audio can be modified to 'Hello everyone, I am XX' and 'Hello, I am XX'" based on the audio text and the editing instruction. The user can select the appropriate editing suggestion according to actual needs.

[0048] In the embodiment of the present application, the audio content can be accurately converted into text, providing a reliable basis for subsequent editing work. At the same time, the audio editing model can perform intelligent analysis and give reasonable editing suggestions, reducing the possibility of human judgment errors.

[0049] In some implementations, the editing instruction includes at least one of a content editing instruction and a style editing instruction.

[0050] In some implementations, the editing suggestions include at least one of editing suggestions for audio content and editing suggestions for audio style.

[0051] In the embodiment of the present application, the audio editing model can understand the user's content editing instructions and style editing instructions. Among them, the content editing instructions include a series of instructions for modifying or supplementing the content spoken by the user.

[0052] Style editing instructions refer to instructions for changing the audio style, where audio styles include but are not limited to music styles (such as classical, rock, jazz, electronic music, etc.), emotional expression styles (such as cheerful, sad, calm, etc.), age styles (such as retro, modern, futuristic, etc.), and specific scene atmospheres (such as movie theater sound effects, outdoor natural sounds), etc. The audio editing model can perform subsequent audio editing work based on the style editing instructions provided by the user, or automatically analyze and match the audio style that best matches the audio to be edited.

[0053] The audio editing model can generate corresponding editing suggestions based on the user's content editing instructions and style editing instructions. Using the above method, the editing of audio content and its style can be realized.

[0054] In some embodiments, the audio editing model modifies the text based on the text and the content editing instructions; and generates editing suggestions for the audio content based on the modified text and the timestamps of each part of the text; and / or,

[0055] The audio editing model determines the domain to which the text corresponds; and based on the domain and style editing instructions, generates editing suggestions tailored to the audio style.

[0056] For editing suggestions for audio content, for example, a piece of audio recorded by a user is "I went to the frying market yesterday and bought some water pots", and the user provides the instruction "Modify the typos in the audio". The audio editing model simultaneously recognizes the audio text, finds the typos in the text, modifies them in the text, and outputs the editing suggestion in combination with the timestamp where the typos are located as "Timestamp: 1703213598, the typo 'frying' can be modified to 'chaoshi'; Timestamp 1703213612, the typo 'pots' can be modified to 'fruits'".

[0057] For editing suggestions for audio style, for example, the user is recording a short course audio, and the audio content may be "Today we will learn about Newton's First Law in physics", and the user provides the instruction "Match the audio with a suitable style". By recognizing the audio text, the audio editing model determines that the audio is used in the education field from the audio text, and according to this field, the audio editing model outputs the style editing suggestion for the audio as "This audio is a teaching audio, and the audio style can be set to 'quiet'".

[0058] By using the above method, the accuracy and editing efficiency of audio editing can be improved. At the same time, this method supports domain-specific and stylized audio editing, enhancing the user experience.

[0059] In some embodiments, the editing material includes at least one of the content to be supplemented and the audio template;

[0060] The middleware determines the corresponding editing material according to the editing suggestion, including:

[0061] The middleware searches for the content to be supplemented from the pre-set content library according to the editing suggestion for the audio content; and / or,

[0062] The middleware searches for the corresponding audio template from the pre-set audio template library according to the editing suggestion for the audio style.

[0063] By using the above method, it is possible to search for and supplement the required content or template from the pre-set content library or audio template library, greatly reducing the time for users to manually search and filter, thereby improving the editing efficiency of audio content.

[0064] In some embodiments, the prompt includes at least one of a first indication, a second indication, and a third indication. Among them, the first indication is used to instruct the audio editing model to modify at least one of the style and background sound of the audio to be edited according to the audio template; the second indication is used to instruct the audio editing model to expand the audio to be edited based on the content to be supplemented; the third indication is used to instruct the audio editing model to correct the errors in the audio to be edited.

[0065] The above method can improve the accuracy of audio editing and enhance the integrity of audio content. At the same time, this method promotes the automated audio editing process and reduces the need for manual intervention.

[0066] The disclosed embodiment also provides an audio editing system, which includes middleware and an audio editing model; wherein the audio editing model can be a general large language model, which is applied to multiple fields. Figure 2 It is a schematic diagram of the content delivery method in the audio editing method proposed in the embodiment of the present disclosure.

[0067] S201, input the audio to be edited and the editing instruction into the middleware.

[0068] After the audio to be edited and the editing instructions are input into the middleware, the middleware can pre-process the audio, such as noise reduction, volume adjustment, etc., and convert the audio to be edited and the editing instructions into a format that can be understood by the audio editing model, so that the audio to be edited and the editing instructions can be input into the audio editing model to perform the corresponding tasks.

[0069] S202: The middleware inputs the audio to be edited and the editing instructions into the audio editing macro model.

[0070] S203: The audio editing model identifies the audio to be edited and generates an audio text.

[0071] Based on the specific content of the audio text and the editing instructions provided by the user, it deeply analyzes all aspects of the audio and generates a series of targeted editing suggestions based on the analysis results, which are fed back to the middleware. These suggestions not only cover how to improve the audio content, but also how to adjust the audio style to meet specific emotional or atmospheric requirements.

[0072] S204: The middleware queries the content library or audio template library for relevant content based on the editing suggestions fed back.

[0073] Specifically, if the editing suggestion includes modification or expansion of the audio content, the middleware uses the modified or expanded content as the query basis to query the content library for relevant content; if the editing suggestion includes modification of the audio template, the middleware uses the modification suggestion of the audio template as the query basis to query the audio template library for relevant templates.

[0074] S205: The content library or the audio template library feeds back the relevant materials queried by the middleware to the middleware.

[0075] S206: The middleware inputs the prompt words and editing materials into the audio editing model.

[0076] S207: The audio editing model processes the audio to be edited according to the received prompt words and editing materials to obtain the edited audio.

[0077] Based on Figure 2 In the specific mode of content transfer in the audio editing method shown, the following examples can be used to deeply understand and practice this process:

[0078] Example 1: A user records an explanatory audio about a historical event, with the content being "During a period of one zhang from 1939 to 1945, at that time, well, people experienced a lot of difficulties, and then, well, they started looking for solutions". From the above audio content, it can be seen that the user wrongly corresponded the time with the historical event, there are typos in the audio, and the sentence expression is not smooth. The user can input the audio editing instruction "Correct the typos in the audio and make this sentence smooth" into the audio editing model.

[0079] After receiving the instruction "Correct the typos in the audio and make this sentence smooth" and identifying the audio content, the audio editing model can provide suggestions for the edited content of the audio as "At timestamp 1703211378, the typo 'zhang' can be changed to 'zhan'; at timestamp 1703211250, modify the historical event corresponding to 1939 - 1945".

[0080] After receiving the editing suggestions provided by the audio editing model as "At timestamp 1703211378, the typo 'zhang' can be changed to 'zhan'; at timestamp 1703211250, modify the historical event corresponding to 1939 - 1945", the middleware queries the historical event that fits the time period of "1939 - 1945" from the content library, and the editing material feedback by the content library to the middleware is "During the period from 1939 to 1945, the world experienced World War II".

[0081] According to the actual situation of this example, the middleware sends the queried editing material and the third instruction to the audio editing model, and the result after being processed by the audio editing model is "During World War II from 1939 to 1945, people encountered unprecedented hardships, and then, they began to actively seek various solutions to cope with the difficulties".

[0082] Example 2: A user records audio content as "We came to AA County today", and the user can provide the instruction "At timestamp 170321256, supplement the location information of AA County" to the audio editing model.

[0083] After receiving the instruction "At timestamp 170321256, supplement the information of AA County" and identifying the audio content, the audio editing model can provide suggestions for the edited content of the audio as "Relevant geographical information can be supplemented before 'AA County'".

[0084] The middleware receives the editing suggestion provided by the audio editing model as "relevant geographical information can be added before 'AA County'". The middleware queries the geographical information about "AA City" from the content library, and the editing material fed back by the content library to the middleware is "AA County, located in FF City, YY Province".

[0085] According to the actual situation of this example, the middleware sends the queried editing material and the second instruction to the audio editing model. The result after processing by the audio editing model is "We came to AA County, FF City, YY Province today."

[0086] Example 3: A user records an audio clip to congratulate a store on its grand opening. The user can input the style editing instruction “change the style of this audio clip to a cheerful style” into the audio editing model.

[0087] After receiving the instruction “change the style of this audio to cheerful style” and identifying the audio content, the audio editing model can make a suggestion for the audio style as “cheerful style is suitable for this audio”.

[0088] The middleware receives the editing suggestion provided by the audio editing model as "cheerful style is suitable for this audio", and the middleware queries the audio template of cheerful style from the audio template library. Since the audio template library feeds back more than one audio template to the middleware, the middleware can evaluate the queried audio template according to the audio content. The evaluation process can be based on multiple factors, such as the similarity between the audio template and the original audio content, the applicability of the audio template, the style matching degree, and the user preference. By comprehensively analyzing these factors, the middleware can intelligently select the audio template that best suits the current audio content.

[0089] According to the actual situation of this example, the middleware sends the queried editing material and the first instruction to the audio editing model, and the audio editing model merges the audio template with the audio to be edited to obtain the edited audio.

[0090] After the above steps, it may happen that the editing material cannot be found. For the query failure, these situations can be recorded, including the query keywords, time and other information, and this information can be fed back to the maintenance personnel to supplement the missing data.

[0091] Under extreme conditions (such as high noise environments), the audio editing model may not be able to recognize the audio recorded by the user, so it is necessary to fine-tune the audio editing model. The model fine-tuning process includes:

[0092] Collect audio data recorded by users under extreme conditions, which covers different extreme conditions. Detailedly annotate the collected audio data, clearly mark the time period of the main speaker, the main speaker's speech content and other information;

[0093] Based on the annotation information, a series of targeted prompts are designed. These prompts can guide the audio editing model to focus on the vocal channel characteristics of the main speaker and prompt the audio editing model to identify the main speaker's speech content. The prompt may include a description of the audio characteristics (such as "extract clear human voice in a noisy environment"), specific instructions for the target channel (such as "focus on the main speaker's vocal channel"), and the desired output format (such as "separate and clarify the main speaker's audio").

[0094] The middleware is used to obtain the editing suggestions of the audio editing model for the audio data under extreme conditions, and the audio data under extreme conditions, the designed prompt and the information provided by the middleware are input into the audio editing model. The model continuously attempts to separate the voice of the main speaker and the speech content of the main speaker from the audio data, and continuously optimizes its internal parameters through careful comparison with the annotation information to achieve a more accurate editing effect. The embodiment of the present disclosure also proposes an audio editing system, Figure 3 is a schematic diagram of the structure of an audio editing system 300 according to an embodiment of the present disclosure, including a middleware 310 and an audio editing model 320: wherein,

[0095] The middleware 310 is used to input the audio to be edited and the editing instruction into the audio editing model, receive the editing suggestion returned by the audio editing model; determine the corresponding editing material according to the editing suggestion, construct the prompt word according to the editing suggestion and the editing material, and input the prompt word and the editing material into the audio editing model;

[0096] The audio editing model 320 is used to determine corresponding editing suggestions based on the audio to be edited and the editing instructions, and to feed back the editing suggestions to the middleware; it is also used to receive prompt words and editing materials, and edit the audio to be edited based on the prompt words and the editing materials to obtain the edited audio.

[0097] In some implementations, the audio editing model 320 is used to:

[0098] Identify the audio to be edited to determine the text corresponding to the audio to be edited;

[0099] Based on the text and the editing instructions, corresponding editing suggestions are determined.

[0100] In some implementations, the editing instruction includes at least one of a content editing instruction and a style editing instruction.

[0101] In some implementations, the editing suggestions include at least one of editing suggestions for audio content and editing suggestions for audio style.

[0102] In some implementations, the audio editing model 320 is used to:

[0103] Based on the text and content editing instructions, modify the text; and based on the modified text and the timestamps of the parts of the text, generate editing suggestions for the audio content; and / or,

[0104] Determine the domain to which the text corresponds; and generate editing suggestions for the audio style based on the domain and style editing instructions.

[0105] In some implementations, the editing material includes at least one of content to be supplemented and an audio template;

[0106] The middleware 310 is used to search for content that needs to be supplemented from a pre-set content library according to the editing suggestion for the audio content; and / or,

[0107] Based on editing suggestions for your audio style, search for corresponding audio templates from a library of pre-set audio templates.

[0108] In some embodiments, the prompt word includes at least one of the first indication, the second indication, and the third indication:

[0109] The first instruction is used to instruct the audio editing model to modify at least one of the style and background sound of the audio to be edited according to the audio template;

[0110] The second instruction is used to instruct the audio editing model to expand the audio to be edited based on the content that needs to be supplemented;

[0111] The third instruction is used to instruct the audio editing model to correct errors in the audio to be edited.

[0112] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0113] In the technical solution disclosed in the present invention, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0114] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0115] Figure 4A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0116] like Figure 4 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0117] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0118] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0119] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An audio editing method, applied to an audio editing system including middleware and an audio editing model, the method comprising: The middleware inputs the audio to be edited and the editing instruction into the audio editing model; The audio editing model determines corresponding editing suggestions based on the audio to be edited and the editing instruction, and feeds back the editing suggestions to the middleware; The middleware determines corresponding editing materials according to the editing suggestion, constructs prompt words according to the editing suggestion and the editing materials, and inputs the prompt words and the editing materials into the audio editing model; The audio editing model edits the audio to be edited based on the prompt word and the editing material to obtain the edited audio.

2. The method according to claim 1, wherein: The audio editing model determines corresponding editing suggestions based on the audio to be edited and the editing instruction, including: The audio editing model identifies the audio to be edited to determine the text corresponding to the audio to be edited; The audio editing model determines the corresponding editing suggestion based on the text and the editing instruction.

3. The method according to claim 2, wherein: The editing instruction includes at least one of a content editing instruction and a style editing instruction.

4. The method according to claim 2 or 3, wherein: The editing suggestions include at least one of editing suggestions for audio content and editing suggestions for audio style.

5. The method according to claim 4, wherein: The audio editing model determines the corresponding editing suggestion based on the text and the editing instruction, including: The audio editing model modifies the text based on the text and the content editing instruction; and generates the editing suggestion for the audio content based on the modified text and the timestamps of each part of the text; and / or, The audio editing model determines the domain to which the text corresponds; and generates the editing suggestion for the audio style according to the domain and the style editing instruction.

6. The method according to claim 5, wherein: The editing material includes at least one of the content to be supplemented and the audio template; The middleware determines corresponding editing materials according to the editing suggestion, including: The middleware searches for the content to be supplemented from a preset content library according to the editing suggestion for the audio content; and / or, The middleware searches for a corresponding audio template from a preset audio template library according to the editing suggestion for the audio style.

7. The method according to claim 6, wherein: The prompt word includes at least one of the following: a first instruction, wherein the first instruction is used to instruct the audio editing model to modify at least one of the style and background sound of the audio to be edited according to the audio template; A second indication, wherein the second indication is used to instruct the audio editing model to expand the audio to be edited based on the content that needs to be supplemented; A third indication is used to instruct the audio editing model to correct errors in the audio to be edited.

8. An audio editing system, the system comprising middleware and an audio editing model; wherein: The middleware is used to input the audio to be edited and the editing instruction into the audio editing model, and receive the editing suggestion returned by the audio editing model; Determine corresponding editing materials according to the editing suggestion, construct prompt words according to the editing suggestion and the editing materials, and input the prompt words and the editing materials into the audio editing model; The audio editing model is used to determine the corresponding editing suggestions based on the audio to be edited and the editing instructions, and to feed back the editing suggestions to the middleware; it is also used to receive the prompt words and the editing materials, and edit the audio to be edited based on the prompt words and the editing materials to obtain the edited audio.

9. The system according to claim 8, wherein: The audio editing model is used to: Identify the audio to be edited to determine the text corresponding to the audio to be edited; Based on the text and the editing instruction, the corresponding editing suggestion is determined.

10. The system according to claim 9, wherein: The editing instruction includes at least one of a content editing instruction and a style editing instruction.

11. The system according to claim 9 or 10, wherein: The editing suggestions include at least one of editing suggestions for audio content and editing suggestions for audio style.

12. The system according to claim 11, wherein: The audio editing model is used to: Based on the text and the content editing instruction, modify the text; and based on the modified text and the timestamps of each part of the text, generate the editing suggestion for the audio content; and / or, Determining the domain to which the text corresponds; And based on the domain and the style editing instruction, the editing suggestion for the audio style is generated.

13. The system according to claim 12, wherein: The editing material includes at least one of the content to be supplemented and the audio template; The middleware is used to: According to the editing suggestion for the audio content, searching for the content to be supplemented from a pre-set content library; and / or, According to the editing suggestion for the audio style, a corresponding audio template is searched from a preset audio template library.

14. The system according to claim 13, wherein: The prompt word includes at least one of the following: a first instruction, wherein the first instruction is used to instruct the audio editing model to modify at least one of the style and background sound of the audio to be edited according to the audio template; A second indication, wherein the second indication is used to instruct the audio editing model to expand the audio to be edited based on the content that needs to be supplemented; A third indication is used to instruct the audio editing model to correct errors in the audio to be edited.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images