Audio Processing Method, Electronic Device, and Computer-Readable Storage Medium

Abstract

This application relates to the field of audio processing, and in particular, to an audio processing method, an electronic device, and a computer-readable storage medium. The method includes: obtaining an audio to be processed and a target theme, where the target theme represents the type of sound included in the audio; performing audio processing on the audio to be processed according to the target theme to obtain an audio segment that matches the target theme; generating a target audio according to the audio segment and a preset background music. Through the above method, it is possible to extract audio segments with editing value, which is beneficial to improving the audio effect of the video editing finished product.

Description

Technical Field

[0001] This application relates to the field of audio processing, and in particular, to an audio processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Using a mobile phone to record every bit of life has become a common function for many end users. The mobile phone can provide a function of automatically editing and generating a recording short film based on a batch of materials, which is convenient for users to share their life records on social platforms such as short videos. During the process of generating the recording short film, the background music will replace the original audio of the materials, resulting in a poor effect of the finished film. Summary of the Invention

[0003] This application provides an audio processing method, an electronic device, and a computer-readable storage medium, which can effectively improve the audio effect of the video editing finished film.

[0004] To achieve the above object, this application adopts the following technical solutions:

[0005] In a first aspect, an audio processing method is provided. The method includes:

[0006] Obtain the audio to be processed and the target theme, where the target theme represents the type of sound included in the audio;

[0007] Perform audio processing on the audio to be processed according to the target theme to obtain an audio segment that matches the target theme;

[0008] Generate a target audio according to the audio segment and the preset background music.

[0009] In the embodiments of this application, an audio segment with editing value can be extracted, and then the original sound of this segment can be intelligently retained during the process of forming the finished film, thereby improving the effect of the "one-key blockbuster" video.

[0010] In an implementation manner of the first aspect, the performing audio processing on the audio to be processed according to the target theme to obtain an audio segment that matches the target theme includes:

[0011] Obtain the first probability corresponding to each audio frame in the audio to be processed according to the trained first model, where the first probability corresponding to the audio frame represents the probability that the audio frame contains an audio signal belonging to the target theme;

[0012] Obtain the audio segment from the audio to be processed according to the first probability corresponding to each audio frame.

[0013] In the embodiment of the present application, a first model is pre-trained. The first model can be a neural network model or other algorithm models capable of implementing audio detection.

[0014] In an implementation manner of the first aspect, obtaining the first probability corresponding to each audio frame in the audio to be processed according to the trained first model includes:

[0015] Input the audio to be processed into the trained first model, and output the second probability corresponding to each audio point in the audio to be processed, where the second probability corresponding to the audio point represents the probability that the audio point contains an audio component belonging to the target theme;

[0016] Calculate the first probability corresponding to each audio frame in the audio to be processed according to the second probability corresponding to each audio point.

[0017] In the embodiment of the present application, training the first model in units of audio points is beneficial to improving the detection accuracy of the first model.

[0018] In an implementation manner of the first aspect, calculating the first probability corresponding to each audio frame in the audio to be processed according to the second probability corresponding to each audio point includes:

[0019] For each audio frame, perform weighted summation processing according to the second probabilities corresponding to the audio points included in the audio frame to obtain the first probability corresponding to the audio frame.

[0020] For example, through the formula Calculate the first probability of each audio frame, where p(f) is the second probability of the f-th audio point and p is the first probability.

[0021] In the embodiment of the present application, the first probability of the audio frame calculated through the weighted summation of the second probabilities of the audio points in the audio frame can reflect the comprehensive level of each audio point, which is beneficial to improving the effect of subsequent audio processing.

[0022] In an implementation manner of the first aspect, calculating the first probability corresponding to each audio frame in the audio to be processed according to the second probability corresponding to each audio point includes:

[0023] Divide at least one frequency band according to the sound frequency corresponding to the target theme;

[0024] Calculate the first probability of the audio frame according to the second probabilities of the audio points included in each frequency band in the audio frame.

[0025] Among them, one way to divide frequency bands is to determine a frequency range according to the maximum and minimum values of the sound frequencies corresponding to the target theme, and divide the frequency range into at least one frequency band.

[0026] Specifically, for each audio frame, calculate the segmentation probability according to the second probability of the audio points belonging to the same frequency band in the audio frame; calculate the first probability according to the calculated segmentation probability corresponding to each frequency band.

[0027] For example, through the formula calculate the first probability of each audio frame, where F1 represents the number of audio points belonging to the first frequency band (such as the low frequency band in the above example) in the audio frame, F2 represents the number of audio points belonging to the second frequency band (such as the middle frequency band in the above example) in the audio frame, and F3 represents the number of audio points belonging to the second frequency band (such as the high frequency band in the above example) in the audio frame. represents the segmentation probability corresponding to the jth frequency band.

[0028] In the embodiments of the present application, by determining the screening thresholds for different frequency bands respectively, it is possible to more accurately distinguish the audio of the target theme from the audio of non-target themes within each frequency band, which is beneficial to improving the accuracy of audio processing.

[0029] In one implementation manner of the first aspect, the obtaining the audio segment from the audio to be processed according to the first probability corresponding to each audio frame includes:

[0030] If the first probability corresponding to the audio frame is greater than the first threshold, then determine the audio frame as an alternative frame;

[0031] Determine the audio segment according to the alternative frames in the audio to be processed.

[0032] In this way, the first threshold can effectively filter out the audio that does not match the target theme, thereby being beneficial to improving the effect of subsequent audio processing.

[0033] In one implementation manner of the first aspect, the determining the audio segment according to the alternative frames in the audio to be processed includes:

[0034] Generate a candidate segment according to each group of consecutive alternative frames in the audio to be processed;

[0035] Determine the first audio duration according to the target theme;

[0036] Determine the target segment from the candidate segments according to the first audio duration;

[0037] Determine the audio segment according to the target segment.

[0038] In the embodiments of the present application, through the first audio duration, candidate segments with shorter durations can be effectively filtered out, thereby effectively reducing the false detection probability of candidate frames and facilitating the improvement of the accuracy of audio processing.

[0039] In one implementation manner of the first aspect, determining the target segment from the candidate segments according to the first audio duration includes:

[0040] If the time interval between the first segment and the second segment is less than the third threshold, then the first segment, the second segment, and the audio frames between the first segment and the second segment are merged into one target segment, where the first segment is one target segment and the second segment is the subsequent target segment of the first segment.

[0041] In the embodiments of the present application, through the third threshold, discrete candidate segments with short interval times can be combined into continuous audio frames, which can effectively reduce the non - continuity influence caused by factors such as sentence pauses, thereby improving the continuity of audio segments and facilitating the improvement of user experience.

[0042] In one implementation manner of the first aspect, determining the audio segment according to the target segment includes:

[0043] Calculate the first audio ratio corresponding to each target segment, where the audio ratio represents the ratio of the audio component belonging to the target theme in the target segment to the audio component not belonging to the target theme;

[0044] Determine the audio segment according to the target segment corresponding to the first audio ratio greater than the fourth threshold.

[0045] In this implementation manner, the first audio ratio can be used to represent the ratio of the audio component belonging to the target theme in the target segment to the audio component not belonging to the target theme. By using this parameter to evaluate the editing value of the target segment, it is beneficial to improve the effect of audio processing.

[0046] In one implementation manner of the first aspect, calculating the first audio ratio corresponding to each target segment includes:

[0047] For each target segment, calculate the second audio ratio of each audio frame in the target segment;

[0048] Calculate the first audio ratio corresponding to the target segment according to the second audio ratio of each audio frame in the target segment.

[0049] After calculating the second audio ratio of each audio frame, the average value of all the second audio frames in the target segment can be calculated as the first audio ratio of the target segment.

[0050] In this implementation manner, the first audio proportion of the target segment can reflect the average level of the audio proportions of each audio frame in the target segment, which is beneficial to improving the audio processing effect.

[0051] In one implementation manner of the first aspect, the determining the audio segment according to the target segment includes:

[0052] Calculating the first loudness corresponding to each target segment;

[0053] Determining the audio segment according to the target segment corresponding to the first loudness greater than the fifth threshold.

[0054] In this implementation manner, the first loudness can be used to evaluate the sound intensity or volume of the target segment, so as to filter out the audio with relatively small sound intensity or volume, which is beneficial to improving the audio processing effect.

[0055] In other implementation manners, the audio segment is determined according to the first audio proportion and the first loudness. Specifically, if the first audio proportion of the target segment is greater than the fourth threshold and the first loudness is greater than the fifth threshold, then the target segment is determined as the audio segment.

[0056] In one implementation manner of the first aspect, the generating the target audio according to the audio segment and the preset background music includes:

[0057] Obtaining the timestamp of the audio segment in the audio to be processed;

[0058] Filtering the background music according to the timestamp to obtain the filtered background music;

[0059] Generating the target audio according to the audio segment and the filtered background music.

[0060] Wherein, the timestamp includes the first moment corresponding to the start moment of the audio segment in the audio to be processed and the second moment corresponding to the end moment of the audio segment in the audio to be processed.

[0061] In the embodiments of the present application, the audio segment and the background music are organically combined according to the timestamp, effectively reducing the pauses between audios, making the generated target audio more fluent, which is beneficial to improving the user experience.

[0062] In one implementation manner of the first aspect, the steps of obtaining the target theme include:

[0063] Identifying the first theme corresponding to the audio to be processed, where the first theme represents the type of sound included in the audio;

[0064] Displaying a first interface, where the first interface includes a first control corresponding to each first theme;

[0065] In response to the user's operation on the first control, determine the first theme corresponding to the first control operated by the user as the target theme.

[0066] In the embodiments of the present application, after identifying the audio theme included in the audio to be processed, the user can independently select the audio theme, which not only enables the audio original sound matching the audio theme selected by the user to be retained in the finished film, but also effectively reduces the probability of audio processing failure, which is beneficial to improving the user experience.

[0067] In one implementation manner of the first aspect, the steps of obtaining the target theme include:

[0068] Display a second interface, where the second interface includes a plurality of preset second themes and second controls corresponding to each second theme, and the second theme represents the type of sound included in the audio;

[0069] In response to the user's operation on the second control, determine the second theme corresponding to the second control operated by the user as the target theme.

[0070] In the embodiments of the present application, the user can independently select the audio theme, so that the audio original sound matching the audio theme selected by the user is retained in the finished film, which is beneficial to improving the user experience.

[0071] In one implementation manner of the first aspect, the method further includes:

[0072] Obtain an audio sample and a noise audio;

[0073] Perform audio mixing processing on the audio sample and the noise audio to obtain a mixed audio;

[0074] Train the first model according to the mixed audio.

[0075] Among them, the noise audio can be Gaussian white noise or random white noise, or an interference audio including the audio theme corresponding to the audio sample, or an interference audio not including the audio theme corresponding to the audio sample.

[0076] One implementation manner of the mixing processing can be to superimpose the corresponding audio points in the audio sample and the noise audio.

[0077] In the embodiments of the present application, by adding the noise audio, the trained first model can more accurately identify the audio frames in the audio that do not belong to the audio theme, thereby effectively improving the detection accuracy of the first model.

[0078] In one implementation manner of the first aspect, the training of the first model according to the mixed audio includes:

[0079] Obtain the first component of each audio point of the mixed audio, where the first component is the audio component belonging to the third theme in the audio point, and the third theme represents the type of sound included in the audio sample;

[0080] Input the mixed audio into the first model, and output the third probability corresponding to each audio point in the mixed audio, where the third probability corresponding to the audio point represents the probability that the audio point contains an audio component belonging to the third theme;

[0081] Calculate a loss value according to the first component of each audio point of the mixed audio and the third probability corresponding to each audio point in the mixed audio;

[0082] Adjust the model parameters of the first model according to the loss value.

[0083] In the embodiments of the present application, through mixing processing, for each audio sample, multiple mixed audios can be constructed by mixing different noisy audios. In the above manner, not only the operation of constructing labels for the audio used to train the first model is reduced, but also data augmentation can be achieved through the mixing processing method, improving the scalability of the model.

[0084] In a second aspect, a chip system is provided. The chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the method according to any one of the first aspect.

[0085] In a third aspect, an electronic device is provided. The electronic device includes a processor, and the processor is configured to run a computer program stored in a memory so that the electronic device implements the method according to any one of the first aspect.

[0086] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method according to any one of the first aspect is implemented.

[0087] In a fifth aspect, a computer program product is provided. When the computer program product runs on an electronic device, the electronic device can implement the method according to any one of the first aspect. Description of the Drawings

[0088] Figure 1 is a schematic process diagram of the "one-key blockbuster" provided by the embodiments of the present application;

[0089] Figure 2 is a schematic structural diagram of an electronic device provided by the embodiments of the present application;

[0090] Figure 3It is a schematic diagram of the application scenario for audio theme selection provided by an embodiment of the present application;

[0091] Figure 4 It is a schematic diagram of the application scenario for audio theme selection provided by an embodiment of the present application;

[0092] Figure 5 It is a schematic flowchart of the audio processing method provided by an embodiment of the present application;

[0093] Figure 6 It is a schematic diagram of the timestamp provided by an embodiment of the present application;

[0094] Figure 7 It is a schematic diagram of the timestamp provided by an embodiment of the present application;

[0095] Figure 8 It is a schematic diagram of the alternative frame provided by an embodiment of the present application. Detailed implementation manners

[0096] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details.

[0097] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0098] It should also be understood that in the embodiments of the present application, "one or more" means one, two, or more than two; "and / or" describes the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0099] In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0100] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that specific features, structures, or characteristics described in connection with that embodiment are included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0101] Using a mobile phone to record every bit of life has become a common function for many end users. Facing a large number of recorded materials, the "one-click blockbuster" function can automatically edit and generate recorded short videos, facilitating users to share their life records on social platforms such as short videos.

[0102] Exemplarily, refer to Figure 1 , which is a process schematic diagram of the "one-click blockbuster" provided by an embodiment of this application. As Figure 1 shown in (a) of [], it is the interface 10 of the "one-click blockbuster". The interface 10 includes a "one-click blockbuster" control 101. In response to the user's operation on the "one-click blockbuster" control 101 (such as clicking, double-clicking, etc.), the mobile phone jumps to the interface 11 shown in (b) of Figure 1 . The interface 11 is a thumbnail interface of the photo gallery. The interface 11 may include thumbnails of pictures, animated pictures, and videos in the mobile phone photo gallery application.

[0103] The user can select one or more materials for editing. As Figure 1 shown in (b) of [], a "select" identifier 102 is displayed on each thumbnail in the interface 11. When not selected, the "select" identifier 102 is in a first form. As shown in the interface 11, the "select" identifier 102 is a centered circle. When selected, the "select" identifier 102 is in a second form. As shown in the interface 12, the "select" identifiers 102 on the thumbnail 103 and the thumbnail 104 are solid circles, and a "√" pattern is marked in the solid circle. Through the change in the form presented by the "select" identifier, the user can understand the current selection situation. Of course, the above is only an example of the form change of the "select" identifier. In actual applications, other forms can also be set, and the embodiment of this application does not specifically limit the form display effect of the "select" identifier.

[0104] The thumbnail interface may include a selection area, and the selected thumbnails are displayed in the selection area. As Figure 1As shown in (b) therein, the interface 11 includes a selection area 105. When the user does not select any material, no thumbnail is displayed in the selection area 105. When the user selects a material, the selected thumbnail of the user is displayed in the selection area 105. As Figure 1 In the interface 12 shown in (c) therein, the thumbnail 103 and the thumbnail 104 are selected. Correspondingly, the thumbnail 103 and the thumbnail 104 are displayed in the selection area 105. The selection area in the thumbnail interface may include editing controls for triggering the mobile phone to edit according to the selected materials. As Figure 1 As shown in (b) therein, the interface 11 includes a "one - key blockbuster" control 106. When no thumbnail is displayed in the selection area 105 (that is, the user does not select any material), the "one - key blockbuster" control 106 is in an inoperable mode. As Figure 1 As shown in (c) therein, when a thumbnail is displayed in the selection area 105 (that is, the user has selected a material), the "one - key blockbuster" control 106 is in an operable mode.

[0105] The mobile phone can generate a video according to the materials selected by the user. As Figure 1 As shown in (c) therein, the user operates the "one - key blockbuster" control 106. In response to this user operation, the mobile phone generates a video according to the currently selected materials and jumps to the interface 13 shown in (d) therein. Figure 1 This interface 13 is an editing interface for the video. The user can edit the generated video through the controls in the interface 13.

[0106] In the embodiments of the present application, "one - key blockbuster" can edit pictures, videos, and animated pictures. The multiple materials selected by the user may only include one type of material, such as all pictures, or all videos, or all animated pictures. The multiple materials selected by the user may also include multiple types of materials. For example, it includes pictures and videos, or includes pictures and animated pictures, or includes videos and animated pictures, or includes pictures, videos, and animated pictures.

[0107] Each effect template in the "one - key blockbuster" has a combination of background music, filters, special effects (such as twinkling stars), transitions, photo frames, stickers, etc., which can be simply referred to as video effects. That is to say, each effect template corresponds to a set of video effects. Correspondingly, when the mobile phone uses the effect template, it can add the corresponding video effects to the video. The mobile phone responds to the user's operation on the "one - key blockbuster" control, uses the effect template to process the selected materials, and forms a finished video.

[0108] However, the videos generated by the current "One - click Blockbuster" do not contain the original video audio, only the background music in the effect template, which greatly reduces the display effect of the finished videos. If the original video audio and the background music are directly superimposed, it may lead to unclear audio mixing and the inability to highlight the importance of certain segments.

[0109] Based on this, the embodiments of the present application provide an audio processing method. In the embodiments of the present application, audio segments with editing value can be extracted, and then the original sound of these segments can be intelligently retained during the process of generating the finished video, thereby improving the effect of the "One - click Blockbuster" video.

[0110] The method provided by the embodiments of the present application can be applied to electronic devices with audio playback functions. Electronic devices include terminal devices, and terminal devices can also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Terminal devices can be mobile phones, smart TVs, wearable devices, tablets (Pad), smart screens, handheld computers, laptop computers, computers with wireless transceiver functions, ultra - mobile personal computers (UMPC), netbooks, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, and so on. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the electronic devices.

[0111] See Figure 2, which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, etc.

[0112] It can be understood that the structure schematically shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0113] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. For example, the processor 110 is used to execute the audio processing method in the embodiment of the present application.

[0114] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching instructions and executing instructions.

[0115] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can hold the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly retrieved from the memory. This avoids repeated accesses and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0116] The external memory 120 generally refers to external storage. In the embodiments of the present application, the external memory refers to the storage other than the internal memory of the electronic device and the cache of the processor, and this storage is generally a non-volatile memory.

[0117] The internal memory 121, which can also be referred to as "memory", can be used to store computer-executable program codes, and the executable program codes include instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.).

[0118] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. In some embodiments, the electronic device 100 displays a user interface through the display screen 194.

[0119] The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0120] The electronic device 100 can implement the shooting function through the ISP, camera 193, video codec, GPU, display screen 194, application processor, etc.

[0121] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV and other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1. Exemplarily, the camera 193 is used to capture an image of a user looking at the display screen 194.

[0122] The electronic device 100 also includes various sensors that can convert various different physical signals into electrical signals. Exemplarily, the pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. The barometric pressure sensor 180C is used to measure barometric pressure. The magnetic sensor 180D includes a Hall sensor. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance through infrared or laser. The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access to application locks, fingerprint photography, fingerprint answering of incoming calls, etc. The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 executes a temperature processing strategy using the temperature detected by the temperature sensor 180J. The bone conduction sensor 180M can acquire vibration signals.

[0123] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 together form a touch screen, also known as the "touch display screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from where the display screen 194 is located.

[0124] Exemplarily, in the embodiments of the present application, the touch sensor 180K can detect a user's click operation on an application icon, and transmit the detected click operation to the application processor to determine that the click operation is used to start or run the application program, and then execute the running operation of the application program.

[0125] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modulation and demodulation processor, and baseband processor, etc.

[0126] The electronic device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, etc. Such as music playback, recording, etc.

[0127] The above is a specific description of the embodiments of the present application taking the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0128] Next, the audio processing method of the embodiments of the present application will be described in detail with reference to the accompanying drawings, taking the electronic device as a mobile phone as an example.

[0129] Different end users have different definitions of audio clips with clipping value (which can be called audio highlights). For example: users who like music tend to focus on audio related to music such as concerts, singing shows, singing, etc. Users who like outdoor sports tend to focus on audio related to nature such as the sound of wind, birdsong, running water, etc. Users who like to shoot daily short videos tend to focus on audio related to human voices, such as the voices in scenarios like chatting, parties, classes, interviews, store visits, etc. To meet the needs of different users, the audio processing method in the embodiments of the present application can set multiple audio themes, and the mobile phone can process the original audio of the material according to the preset audio themes.

[0130] It can be understood that in the embodiments of the present application, an audio theme represents the type of sound included in the audio.

[0131] In some embodiments, a default audio theme can be set. In the implementation process, when the user selects a material, the mobile phone processes the original audio of the material according to the default audio theme to extract the audio highlights related to the audio theme from the original audio. For example, if the human voice is set as the default audio theme, the mobile phone extracts the audio highlights related to the human voice from the original audio of the material selected by the user.

[0132] In other embodiments, multiple audio themes can be preset. When "one - click video creation" is started, the user can independently select an audio theme from the preset multiple audio themes. Correspondingly, the mobile phone processes the original audio of the material selected by the user according to the audio theme selected by the user to extract the audio highlights related to the audio theme selected by the user from the original audio.

[0133] Exemplarily, see Figure 3 , which is a schematic diagram of the application scenario of audio theme selection provided by the embodiments of the present application. As shown in Figure 3 , the interface 31 shown in (a) includes a "one - click blockbuster" control 311. When the user selects a material and operates the "one - click blockbuster" control 311, in response to this user operation, the mobile phone jumps to the interface 32 shown in (b) of Figure 3 . This interface 32 (the second interface) includes selection controls 312 (the second controls) corresponding to multiple audio themes (the second themes) respectively. As shown in Figure 3 , in (b), the audio themes shown in the interface 32 include "human voice", "music" and "wind sound", and the selection controls 312 corresponding to each audio theme are default to the unselected state. The user can operate the selection controls 312 to select an audio theme. As shown in Figure 3 , in (b), when the user operates the selection control 312 of the "human voice" theme, in response to this user operation, the mobile phone jumps to Figure 3The interface 33 shown in (c) in [description]. As shown in the interface 33, the selection control 312 corresponding to the "human voice" theme has become the selected state. The interface 33 includes a "confirm" control 313. When the user operates the "confirm" 313, in response to this user operation, the mobile phone performs audio processing on the audio original of the material selected by the user according to the audio theme "human voice" selected by the user, so as to extract the audio highlight segments related to the "human voice" in the audio original.

[0134] Figure 3 In the embodiment, it is equivalent to the user selecting the audio theme of interest (changing from the unselected state to the selected state) through the operation, and then the mobile phone performs audio processing on the audio original according to the audio theme selected by the user.

[0135] It should be noted that Figure 3 This is only an example of the selection interface of the audio theme. In other application scenarios, there may be more or fewer audio themes. The embodiments of the present application do not specifically limit the type of the audio theme. In addition, in actual applications, the user can select one or more audio themes, and the embodiments of the present application do not specifically limit this.

[0136] In some other embodiments, multiple audio themes can be preset. After starting the "one-click video compilation", the mobile phone first performs audio recognition on the audio original of the material selected by the user to obtain the recognized audio theme; then provides the recognized audio theme for the user to select; after the user selects a certain audio theme, the mobile phone then performs audio processing on the audio original of the material selected by the user according to the audio theme selected by the user, so as to extract the audio highlight segments related to the audio theme selected by the user in the audio original.

[0137] Exemplarily, referring to Figure 4 , it is a schematic diagram of the application scenario of audio theme selection provided by the embodiments of the present application. As Figure 4 shown in (a) in [description], the interface 41 includes a "one-click blockbuster" control 411. When the user selects the material and operates the "one-click blockbuster" control 411, in response to this user operation, the mobile phone performs audio recognition on the audio original of the material already selected by the user and jumps to the interface 42 shown in (b) in [description]. The interface 42 (the first interface) includes at least one audio theme (the first theme) recognized by the mobile phone from the audio original and the corresponding selection controls 412 (the first controls) respectively. As Figure 4 shown in (b) in [description], the audio themes shown in the interface 42 include "human voice" and "music". The selection controls 412 of the recognized audio themes are default to the selected state. The user can operate the selection controls 412 to select the audio theme. As Figure 4 shown in (b) in [description], when the user operates the selection control 412 of the "human voice" theme (the target theme), in response to this user operation, the mobile phone jumps to Figure 4 as shown in (c) in [description].Figure 4 Interface 43 shown in (c) in FIG. As shown in interface 43, the selection control 412 corresponding to the "voice" theme becomes unselected. Interface 43 includes a "confirmation" control 413. When the user operates "confirmation" 413, in response to the user operation, the mobile phone performs audio processing on the audio soundtrack of the material selected by the user according to the audio theme "music" selected by the user, so as to extract the audio highlight segment related to "music" in the audio soundtrack.

[0138] Figure 4 In the example, it is equivalent to the user deleting the unnecessary audio theme through operation (changing from the selected state to the unselected state), and the mobile phone processes the original audio sound according to the retained (remaining selected) audio theme.

[0139] It should be noted that Figure 4 This is only an example of an audio theme selection interface. In other application scenarios, the mobile phone can identify different audio themes based on different audio original sounds. The embodiment of the present application does not specifically limit the type of audio theme. Secondly, in actual applications, the user can select one, multiple audio themes, or not select an audio theme. The embodiment of the present application does not specifically limit this. For example, in interface 43, the user can operate the selection controls corresponding to "voice" and "music" to an unselected state, which is equivalent to the user not selecting an audio theme. In this case, the mobile phone does not need to extract the audio highlight segments in the audio original sound. In addition, Figure 4 In the embodiment, after the mobile phone recognizes the audio theme, the selection control of the recognized audio theme in the interface 42 is in the selected state by default. In other application scenarios, the selection control of the recognized audio theme may also be in the unselected state by default. After the user operates the selection control of a certain audio theme, the selection control is switched to the selected state. This embodiment of the application is not specifically limited to this.

[0140] In other application scenarios, such as Figure 3 As shown in (c) of FIG. 1 , when the user selects an audio theme and operates the “confirm” control 313, in response to the user operation, the mobile phone can perform audio recognition based on the original audio of the material selected by the user and jump to the following Figure 4 The interface 42 shown in (b) in the figure allows the user to modify the audio theme through the interface 42. In this way, when the audio theme initially selected by the user does not exist in the original audio sound, the audio theme existing in the original audio sound can be automatically identified to allow the user to modify the selection, which effectively reduces the probability of audio processing failure and helps to improve the user experience.

[0141] In one implementation, a method for a mobile phone to identify an audio theme according to an original audio sound includes:

[0142] Obtain the trained audio recognition model; input the original audio into the trained audio recognition model, and output the audio theme corresponding to the original audio.

[0143] In this implementation method, it is necessary to pre-train an audio recognition model. In one example, the training process of the audio recognition model may include: obtaining multiple audio samples, each audio sample corresponding to a true label, and the true label is used to represent the audio theme to which the audio sample belongs; using the multiple audio samples to train the audio recognition model. Specifically, input the audio sample into the audio recognition model, and output the confidence that the audio sample belongs to the audio theme; calculate the loss value of the model according to the confidence; if the loss value is less than or equal to the preset loss, determine the current audio recognition model as the trained audio recognition model; if the loss value is greater than the preset loss, adjust the model parameters of the audio recognition model according to the loss value to obtain the adjusted audio recognition model, and continue to train the adjusted audio recognition model according to the audio sample until the loss value of the model is less than or equal to the preset loss, and obtain the trained audio recognition model.

[0144] It should be noted that the above is only an example of training the audio recognition model. In actual applications, other training methods can also be used, such as controlling the number of iterations, etc. The embodiments of the present application do not specifically limit the training process of the model.

[0145] It can be understood that the premise of the audio processing method in the embodiments of the present application is that the selected material by the user includes the original audio. For example, the selected material by the user may include pictures and videos, or may only include videos. If the selected materials by the user are all pictures, the mobile phone cannot recognize the audio theme of the original audio.

[0146] Based on the above selected audio theme, the following introduces the specific method for the mobile phone to perform audio processing on the original audio according to the audio theme to extract the audio highlight segments related to the selected audio theme in the original audio.

[0147] See Figure 5 , which is a schematic flowchart of the audio processing method provided by the embodiments of the present application. As an example rather than a limitation, as Figure 5 shown, the audio processing method may include the steps:

[0148] S501, obtain the audio to be processed and the target theme.

[0149] Among them, the target theme represents the type of sound included in the audio. As described in the above embodiment, one way to obtain the target theme is to set the default audio theme as the target theme; another way is to preset multiple audio themes, and the user selects the target theme from the multiple preset themes (such as Figure 3Example); in another way, the mobile phone identifies the audio theme of the audio to be processed, and the user selects a target theme from the identified audio themes (such as Figure 4 Example).

[0150] In the embodiments of the present application, the audio to be processed may be the original audio contained in the materials selected by the user. For example, in the process of "one-click blockbuster", if the user selects a video, the audio to be processed is the original audio in the video; if the user selects multiple videos, the audio to be processed is the audio obtained by splicing the original audio of each of the multiple videos according to the user's selection order.

[0151] S502. Perform audio processing on the audio to be processed according to the target theme to obtain an audio segment matching the target theme.

[0152] In the embodiments of the present application, the audio segment matching the target theme refers to Figure 3 or Figure 4 the audio highlight segment described in the example.

[0153] For the specific implementation manner of step S502, please refer to the description in the following embodiments.

[0154] S503. Generate a target audio according to the audio segment and a preset background music.

[0155] For example, the preset background music is the background music in the effect template of "one-click blockbuster".

[0156] In the embodiments of the present application, the duration of the preset background music is the same as the total playing duration of the materials selected by the user.

[0157] In some embodiments, step S503 may include:

[0158] Obtain the time stamp of the audio segment in the audio to be processed;

[0159] Filter the background music according to the time stamp to obtain the filtered background music;

[0160] Generate a target audio according to the audio segment and the filtered background music.

[0161] The time stamp includes the first moment corresponding to the start moment of the audio segment in the audio to be processed and the second moment corresponding to the end moment of the audio segment in the audio to be processed.

[0162] In some implementation manners, if the user selects multiple videos, the total playing duration is the sum of the playing durations of the multiple videos. Correspondingly, the duration of the audio to be processed is the total playing duration of the multiple videos, that is, the duration of the audio to be processed is the same as the duration of the preset background music.

[0163] As can be seen from the embodiments of the specific implementation manner of S502 below, the audio segment includes at least one audio frame in the audio to be processed. Optionally, in this implementation manner, the time stamp of the audio segment can be determined according to the time stamps of the audio frames included in the audio segment in the audio to be processed.

[0164] For example, referring to Figure 6 , it is a schematic diagram of the time stamp provided by the embodiments of the present application. Assume that the total duration of the audio to be processed is 5s and it includes 50 audio frames of 100ms each. Among them, the audio segment is the 2nd - 5th audio frames. The time stamp corresponding to the 2nd audio frame is the moment corresponding to the start moment of the 2nd audio frame in the audio to be processed (such as the duration of t1 relative to t0 shown in Figure 6 ), that is, 100ms (the first moment), and the time stamp corresponding to the 5th audio frame is the moment corresponding to the end moment of the 5th audio frame in the audio to be processed (such as the duration of t2 relative to t0 shown in Figure 6 ), that is, 500ms (the second moment). Therefore, the time stamp of the audio segment is 100 - 500ms.

[0165] In some other implementation manners, if the user selects multiple pictures and a video, the total playback duration is the sum of the display durations of the multiple pictures and the playback duration of one video. Correspondingly, the duration of the audio to be processed is the playback duration of one video in the materials selected by the user, that is, the duration of the audio to be processed is less than the duration of the preset background music.

[0166] Optionally, in this implementation manner, the time stamp of the audio to be processed within the total time period of the material playback can be determined first, the time stamp of the audio segment in the audio to be processed can be determined, and then the time stamp of the audio segment within the total time period of the material playback can be calculated according to the time stamp corresponding to the audio to be processed and the time stamp of the audio segment in the audio to be processed.

[0167] For example, referring to Figure 7 , it is a schematic diagram of the time stamp provided by the embodiments of the present application. Assume that the total playback duration of the material is 7s, among which, pictures are played in the first 2s and a video is played in the last 5s. Among them, the video includes 50 audio frames of 100ms each, and the audio segment is the 2nd - 5th audio frames. The time stamp of the audio to be processed within the total time period of the material playback is the moment corresponding to the start moment of the audio to be processed within the total time period of the material playback, such as the duration of t3 relative to t0 shown in Figure 7 , which is 2s. The time stamp of the audio segment in the audio to be processed includes the moment corresponding to the start moment of the audio segment in the audio to be processed (such as the duration of t4 relative to t3 shown in Figure 7 , which is 100ms), and the moment corresponding to the end moment of the audio segment in the audio to be processed (such as Figure 7The duration t5_t3 of t5 relative to t3 shown is 500 ms). Determine the timestamps of the audio segment within the total time period of the material playback according to the above timestamps, that is, including the moment corresponding to the start time of the audio segment within the total time period of the material playback (such as Figure 7 The duration t4_t3 + t3_t0 of t4 relative to t0 shown is 2.01 s), and the moment corresponding to the end time of the audio segment within the total time period of the material playback (such as Figure 7 The duration t5_t3 + t3_t0 of t5 relative to t0 shown is 2.05 s).

[0168] In the embodiments of the present application, the step of filtering the background music is: deleting the audio corresponding to the timestamps of the audio segment in the background music, so that there is no audio in the time period corresponding to the timestamps of the audio segment in the filtered background music. As an example in the above first implementation manner, if the timestamps of the audio segment are 100 - 500 ms, then delete the audio with timestamps 100 - 500 ms in the background music to obtain the filtered background music.

[0169] It should be noted that in the embodiments of the present application, the target segments included in the audio segment may be discontinuous. Correspondingly, filtering the background music means deleting the audio corresponding to the timestamps of each target segment of the audio segment in the background music.

[0170] In the embodiments of the present application, the step of generating the target audio according to the audio segment and the filtered background music may superimpose the audio segment and the filtered background music. Since there is no audio in the time period corresponding to the timestamps of the audio segment in the filtered background music, after audio superposition, the audio segment can fill the no-audio time period in the filtered background music.

[0171] In the embodiments of the present application, the audio segment and the background music are organically combined according to the timestamps, effectively reducing the pauses between audios, making the generated target audio smoother, and facilitating the improvement of user experience.

[0172] The following introduces the specific implementation manner of audio processing in step S502.

[0173] In some embodiments, step S502 may include:

[0174] S5021, obtaining the first probability corresponding to each audio frame in the to-be-processed audio according to the trained first model.

[0175] S5022, obtaining the audio segment from the to-be-processed audio according to the first probability corresponding to each audio frame.

[0176] Among them, the first probability corresponding to the audio frame represents the probability that the audio frame contains an audio signal belonging to the target theme.

[0177] In the embodiments of the present application, a first model is pre-trained. The first model can be a neural network model or other algorithm models capable of implementing audio detection.

[0178] In some implementation manners, the process of training the first model may include:

[0179] Obtain audio samples; train the first model according to the audio samples.

[0180] Among them, each audio sample corresponds to an audio theme (the third theme). The first model trained by the audio samples can be used to detect the first probability corresponding to each audio frame in the audio to be processed.

[0181] In order to improve the detection accuracy of the first model, in some other implementations, the process of training the first model may include:

[0182] Obtain audio samples and noise audio;

[0183] Perform audio mixing processing according to the audio samples and the noise audio to obtain mixed audio;

[0184] Train the first model according to the mixed audio.

[0185] Among them, the noise audio can be Gaussian white noise or random white noise, or interference audio containing the audio theme corresponding to the audio sample, or interference audio not containing the audio theme corresponding to the audio sample.

[0186] One implementation manner of the mixing process can be to superimpose the corresponding audio points in the audio sample and the noise audio. For example, the f-th audio point x(f) in the first audio frame of the mixed audio includes a first component s(f) and a second component n(f), satisfying x(f)=s(f)+n(f). Among them, the first component is the f-th audio point in the first audio frame of the audio sample, and the second component is the f-th audio point in the first audio frame of the noise audio.

[0187] In the embodiments of the present application, by adding noise audio, the trained first model can more accurately identify the audio frames in the audio that do not belong to the audio theme, thereby effectively improving the detection accuracy of the first model.

[0188] In some training methods, the first model can be trained in units of audio frames. Specifically, input the mixed audio into the first model, and output the first probability corresponding to each audio frame in the mixed audio; then calculate the loss value according to the first probability and the audio frames marked in the audio sample; adjust the model parameters of the first model according to the loss value.

[0189] Since an audio frame is composed of multiple audio points (i.e., audio signal points), in order to improve the model accuracy, in some other training methods, the first model can be trained with audio points as the unit.

[0190] Specifically, input the mixed audio into the first model, and output the third probability corresponding to each audio point in the mixed audio; calculate the loss value according to the first component of each audio point of the audio sample and the third probability corresponding to each audio point in the mixed audio; adjust the model parameters of the first model according to the loss value. Wherein, the third probability corresponding to the audio point represents the probability that the audio point contains an audio component belonging to the audio theme corresponding to the audio sample.

[0191] Continuing the example of the above mixing process, optionally, the loss function can be set as:

[0192]

[0193] Wherein, F is the total number of audio points included in each audio frame, and p(f) represents the third probability corresponding to the f-th audio point.

[0194] Correspondingly, the process of adjusting the model parameters of the first model according to the loss value is: calculate the loss value L according to the above formula; if the loss value L is less than or equal to the preset loss, then use the current first model as the trained first model; if the loss value L is greater than the preset loss, then adjust the model parameters of the first model according to the loss value L to obtain the adjusted first model; continue to train the first model according to the next audio sample until the loss value L of the first model is less than or equal to the preset loss.

[0195] It should be noted that during the training process of the first model, audio samples of different audio themes can be obtained, and the first model can be trained according to these audio samples so that the first model can detect different audio themes in the audio to be processed.

[0196] In the embodiments of the present application, through the mixing process, for each audio sample, multiple mixed audios can be constructed by mixing different noise audios. In the above way, not only the operation of the audio construction label for training the first model is reduced, but also data augmentation can be achieved through the mixing process, improving the scalability of the model.

[0197] After obtaining the trained first model, obtain the first probability corresponding to each audio frame in the audio to be processed according to S5021. In one implementation, input the audio to be processed into the trained first model, output the second probability corresponding to each audio point in the audio to be processed, and calculate the first probability corresponding to each audio frame in the audio to be processed according to the second probability corresponding to each audio point.

[0198] Among them, the second probability corresponding to the audio point represents the probability that the audio point contains an audio component belonging to the target theme.

[0199] In one implementation of calculating the first frequency, for each audio frame, weighted summation processing is performed according to the second probability corresponding to the audio points included in the audio frame, and the first probability corresponding to the audio frame is obtained. For example, through the formula Calculate the first probability of each audio frame, where p is the first probability.

[0200] In another implementation of calculating the first frequency, at least one frequency band is divided according to the sound frequency corresponding to the target theme; the first probability of the audio frame is calculated according to the second probability of the audio points included in each frequency band in the audio frame.

[0201] Among them, one way to divide the frequency band is to determine the frequency range according to the maximum and minimum values of the sound frequency corresponding to the target theme, and divide the frequency range into at least one frequency band.

[0202] For example, if the maximum value of the sound frequency of the target theme is 12 kHz and the minimum value is 80 Hz, the frequency range (80, 12000) can be divided into a low frequency band (80, 150), a middle frequency band (150, 500), and a high frequency band three frequency bands (500, 12000).

[0203] Specifically, for each audio frame, the segment probability is calculated according to the second probability of the audio points belonging to the same frequency band in the audio frame; the first probability is calculated according to the calculated segment probability corresponding to each frequency band.

[0204] For example, through the formula Calculate the first probability of each audio frame, where F1 represents the number of audio points belonging to the first frequency band (such as the low frequency band in the above example), F2 represents the number of audio points belonging to the second frequency band (such as the middle frequency band in the above example), and F3 represents the number of audio points belonging to the second frequency band (such as the high frequency band in the above example). Represents the segment probability corresponding to the j-th frequency band.

[0205] In the embodiments of the present application, screening thresholds are respectively determined according to different frequency bands, which can more accurately distinguish the audio of the target theme and the audio of non-target themes within each frequency band, and is conducive to improving the accuracy of audio processing.

[0206] After obtaining the first probability of each audio frame, execute S5022, and obtain the audio segment from the audio to be processed according to the first probability corresponding to each audio frame.

[0207] In one implementation of S5022: If the first probability corresponding to the audio frame is greater than the first threshold, the audio frame is determined as an alternative frame; and the audio segment is determined according to the alternative frames in the audio to be processed.

[0208] Affected by certain transient noises and algorithm convergence, there may be some alternative frames misdetected. To reduce the misdetection rate, in some embodiments, the step of determining the audio segment according to the alternative frames in the audio to be processed in step S5022 may include:

[0209] I. Generate a candidate segment according to each group of consecutive alternative frames in the audio to be processed.

[0210] Among them, consecutive alternative frames refer to two or more alternative frames that are adjacent audio frames, that is, the end time of the previous alternative frame is the start time of the next alternative frame. If the audio frames before and after an alternative frame are not alternative frames, then this alternative frame can be regarded as a group of consecutive alternative frames.

[0211] Exemplarily, referring to Figure 8 , it is a schematic diagram of alternative frames provided by an embodiment of the present application. As Figure 8 shown, the audio to be processed includes 50 audio frames, among which, the 1st - 5th audio frames, the 7th - 9th audio frames, and the 49th audio frame are alternative frames. The 1st - 5th audio frames are consecutive alternative frames, then the 1st - 5th audio frames form a candidate segment clip1; the 7th - 9th alternative frames form a candidate segment clip2; the 49th audio frame forms a candidate segment clip3.

[0212] II. Determine the first audio duration according to the target theme.

[0213] The audio features of different target themes are different. Determining the first audio duration according to the target theme is conducive to screening out audio segments that are more in line with the target theme. For example, if the target theme is human voice, generally the duration of a single syllable is 100 - 200 ms, and the first audio duration can be set to 100 ms.

[0214] III. Determine the target segment from the candidate segments according to the first audio duration.

[0215] In some implementations, if the audio duration of the candidate segment is less than or equal to the first audio duration, then delete this candidate segment.

[0216] Continue Figure 8In the example in , if the target theme is human voice, the duration of the first audio is 100 ms. Among the three candidate segments, the audio duration of candidate segment clip1 is the total duration of the 1st - 5th audio frames, which is 500 ms; the audio duration of candidate segment clip2 is the duration of the 7th - 9th audio frames, which is 300 ms; the audio duration of candidate segment clip3 is the total duration of the 49th audio frame, which is 100 ms. Since the audio duration of candidate segment clip3 is equal to the duration of the first audio, candidate segment clip3 is deleted, and candidate segments clip1 and clip2 are retained.

[0217] In the embodiments of the present application, through the duration of the first audio, candidate segments with shorter durations can be effectively filtered out, thereby effectively reducing the false detection probability of the alternative frames, which is conducive to improving the accuracy of audio processing.

[0218] In order to ensure the continuity of the highlight audio segments, in some implementation manners, if the time interval between the first segment and the second segment is less than the third threshold, the first segment, the second segment, and the audio frames between the first segment and the second segment are merged into a target segment, where the first segment is a target segment, and the second segment is the next target segment after the first segment.

[0219] Among them, the third threshold can be determined according to the target theme. For example, if the target theme is human voice, the duration of a single Chinese character syllable is generally 100 - 200 ms, and the third threshold can be set to 150 ms. It should be noted that the above is an example of the third threshold. In practical applications, for the same target theme, the corresponding duration of the first audio and the third threshold can be the same or different, and the embodiments of the present application do not make specific limitations on this.

[0220] Continue Figure 8 In the example in , candidate segment clip1 is equivalent to the first segment, candidate segment clip2 is equivalent to the second segment, and the time interval between candidate segment clip1 and candidate segment clip2 is two audio frames, that is, 100 ms, which is less than the third threshold of 150 ms corresponding to the human voice target theme. Therefore, candidate segment clip1, candidate segment clip2, and the audio frames between candidate segment clip1 and candidate segment clip2 are merged into a target segment, that is, the target segment is the 1st - 9th audio frames.

[0221] In the embodiments of the present application, through the third threshold, discrete candidate segments with short interval times can be combined into continuous audio frames, which can effectively reduce the non - continuity impact caused by factors such as pauses in sentences, thereby improving the continuity of the audio segments and being conducive to enhancing the user experience.

[0222] IV. Determine the audio segment according to the target segment.

[0223] In some implementation manners, step IV includes: calculating a first audio proportion corresponding to each of the target segments, and determining the audio segment according to the target segments corresponding to the first audio proportions greater than a fourth threshold.

[0224] Wherein, the audio proportion represents the ratio of the audio component belonging to the target theme in the target segment to the audio component not belonging to the target theme.

[0225] Optionally, for each target segment, calculate a second audio proportion of each audio frame in the target segment; according to the second audio proportion of each audio frame in the target segment, calculate the first audio proportion corresponding to the target segment.

[0226] Specifically, the second audio proportion of each audio frame can be calculated by the following formula:

[0227]

[0228] After calculating the second audio proportion of each audio frame, the average value of all the second audio frames in the target segment can be calculated as the first audio proportion of the target segment.

[0229] In this implementation manner, the first audio proportion can be used to represent the ratio of the audio component belonging to the target theme in the target segment to the audio component not belonging to the target theme, and the editing value of the target segment can be evaluated through this parameter, which is beneficial to improving the audio processing effect.

[0230] In some other implementation manners, step IV includes: calculating a first loudness corresponding to each of the target segments; determining the audio segment according to the target segments corresponding to the first loudness greater than a fifth threshold.

[0231] Loudness refers to the intensity or volume of sound.

[0232] In this implementation manner, the first loudness can be used to evaluate the sound intensity or volume of the target segment, so as to filter out the audio with a small sound intensity or a small volume, which is beneficial to improving the audio processing effect.

[0233] In some other implementation manners, step IV includes: determining the audio segment according to the first audio proportion and the first loudness. Specifically, if the first audio proportion of the target segment is greater than the fourth threshold and the first loudness is greater than the fifth threshold, then the target segment is determined as the audio segment.

[0234] It should be noted that in the embodiments of the present application, only two evaluation indicators, namely the audio proportion and the loudness, are shown. In practical applications, other evaluation indicators can also be used to screen the target segments, and the embodiments of the present application do not make specific limitations thereto.

[0235] It should be understood that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0236] The embodiments of the present application also provide a computer-readable storage medium storing a computer program, which when executed by a processor can implement the steps in the above-mentioned method embodiments.

[0237] The embodiments of the present application also provide a computer program product, which when running on an electronic device enables the electronic device to implement the steps in the above-mentioned method embodiments.

[0238] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above embodiments of the method of the present application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps in the above-mentioned method embodiments. Among them, the computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code to a first device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0239] The embodiments of the present application also provide a chip system, which includes a processor. The processor is coupled to a memory, and the processor executes the computer program stored in the memory to implement the steps in any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0240] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0241] Those of ordinary skill in the art will appreciate that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application. Finally, it should be noted that the above description is only a specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims described above.

Claims

1. An audio processing method, characterized in that, Including: Obtain the audio to be processed and the target theme, where the target theme represents the type of sound contained in the audio; Perform audio processing on the audio to be processed according to the target theme to obtain an audio segment matching the target theme; Generate a target audio according to the audio segment and a preset background music; The performing audio processing on the audio to be processed according to the target theme to obtain an audio segment matching the target theme includes: Generate a candidate segment according to each group of consecutive alternative frames in the audio to be processed; the first probability corresponding to the alternative frame is greater than a first threshold; wherein, the first probability corresponding to the audio frame in the audio to be processed represents the probability that the audio frame contains an audio signal belonging to the target theme; Determine a first audio duration according to the target theme; If the time interval between a first segment and a second segment is less than a third threshold, then merge the first segment, the second segment, and the audio frames between the first segment and the second segment into a target segment, where the first segment is a target segment and the second segment is the next target segment after the first segment; Determine the audio segment according to the target segment.

2. The method according to claim 1, wherein The performing audio processing on the audio to be processed according to the target theme to obtain an audio segment matching the target theme includes: Obtain the first probability corresponding to each audio frame in the audio to be processed according to a trained first model, where the first probability corresponding to the audio frame represents the probability that the audio frame contains an audio signal belonging to the target theme; Obtain the audio segment from the audio to be processed according to the first probability corresponding to each audio frame.

3. The method according to claim 2, wherein The obtaining the first probability corresponding to each audio frame in the audio to be processed according to a trained first model includes: Input the audio to be processed into the trained first model, and output the second probability corresponding to each audio point in the audio to be processed, where the second probability corresponding to the audio point represents the probability that the audio point contains an audio component belonging to the target theme; Calculate the first probability corresponding to each audio frame in the audio to be processed according to the second probability corresponding to each audio point.

4. The method according to claim 3, characterized in that The calculating the first probability corresponding to each audio frame in the audio to be processed according to the second probability corresponding to each audio point includes: For each audio frame, perform weighted summation processing according to the second probabilities corresponding to the audio points included in the audio frame to obtain the first probability corresponding to the audio frame.

5. The method according to claim 3, wherein The calculating the first probability corresponding to each audio frame in the audio to be processed according to the second probability corresponding to each audio point includes: Divide at least one frequency band according to the sound frequency corresponding to the target theme; Calculate the first probability of the audio frame according to the second probabilities of the audio points included in each frequency band in the audio frame.

6. The method according to claim 1, characterized in that The determining the audio segment according to the target segment includes: Calculate the first audio ratio corresponding to each of the target segments, where the audio ratio represents the ratio of the audio component belonging to the target theme in the target segment to the audio component not belonging to the target theme; Determine the audio segment based on the target segments corresponding to the first audio ratio greater than the fourth threshold.

7. The method according to claim 6, characterized in that The calculating the first audio ratio corresponding to each of the target segments includes: For each of the target segments, calculate the second audio ratio of each audio frame in the target segment; Calculate the first audio ratio corresponding to the target segment according to the second audio ratio of each audio frame in the target segment.

8. The method according to any one of claims 1 to 7, characterized in that, The determining the audio segment based on the target segment includes: Calculate the first loudness corresponding to each of the target segments; Determine the audio segment based on the target segments corresponding to the first loudness greater than the fifth threshold.

9. The method according to claim 1, wherein The generating the target audio based on the audio segment and the preset background music includes: Obtain the timestamp of the audio segment in the audio to be processed; Filter the background music according to the timestamp to obtain the filtered background music; Generate the target audio based on the audio segment and the filtered background music.

10. The method according to claim 1, wherein The steps of obtaining the target theme include: Identify the first theme corresponding to the audio to be processed, where the first theme represents the type of sound included in the audio; Display a first interface, where the first interface includes a first control corresponding to each of the first themes; In response to the user's operation on the first control, determine the first theme corresponding to the first control operated by the user as the target theme.

11. The method according to claim 1, wherein The steps of obtaining the target theme include: Display a second interface, where the second interface includes a preset plurality of second themes and a second control corresponding to each of the second themes, and the second theme represents the type of sound included in the audio; In response to the user's operation on the second control, determine the second theme corresponding to the second control operated by the user as the target theme.

12. The method according to claim 2, wherein The method further includes: Obtain an audio sample and a noise audio; Perform audio mixing processing according to the audio sample and the noise audio to obtain a mixed audio; Train the first model according to the mixed audio.

13. The method according to claim 12, characterized in that, The training the first model according to the mixed audio includes: Obtain the first component of each audio point of the mixed audio, where the first component is the audio component belonging to the third theme in the audio point, and the third theme represents the type of sound included in the audio sample; Input the mixed audio into the first model, and output the third probability corresponding to each audio point in the mixed audio, where the third probability corresponding to the audio point represents the probability that the audio point includes an audio component belonging to the third theme; Calculate a loss value according to the first component of each audio point of the mixed audio and the third probability corresponding to each audio point in the mixed audio; Adjust the model parameters of the first model according to the loss value.

14. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in any one of claims 1 to 13 is implemented.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 13 is implemented.

Images