Audio processing method and electronic device

Abstract

The present application provides an audio processing method and an electronic device. In a scenario where an audio analysis algorithm is used to analyze audio and obtain analysis results. It is used to perform sampling decoding and other processing on the encoded audio, speed up the processing of the encoded audio into audio adapted to the audio analysis algorithm, and then input it into the audio analysis algorithm for audio analysis to obtain analysis results. Using sampling decoding can make the encoded audio decoded quickly, thereby speeding up the speed of obtaining analysis results from the encoded audio.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to an audio processing method and an electronic device. Background Art

[0002] Audio analysis is performed based on audio, and the analysis results can be applied in multiple fields. These fields include but are not limited to card point video generation, speech recognition, etc. The audio analysis processing corresponding to different fields is different, and the analysis results obtained are also different. For example, when generating a card point video, the music node in the audio is analyzed, and then the video picture is adjusted based on the music node so that the scene corresponds to the music node to achieve the card point and obtain the card point video. The music node of the audio is a timestamp with a special beat in the audio. The special beat is not limited to: a beat with a strong sense of rhythm, a beat with transition (from a stronger rhythm to a weaker rhythm or from a weaker rhythm to a stronger rhythm), etc.

[0003] For the convenience of storage and transmission, the audio (to be analyzed) stored in the electronic device is usually coded audio. In order to analyze the audio more accurately, when performing audio analysis processing, it is usually necessary to decode the audio and use the decoded audio to perform audio analysis.

[0004] However, the decoding process usually consumes a lot of time. In the scenario where the analysis results need to be obtained as soon as possible, how to obtain the analysis results based on the encoded audio as quickly as possible is worth discussing. Summary of the invention

[0005] The present application provides an audio processing method and electronic device, in a scenario where an audio analysis algorithm is used to analyze the encoded audio and obtain the analysis result. The method is used to speed up the processing of the encoded audio into audio adapted to the audio analysis algorithm, and then input it into the audio analysis algorithm for audio analysis to obtain the analysis result.

[0006] In a first aspect, the present application provides an audio processing method, the method comprising: an electronic device divides a first audio into N sub-audios; N is a positive integer; the first audio is sampled and decoded by N audio decoders to obtain a second audio; wherein the i-th audio decoder among the N audio decoders is used to sample and decode the i-th sub-audio in the first audio; the sampling decoding includes decoding 1 / H of each audio frame in the i-th sub-audio; the number of sampling points contained in the second audio is 1 / H of the number of sampling points of the first audio; H is an integer greater than or equal to 2; the value of i is 1 to N; audio analysis processing is performed on the second audio to obtain an analysis result.

[0007] In the above embodiment, the first audio may be the audio 1 in the embodiment. The second audio may be the audio 2 (decoded audio) in the embodiment. Sampling decoding can speed up the decoding of the audio, thereby speeding up the speed of obtaining the analysis result from the encoded audio.

[0008] In combination with the first aspect, in some embodiments, the second audio is subjected to audio analysis processing to obtain an analysis result, which specifically includes: when the number of sampling points contained in a unit time length of the second audio is different from the first sampling points, the electronic device resamples the second audio to obtain a third audio; the number of sampling points contained in a unit time length of the third audio is the same as the first sampling points; the first sampling points are the number of sampling points contained in the unit time length of the input audio of the adapted audio analysis algorithm; the electronic device inputs the third audio into the audio analysis algorithm to obtain the analysis result.

[0009] In the above embodiment, the third audio may be audio 3 in the embodiment (the resampled audio). Because sampling decoding is performed, the sampling rate of the second audio is reduced, and usually the number of sampling points contained in a unit time length is different from the number of first sampling points. Therefore, resampling is required to obtain the third audio adapted to the audio analysis algorithm.

[0010] In combination with the first aspect, in some embodiments, the method also includes: the electronic device also adjusts audio parameters of the second audio other than the number of sampling points contained in the unit time length, and the audio parameters include at least one of the number of channels and bit depth corresponding to the second audio.

[0011] In combination with the first aspect, in some embodiments, the method also includes: the electronic device performs deduplication processing on the decoded N sub-audio segments; merges the deduplicated N sub-audio segments according to timestamps, and uses the merged audio as the second audio; wherein the deduplication processing includes: determining that there is overlap between the decoded e-th sub-audio segment and the decoded e+1-th sub-audio segment; removing the overlapping part from the decoded e-th sub-audio segment or the decoded e+1-th sub-audio segment; the value of e is 1 to N-1.

[0012] In the above embodiment, before dividing audio 1 (encoded) into N sub-audio segments for decoding, due to the low precision of the algorithm used, two adjacent sub-audio segments in the N sub-audio segments may be divided to have some duplication, and thus the two adjacent sub-audio segments also have overlapping parts after decoding. Therefore, before merging the decoded frames in the N audio queues to obtain the decoded audio 1, it is necessary to perform deduplication processing, and then merge the N decoded sub-audio segments after deduplication according to the timestamps, and use the merged audio as the decoded audio 1.

[0013] In combination with the first aspect, in some embodiments, the audio analysis algorithm is a node analysis algorithm, and the analysis result is an audio node in the first audio.

[0014] In combination with the first aspect, in some embodiments, before the electronic device divides the first audio into N sub-audios, the method also includes: the electronic device displays a first interface, the first interface includes a selected video and a first control; in response to an operation on the first control, the electronic device determines the audio corresponding to the video as the first audio.

[0015] In the above embodiment, the first control is an entry for automatically analyzing audio nodes based on a node analysis algorithm, and may be the production control 121 involved in the following embodiments.

[0016] In combination with the first aspect, in some embodiments, the method also includes: after obtaining the audio node in the first audio, the electronic device adjusts at least one video segment in the video based on the audio node in the first audio to obtain an adjusted video; the adjusted video includes the first audio, and the original audio in the adjusted video is suppressed; the audio node of the first audio in the adjusted video is partially or completely aligned with the picture transition in the video.

[0017] In the above embodiment, the method for automatically analyzing audio nodes based on the node analysis algorithm is applicable to the scenario of generating a stuck point video based on the selected video, so as to facilitate the intelligent generation of the stuck point video.

[0018] In combination with the first aspect, in some embodiments, when the first audio is sampled and decoded through N audio decoders, the method also includes: the electronic device creates a read thread and a write thread of the i-th audio decoder; when the read thread determines that the first input buffer is idle, writes 1 / H of the first audio frame to the first input buffer; the first audio frame is a frame of audio in the i-th sub-audio; the i-th audio decoder obtains 1 / H of the first audio frame from the first input buffer for decoding, and writes the decoded first audio frame to the first output buffer; when the read thread determines that the first input buffer is idle, the write thread determines that there is a decoded second audio frame in the second output buffer, and stores the decoded second audio frame to the i-th audio frame queue and releases the second output buffer; the second audio frame is an audio frame before the first audio frame.

[0019] In the above embodiment, the reading process can be set in one thread (reading thread), but the writing process can be set in another thread (writing thread), that is, the reading process and the writing process are parallel. In this way, the decoding speed can be accelerated.

[0020] In combination with the first aspect, in some embodiments, the method also includes: when the timestamp of the audio frame read by the reading thread is equal to the end timestamp of the i-th sub-audio, the electronic device merges the audio frames in the i-th audio frame queue according to the timestamp to obtain the decoded i-th sub-audio.

[0021] In combination with the first aspect, in some embodiments, when the audio analysis algorithm is a node analysis algorithm, the electronic device inputs the third audio into the audio analysis algorithm to obtain the analysis result, which specifically includes: the electronic device divides the third audio into R sub-audio segments; there is overlapping audio between the end of the jth sub-audio segment and the beginning of the j+1th sub-audio segment in the R sub-audio segments; the value of j ranges from 1 to R-1; the degree of overlap is related to the audio analysis algorithm; R is greater than or equal to 2; the R sub-audio segments are analyzed based on the node analysis algorithm to obtain the audio nodes corresponding to each of the R sub-audio segments.

[0022] In the above embodiment, the overlap degree is related to the node analysis algorithm, which means that when the node analysis algorithm analyzes the audio (audio a) corresponding to a timestamp, it uses the previous audio (audio b) and the following audio (audio c) of the audio a to perform comparative analysis with the audio a to determine whether the timestamp corresponding to the audio a is an audio node. Therefore, if it is necessary to obtain all the audio nodes in the resampled audio 1, the length of the overlapped part of the jth sub-audio and the j+1th sub-audio is at least equal to (or equal to): the length of audio a+audio b+audio c.

[0023] In combination with the first aspect, in some embodiments, the audio nodes corresponding to the R segment sub-audios are merged to obtain the audio nodes in the first audio.

[0024] In combination with the first aspect, in some embodiments, before merging the audio nodes corresponding to the R sub-audios, the method further includes: deleting the audio nodes in the j-th sub-audio that are repeated in the j+1-th sub-audio.

[0025] In the above embodiment, since there are overlapping audios in the R segment sub-audios, overlapping audio syllables may appear in the audio nodes corresponding to the R segment sub-audios. In some possible cases, before merging, the audio nodes in the j-th segment sub-audio that are repeated in the j+1-th segment sub-audio may be deleted. In this way, the audio nodes in the obtained audio 1 will no longer have repeated nodes.

[0026] In combination with the first aspect, in some embodiments, when the length of the first audio is less than a threshold, N is equal to 1; when the length of the first audio is greater than or equal to the threshold, N is greater than or equal to 2.

[0027] In the above embodiment, for a relatively long audio 1, at least two audio decoders may be used to align and decode the audio 1 to further speed up the decoding speed.

[0028] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors are used to call the computer instructions to enable the electronic device to execute: dividing a first audio into N sub-audio segments; N is a positive integer; sampling and decoding the first audio through N audio decoders to obtain a second audio; wherein the i-th audio decoder among the N audio decoders is used to sample and decode the i-th sub-audio segment in the first audio; the sampling decoding includes decoding 1 / H of each audio frame in the i-th sub-audio segment; the number of sampling points contained in the second audio is 1 / H of the number of sampling points of the first audio; H is an integer greater than or equal to 2; the value of i is 1 to N; audio analysis processing is performed on the second audio to obtain an analysis result.

[0029] In combination with the second aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to enable the electronic device to execute: when the number of sampling points contained in the unit time length of the second audio is different from the first sampling points, the electronic device resamples the second audio to obtain a third audio; the number of sampling points contained in the unit time length of the third audio is the same as the first sampling points; the first sampling points are the number of sampling points contained in the unit time length of the input audio of the adapted audio analysis algorithm; the electronic device inputs the third audio into the audio analysis algorithm to obtain the analysis result.

[0030] In combination with the second aspect, in some embodiments, the one or more processors are also used to call the computer instructions to cause the electronic device to execute: the electronic device also adjusts the audio parameters of the second audio other than the number of sampling points contained in the unit time length, and the audio parameters include at least one of the number of channels and bit depth corresponding to the second audio.

[0031] In combination with the second aspect, in some embodiments, the one or more processors are also used to call the computer instructions to enable the electronic device to execute: the electronic device performs deduplication processing on the decoded N sub-audio segments; merges the deduplicated N sub-audio segments according to timestamps, and uses the merged audio as the second audio; wherein the deduplication processing includes: determining that there is overlap between the decoded e-th sub-audio segment and the decoded e+1-th sub-audio segment; removing the overlapping part from the decoded e-th sub-audio segment or the decoded e+1-th sub-audio segment; the value of e is 1 to N-1.

[0032] In combination with the second aspect, in some embodiments, the one or more processors are also used to call the computer instructions to cause the electronic device to execute: before the electronic device divides the first audio into N sub-audios, the electronic device displays a first interface, and the first interface includes a selected video and a first control; in response to the operation of the first control, the electronic device determines the audio corresponding to the video as the first audio.

[0033] In combination with the second aspect, in some embodiments, the one or more processors are also used to call the computer instructions to cause the electronic device to execute: after obtaining the audio node in the first audio, the electronic device adjusts at least one video segment in the video based on the audio node in the first audio to obtain an adjusted video; the adjusted video includes the first audio, and the original audio in the adjusted video is suppressed; the audio node of the first audio in the adjusted video is partially or completely aligned with the picture transition in the video.

[0034] In conjunction with the second aspect, in some embodiments, the one or more processors are also used to call the computer instruction so that the electronic device executes: when sampling and decoding the first audio through N audio decoders, the electronic device creates a read thread and a write thread of the i-th audio decoder; when the read thread determines that the first input buffer is idle, write 1 / H of the first audio frame to the first input buffer; the first audio frame is a frame of audio in the i-th sub-audio; the i-th audio decoder obtains 1 / H of the first audio frame from the first input buffer for decoding, and writes the decoded first audio frame to the first output buffer; when the read thread determines that the first input buffer is idle, the write thread determines that there is a decoded second audio frame in the second output buffer, and stores the decoded second audio frame to the i-th audio frame queue and releases the second output buffer; the second audio frame is an audio frame before the first audio frame. In the second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method implemented in the first aspect.

[0035] In combination with the second aspect, in some embodiments, the one or more processors are also used to call the computer instruction to cause the electronic device to execute: when the timestamp of the audio frame read by the reading thread is equal to the end timestamp of the i-th sub-audio, the electronic device merges the audio frames in the i-th audio frame queue according to the timestamp to obtain the decoded i-th sub-audio.

[0036] In combination with the second aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to enable the electronic device to execute: when the audio analysis algorithm is a node analysis algorithm, the electronic device divides the third audio into R sub-audio segments; there is overlapping audio between the end of the jth sub-audio segment and the beginning of the j+1th sub-audio segment in the R sub-audio segments; the value of j ranges from 1 to R-1; the degree of overlap is related to the audio analysis algorithm; R is greater than or equal to 2; based on the node analysis algorithm, the R sub-audio segments are analyzed to obtain the audio nodes corresponding to each of the R sub-audio segments.

[0037] In combination with the second aspect, in some embodiments, the one or more processors are further used to call the computer instruction to cause the electronic device to execute: merging the audio nodes corresponding to each of the R segment sub-audios to obtain the audio nodes in the first audio.

[0038] In combination with the second aspect, in some embodiments, the one or more processors are also used to call the computer instruction to cause the electronic device to execute: before merging the audio nodes corresponding to the R segments of sub-audio, deleting the audio nodes in the jth sub-audio that are repeated in the j+1th sub-audio.

[0039] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on an electronic device, enable the electronic device to execute the method implemented in the first aspect.

[0040] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method implemented in the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method implemented in the first aspect.

[0042] It is understandable that the electronic device provided in the second aspect, the computer storage medium provided in the third aspect, the chip system provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, other beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A set of exemplary user interfaces for an electronic device to generate a stuck point video is shown;

[0044] Figure 2A software structure framework diagram for obtaining an audio node based on an audio processing method is shown;

[0045] Figure 3 An exemplary module interaction diagram involved in obtaining an audio node based on an audio processing method is shown;

[0046] Figure 4 A schematic diagram showing deduplication processing of the end of the e-th sub-audio segment and the e+1-th sub-audio segment;

[0047] Figure 5 An exemplary flow chart involved in parallel analysis is shown;

[0048] Figure 6 An exemplary flow chart for deduplication of audio nodes is shown;

[0049] Fig. 7A as well as Figure 7B The process of decoding when the reading process and the writing process are serial is shown;

[0050] Fig. 8A as well as Figure 8B The process of decoding when the reading process and the writing process are serial is shown;

[0051] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0053] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0054] The audio processing method provided in the present application is used to accelerate the processing of encoded audio into audio compatible with the audio analysis algorithm in a scenario where an audio analysis algorithm is used to perform analysis based on audio (encoded) to obtain analysis results, and then input the audio into the audio analysis algorithm for audio analysis to obtain analysis results.

[0055] Adapting to the audio analysis algorithm here means that the properties of the processed audio are the same as the input audio properties of the audio analysis algorithm. In this way, the audio analysis algorithm can analyze the audio and obtain the analysis result. In the audio processing method, the input audio properties of the audio analysis algorithm at least include that the input audio is decoded audio. Other properties may also be included, such as the number of sampling points contained in the input audio per unit time length is 1, etc., which is not limited in the embodiments of the present application.

[0056] Hereinafter, the audio analysis algorithm used is a node analysis algorithm, and the analysis result obtained is an audio node. The actual audio analysis algorithm may also be other algorithms, such as a speech recognition algorithm, etc., which is not limited in the present application.

[0057] The music node of the audio refers to the timestamp with a special beat in the audio. The special beat includes but is not limited to: a beat with a strong sense of rhythm, a beat with transition (from a stronger beat to a weaker beat or from a weaker beat to a stronger beat), etc.

[0058] The encoded audio is recorded as audio 1. Here, the method of speeding up the processing of audio 1 into audio adapted to the node analysis algorithm includes but is not limited to at least one of the following methods.

[0059] Accelerated decoding. Accelerated decoding means that before audio 1 is input into the node analysis algorithm, the electronic device uses N audio decoders to sample and decode audio 1 to obtain decoded audio 1 (recorded as audio 2). Among them, the i-th audio decoder among the N audio decoders is used to sample and decode the i-th sub-audio in audio 1. Sampling decoding includes: decoding 1 / H of each audio frame in the i-th sub-audio. The number of sampling points contained in audio 2 is 1 / H of the number of sampling points of audio 1, where H is an integer greater than or equal to 2. The value of i is 1 to N. For details about accelerated decoding, please refer to the following description of steps S12-S16 and steps S171-S173, which will not be repeated here.

[0060] Since it is sampling decoding, the speed of decoding the audio 1 (encoded) can be accelerated.

[0061] Wherein, N is a positive integer. When the value of N is greater than or equal to 2, at least two audio decoders are used to decode the audio 1 in parallel, which can further speed up the decoding speed of the audio 1.

[0062] In some possible cases, the value of N can be set to an integer greater than or equal to 2.

[0063] In some other possible cases, the value of N can be determined according to the length of audio 1 (expressed as the length of time): when the length of audio 1 is less than the threshold, N is equal to 1. When the length of audio 1 is greater than or equal to the threshold, N is greater than or equal to 2. The threshold can be 4 minutes or other values, such as 3 minutes, etc., which is not limited in the embodiments of the present application.

[0064] Parallel analysis. Parallel analysis means that after the electronic device processes audio 1 into audio that is compatible with the audio analysis algorithm (recorded as audio 3), audio 3 is divided into R segments of sub-audio, and the R segments of sub-audio are analyzed based on the node analysis algorithm to obtain the audio nodes corresponding to each of the R segments of sub-audio. Then the audio nodes corresponding to each of the R segments of sub-audio are merged to obtain the audio nodes in audio 1. There is overlapping audio at the end of the jth sub-audio and the beginning of the j+1th sub-audio in the R segments of sub-audio, and the degree of overlap is related to the node analysis algorithm. Among them, R is greater than or equal to 2, and the value of j ranges from 1 to R-1.

[0065] Here, since there are overlapping audios in the R segment sub-audios, overlapping audio nodes may appear in the audio nodes corresponding to the R segment sub-audios. In some possible cases, before merging, the audio nodes in the jth segment sub-audio that are repeated in the j+1th segment sub-audio can be deleted. In this way, there will be no repeated nodes in the audio nodes of the obtained audio 1.

[0066] It should be noted here that when audio 2 (decoded audio 1) is already audio adapted to the node analysis algorithm, audio 3 is audio 2.

[0067] When audio 2 (decoded audio 1) is not audio adapted to the node analysis algorithm, the electronic device can also resample audio 2, change the audio properties, and obtain audio 3 adapted to the node analysis algorithm. For example, when the number of sampling points contained in audio 2 per unit time length is not equal to sampling point number 1, but the input audio requires that the number of sampling points contained in the unit time length is equal to sampling point number 1, the electronic device can resample audio 2 to obtain audio 3.

[0068] Among them, resampling audio 2 to obtain audio 3 may include: when the number of sampling points contained in audio 2 per unit time length is less than sampling point number 1, the electronic device increases the number of sampling points contained in audio 2 per unit time length to sampling point number 1 by interpolation. When the number of sampling points contained in audio 2 per unit time length is greater than sampling point number 1, the electronic device reduces the number of sampling points contained in audio 2 per unit time length to sampling point number 1 by interception.

[0069] For details of the parallel analysis, please refer to the following description of step S201, step S202, and step S211-step S213, which will not be repeated here.

[0070] After obtaining the audio node, the video picture can be adjusted based on the music node so that the scene corresponds to the music node to achieve the card point and obtain the card point video.

[0071] A card point video can be understood as matching the scene in the video (shown through the screen) with the sound in the video, and using the corresponding sound to emphasize the scene effect. At the card point scene of the card point video, you can selectively add a low volume to highlight the human voice, add a high volume to set the atmosphere through music, or add other special effects. This makes the video more attractive and impactful, so that the audience can observe and appreciate the details or content in the video more attentively, enhancing the overall viewing experience.

[0072] Figure 1 A set of exemplary user interfaces for an electronic device to generate a stuck point video is shown.

[0073] The electronic device can generate a card point video in the gallery application. Figure 1 As shown, the user interface 11 may be a material selection interface provided by the gallery application, and the user interface 11 may include some or all of the materials (videos or photos). In response to the user selecting a material (such as video 111 and video 112), the electronic device may display the user interface 12 including a production control 121. The production control 121 is the entry for generating a card point video.

[0074] In response to the operation of the production control 121, the electronic device can display the start production interface (such as user interface 13) and start generating the card point video based on the audio processing method. The process includes: the electronic device can select the audio corresponding to the material. Then the audio corresponding to the material (equivalent to audio 1) is at least accelerated decoding processing to obtain audio adapted to the node analysis algorithm. Then the node analysis is performed to obtain the audio node in audio 1. After obtaining the audio node, the electronic device adjusts at least one video clip in the selected material based on the audio node in audio 1 to obtain the adjusted video. The adjusted video includes audio 1, and the audio node of audio 1 in the adjusted video is partially or completely aligned with the picture transition in the video. In some possible cases, the original audio in the adjusted video can be suppressed, or it can also be retained, which is not limited in the embodiments of the present application.

[0075] refer to Figure 1The user interface 14 shown in FIG. 1 is an exemplary user interface after the card point video is generated. Optionally, the electronic device may display an identifier 141 corresponding to the audio node in the user interface 14 to prompt the user that there is an audio node and corresponds to a scene transition.

[0076] It should be noted here that the aforementioned Figure 1 It is a set of exemplary user interfaces for generating stuck point videos. In actual situations, it can also be other user interfaces, in which icons, controls or applications for generating stuck point videos can be changed. It should not constitute a limitation on the embodiments of the present application.

[0077] The following describes a software structure framework diagram involved in an electronic device obtaining an audio node based on an audio processing method.

[0078] refer to Figure 2 As shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software framework is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer, and the kernel layer.

[0079] The application layer can include a series of application packages.

[0080] like Figure 2 As shown, the application layer may include a video production application (such as the gallery application mentioned above).

[0081] refer to Figure 2 At (1), the video production application can provide the following file parsing module with audio (eg, audio 1) corresponding to the selected material.

[0082] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0083] The application framework layer may include a file parsing module, an audio decoding module, a cache merging module, and a resampling module.

[0084] refer to Figure 2 At (2), the file parsing module can be used to analyze the audio 1, determine the audio format of the audio 1 and the number N of audio decoders for decoding the audio 1. The audio format of the audio 1 and N are sent to the audio decoding module.

[0085] The file parsing module is also used to obtain the i-th sub-audio of audio 1, divide the i-th sub-audio into Mi frames, and gradually transmit 1 / H of each audio frame to the i-th audio decoder.

[0086] The audio decoding module may create and configure N audio decoders suitable for decoding audio 1 based on the audio format of audio 1. Then the N audio decoders are enabled for sampling and decoding audio 1. The ith audio decoder is used to gradually decode 1 / H of each audio frame in the ith sub-audio.

[0087] The cache merging module caches the decoded frame corresponding to the i-th sub-audio into the i-th audio queue, and merges the decoded frames in the N audio queues to obtain the decoded audio 1 (audio 2).

[0088] refer to Figure 2 At (3), the resampling module is used to resample the audio 2 to obtain the resampled audio 1, and the resampled audio 1 is the audio (audio 3) adapted to the node analysis algorithm.

[0089] The hardware abstraction layer (HAL) is an interface layer located between the operating system kernel layer and other layers of the electronic device (such as the local service layer). Its purpose is to abstract the hardware and provide a virtual hardware platform for the operating system.

[0090] The hardware abstraction layer may include a node analysis module.

[0091] refer to Figure 2 In (4), the node analysis module is used to analyze the audio 1 adapted to the node analysis algorithm in parallel based on the node analysis algorithm to obtain the audio nodes in the audio 1.

[0092] refer to Figure 2 At (5), after the video production application receives the audio node transmitted by the node analysis module, it can adjust the video picture based on the music node so that the scene corresponds to the music node to achieve the card point and obtain the card point video.

[0093] The kernel layer is the layer between hardware and software. The kernel layer can also be called the driver layer.

[0094] The kernel layer may include display drivers.

[0095] The display driver can drive the screen to display the user interface involved in making the card point video, such as the user interface 11-user interface 14 mentioned above.

[0096] Figure 3 An exemplary module interaction diagram involved in obtaining an audio node based on an audio processing method is shown.

[0097] Combine the following Figure 2 as well as Figure 3 Describe the module interaction process involved in the generation of audio nodes by an electronic device. The process can refer to the following description of steps S11 to S19, step S20 and step S21.

[0098] Among them, step S11-step S19 describes the process of the electronic device sampling, decoding and resampling the encoded audio 1 to obtain the audio (audio 3) adapted to the node analysis algorithm. Step S20 and step S21 describe the process of the electronic device performing parallel analysis on audio 3 based on the node analysis algorithm to obtain the audio node.

[0099] First, based on steps S11 to S19, the process of an electronic device sampling, decoding and resampling the encoded audio 1 to obtain the audio 1 adapted to the node analysis algorithm is described.

[0100] S11. Application 1 selects audio 1 corresponding to the material.

[0101] The material involved in step S11 is the material to be edited selected by the electronic device through the application 1. An exemplary user interface involved in selecting the material can refer to the aforementioned Figure 1 The user interface 11 is shown in FIG.

[0102] In response to an operation on the production control (eg, clicking the production control), the application 1 selects the audio 1 corresponding to the material. The production control involved here can refer to the aforementioned Figure 1 Production controls 121 shown in FIG.

[0103] Application 1 selects audio 1 corresponding to the material, including but not limited to: selecting audio with a total time close to or equal to that of the material as audio 1. Or selecting a default audio as audio 1 corresponding to the material.

[0104] After selecting the audio 1 corresponding to the material, the application 1 sends the path of the audio 1 to the file parsing module. The file parsing module can obtain the audio 1 (encoded) in the memory of the electronic device based on the path of the audio 1. Then, the following step S12 is performed to analyze the length of the audio 1 to determine the number of audio decoders required to decode the audio 1.

[0105] S12. The file parsing module determines the number N of audio decoders according to the length of the audio.

[0106] The length of the audio here can be understood as the length of audio 1.

[0107] When the length of the audio 1 is less than the threshold, N is equal to 1. When the length of the first audio is greater than or equal to the threshold, N is greater than or equal to 2.

[0108] The threshold value may be set to 4 minutes, or other values, such as 3 minutes, etc. The threshold value may be set based on the processing capability of the electronic device, and the embodiment of the present application does not limit this.

[0109] S13. The file parsing module obtains the audio format, including the encoding format and audio parameters.

[0110] Audio format refers to the way audio 1 is stored in electronic devices, which includes encoding format and audio parameters.

[0111] The encoding format of audio 1 refers to an algorithm for encoding a continuous analog audio signal into audio 1. The encoding format of audio 1 includes but is not limited to one of MP3, AAC, WAV or FLAC, etc., and the embodiment of the present application does not limit this.

[0112] For emphasis, the encoding format of audio 1 may be referred to as encoding format 1 (eg, MP3) hereinafter, and the audio parameters of audio 1 may be referred to as audio parameters 1 .

[0113] Audio parameter 1 includes parameters such as sampling rate, number of channels, and bit depth corresponding to audio 1.

[0114] The sampling rate corresponding to audio 1 refers to the rate at which a continuous analog audio signal is sampled to obtain audio 1. It can also be understood as the number of sample points (sampling points for short) collected per unit time (per second), or the number of sampling points contained in the unit time of audio 1. Common sampling rates include 44.1kHz, 48kHz, etc.

[0115] The number of channels corresponding to audio 1 indicates the number of channels of audio 1. Common channel numbers include mono, stereo, and surround. Mono has only one channel, while stereo has left and right channels, and surround has more channels.

[0116] The bit depth corresponding to audio 1 refers to the number of binary bits used to store the sampling points. Common bit depths include 8 and 16. 16-bit depth means that 16 binary bits can be used to represent the value of the sampling point, which means that the value range of the sampling point is 2 16 discrete values.

[0117] After determining the audio format and the number of audio decoders N, the file parsing module may send the audio format of audio 1 and the value of N to the audio decoding module. After receiving the audio format and the value of N, the audio decoding module may execute the following step S14 to start creating an audio decoder.

[0118] S14. The audio decoding module creates N audio decoders that support encoding format decoding.

[0119] Creating N audio decoders that support encoding format decoding means that the audio decoders can read audio 1 encoded according to encoding format 1 (such as MP3) and decompress the audio 1 to obtain decoded audio 1.

[0120] S15. The audio decoding module configures the N audio decoders according to the audio parameters.

[0121] Configuring N audio decoders means setting the parameters of the audio decoders according to the audio parameters 1 (sampling rate, number of channels, and bit depth) of the audio 1 (encoded) to adapt to the specific audio parameters of the audio 1 (encoded), so that the audio decoders can correctly parse the audio 1 (encoded) with the audio parameters 1 and output the decoded audio 1 with the audio parameters 1.

[0122] S16. The audio decoding module starts the N decoders.

[0123] Starting the N decoders means that the audio 1 (encoded) can be sampled and decoded based on the N decoders created and configured above.

[0124] It should be noted here that when using N audio decoders to decode audio 1 (encoded), audio 1 is divided into N sub-audio segments. One sub-audio segment is decoded using one audio decoder, and the N audio decoders are decoded in parallel. In particular, when N is equal to 1, audio 1 is also a sub-audio segment, and audio 1 can be decoded using one audio decoder. The process of using the i-th audio decoder to decode the i-th sub-audio segment can refer to steps S171-S173. Among them, the value of i is from 1 to N, and the N audio decoders decode in parallel.

[0125] In step S171-step S173, the i-th audio decoder is also configured with X1 input buffers and corresponding X1 output buffers when decoding. The input buffer is used to temporarily record the 1 / H of the audio frame to be decoded, and the output buffer is used to temporarily record the audio (decoded frame) after the 1 / H frame is decoded. The decoded frame in the output buffer will be recorded in the i-th audio queue by the buffer merging module. X1 is an integer greater than or equal to 1.

[0126] S171. The file parsing module obtains the i-th sub-audio of audio 1, divides the i-th sub-audio into Mi frames, and gradually transmits 1 / H of each audio frame to the i-th audio decoder.

[0127] The file parsing module may first record 1 / H of each audio frame into an input buffer, and send the sequence number of the input buffer to the i-th audio decoder.

[0128] S172. The file parsing module uses the i-th audio decoder to gradually decode the 1 / H of each audio frame to obtain the audio (decoded frame) after the 1 / H frame is decoded.

[0129] The i-th audio decoder finds the corresponding input buffer based on the sequence number involved in the aforementioned step S171 and reads the data therein for decoding to obtain the audio (decoded frame) after 1 / H frame decoding, and reads the decoded frame into the output buffer.

[0130] S173. The cache merging module caches the decoded frame corresponding to the i-th sub-audio into the i-th audio queue.

[0131] The cache merging module writes the decoded frames of the output buffer into the i-th audio queue.

[0132] It should be noted here that in the aforementioned steps S171-S173, the number of input buffers and output buffers is limited. Usually X1 input buffers cannot accommodate all the data to be decoded in the i-th sub-audio. Therefore, during the decoding process, after the data in the input buffer is transmitted to the audio decoder, the input buffer will be empty and the 1 / H of the next audio frame can be read in. Similarly, after the data in the output buffer is written to the i-th audio queue, the output buffer will be empty, and the new 1 / H frame of decoded audio can be put in.

[0133] Here, 1 / H of each audio frame in audio 1 is read into the input buffer. The process of then decoding the data in the input buffer and reading the decoded frame into the output buffer is called the read process. The process of reading the data in the output buffer, writing it to the audio queue and releasing the output buffer is called the write process. It can also be understood that the read process is the process of reading 1 / H of the audio frame, decoding it, and temporarily recording the decoded frame to the output buffer to wait for entering the audio queue. The write process is the process of writing the decoded frame in the output buffer to the audio frame queue to wait for merging to obtain the decoded audio 1.

[0134] In some possible cases, the reading process and the writing process can be set in one thread (reading and writing thread), that is, the reading process and the writing process are serial. In this way, since only one thread is involved in decoding, it is easy to manage the decoding process. For the decoding process when the reading process and the writing process are serial, please refer to the following Fig. 7A as well as Figure 7B The description of the relevant content will not be repeated here.

[0135] In some possible cases, the reading process can be set in one thread (reading thread), but the writing process can be set in another thread (writing thread), that is, the reading process and the writing process are parallel. In this way, the decoding speed can be accelerated. For the decoding process when the reading process and the writing process are parallel, please refer to the following Fig. 8A as well as Figure 8B The description of the relevant content will not be repeated here.

[0136] After N audio decoders complete decoding and obtain N audio queues, the cache merging module can execute the following step S18 to merge the decoded frames in the N audio queues to obtain the decoded audio 1.

[0137] S18. The cache merging module merges the decoded frames in the N audio queues to obtain decoded audio 1, and the sampling rate corresponding to the decoded audio 1 is equal to 1 / H of audio 1.

[0138] The sampling rate corresponding to the decoded audio 1 (audio 2) is equal to 1 / H of the encoded audio 1, which means that the number of sampling points contained in audio 2 is 1 / H of the number of sampling points of audio 1 (encoded).

[0139] In some possible cases, before dividing audio 1 (encoded) into N sub-audio segments for decoding, due to the low precision of the algorithm used, two adjacent sub-audio segments in the N sub-audio segments may be divided to overlap, and thus the two adjacent sub-audio segments also have overlapping parts after decoding. Therefore, before merging the decoded frames in the N audio queues to obtain the decoded audio 1, deduplication processing is required, and then the N decoded sub-audio segments after deduplication are merged according to the timestamps, and the merged audio is used as the decoded audio 1.

[0140] Here, we take two adjacent sub-audios, the e-th sub-audio and the e+1-th sub-audio, as an example. Figure 4 The deduplication process is described below. The two adjacent sub-audio segments are called the decoded e-th sub-audio segment and the decoded e+1-th sub-audio segment after decoding. The value of e is 1 to N-1.

[0141] like Figure 4 As shown in (1), the end of the e-th audio segment and the beginning of the e+1-th audio segment overlap. If deduplication is not performed, Figure 4 As shown in (2), there will be overlap in the decoded audio 1. If deduplication is performed, such as Figure 4 As shown in (3), the cache merging module can compare the e-th sub-audio segment and the e+1-th sub-audio segment to determine the location of the overlapping portion. Then, the overlapping portion is removed from the decoded e-th sub-audio segment or the decoded e+1-th sub-audio segment. Figure 4As shown in (4), the overlapping part of the decoded e-th audio segment is deleted. Figure 4 As shown in (5), the e-th audio segment and the decoded e+1-th audio segment are then merged. There is no overlapping part in the merged audio.

[0142] S19. The resampling module configures resampling parameters, resamples the decoded audio 1, and obtains the resampled audio 1 adapted to the node analysis algorithm.

[0143] It should be noted that step S19 is optional. When the decoded audio 1 is audio adapted to the node analysis algorithm, step S19 does not need to be performed.

[0144] Step S19 is only executed when the decoded audio 1 (audio 2) is not audio that is compatible with the node analysis algorithm.

[0145] In step S19, the decoded audio 1 may be resampled to change the audio properties, thereby obtaining the resampled audio 1 (audio 3) adapted to the node analysis algorithm.

[0146] For example, when the number of sampling points contained in the unit time length of audio 2 is not equal to the number of sampling points 1, but the input audio requires that the number of sampling points contained in the unit time length is equal to the number of sampling points 1, the electronic device can resample the number of sampling points contained in the unit time length of audio 2 to the number of sampling points 1, and obtain audio 3. Among them, the number of sampling points 1 is the number of sampling points contained in the unit time length of the input audio corresponding to the node analysis algorithm.

[0147] The purpose of resampling is to change the properties (audio parameters) of audio 2 to obtain audio 3. The properties here include the number of sampling points contained in the unit time length, bit depth, number of channels, etc., which are not limited in the embodiment of the present application.

[0148] For other descriptions about resampling, please refer to the aforementioned related content and will not be repeated here.

[0149] After obtaining the audio (audio 3) adapted to the node analysis algorithm, the audio 3 is analyzed in parallel based on the node analysis algorithm. The process of obtaining the audio nodes can refer to the following steps S20 and S21.

[0150] Combine the following Figure 5 Step S20 and step S21 are described in detail.

[0151] S20. The resampling module sends the resampled audio 1 to the node analysis module in parallel.

[0152] The resampling module sends the resampled audio 1 to the node analysis module, including: dividing the resampled audio 1 into R segments of sub-audio. In the R segments of sub-audio, there is overlapping audio at the end of the jth sub-audio and the beginning of the j+1th sub-audio, and the degree of overlap is related to the node analysis algorithm. Wherein, R is greater than or equal to 2, and j ranges from 1 to R-1.

[0153] The overlap degree is related to the node analysis algorithm, which means that when the node analysis algorithm analyzes the audio (audio a) corresponding to a timestamp, it will use the previous audio (audio b) and the following audio (audio c) of the audio a to perform comparative analysis with the audio a to determine whether the timestamp corresponding to the audio a is an audio node. Therefore, if you need to obtain all the audio nodes in the resampled audio 1, the length of the overlap between the jth sub-audio and the j+1th sub-audio is at least equal to (or equal to): the length of audio a + audio b + audio c.

[0154] Then, if Figure 5 As shown, step S201 and step S202 are executed in parallel to send the R segment sub-audio to the beat analysis algorithm for decoding.

[0155] S201. Send the jth sub-audio in the resampled audio 1.

[0156] S202. Send the j+1th sub-audio segment in the resampled audio 1.

[0157] Subsequently, step S21 is executed to perform node analysis in parallel on each sub-audio segment in the resampled audio 1 using a beat analysis algorithm.

[0158] S21. The resampling module performs node analysis on the resampled audio 1.

[0159] For the description of step S21, please refer to Figure 5 Steps S211-S213 shown in FIG.

[0160] S211. After receiving the j-th sub-audio segment, perform node analysis on the j-th sub-audio segment to obtain the audio node corresponding to the j-th sub-audio segment.

[0161] S212. After receiving the j+1th sub-audio segment, perform node analysis on the j+1th sub-audio segment to obtain an audio node corresponding to the j+1th sub-audio segment.

[0162] S213. Merge the audio nodes corresponding to the R audio segments, and delete one of the two audio nodes with duplicate timestamps to obtain T audio nodes corresponding to audio 1.

[0163] Since the jth sub-audio segment and the j+1th sub-audio segment in the R sub-audio segment have overlapping audio when the resampled audio 1 is divided into R sub-audio segments, the audio nodes in the jth sub-audio segment and the audio nodes in the j+1th sub-audio segment may have repeated audio nodes. The repeated audio nodes come from the overlapping audio.

[0164] Before merging the audio nodes corresponding to the R sub-audios, it is necessary to delete the audio nodes repeated in the j-th sub-audio and the j+1-th sub-audio to obtain T audio nodes corresponding to audio 1.

[0165] refer to Figure 6 Here, the R segment sub-audio includes sub-audio 1, sub-audio 2, and sub-audio 3 as an example for explanation. The audio nodes corresponding to sub-audio 1 include [011, 014, 016, 021]. The audio nodes corresponding to sub-audio 2 include [021, 026, 030, 034]. The audio nodes corresponding to sub-audio 3 include [034, 037, 044]. In fact, the audio node corresponding to sub-audio 1 and the audio node corresponding to sub-audio 2 have the same audio node 021, so the audio node 021 in sub-audio 1 is deleted. Then the audio node corresponding to sub-audio 1 is updated to [011, 014, 016]. The audio node corresponding to sub-audio 2 and the audio node corresponding to sub-audio 3 have the same audio node 034, so the audio node 034 in sub-audio 2 is deleted. Then the audio node corresponding to sub-audio 2 is updated to [021, 026, 030]. Here, sub-audio 3 is already the last segment of the sub-audio in the R segment sub-audio, and its corresponding audio node remains unchanged. Subsequently, after merging the audio nodes corresponding to sub-audio 1 (updated), the audio nodes corresponding to sub-audio 2 (updated), and the audio nodes corresponding to sub-audio 3, T audio nodes corresponding to the audio can be obtained [011, 014, 016, 021, 026, 030, 034, 037, 044].

[0166] After obtaining the T audio nodes corresponding to audio 1, the node analysis algorithm can send the T audio nodes corresponding to audio 1 to application 1. Subsequently, application 1 can process the material based on the audio nodes to obtain the stuck point video. For the process of obtaining the stuck point video, please refer to the above-mentioned related content, which will not be repeated here.

[0167] Let's take N=2 as an example. Fig. 7A as well as Figure 7B The above-mentioned decoding process when the reading process and the writing process are serially performed after the N audio decoders are turned on is described in detail.

[0168] For the description of this process, reference may be made to the following description of step S31, step S32, step S33a-step S43a, and step S33b-step S43b.

[0169] S31. The audio decoding module creates and configures two audio decoders.

[0170] S32. The audio decoding module starts the two audio decoders.

[0171] The two audio decoders include audio decoder 1 and audio decoder 2 .

[0172] The content involved in step S31 is the same as that involved in the aforementioned step S14 and step S15, and the content involved in step S32 is the same as that involved in the aforementioned step S16. Please refer to the aforementioned description of the relevant content, which will not be repeated here.

[0173] After starting the two audio decoders, the audio decoding module executes step S33a to create a read / write thread 1 for audio decoder 1, and executes step S33b to create a read / write thread 2 for audio decoder 2.

[0174] After the read / write thread 1 is created, the audio decoder 1 can decode the first 1 / 2 of the content in the audio 1 based on the read / write thread 1. And, after the read / write thread 2 is created, the audio decoder 2 can decode the last 1 / 2 of the content in the audio 1 based on the read / write thread 2. The decoding processes of the two audio decoders are similar here, except that the objects to be decoded are different. The following text takes the decoding process of the audio decoder 1 based on the read / write thread 1 as an example for explanation. The decoding process of the audio decoder can refer to this description and will not be repeated.

[0175] The process of audio decoder 1 decoding the first 1 / 2 of the content in audio 1 based on read / write thread 1 can refer to the description of the following steps S34a to S43a.

[0176] S34a. The file parsing module determines whether there is an idle input buffer.

[0177] Idle means there is no data in the input buffer.

[0178] As long as it is determined that there is still an idle input buffer, the read / write thread 1 executes the following step S35a through the file parsing module to enter the reading process.

[0179] After determining that there is no idle input buffer, the read / write thread 1 executes the following step S40a through the buffer merging module to enter the writing process.

[0180] S35a. The file parsing module determines an idle input buffer 1.

[0181] The file parsing module may first obtain the sequence number of the idle input buffer, and find the idle input buffer based on the sequence number. For the sake of distinction, the idle input buffer is referred to as input buffer 1 for illustration.

[0182] The way in which the file parsing module obtains the sequence number includes but is not limited to: the file parsing module can inquire the audio decoder 1 about the idle input buffer sequence number, or the audio decoder 1 can send the idle input buffer sequence number to the file parsing module.

[0183] S36a. Check whether the timestamp of the read audio frame is the middle time of audio 1.

[0184] When the timestamp of the read audio frame does not reach the middle time of audio 1, the read-write thread 1 executes the following step S37a through the file parsing module to continue the reading process.

[0185] When the timestamp of the read audio frame reaches the middle time of audio 1, the read-write thread 1 executes the following step S40a through the buffer merging module to enter the writing process.

[0186] It should be noted here that the timestamp of the first frame of audio read by audio decoder 1 is the start time of audio 1.

[0187] S37a. The file parsing module reads 1 / H of the audio frame into the idle input buffer 1.

[0188] It should be understood that the file parsing module obtains the 1 / H of each audio frame from the front to the back according to the timestamp, and then puts it into the input buffer. The 1 / H of the audio frame that enters the input buffer first will be decoded first, then read into the output buffer and then written into the audio frame queue. In this way, the audio is guaranteed not to be out of order during the decoding process.

[0189] S38a. The file parsing module sends the sequence number 1 of the input buffer 1 to the audio decoder 1.

[0190] S39a. The audio decoder 1 reads the 1 / H frame audio from the corresponding buffer based on the sequence number 1 for decoding, and sends the 1 / H frame decoded audio to the output buffer.

[0191] The aforementioned steps S35a, S37a to S39a describe the reading process. For the description of the writing process, reference may be made to the description of the following steps S40a to S42a.

[0192] S40a. The cache merging module determines whether there is an output buffer with data.

[0193] Among them, the way in which the file parsing module determines whether there is an output buffer with data includes but is not limited to: the file parsing module can inquire the audio decoder 1 about the sequence number of the idle output buffer, or the audio decoder 1 can send the sequence number of the idle output buffer to the file parsing module.

[0194] When it is determined that there is no output buffer with data, the following step S35a is executed to enter the reading process.

[0195] When it is determined that there is an output buffer with data, the aforementioned step S41a is executed to continue the writing process.

[0196] S41a. The cache merging module determines the output buffer 1 having data and obtains the decoded audio of the 1 / H frame therein, and writes the obtained decoded audio of the 1 / H frame into the first audio queue.

[0197] Output buffer 1 is the output buffer with data.

[0198] S42a. Check whether the output buffer 1 has an end mark.

[0199] If there is no end mark in the output buffer 1, it means that the decoding process is not completed, and the above step S40a is continued to be executed to re-determine whether to re-enter the reading process or continue the writing process.

[0200] If there is an end mark in the output buffer 1, it means that the decoding process has been completed, and the following step S43a is executed to end the reading thread.

[0201] S43a. End read / write thread 1.

[0202] The process of audio decoder 2 decoding the last 1 / 2 of the content in audio 2 based on read-write thread 2 can refer to the following steps S34b to S43b.

[0203] S34b. The file parsing module determines whether there is an idle input buffer.

[0204] S35b. The file parsing module determines an idle input buffer 2.

[0205] S36b. Check whether the timestamp of the read audio frame is the end time of audio 1.

[0206] When the timestamp of the read audio frame is not the end time of audio 1, the read-write thread 1 executes the following step S37b through the file parsing module to continue the reading process.

[0207] It should be noted here that the timestamp of the first frame of audio read by audio decoder 2 is the middle time of audio 1.

[0208] When the timestamp of the read audio frame is the end time of audio 1, the read-write thread 1 executes the following step S40b through the buffer merging module to enter the writing process.

[0209] S37b. The file parsing module reads 1 / H of the audio frame into the idle input buffer 2.

[0210] S38b. The file parsing module sends the sequence number 2 of the input buffer 2 to the audio decoder 2.

[0211] S39b. The audio decoder 1 reads the 1 / H frame audio from the corresponding buffer based on the sequence number 2 for decoding, and sends the 1 / H frame decoded audio to the output buffer.

[0212] S40b. The cache merging module determines whether there is an output buffer with data.

[0213] S41b. The cache merging module determines the output buffer 2 having data and obtains the decoded audio of the 1 / H frame therein, and writes the obtained decoded audio of the 1 / H frame into the second audio queue.

[0214] S42b. Check whether the output buffer 2 has an end mark.

[0215] S43b. End read / write thread 2.

[0216] Steps S34b to S43b are similar to the previous steps S34a to S43a, and the numbers can be modified accordingly: for example, audio decoder 1 is changed to audio decoder 2, input buffer 1 is changed to input buffer 2, etc. The description of the previous steps S34a to S43a can be referred to, and will not be repeated here.

[0217] Based on the aforementioned Fig. 7A From the content in , we can conclude that when the reading process and the writing process are serial, if the input buffer is free, the decoded frame in the output buffer cannot be written into the audio frame queue. Conversely, if there is an output buffer with data, the 1 / H of the audio frame cannot be read into the input buffer for decoding.

[0218] refer to Figure 7B, in the scenario where audio decoder 1 decodes the first 1 / 2 of audio 1, and audio decoder 2 decodes the second 1 / 2 of audio 1. When the reading process and the writing process are serial, for audio decoder 1, input buffer 13 is an input buffer with data. The data in input buffer 11 and input buffer 12 (1 / H of the audio frame) have been transmitted to the decoder for decoding, and the decoded audio frames have been read into output buffer 11 and output buffer 12 respectively. Now input buffer 11 and input buffer 12 are both idle input buffers. Due to the existence of idle input buffers, even if output buffer 11 and output buffer 12 already have data, it still cannot be written into audio frame queue 1. For audio decoder 2, output buffer 21 and output buffer 23 are both output buffers with data. Even if input buffer 21, input buffer 22 and input buffer 23 are all idle input buffers, data cannot be read into input buffer 21-input buffer 23.

[0219] It can be seen that when the reading process and the writing process are serial, the reading process and the writing process are mutually restricted, which has an impact on the decoding speed. Figure 7B In the example, the output buffer 21 is an idle output buffer, but data cannot be read into the input buffers 21 to 23 to be decoded and read into the output buffer 21.

[0220] Reference again Figure 7B After the read-write thread 1 and the read-write thread 2 are both finished, the cache merging module can merge the data in the audio frame queue 1 and the audio frame queue 2 to obtain the decoded audio 1.

[0221] Let's take N=2 as an example. Fig. 8A as well as Figure 8B The above-mentioned decoding process when the reading process and the writing process are carried out in parallel after the N audio decoders are turned on is described in detail.

[0222] For the description of this process, reference may be made to the following description of step S31, step S32, and steps S331 to S334.

[0223] S31. The audio decoding module creates and configures two audio decoders.

[0224] S32. The audio decoding module starts the two audio decoders.

[0225] The two audio decoders include audio decoder 1 and audio decoder 2 .

[0226] The content involved in step S31 is the same as that involved in the aforementioned step S14 and step S15, and the content involved in step S32 is the same as that involved in the aforementioned step S16. Please refer to the aforementioned description of the relevant content, which will not be repeated here.

[0227] After starting the two audio decoders, the audio decoding module executes step S331 to create a read thread 1 for audio decoder 1 and executes step S332 to create a write thread 1 for audio decoder 1. Also, the audio decoding module executes step S333 to create a read thread 2 for audio decoder 2 and executes step S334 to create a write thread 2 for audio decoder 2.

[0228] After the read thread 1 and the write thread 1 are created, the audio decoder 1 can decode the first 1 / 2 of the content in the audio 1 in parallel based on the read thread 1 and the write thread 1. And, after the read thread 2 and the write thread 2 are created, the audio decoder 2 can decode the last 1 / 2 of the content in the audio 1 in parallel based on the read thread 2 and the write thread 2. The decoding processes of the two audio decoders here are similar, except that the objects being decoded are different. The decoding process of the audio decoder 1 based on the read thread 1 and the write thread 1 is used as an example to illustrate below. The decoding process of the audio decoder can refer to this description and will not be repeated here.

[0229] For the contents of the reading process performed by the audio decoder 1 based on the reading thread 1, please refer to the description of the following steps S331a to S331e. For the contents of the writing process performed by the writing thread 1, please refer to the description of the following steps S332a to S332e.

[0230] First, based on steps S331a to S331e, the contents involved in the reading process performed by the audio decoder 1 based on the reading thread 1 are described.

[0231] S331a. The file parsing module determines that there is an idle input buffer 1.

[0232] The method of determining whether there is an idle input buffer 1 in step S331a is the same as the process of determining an idle input buffer 1 in step S35. Please refer to the above related content and will not be repeated here.

[0233] S331b. Check whether the timestamp of the read audio frame is the middle time of audio 1.

[0234] When the timestamp of the read audio frame does not reach the middle time of audio 1, reading thread 1 executes the following step S331c through the file parsing module to continue the reading process.

[0235] When the timestamp of the read audio frame reaches the middle time of audio 1, the reading thread 1 is terminated.

[0236] It should be noted here that the timestamp of the first frame of audio read by audio decoder 1 is the start time of audio 1.

[0237] S331c. The file parsing module reads 1 / H of the audio frame into the idle input buffer 1.

[0238] The description of step S331c can refer to the aforementioned step S37a and will not be repeated here.

[0239] S331d. The file parsing module sends the sequence number 1 of the input buffer 1 to the audio decoder 1.

[0240] S331e. The audio decoder 1 reads the 1 / H frame audio from the corresponding buffer based on the sequence number 1 for decoding, and sends the decoded audio of the 1 / H frame to the output buffer.

[0241] Based on steps S332a to S332e, the contents involved in the writing process of the audio decoder 1 based on the writing thread 1 are described.

[0242] S332a. The cache merging module determines the output buffer 1 that has data.

[0243] The content involved in step S332a is the same as the process of determining the output buffer 1 with data in the aforementioned step S40a. Please refer to the relevant content in the aforementioned step S40a, and it will not be repeated here.

[0244] S332b. Check whether the output buffer 1 has an end mark.

[0245] If there is an end mark in output buffer 1, it means that the writing process is completed, and then writing thread 1 is terminated.

[0246] If there is no end mark in the output buffer 1, it means that the writing process is not completed, and the following step S332c is executed to continue the writing process.

[0247] S332c. The cache merging module obtains the decoded audio of the 1 / H frame from the output buffer 1.

[0248] S332d. The cache merging module writes the obtained 1 / H frame decoded audio into the first audio queue.

[0249] S332e. The cache merging module releases the output buffer 1 and sends the sequence number of the output buffer 1 to the audio decoder 1.

[0250] The purpose of sending the serial number of the released output buffer 1 to the audio decoder 1 is to make the audio decoder 1 record the state of the output buffer 1 as no data (idle) so as to record the decoded frame into the idle output buffer 1.

[0251] After both the reading thread and the writing thread are finished, it is determined that the audio decoder 1 completes decoding of the first 1 / 2 of the content of the audio 1 .

[0252] The contents of the reading process performed by the audio decoder 2 based on the reading thread 2 can refer to the description of the following steps S333a to S333e. The contents of the writing process performed by the writing thread 2 can refer to the description of the following steps S334a to S334e.

[0253] First, based on steps S333a to S333e, the contents of the reading process performed by the audio decoder 2 based on the reading thread 2 are described.

[0254] S333a. The file parsing module determines that there is an idle input buffer 2.

[0255] S333b. Check whether the timestamp of the read audio frame is the end time of audio 1.

[0256] When the timestamp of the read audio frame does not reach the end time of audio 1, the reading thread 2 executes the following step S333c through the file parsing module to continue the reading process.

[0257] When the timestamp of the read audio frame reaches the end time of audio 1, reading thread 2 is terminated.

[0258] It should be noted here that the timestamp of the first frame of audio read by audio decoder 2 is the middle time of audio 1.

[0259] S333c. The file parsing module reads 1 / H of the audio frame into the idle input buffer 2.

[0260] S333d. The file parsing module sends the sequence number 2 of the input buffer 2 to the audio decoder 2.

[0261] S333e. The audio decoder 2 reads the 1 / H frame audio from the corresponding buffer based on the sequence number 2 for decoding, and sends the decoded audio of the 1 / H frame to the output buffer.

[0262] Steps S333a to S333e are similar to the previous steps S331a to S331e, and the numbers can be modified accordingly: for example, audio decoder 1 is changed to audio decoder 2, input buffer 1 is changed to input buffer 2, etc. The description of the previous steps S331a to S331e can be referred to, and will not be repeated here.

[0263] Then, based on steps S334a to S334e, the content of the writing process performed by the audio decoder 2 based on the writing thread 2 is described.

[0264] S334a. The cache merging module determines that output buffer 2 has data.

[0265] S334b. Check whether the output buffer 2 has an end mark.

[0266] If there is an end mark in the output buffer 2, it means that the writing process is completed, and the writing thread 2 is terminated.

[0267] If there is no end mark in the output buffer 2, it means that the writing process is not completed, and the following step S334c is executed to continue the writing process.

[0268] S334c. The cache merging module obtains the decoded audio of the 1 / H frame from the output buffer 2.

[0269] S334d. The cache merging module writes the obtained 1 / H frame decoded audio into the second audio queue.

[0270] S334e. The cache merging module releases the output buffer 2 and sends the sequence number of the output buffer 2 to the audio decoder 1.

[0271] Steps S334a to S334e are similar to the previous steps S332a to S332e, and the numbers can be modified accordingly: for example, audio decoder 1 is changed to audio decoder 2, input buffer 1 is changed to input buffer 2, etc. The description of the previous steps S332a to S332e can be referred to, and will not be repeated here.

[0272] Steps S333a to S333e are similar to the previous steps S331a to S331e, and the numbers can be modified accordingly: for example, audio decoder 1 is changed to audio decoder 2, input buffer 1 is changed to input buffer 2, etc. The description of the previous steps S331a to S331e can be referred to, and will not be repeated here.

[0273] Based on the aforementioned Fig. 8A From the content in, we can conclude that when the reading process and the writing process are in parallel, the reading process and the writing process do not interfere with each other: if the input buffer is free, 1 / H of the audio frame can be read in for decoding. If there is data in the output buffer, the decoded frame in the output buffer can be written to the audio frame queue.

[0274] refer to Figure 8B, in the scenario where audio decoder 1 decodes the first 1 / 2 of audio 1, and audio decoder 2 decodes the second 1 / 2 of audio 1. When the reading process and the writing process are in parallel, for audio decoder 1, input buffer 13 is an input buffer with data. The data in input buffer 11 and input buffer 12 (1 / H of the audio frame) have been transmitted to the decoder for decoding, and the decoded audio frames have been read into output buffer 11 and output buffer 12 respectively. Now input buffer 11 and input buffer 12 are both idle input buffers. Compare Figure 7B , even if the input buffer 11 and the output buffer 12 are idle, the data in the output buffer 11 can still be written to the audio frame queue 1, and the data in the output buffer 12 can still be written to the audio frame queue 2. For the audio decoder 2, the output buffer 21 and the output buffer 23 are both output buffers with data, and the input buffer 21, the input buffer 22 and the input buffer 23 are all idle input buffers. Figure 7B When writing the data in the output buffer 21 and the output buffer 23 to the audio frame queue 2, data can also be read into the input buffer 21-input buffer 23. In this way, the audio decoder can decode the data in the input buffer 11 and write it into the idle output buffer 21.

[0275] Reference again Figure 8B After the read thread 1 and the write thread 1 as well as the read thread 2 and the write thread 2 are finished, the cache merging module can merge the data in the audio frame queue 1 and the audio frame queue 2 to obtain the decoded audio 1.

[0276] Based on the above Figure 7B as well as Figure 8B It can be seen that Fig. 8A The solution of parallel reading and writing threads shown in Fig. 7A The serial solution of the reading process and writing process shown in can speed up the decoding speed. However, if there is a situation where the audio decoder fails and the decoding is suspended, the reading thread and the writing thread corresponding to the audio decoder of the failure need to stop working at the same time. Otherwise, some decoding errors will be caused: for example, if the reading thread continues to read data in the input buffer and reports the sequence number of the input buffer to the audio decoder, the audio decoder cannot check the sequence number of the input buffer of the data that has been read, and the situation of leaking decoding will be caused during subsequent recovery. If the writing thread continues to write the data in the output buffer into the audio frame queue, and releases the output buffer and sends the sequence number of the released output buffer to the audio decoder, the audio decoder cannot check it either.

[0277] It should be noted here that the aforementioned Fig. 8AIn the example, the read thread 1 and the write thread 1 corresponding to the audio decoder 1 are two parallel threads, and the execution of the two threads is independent. The steps in the read thread 1 (including steps S331a-S331e) and the write thread 1 (including steps S332a-S332e) have no sequence, for example, there is no sequence difference between step S331a and step S332a. The read thread 2 and the write thread 2 corresponding to the audio decoder 2 are also two parallel threads, and the execution of the two threads is independent. The steps in the read thread 2 (including steps S333a-S333e) and the write thread 2 (including steps S334a-S334e) have no sequence, for example, there is no sequence difference between step S331a and step S332a.

[0278] It should also be noted that the aforementioned Figure 3 There is no order of execution between step S12 and step S13. Figure 5 Step S201 and step S202 are executed in parallel, and step S211 and step S21 are also executed in parallel.

[0279] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0280] The following is a detailed description of the embodiment using an electronic device as an example. It should be understood that the electronic device may have more than Fig. 9 More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Fig. 9 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0281] The electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194 and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0282] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0283] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0284] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0285] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0286] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, etc.

[0287] It is understandable that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0288] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0289] The power management module 141 is used to connect the battery 142 , the charging management module 140 and the processor 110 .

[0290] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0291] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas.

[0292] The mobile communication module 150 may provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0293] A modem processor may include a modulator and a demodulator.

[0294] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), infrared technology (IR), etc. for application in electronic devices.

[0295] In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include the global system for mobile communications (GSM), general packet radio service (GPRS), etc.

[0296] The electronic device realizes the display function through a GPU, a display screen 194, and an application processor.

[0297] The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0298] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be made of an organic light-emitting diode (OLED).

[0299] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0300] The ISP is used to process data fed back by the camera 193 .

[0301] The camera 193 is used to capture still images or videos. The object is projected onto the photosensitive element by generating an optical image through the lens.

[0302] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0303] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0304] NPU is a neural-network (NN) computing processor. It can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn.

[0305] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0306] The external memory interface 120 may be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100 .

[0307] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0308] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.

[0309] Non-volatile memory may include disk storage devices and flash memory.

[0310] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle, and can be divided into single-level cell (SLC), multi-level cell (MLC), etc. according to the potential level of the storage cell.

[0311] The pressure sensor 180A is used to sense pressure signals and convert the pressure signals into electrical signals.

[0312] The gyro sensor 180B may be used to determine the motion posture of the electronic device.

[0313] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0314] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device in various directions (generally three axes).

[0315] Distance sensor 180F, used to measure distance. Electronic devices can measure distance through infrared or laser.

[0316] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.

[0317] The ambient light sensor 180L is used to sense the brightness of ambient light.

[0318] The fingerprint sensor 180H is used to collect fingerprints. Electronic devices can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0319] The temperature sensor 180J is used to detect the temperature.

[0320] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device, which is different from the position of the display screen 194.

[0321] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0322] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0323] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0324] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or removing the SIM card from the SIM card interface 195 .

[0325] In the embodiment of the present application, the processor 110 can call the computer instructions stored in the internal memory 121 to enable the electronic device to execute the method in the embodiment of the present application.

[0326] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0327] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0328] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0329] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0330] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be separated from the processor, which is not limited in the embodiment of the present application.

[0331] Exemplarily, the memory can be a non-transitory processor, such as a read-only memory ROM, which can be integrated with the processor on the same chip or can be set separately on different chips. The embodiments of the present application do not specifically limit the type of memory and the setting method of the memory and the processor.

[0332] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0333] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0334] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0335] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0336] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0337] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0338] The terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0339] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0340] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. An audio processing method, characterized in that: The method comprises: The electronic device divides the first audio into N sub-audio segments, where N is a positive integer; The N audio decoders are used to sample and decode the N sub-audio segments in the first audio respectively to obtain the second audio; the i-th audio decoder among the N audio decoders is used to sample and decode the i-th sub-audio segment in the first audio; the sampling decoding includes decoding 1 / H of each audio frame in the i-th sub-audio segment; the number of sampling points contained in the second audio is 1 / H of the number of sampling points of the first audio; H is an integer greater than or equal to 2; the value of i is 1 to N; The process of sampling and decoding the i-th audio sub-audio by the i-th audio decoder includes: dividing the i-th audio sub-audio into at least one frame; transmitting 1 / H of each audio frame in the at least one frame to the i-th audio decoder frame by frame; and decoding 1 / H of each audio frame frame by frame using the i-th audio decoder; When the number of sampling points contained in the unit time length of the second audio is different from the number of first sampling points, the electronic device resamples the second audio to obtain a third audio; the number of sampling points contained in the unit time length of the third audio is the same as the number of first sampling points; the first sampling points are the number of sampling points contained in the unit time length of the input audio of the node analysis algorithm; The electronic device performs audio analysis processing on the third audio through a node analysis algorithm to obtain an audio node in the first audio; the audio node is a timestamp with a special beat in the first audio.

2. The method according to claim 1, characterized in that When performing the resampling, the method further includes: The electronic device further adjusts audio parameters of the second audio except the number of sampling points contained in a unit time length, and the audio parameters include at least one of the number of channels and the bit depth corresponding to the second audio.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The electronic device performs deduplication processing on the decoded N sub-audios; Merge the N deduplicated sub-audios according to the timestamps, and use the merged audio as the second audio; Among them, the deduplication processing includes: determining that there is overlap between the decoded e-th sub-audio and the decoded e+1-th sub-audio; removing the overlapping part from the decoded e-th sub-audio or the decoded e+1-th sub-audio; the value of e is 1 to N-1.

4. The method according to any one of claims 1 to 3, characterized in that Before the electronic device divides the first audio into N sub-audios, the method further includes: The electronic device displays a first interface, wherein the first interface includes a selected video and a first control; In response to an operation on the first control, the electronic device determines the audio corresponding to the video as the first audio.

5. The method according to claim 4, characterized in that The method further comprises: After obtaining the audio nodes in the first audio, the electronic device adjusts at least one video segment in the video based on the audio nodes in the first audio to obtain an adjusted video; the adjusted video includes the first audio, and the original audio in the adjusted video is suppressed; the audio nodes of the first audio in the adjusted video are partially or completely aligned with the picture transitions in the video.

6. The method according to claim 2, characterized in that When the first audio is sampled and decoded by N audio decoders, the method further includes: The electronic device creates a read thread and a write thread of the i-th audio decoder; When the reading thread determines that the first input buffer is idle, write 1 / H of the first audio frame into the first input buffer; the first audio frame is an audio frame in the i-th sub-audio segment; The i-th audio decoder obtains I / H of the first audio frame from the first input buffer for decoding, and writes the decoded first audio frame into the first output buffer; When the read thread determines that the first input buffer is idle, the write thread determines that there is a decoded second audio frame in the second output buffer, stores the decoded second audio frame in the i-th audio frame queue and releases the second output buffer; the second audio frame is an audio frame before the first audio frame.

7. The method according to claim 6, characterized in that The method further comprises: When the timestamp of the audio frame read by the reading thread is equal to the end timestamp of the i-th sub-audio, the electronic device merges the audio frames in the i-th audio frame queue according to the timestamps to obtain the decoded i-th sub-audio.

8. The method according to any one of claims 1 to 7, characterized in that The electronic device performs audio analysis processing on the third audio by using a node analysis algorithm to obtain audio nodes in the first audio, specifically including: The electronic device divides the third audio into R sub-audio segments; there is overlapping audio between the end of the jth sub-audio segment and the beginning of the j+1th sub-audio segment in the R sub-audio segments; the value of j ranges from 1 to R-1; the degree of overlap is related to the audio analysis algorithm; R is greater than or equal to 2; The R segment sub-audios are analyzed based on the node analysis algorithm to obtain audio nodes corresponding to the respective R segment sub-audios.

9. The method according to claim 8, characterized in that The method further includes: merging the audio nodes corresponding to the R segment sub-audios to obtain the audio nodes in the first audio.

10. The method according to claim 9, characterized in that Before merging the audio nodes corresponding to the R segment sub-audios, the method further includes: Delete the audio nodes in the j-th audio sub-segment that are repeated in the j+1-th audio sub-segment.

11. The method according to any one of claims 1 to 10, characterized in that When the length of the first audio is less than a threshold, N is equal to 1; when the length of the first audio is greater than or equal to the threshold, N is greater than or equal to 2.

12. An electronic device, characterized in that: include: One or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1-11.

13. A computer-readable storage medium comprising computer instructions, characterized in that: When the computer instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 11.

14. A chip system, the chip system is applied to electronic equipment, characterized in that: The chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-11.

Images