Audio processing method and device, video processing method and device, medium and electronic equipment

Abstract

The invention provides an audio processing method, a video processing method, an audio processing device, a video processing device, a computer readable storage medium and electronic equipment, and relates to the technical field of audio processing. The audio processing method comprises the following steps: performing initial feature extraction operation on audio data to obtain a first audio feature; performing a processing process based on feature splitting on a time domain on the first audio feature to obtain a second audio feature; enhancing the difference between the voice feature and the non-voice feature in the second audio feature based on an attention mechanism to obtain a third audio feature; and performing full connection mapping operation on the third audio feature to obtain a voice detection result of the audio data. The accuracy of voice detection can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio processing, and in particular to an audio processing method, a video processing method, an audio processing device, a video processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the development of terminal technology, users' demand for albums is no longer limited to reviewing photos and videos, and the demand for multimodal intelligent editing is also increasing. In this context, Voice Activity Detection (VAD) technology plays an important role. As a preprocessing technology for audio intelligent processing, it can automatically divide the frames of audio signals into voice and non-voice. In addition, the effects of speech recognition, voice detection, voice separation and other technologies also rely on the processing results of voice activity detection.

[0003] Currently, active voice detection may have problems such as low detection accuracy and poor voice recognition effect. Summary of the invention

[0004] The present disclosure provides an audio processing method, a video processing method, an audio processing device, a video processing device, a computer-readable storage medium and an electronic device, thereby overcoming the problem of low accuracy of active voice detection at least to a certain extent.

[0005] According to a first aspect of the present disclosure, there is provided an audio processing method, comprising: performing an initial feature extraction operation on audio data to obtain a first audio feature; performing a processing process based on feature splitting in the time domain on the first audio feature to obtain a second audio feature; enhancing the difference between speech features and non-speech features in the second audio feature based on an attention mechanism to obtain a third audio feature; and performing a fully connected mapping operation on the third audio feature to obtain a speech detection result of the audio data.

[0006] According to a second aspect of the present disclosure, a video processing method is provided, comprising: acquiring a video, extracting audio data in the video; and performing voice detection on the audio data using the above audio processing method to determine a voice segment in the video.

[0007] According to a third aspect of the present disclosure, an audio processing device is provided, comprising: a first feature determination module, used to perform an initial feature extraction operation on audio data to obtain a first audio feature; a second feature determination module, used to perform a processing process based on feature splitting in the time domain on the first audio feature to obtain a second audio feature; a third feature determination module, used to enhance the difference between speech features and non-speech features in the second audio feature based on an attention mechanism to obtain a third audio feature; and a speech detection module, used to perform a fully connected mapping operation on the third audio feature to obtain a speech detection result of the audio data.

[0008] According to a fourth aspect of the present disclosure, a video processing device is provided, comprising: an audio extraction module for acquiring a video and extracting audio data in the video; and a segment determination module for performing voice detection on the audio data using the above-mentioned audio processing method to determine the voice segment in the video.

[0009] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned audio processing method or video processing method is implemented.

[0010] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising a processor; and a memory for storing one or more programs, wherein when the one or more programs are executed by the processor, the processor implements the above-mentioned audio processing method or video processing method.

[0011] In the technical solutions provided in some embodiments of the present disclosure, a processing process based on feature splitting in the time domain is performed on the first audio feature extracted from the audio data to obtain a second audio feature, the difference between speech and non-speech is enhanced based on the attention mechanism to obtain a third audio feature, and a fully connected mapping operation is performed on the third audio feature to obtain a speech detection result of the audio data. On the one hand, through the processing process based on feature differential in the time domain, multi-level features in the time domain can be effectively determined, which helps to improve the accuracy of speech detection; on the other hand, the disclosed solution combines the attention mechanism, which can enhance the difference between speech features and non-speech features, provide a good data basis for subsequent classification operations, and further improve the accuracy of speech detection.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0014] Figure 1 A schematic diagram showing an exemplary system architecture of an audio processing solution according to an embodiment of the present disclosure.

[0015] Figure 2 A schematic diagram of the input and output of the audio processing solution of an embodiment of the present disclosure is shown.

[0016] Figure 3 The flowchart of the audio processing method according to the exemplary embodiment of the present disclosure is schematically shown.

[0017] Figure 4 A schematic diagram showing the correction of speech detection results according to an embodiment of the present disclosure is shown.

[0018] Figure 5 The model architecture diagram of the audio processing model of an embodiment of the present disclosure is schematically shown.

[0019] Figure 6 The model architecture diagram of the time domain feature splitting processing unit corresponding to the time domain feature splitting sub-processing process of the embodiment of the present disclosure is schematically shown.

[0020] Figure 7 A schematic diagram of a process of generating training data for training an audio processing model according to an embodiment of the present disclosure is shown.

[0021] Figure 8 The flowchart of a video processing method according to an exemplary embodiment of the present disclosure is schematically shown.

[0022] Fig. 9 The flowchart of applying the exemplary scheme of the present disclosure to construct wonderful memories of children's voices is schematically shown.

[0023] Fig.10 A block diagram of an audio processing device according to an exemplary embodiment of the present disclosure is schematically shown.

[0024] Fig.11 A block diagram of an audio processing device according to another exemplary embodiment of the present disclosure is schematically shown.

[0025] Fig.12 The block diagram schematically shows a video processing device according to an exemplary embodiment of the present disclosure.

[0026] Fig.13A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0028] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0029] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation. In addition, all the terms "first", "second", "third", "fourth", etc. below are only for the purpose of distinction and should not be used as limitations of the present disclosure.

[0030] In some audio scenarios based on model processing, due to the variety of audio scenarios and the possibility of low signal-to-noise ratio, the current model is not robust enough. In addition, for long-term audio data, the modeling capabilities of existing models (such as convolutional neural networks) are difficult to achieve ideal results, resulting in poor speech detection results for audio data.

[0031] In order to at least improve the accuracy of voice detection of audio data, the present disclosure provides a new audio processing solution. Unless otherwise specified, the voice detection mentioned in the embodiments of the present disclosure refers to active voice detection, that is, detecting the voice part and the non-voice part in the audio data.

[0032] Figure 1 A schematic diagram showing an exemplary system architecture of an audio processing solution of an embodiment of the present disclosure is shown. Figure 1 As shown, the system architecture may include a terminal device 110 and a server 120. The terminal device 110 may interact with the server 120 via a network, and the network may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0033] The terminal device 110 may include but is not limited to a smart phone, a tablet computer, a smart wearable device, a personal computer, etc. The present disclosure does not limit the type of the terminal device 110. The server 120 may be a single server or a server cluster composed of multiple servers.

[0034] The audio processing scheme of the embodiment of the present disclosure may be executed by the terminal device 110 , by the server 120 , or jointly by the terminal device 110 and the server 120 .

[0035] In an example in which the audio processing solution of the embodiment of the present disclosure is executed by the terminal device 110, the terminal device 110 may analyze and process the audio data to determine a speech detection result corresponding to the audio data.

[0036] In an example where the audio processing solution of the embodiment of the present disclosure is executed by the server 120, the terminal device 110 can send audio data to the server 120, and the server 120 analyzes and processes the audio data to determine a speech detection result corresponding to the audio data.

[0037] The audio processing scheme of the disclosed embodiment can be implemented based on an audio processing model. In this case, the server 120 can train the model using training data to obtain a trained audio processing model. Subsequently, the server 120 can send the trained audio processing model to the terminal device 110, whereby the terminal device 110 can use the trained audio processing model to process the audio data currently to be analyzed to determine the speech detection result corresponding to the audio data.

[0038] In addition, the server 120 may also directly use the trained audio processing model for inference to determine the speech detection result corresponding to the audio data.

[0039] In order to improve the speech detection effect, the server 120 can pre-process the training data, such as adding noise and / or reverberation, and performing spectrum enhancement, thereby improving the robustness of the model and making the speech detection more accurate.

[0040] For the convenience of description, the following is an example of a terminal device executing the audio processing solution of the embodiment of the present disclosure.

[0041] Figure 2 The input and output diagram of the audio processing solution of the embodiment of the present disclosure is shown. Figure 2 The input of the audio processing method disclosed in the present invention is audio data, and the output is speech detection result.

[0042] First, the terminal device can obtain audio data and perform an initial feature extraction operation on the audio data to obtain a first audio feature. Next, the terminal device can perform a processing process based on feature splitting in the time domain on the first audio feature to obtain a second audio feature. Subsequently, the terminal device can enhance the difference between speech features and non-speech features in the second audio feature based on an attention mechanism to obtain a third audio feature. Then, the terminal device can perform a fully connected mapping operation on the third audio feature to obtain a speech detection result of the audio data.

[0043] like Figure 2 It should be noted that the speech detection result is for each frame in the audio data. For example, the audio frame containing speech is recorded as 1, and the audio frame containing non-speech is recorded as 0; or, the audio frame containing non-speech is recorded as 1, and the audio frame containing speech is recorded as 0.

[0044] The audio processing method according to the embodiment of the present disclosure is described below.

[0045] Figure 3 The flowchart of the audio processing method according to the exemplary embodiment of the present disclosure is schematically shown. Figure 3 , the audio processing method may include the following steps:

[0046] S32. Perform an initial feature extraction operation on the audio data to obtain a first audio feature.

[0047] In an exemplary embodiment of the present disclosure, the audio data is the data currently to be subjected to voice detection and analysis. The present disclosure does not limit the source of the audio data. For example, the audio data may be audio data obtained by a terminal device recording the environment in which it is located. For another example, the audio data may also be audio data generated by a terminal device performing an audio extraction operation on a video, and the video may be a video in an album of the terminal device. For another example, the audio data may be data synthesized by software.

[0048] The audio data may be generated by the terminal device. In addition, the terminal device may also obtain the audio data from other devices.

[0049] After determining the audio data, the terminal device may perform an initial feature extraction operation on the audio data to obtain a first audio feature.

[0050] According to some embodiments of the present disclosure, the terminal device can extract Fbank (Filter Bank) features of audio data. The method of extracting Fbank features conforms to the auditory principle of the human ear, and the Fbank features of audio data can be extracted through operations such as pre-emphasis, framing, windowing, short-time Fourier transform, and Mel filtering, and the present disclosure does not limit this.

[0051] In addition, the present disclosure does not impose any restrictions on the extracted feature dimension, frame length, frame shift, etc. For example, an Fbank feature with 80 dimensions, a frame length of 25 ms, and a frame shift of 10 ms may be extracted.

[0052] The terminal device may determine the extracted Fbank feature as the first audio feature.

[0053] It should be understood that the Fbank feature is only an exemplary description, and the first audio feature of the present disclosure may also be an MFCC (Mel-scale Frequency Cepstral Coefficients) feature, a PNCC (Power-Normalized Cepstral Coefficients) feature, or the like.

[0054] According to some other embodiments of the present disclosure, first, the terminal device may perform an initial feature extraction operation on the audio data to obtain an initial feature, wherein the initial feature may be, for example, the above-mentioned Fbank feature, MFCC feature or PNCC feature.

[0055] Next, the terminal device may perform convolution processing on the initial feature to obtain the first audio feature. The convolution processing in the embodiment of the present disclosure may include a combination of a one-dimensional convolution operation (Conv1d), a linear rectification operation (Relu) and a batch normalization operation (BN).

[0056] It can be understood that the first audio feature mentioned in the present disclosure is generally an audio feature sequence.

[0057] S34. Perform a processing procedure based on feature splitting in the time domain on the first audio feature to obtain a second audio feature.

[0058] In an exemplary embodiment of the present disclosure, the processing process based on feature splitting in the time domain may refer to a process of processing the first audio feature including a feature splitting operation in the time series.

[0059] The processing process based on feature splitting in the time domain may include at least one level of time domain feature splitting sub-processing process. That is, the terminal device may perform at least one level of time domain feature splitting sub-processing process on the first audio feature, and generate the second audio feature based on the output result of the at least one level of time domain feature splitting sub-processing process.

[0060] According to some embodiments of the present disclosure, the terminal device performs two or more levels of time domain feature splitting sub-processes on the first audio feature, and these sub-processes are configured in a cascade manner in terms of the algorithm, that is, the output of each level of time domain feature splitting sub-process is the input of the next level of time domain feature splitting sub-process. In addition, the algorithm execution process of each level of time domain feature splitting sub-process may be the same or different, but each level of time domain feature splitting sub-process includes feature splitting operations on the time series.

[0061] In this case, the terminal device can fuse the output results of the time domain feature splitting sub-processes at each level, and determine the second audio feature according to the fusion result. Thus, the fusion of multiple layers of features at different levels is achieved, which can improve the accuracy of the detection result.

[0062] The time domain feature splitting sub-processing process of the embodiment of the present disclosure is exemplarily described below.

[0063] First, the terminal device can perform a feature splitting operation in the time domain on the audio features input into the time domain feature splitting subprocess to obtain a plurality of first intermediate features. In order to ensure the stability of the algorithm processing, the audio features can be split at equal intervals in the time series, that is, the time period corresponding to each first intermediate feature is of the same length.

[0064] According to some embodiments of the present disclosure, before performing the feature splitting operation, the audio features may be subjected to convolution processing. That is, the terminal device may perform convolution processing on the audio features input into the time domain feature splitting subprocess, and perform a feature splitting operation in the time domain on the features obtained after the convolution processing to obtain a plurality of first intermediate features.

[0065] Next, the terminal device may perform a convolution operation on the plurality of first intermediate features to obtain a plurality of second intermediate features, wherein the second intermediate features may correspond one-to-one to the first intermediate features.

[0066] Specifically, the terminal device may use the first first intermediate feature in the time domain as the first second intermediate feature. The terminal device may perform convolution processing on the second first intermediate feature in the time domain to obtain the second second intermediate feature. The terminal device may add the i-th first intermediate feature and the i-1-th second intermediate feature in the time domain, and perform convolution processing on the added features to obtain the i-th second intermediate feature. Wherein, i is a positive integer greater than or equal to 3.

[0067] Subsequently, the terminal device may concatenate the multiple second intermediate features obtained to obtain a third intermediate feature.

[0068] Then, the terminal device can perform feature enhancement on the third intermediate feature to obtain the audio feature output by the time domain feature splitting sub-process.

[0069] According to some embodiments of the present disclosure, the terminal device can directly output the enhanced features.

[0070] According to other embodiments of the present disclosure, the terminal device may perform feature enhancement on the third intermediate feature to obtain a fourth intermediate feature, and add the fourth intermediate feature to the audio feature of the input time domain feature splitting subprocess to obtain the audio feature output by the time domain feature splitting subprocess. This process can be implemented through a model, whereby, by adding the input and output, the problem of model degradation can be solved, which helps the model to converge quickly.

[0071] In addition, before feature enhancement, the third intermediate feature may be convolved, and the feature obtained after the convolution may be enhanced to obtain a fourth intermediate feature.

[0072] The present disclosure does not limit the specific method of feature enhancement. For example, the terminal device can determine the importance of each feature channel in the third intermediate feature and configure a weight for each feature channel according to the importance of each feature channel. In this way, features useful for audio processing tasks can be enhanced and features that are useless or not very useful for audio processing tasks can be suppressed.

[0073] It should be noted that the present disclosure does not impose any restriction on the number of levels of the time domain feature splitting sub-process, which may be related to the processing capability of the terminal device and the complexity of the audio processing task.

[0074] S36. Enhance the difference between speech features and non-speech features in the second audio feature based on the attention mechanism to obtain a third audio feature.

[0075] The attention mechanism configured in the embodiment of the present disclosure can enhance the difference between speech features and non-speech features. Specifically, the attention conversion process of the present disclosure can be implemented in combination with the Encoder-Decoder framework, and the present disclosure does not limit this.

[0076] S38. Perform a fully connected mapping operation on the third audio feature to obtain a speech detection result of the audio data.

[0077] In an exemplary embodiment of the present disclosure, the speech detection result of the audio data obtained includes the speech detection result of each audio frame in the audio data. The speech detection result characterizes the detection result of whether the audio frame is speech or non-speech. For example, an audio frame containing speech is recorded as 1, and an audio frame containing non-speech is recorded as 0; or, an audio frame containing non-speech is recorded as 1, and an audio frame containing speech is recorded as 0.

[0078] Since the speech detection result is obtained for each audio frame, a segment of speech may be divided into discrete parts, which may cause the speech detection result to be uneven, thus affecting the accuracy of the speech detection result.

[0079] In order to solve this problem, the embodiment of the present disclosure also provides a smoothing processing method.

[0080] First, the terminal device performs a fully connected mapping operation on the third audio feature to determine a first speech detection result for each audio frame in the audio data. Next, the terminal device can modify the first speech detection result using the first frame number threshold and the overlapped frame number threshold to determine a second speech detection result for each audio frame in the audio data, and output the second speech detection result as the final detection result.

[0081] The first frame number threshold is a frame number requirement set by switching from a voice state to a non-voice state, and the second frame number threshold is a frame number requirement set by switching from a non-voice state to a voice state. It can be understood that the first frame number threshold and the second frame number threshold are both the minimum frame number requirements. The first frame number threshold can be the same as the second frame number threshold, or can be configured to be different.

[0082] Figure 4 A schematic diagram showing the correction of speech detection results according to an embodiment of the present disclosure is shown.

[0083] by Figure 4 For example, if non-speech is recorded as 0, speech is recorded as 1, and the first frame number threshold and the second frame number threshold are both configured as 3, an effect of converting the first speech detection result into the second speech detection result can be achieved.

[0084] By smoothing the speech detection results based on the frame number threshold disclosed in the present invention, the accuracy of the test can be improved.

[0085] In addition, in addition to the above-mentioned smoothing method based on the frame number threshold, a smoothing process may also be performed by using a method such as a median filter, which is not limited in the present disclosure.

[0086] Based on the above exemplary audio processing method of the present disclosure, accurate active voice detection can be achieved, providing a high-quality data foundation for subsequent algorithms such as speech recognition and human voice segmentation.

[0087] The present disclosure can implement part of the above audio processing method through an audio processing model. The audio processing model used in the present disclosure is described below.

[0088] The audio processing module of the disclosed embodiment may include a time domain feature splitting processing module, an attention layer, and a fully connected layer. Specifically, the time domain feature splitting processing module may be used to perform a processing process based on feature splitting in the time domain on the first audio feature to obtain a second audio feature. The attention layer may be used to enhance the difference between speech features and non-speech features in the second audio feature based on the attention mechanism to obtain a third audio feature. The fully connected layer may be used to perform a fully connected mapping operation on the third audio feature.

[0089] Figure 5 The model architecture diagram of the audio processing model of the embodiment of the present disclosure is schematically shown. Figure 5 The audio processing model 5 may include a convolution unit 51, a time domain feature splitting processing unit 52, a time domain feature splitting processing unit 53, a time domain feature splitting processing unit 54, a convolution unit 55, an attention layer 56 and a fully connected layer 57.

[0090] Among them, both the convolution unit 51 and the convolution unit 55 can perform the above-mentioned convolution processing, that is, they are a combination of one-dimensional convolution layer, linear rectification layer and batch normalization layer operation. For the convolution unit 55, the outputs of each time domain feature splitting processing unit are accumulated as the input of the convolution unit 55, thereby realizing the fusion of multiple layers of features, which helps to improve the accuracy of speech detection.

[0091] The attention layer 56 can be configured as a combination of attention (Attention) and batch normalization (BN), and the attention mechanism is directly applied to the time series dimension, so that the model can better learn the difference between speech and non-speech in long time series.

[0092] The fully connected layer 57 can map the output of the previous layer to the output node. There are two output nodes corresponding to the speech detection of the present disclosure, which represent the speech and non-speech states respectively.

[0093] The time domain feature splitting processing unit 52, the time domain feature splitting processing unit 53, and the time domain feature splitting processing unit 54 may be included in the time domain feature splitting processing module. However, it should be noted that the number of time domain feature splitting processing units included in the time domain feature splitting processing module may be one or more. It should be noted that each time domain feature splitting processing unit corresponds to executing the above-mentioned time domain feature splitting sub-process.

[0094] Reference below Figure 6 The time domain feature splitting processing unit disclosed in the present invention is described.

[0095] refer to Figure 6 , the first convolution unit is used to transform the feature channels of the audio features input to the time domain feature splitting processing unit. The output of this layer is x. It can be understood that x is an audio feature sequence. Subsequently, x is split into 4 subsequences in the time series, recorded as x1, x2, x3 and x4.

[0096] Next, x1 remains unchanged, so y1 = x1. Perform one-dimensional convolution processing of the convolution unit on x2, x3, and x4 respectively, wherein the calculation result of the previous subsequence in time sequence can be added to the current subsequence, and then convolution processing is performed, thereby generating y2, y3, and y4. Then, y1, y2, y3, and y4 are concatenated to generate y.

[0097] A one-dimensional convolution process of a convolution unit is performed on y to transform the feature dimension of the audio feature to the feature dimension before being input into the time domain feature splitting processing unit.

[0098] Subsequently, the features are input into a compression excitation unit to achieve data enhancement. The compression excitation unit may include a compression network and an excitation network.

[0099] For example, the dimension of the audio feature input to the unit is recorded as T*C, where T represents the time dimension and C represents the number of channels of the feature. In this case, the compression network can compress the audio feature of T*C into an audio feature of 1*C, which can be achieved by averaging the time dimension. After the audio feature of T*C is compressed into an audio feature of 1*C, it is equivalent to the one-dimensional parameter obtaining the previous global field of view and a wider perception area. The excitation model can include a fully connected layer for predicting the importance of each channel of the 1*C audio feature, obtaining the importance prediction results of different channels, and then applying the importance prediction results to the corresponding channels. In this way, features that are useful for audio processing tasks can be enhanced and features that are useless or less useful for audio processing tasks can be suppressed.

[0100] like Figure 6 As shown, the audio features output by the compression excitation unit can be added to the audio features input to the time domain feature splitting processing unit to form the final output of the time domain feature splitting processing unit. This connection method can be called a skip connection, which can solve the problem of network degradation and also make the model converge quickly.

[0101] For the audio processing model of the embodiment of the present disclosure, in order to improve its robustness, the present disclosure also provides a solution for preprocessing the training data.

[0102] Specifically, when training the audio processing model, original training data may be acquired in advance, and noise and / or reverberation may be added to the original training data to obtain intermediate training data. Subsequently, an initial feature extraction operation may be performed on the intermediate training data to obtain a first training audio feature. Then, the audio processing model may be trained using the first training audio feature.

[0103] In some embodiments of the present disclosure, the first training audio features may be directly used as data for training the audio processing model.

[0104] In some other embodiments of the present disclosure, the first training audio feature may be subjected to spectral enhancement processing to obtain a second training audio feature, and the audio processing model may be trained using the second training audio feature.

[0105] Figure 7 FIG. 1 is a schematic diagram showing a process of generating training data for training an audio processing model according to an embodiment of the present disclosure. Figure 7 There is no restriction on the order of the operation steps included in the data preprocessing. For example, reverberation may be added first and then noise may be added.

[0106] For adding noise, you can use the open source MUSAN dataset, which contains three types of noise: speech, music, and noise. The specific method of adding noise is: randomly select a noise for each training data to add noise. For speech noise, the signal-to-noise ratio of the added noise is 0dB-15dB, which can be randomly selected; for music noise, the signal-to-noise ratio of the added noise is 5dB-15dB, which can be randomly selected; for noise noise, the signal-to-noise ratio of the added noise is 10dB-20dB, which can be randomly selected. You can also add more than two types of noise at the same time. This dataset covers many types of noise, and the noise addition process covers different signal-to-noise ratios. Therefore, it can simulate complex scenes in reality to a certain extent and improve the accuracy of the model.

[0107] For adding reverberation, you can use the open-source RIRS dataset. The specific method is: randomly select a reverberation data and add reverberation to the training data. This dataset covers a lot of reverberation data. After adding reverberation, it can better simulate the reverberation scene of real users and improve the accuracy of the model.

[0108] For feature extraction, the extraction method such as Fbank feature as described above may be used.

[0109] For spectrum enhancement, 0-5 frames can be randomly masked in the time dimension; 0-10 signals can be randomly masked in the frequency domain dimension. Masking can mean setting all data to 0, for example. Spectral enhancement can effectively improve the robustness of the model.

[0110] refer to Figure 7 After data preprocessing, the original training data generates target training data. The target training data can be used alone or in combination with the original training data as training samples to train the audio processing model of the embodiment of the present disclosure.

[0111] Based on the above audio processing process, a speech detection result of the audio data can be obtained. Next, the speech detection result can be used to execute subsequent algorithms.

[0112] Figure 8 The flowchart of the video processing method according to an exemplary embodiment of the present disclosure is schematically shown. Figure 8 , the video processing method may include the following steps:

[0113] S82. Obtain the video and extract the audio data in the video.

[0114] In an exemplary embodiment of the present disclosure, the video may be from an album of a terminal device, and the terminal device may extract audio data from the video.

[0115] S84. Perform voice detection on the audio data using an audio processing method to determine the voice segment in the video.

[0116] After the audio data is determined, the terminal device can use the above-mentioned audio processing method of the present disclosure to perform voice detection on the audio data to determine the voice segment contained in the video.

[0117] Next, the terminal device may also analyze the voice segment, and if the voice segment contains multiple voices, separate the voice data of the target voice from the voice segment. The target voice may be a voice specified by the user.

[0118] Subsequently, the terminal device may generate a voice segment and / or a video segment corresponding to the target voice using the voice data of the target voice.

[0119] Specifically, in an embodiment of generating a speech segment corresponding to a target human voice, scattered speech data of the target human voice may be concatenated to generate a speech segment corresponding to the target human voice.

[0120] In an embodiment of generating a video clip corresponding to a target human voice, an image frame aligned with the speech data of the target human voice can be determined from the video, and the speech data of the target human voice and the determined image frame can be spliced ​​separately. After time alignment, a video clip corresponding to the target human voice can be generated.

[0121] In addition, for the video clip, the image frame displayed can also be independent of the original video. In other words, the speech of the target person's voice can be combined with other image frames to generate an original video clip containing the target person's voice.

[0122] Fig. 9 The flowchart of applying the exemplary scheme of the present disclosure to construct wonderful memories of children's voices is schematically shown.

[0123] In step S902, the terminal device may obtain videos from the user's photo album.

[0124] In step S904, the terminal device may perform active voice detection using the audio processing solution of the embodiment of the present disclosure.

[0125] In step S906, the terminal device may perform vocal segmentation and clustering to divide different vocals in the video.

[0126] In step S908, the terminal device may detect each voice data to determine the child's voice.

[0127] In step S910, the terminal device may construct wonderful memories of children's voices based on the voice data of the children's voices. The memories may be embodied in the form of sound or sound plus image, and the present disclosure does not impose any limitation on this.

[0128] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0129] Furthermore, this example implementation also provides an audio processing device.

[0130] Fig.10 FIG. 1 schematically shows a block diagram of an audio processing device according to an exemplary embodiment of the present disclosure. Fig.10 According to an exemplary embodiment of the present disclosure, the audio processing device 10 may include a first feature determination module 101 , a second feature determination module 103 , a third feature determination module 105 and a speech detection module 107 .

[0131] Specifically, the first feature determination module 101 can be used to perform an initial feature extraction operation on the audio data to obtain a first audio feature; the second feature determination module 103 can be used to perform a processing process based on feature splitting in the time domain on the first audio feature to obtain a second audio feature; the third feature determination module 105 can be used to enhance the difference between speech features and non-speech features in the second audio feature based on an attention mechanism to obtain a third audio feature; the speech detection module 107 can be used to perform a fully connected mapping operation on the third audio feature to obtain a speech detection result of the audio data.

[0132] According to an exemplary embodiment of the present disclosure, the processing process based on feature splitting in the time domain includes at least one level of time domain feature splitting sub-processing process. In this case, the second feature determination module 103 can be configured to perform: performing at least one level of time domain feature splitting sub-processing process on the first audio feature; generating the second audio feature based on the output result of at least one level of time domain feature splitting sub-processing process.

[0133] According to an exemplary embodiment of the present disclosure, in the case where two or more levels of time domain feature splitting sub-processes are performed on the first audio feature, the output of each level of time domain feature splitting sub-processing is the input of the next level of time domain feature splitting sub-processing. In this case, the process of generating the second audio feature based on the output results of the two or more levels of time domain feature splitting sub-processing processes by the second feature determination module 103 can be configured to perform: fusing the output results of the time domain feature splitting sub-processing processes at each level; and determining the second audio feature according to the fusion results.

[0134] According to an exemplary embodiment of the present disclosure, the time domain feature splitting sub-processing process performed by the second feature determination module 103 includes: performing feature splitting operations in the time domain on the audio features of the input time domain feature splitting sub-processing process to obtain multiple first intermediate features; performing convolution processing on the multiple first intermediate features to obtain multiple second intermediate features; splicing the multiple second intermediate features to obtain a third intermediate feature; and performing feature enhancement on the third intermediate feature to obtain the audio features output by the time domain feature splitting sub-processing process.

[0135] According to an exemplary embodiment of the present disclosure, the second feature determination module 103 can be configured to perform: convolution processing on the audio features of the input time domain feature splitting sub-process, and performing feature splitting operations in the time domain on the features obtained after the convolution processing to obtain multiple first intermediate features.

[0136] According to an exemplary embodiment of the present disclosure, the second feature determination module 103 can be configured to execute: taking the first first intermediate feature in the time domain as the first second intermediate feature; performing convolution processing on the second first intermediate feature in the time domain to obtain the second second intermediate feature; adding the i-th first intermediate feature and the i-1-th second intermediate feature in the time domain, and performing convolution processing on the added features to obtain the i-th second intermediate feature; wherein i is a positive integer greater than or equal to 3.

[0137] According to an exemplary embodiment of the present disclosure, the second feature determination module 103 can be configured to perform: feature enhancement on the third intermediate feature to obtain a fourth intermediate feature; adding the fourth intermediate feature to the audio feature of the input time domain feature splitting sub-processing process to obtain the audio feature output by the time domain feature splitting sub-processing process.

[0138] According to an exemplary embodiment of the present disclosure, the second feature determination module 103 may be configured to execute: performing convolution processing on the third intermediate feature, and performing feature enhancement on the feature obtained after the convolution processing to obtain a fourth intermediate feature.

[0139] According to an exemplary embodiment of the present disclosure, the second feature determination module 103 may be configured to execute: determining the importance of each feature channel in the third intermediate feature; and configuring a weight for each feature channel according to the importance of each feature channel.

[0140] According to an exemplary embodiment of the present disclosure, the speech detection module 107 can be configured to perform: performing a fully connected mapping operation on the third audio feature to determine the speech detection result of each audio frame in the audio data; wherein the speech detection result characterizes the detection result of whether the audio frame is speech or non-speech.

[0141] According to an exemplary embodiment of the present disclosure, the speech detection module 107 can be configured to perform: performing a fully connected mapping operation on the third audio feature to determine a first speech detection result for each audio frame in the audio data; modifying the first speech detection result using a first frame number threshold and a second frame number threshold to determine a second speech detection result for each audio frame in the audio data; wherein the first frame number threshold is a frame number requirement set for switching from a speech state to a non-speech state, and the second frame number threshold is a frame number requirement set for switching from a non-speech state to a speech state.

[0142] According to an exemplary embodiment of the present disclosure, the first feature determination module 101 may be configured to execute: performing an initial feature extraction operation on the audio data to obtain an initial feature; and performing a convolution process on the initial feature to obtain a first audio feature.

[0143] According to an exemplary embodiment of the present disclosure, the above-mentioned processing process can be implemented by remembering the audio processing model. In this case, reference Fig.11 Compared with the audio processing device 10 , the audio processing device 11 may further include a model training module 111 .

[0144] Specifically, the model training module 111 can be configured to execute: pre-acquiring original training data; adding noise and / or reverberation to the original training data to obtain intermediate training data; performing initial feature extraction operations on the intermediate training data to obtain first training audio features; and training the audio processing model using the first training audio features.

[0145] According to an exemplary embodiment of the present disclosure, the model training module 111 may also be configured to execute: performing spectral enhancement processing on the first training audio feature to obtain a second training audio feature; and training the audio processing model using the second training audio feature.

[0146] Furthermore, this example implementation also provides a video processing device.

[0147] Fig.12 FIG. 1 schematically shows a block diagram of an audio processing device according to an exemplary embodiment of the present disclosure. Fig.12 , the audio processing device 12 according to an exemplary embodiment of the present disclosure may include an audio extraction module 121 and a segment determination module 123 .

[0148] Specifically, the audio extraction module 121 can be used to acquire a video and extract audio data from the video; the segment determination module 123 can be used to perform voice detection on the audio data using the above audio processing method to determine the voice segment in the video.

[0149] According to an exemplary embodiment of the present disclosure, the segment determination module 123 can also be configured to perform: when a speech segment contains multiple human voices, separating the speech data of the target human voice from the speech segment; using the speech data of the target human voice to generate a speech segment and / or video segment corresponding to the target human voice.

[0150] Since the functional modules of the audio processing device and the video processing device in the embodiment of the present disclosure are the same as those in the above method embodiment, they will not be described in detail here.

[0151] Fig.13 Schematic diagram of an electronic device suitable for implementing the exemplary embodiment of the present disclosure is shown. The terminal device of the exemplary embodiment of the present disclosure may be configured as follows Fig.13 It should be noted that Fig.13 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0152] The electronic device of the present disclosure includes at least a processor and a memory, and the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the audio processing method and / or video processing method of the exemplary embodiment of the present disclosure.

[0153] Specifically, Fig.13 As shown, the electronic device 130 may include: a processor 1310, an internal memory 1321, an external memory interface 1322, a Universal Serial Bus (USB) interface 1330, a charging management module 1340, a power management module 1341, a battery 1342, an antenna 1, an antenna 2, a mobile communication module 1350, a wireless communication module 1360, an audio module 1370, a sensor module 1380, a display screen 1390, a camera module 1391, an indicator 1392, a motor 1393, a button 1394, and a Subscriber Identification Module (SIM) card interface 1395, etc. The sensor module 1380 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0154] It is to be understood that the structure illustrated in the embodiment of the present disclosure does not constitute a specific limitation on the electronic device 130. In other embodiments of the present disclosure, the electronic device 130 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0155] The processor 1310 may include one or more processing units, for example, the processor 1310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. In addition, a memory may be provided in the processor 1310 for storing instructions and data.

[0156] The electronic device 130 can realize the shooting function through the ISP, the camera module 1391, the video codec, the GPU, the display screen 1390 and the application processor. In some embodiments, the electronic device 130 may include 1 or N camera modules 1391, where N is a positive integer greater than 1. If the electronic device 130 includes N cameras, one of the N cameras is a main camera.

[0157] The internal memory 1321 can be used to store computer executable program codes, which include instructions. The internal memory 1321 can include a program storage area and a data storage area. The external memory interface 1322 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 130.

[0158] The present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device.

[0159] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0160] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0161] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the embodiments of the present disclosure.

[0162] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0164] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0165] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0166] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0167] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0168] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that: include: Performing an initial feature extraction operation on the audio data to obtain a first audio feature; Performing a processing process based on feature splitting in the time domain on the first audio feature to obtain a second audio feature; enhancing the difference between speech features and non-speech features in the second audio feature based on an attention mechanism to obtain a third audio feature; A fully connected mapping operation is performed on the third audio feature to obtain a speech detection result of the audio data.

2. The audio processing method according to claim 1, characterized in that: The processing process based on time domain feature splitting includes at least one level of time domain feature splitting sub-processing process; wherein, performing the processing process based on time domain feature splitting on the first audio feature to obtain the second audio feature includes: Performing at least one level of time domain feature splitting sub-process on the first audio feature; The second audio feature is generated based on the output result of the at least one level of time domain feature splitting sub-process.

3. The audio processing method according to claim 2, characterized in that: In the case where two or more levels of time domain feature splitting sub-processes are performed on the first audio feature, the output of each level of time domain feature splitting sub-processes is the input of the next level of time domain feature splitting sub-processes; wherein generating the second audio feature based on the output results of the two or more levels of time domain feature splitting sub-processes includes: The output results of each level of time domain feature splitting sub-process are integrated; The second audio feature is determined according to the fusion result.

4. The audio processing method according to claim 2, characterized in that: The time domain feature splitting sub-process comprises: Performing a feature splitting operation in the time domain on the audio features input into the time domain feature splitting subprocess to obtain a plurality of first intermediate features; Performing convolution processing on the plurality of first intermediate features to obtain a plurality of second intermediate features; splicing the plurality of second intermediate features to obtain a third intermediate feature; Feature enhancement is performed on the third intermediate feature to obtain the audio feature output by the time domain feature splitting sub-process.

5. The audio processing method according to claim 4, characterized in that: Performing a feature splitting operation in the time domain on the audio features input to the time domain feature splitting subprocess to obtain a plurality of first intermediate features, including: The audio features input into the time domain feature splitting subprocess are subjected to convolution processing, and the features obtained after the convolution processing are subjected to feature splitting operation in the time domain to obtain a plurality of first intermediate features.

6. The audio processing method according to claim 4, characterized in that: Performing convolution processing on the plurality of first intermediate features to obtain a plurality of second intermediate features, comprising: The first first intermediate feature in the time domain is used as the first second intermediate feature; Performing convolution processing on the second first intermediate feature in the time domain to obtain a second second intermediate feature; Add the i-th first intermediate feature and the i-1-th second intermediate feature in the time domain, and perform convolution processing on the added features to obtain the i-th second intermediate feature; Here, i is a positive integer greater than or equal to 3.

7. The audio processing method according to claim 4, characterized in that: Performing feature enhancement on the third intermediate feature to obtain the audio feature output by the time domain feature splitting sub-process includes: Performing feature enhancement on the third intermediate feature to obtain a fourth intermediate feature; The fourth intermediate feature is added to the audio feature input to the time domain feature splitting sub-processing process to obtain the audio feature output by the time domain feature splitting sub-processing process.

8. The audio processing method according to claim 7, characterized in that: Performing feature enhancement on the third intermediate feature to obtain a fourth intermediate feature, comprising: The third intermediate feature is subjected to convolution processing, and the feature obtained after the convolution processing is subjected to feature enhancement to obtain a fourth intermediate feature.

9. The audio processing method according to claim 4, characterized in that: Performing feature enhancement on the third intermediate feature includes: Determining the importance of each feature channel in the third intermediate feature; A weight is configured for each of the feature channels according to the importance of each feature channel.

10. The audio processing method according to claim 1, characterized in that: Performing a fully connected mapping operation on the third audio feature to obtain a speech detection result of the audio data includes: A fully connected mapping operation is performed on the third audio feature to determine a speech detection result of each audio frame in the audio data; wherein the speech detection result represents a detection result of whether the audio frame is speech or non-speech.

11. The audio processing method according to claim 10, characterized in that: Performing a fully connected mapping operation on the third audio feature to determine a speech detection result of each audio frame in the audio data includes: Performing a fully connected mapping operation on the third audio feature to determine a first speech detection result for each audio frame in the audio data; Modifying the first speech detection result using a first frame number threshold and a second frame number threshold to determine a second speech detection result for each audio frame in the audio data; The first frame number threshold is a frame number requirement set for switching from a voice state to a non-voice state, and the second frame number threshold is a frame number requirement set for switching from a non-voice state to a voice state.

12. The audio processing method according to claim 1, characterized in that: Performing an initial feature extraction operation on the audio data to obtain a first audio feature includes: Performing an initial feature extraction operation on the audio data to obtain initial features; The initial feature is convolved to obtain a first audio feature.

13. The audio processing method according to any one of claims 1 to 12, characterized in that: Implementing the audio processing method through an audio processing model; The audio processing model includes: A time domain feature splitting processing module, used for performing a processing process based on feature splitting in the time domain on the first audio feature to obtain a second audio feature; an attention layer, configured to enhance the difference between speech features and non-speech features in the second audio features based on an attention mechanism, so as to obtain a third audio feature; A fully connected layer is used to perform a fully connected mapping operation on the third audio feature.

14. The audio processing method according to claim 13, characterized in that: The audio processing method further comprises: Get the original training data in advance; Adding noise and / or reverberation to the original training data to obtain intermediate training data; Performing an initial feature extraction operation on the intermediate training data to obtain a first training audio feature; The audio processing model is trained using the first training audio features.

15. The audio processing method according to claim 14, characterized in that: Training the audio processing model using the first training audio feature includes: Performing spectral enhancement processing on the first training audio feature to obtain a second training audio feature; The audio processing model is trained using the second training audio features.

16. A video processing method, characterized in that: include: Acquire a video, and extract audio data from the video; Perform voice detection on the audio data using the audio processing method described in any one of claims 1 to 15 to determine the voice segment in the video.

17. The video processing method according to claim 16, characterized in that: The video processing method further includes: When the speech segment contains multiple human voices, separating speech data of a target human voice from the speech segment; The speech data of the target human voice is used to generate a speech segment and / or a video segment corresponding to the target human voice.

18. An audio processing device, characterized in that: include: A first feature determination module, used to perform an initial feature extraction operation on the audio data to obtain a first audio feature; A second feature determination module, configured to perform a processing process based on feature splitting in the time domain on the first audio feature to obtain a second audio feature; A third feature determination module, configured to enhance the difference between speech features and non-speech features in the second audio feature based on an attention mechanism to obtain a third audio feature; The speech detection module is used to perform a fully connected mapping operation on the third audio feature to obtain a speech detection result of the audio data.

19. A video processing device, characterized in that: include: An audio extraction module, used to acquire a video and extract audio data from the video; A segment determination module is used to perform speech detection on the audio data using the audio processing method described in any one of claims 1 to 15 to determine the speech segment in the video.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the audio processing method according to any one of claims 1 to 15 or the video processing method according to claim 16 or 17 is implemented.

21. An electronic device, characterized in that: include: processor; A memory for storing one or more programs, which, when executed by the processor, enables the processor to implement the audio processing method according to any one of claims 1 to 15 or the video processing method according to claim 16 or 17.

Images