Time-frequency conversion method and device and electronic equipment

Abstract

The invention discloses a time-frequency transformation method and device and electronic equipment, and belongs to the technical field of signal processing, and the time-frequency transformation method comprises the steps: obtaining to-be-processed time domain data; the time domain data comprises a sampling point data sequence formed by a plurality of sampling point data on a time domain; positioning a first time domain interval to be processed currently on the time domain data; performing time domain to frequency domain processing on each sampling point data in the first time domain interval to obtain first frequency domain data corresponding to each sampling point data in the first time domain interval; wherein if the first time domain interval corresponds to a second time domain interval which is adjacent to the first time domain interval on the time domain data, the first time domain interval and the second time domain interval have partially overlapped sampling point data; the first frequency domain data is frequency domain data obtained by applying a target increment to second frequency domain data corresponding to each sampling point data in the second time domain interval.

Description

Technical Field

[0001] The present application belongs to the field of signal processing technology, and in particular relates to a time-frequency conversion method, device and electronic equipment. Background Art

[0002] The time-domain-to-frequency-domain processing of data signals includes single-frequency and multi-frequency cases. In both cases, the data is divided into intervals for calculation. The sampling point interval formed by N consecutive sampling points of the data signal is regarded as a frame to calculate the frequency domain information once. In order to avoid truncation of data features (such as speech features) at both ends of the interval, a part of overlapping interval is usually added between each two adjacent intervals. However, this will lead to a significant increase in time complexity and calculation amount. For example, if the frame shift of an interval of length N is changed from N sampling points to N / 2 sampling points (that is, the overlapping interval contains N / 2 sampling points), the calculation amount will become twice that of the original frame shift of N. If the frame shift is 1 sampling point (that is, the overlapping interval contains N-1 sampling points), the calculation amount will become N times that of the original frame shift of N. The time complexity also increases accordingly, which seriously affects the processing efficiency of time-frequency transformation. Summary of the invention

[0003] To this end, this application discloses the following technical solutions:

[0004] A time-frequency transformation method, comprising:

[0005] Obtaining time domain data to be processed; the time domain data includes a sampling point data sequence formed by a plurality of sampling point data in the time domain;

[0006] Locating a first time domain interval currently to be processed on the time domain data;

[0007] Performing time-domain-to-frequency-domain processing on each sampling point data in the first time domain interval to obtain first frequency-domain data corresponding to each sampling point data in the first time domain interval;

[0008] Among them, if the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, there are partially overlapping sampling point data between the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

[0009] Optionally, the locating the first time domain interval currently to be processed on the time domain data includes one of the following:

[0010] Locating a time domain interval of a predetermined interval length from the head of the sampling point data sequence to obtain the first time domain interval;

[0011] The most recently processed time domain interval is slid on the sampling point data sequence in a time sequence by a predetermined step length to obtain the first time domain interval; the first time domain interval includes a plurality of continuous sampling point data corresponding to the predetermined interval length, and the predetermined step length is less than the predetermined interval length.

[0012] Optionally, performing time-domain-to-frequency-domain processing on each sampling point data in the first time domain interval to obtain first frequency domain data corresponding to each sampling point data in the first time domain interval includes:

[0013] If the first time domain interval does not correspond to a second time domain interval adjacent to the first time domain interval in the time domain data, converting each sampling point data in the first time domain interval into one or more frequency domains to obtain first frequency domain data in the one or more frequency domains;

[0014] If the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, determine a target increment of frequency domain data corresponding to the sampling point data in the first time domain interval relative to the second frequency domain data, and apply the target increment to the second frequency domain data to obtain the first frequency domain data;

[0015] The target increment includes at least first difference data between frequency domain data components corresponding to non-overlapping sampling point data between the first time domain interval and the second time domain interval.

[0016] Optionally, determining a target increment of frequency domain data corresponding to the sampling point data in the first time domain interval relative to the second frequency domain data includes:

[0017] Without applying a window function to the sampling point data, determining first difference data between frequency domain data components corresponding to non-overlapping sampling point data of the first time domain interval and the second time domain interval as the target increment;

[0018] When a window function is applied to the sampling point data, first difference data between frequency domain data components corresponding to non-overlapping sampling point data of the first time domain interval and the second time domain interval, and second difference data between frequency domain data components corresponding to overlapping sampling point data are determined, and the target increment includes the first difference data and the second difference data.

[0019] Optionally, when a window function is applied to the sampling point data, determining second difference data between frequency domain data components respectively corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval includes:

[0020] Performing a difference operation on frequency domain data components respectively corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval;

[0021] If the result of the difference operation is not 0, recursively perform frequency domain data difference operation on the overlapping interval formed by the overlapping sampling point data and the previous adjacent interval of the overlapping interval until the end condition is met, and the difference operation result of the frequency domain data components corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval respectively when the end condition is met is determined as the second difference data.

[0022] Optionally, if the frequency domain in the time-frequency transform is a single frequency domain, after obtaining the first frequency domain data corresponding to each sampling point data in the first time domain interval, the method further includes:

[0023] Determining whether the frequency domain characteristics of the first frequency domain data meet a first characteristic condition;

[0024] If the first characteristic condition is met, the right boundary of the first time domain interval is increased, and it is determined whether there is target sampling point data with corresponding frequency domain characteristics that meet the second characteristic condition in the time domain interval obtained after increasing the right boundary; if so, the right boundary of the first time domain interval is updated to the target sampling point data, and the first frequency domain data corresponding to each sampling point data in the first time domain interval after the update is determined; and when the first time domain interval is determined next time, the left boundary of the first time domain interval is determined based on the target sampling point data, and the right boundary of the first time domain interval is determined based on the predetermined interval length.

[0025] Optionally, determining the left boundary of the first time domain interval based on the target sampling point data includes:

[0026] Determining a plurality of left boundaries for the first time domain interval based on the target sampling point data and the maximum overlapping length of the sampling points;

[0027] The maximum overlapping length of sampling points is used to represent the maximum overlapping number of sampling point data between the first time domain interval and the time domain interval that is adjacent to the first time domain interval in the time domain data.

[0028] Optionally, obtaining the time domain data to be processed includes:

[0029] A sampling point data sequence formed by a plurality of sampling point data of the audio data in the time domain is obtained by sampling the audio data in the time domain.

[0030] A time-frequency conversion device, comprising:

[0031] An acquisition module, used to obtain time domain data to be processed; the time domain data includes a sampling point data sequence formed by a plurality of sampling point data in the time domain;

[0032] A positioning module, used for positioning a first time domain interval to be processed on the time domain data;

[0033] A processing module, configured to perform time-domain to frequency-domain processing on each sampling point data within the first time domain interval to obtain first frequency domain data corresponding to each sampling point data within the first time domain interval;

[0034] Among them, if the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, there are partially overlapping sampling point data between the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

[0035] An electronic device, comprising:

[0036] A collection device, used for collecting sampling point data;

[0037] A processor for performing the following processing:

[0038] Obtaining time domain data to be processed; the time domain data includes a sampling point data sequence formed by a plurality of sampling point data in the time domain;

[0039] Locating a first time domain interval currently to be processed on the time domain data;

[0040] Performing time-domain-to-frequency-domain processing on each sampling point data in the first time domain interval to obtain first frequency-domain data corresponding to each sampling point data in the first time domain interval;

[0041] Among them, if the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, there are partially overlapping sampling point data between the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

[0042] A storage medium carries one or more computer instruction sets, and when the one or more computer instruction sets are executed by an electronic device, the electronic device can implement the time-frequency conversion method as described in any one of the above items. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 It is a flow chart of the time-frequency conversion method provided by this application;

[0045] Figure 2 It is a schematic diagram of the time complexity of single-frequency discrete Fourier transform in the case of no windowing and no overlapping intervals in the known technology;

[0046] Figure 3 It is a schematic diagram of the butterfly operation of the fast Fourier transform;

[0047] Figure 4 It is a schematic diagram of the time complexity of multi-frequency fast Fourier transform in the case of no windowing and no overlapping intervals in the known technology;

[0048] Figure 5 It is a schematic diagram of the time complexity of multi-frequency fast Fourier transform in the case of overlapping intervals without windowing in the known technology;

[0049] Figure 6 It is a schematic diagram of the time complexity of multi-frequency fast Fourier transform in the case of overlapping window intervals in the known technology;

[0050] Figure 7 It is another flow chart of the time-frequency conversion method provided by the present application;

[0051] Figure 8 It is a structural diagram of the time-frequency conversion device provided by the present application;

[0052] Fig. 9 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The embodiments of the present application disclose a time-frequency conversion method, device and electronic device, which are used to reduce the time complexity of the time-frequency conversion and improve the efficiency of the time-frequency conversion. The time-frequency conversion method provided by the embodiments of the present application can be applied to electronic devices in many general or special computing device environments or configurations, such as personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, etc.

[0055] See also Figure 1As shown in the method flow chart, the time-frequency transformation method provided in the embodiment of the present application may at least include the following steps 101 to 103, and these steps are described in detail below.

[0056] Step 101: Obtain time domain data to be processed; the time domain data includes a sampling point data sequence formed by a plurality of sampling point data in the time domain.

[0057] Optionally, a sampling point data sequence formed by a plurality of sampling point data of the audio data in the time domain may be obtained by sampling the audio data in the time domain, and the sampling point data sequence may be used as the time domain data to be processed.

[0058] The sampling point data in the sampling point data sequence are arranged in sequence according to the sampling order, and the sampling order is consistent with the timing characteristics of the sampled data, such as the audio data. That is, the sampling time of the data with the earlier timing in the sampled data, such as the audio data, is earlier than the sampling time of the data with the later timing.

[0059] In practical applications, the time-frequency domain transformation is not limited to audio processing scenarios, and the audio transformation solution of the embodiment of the present application can also be applied to other scenarios such as video processing, but is not limited to it.

[0060] Furthermore, the time-frequency transformation scheme of the embodiment of the present application can be applied to real-time or non-real-time data processing scenarios, such as real-time voice processing scenarios or non-real-time voice processing scenarios, etc., without limitation, and can be determined according to actual application requirements.

[0061] Step 102: Locate a first time domain interval to be processed on the time domain data.

[0062] The time domain to frequency domain processing of data signals includes single-frequency and multi-frequency cases. In both cases, the data is divided into intervals for calculation, and the sampling point interval formed by N consecutive sampling points of the data signal is regarded as a frame to calculate the frequency domain information once.

[0063] Based on this, in this step, specifically, starting from the head of the sampling point data sequence contained in the time domain data to be processed, the frames can be divided in sequence, each frame contains N consecutive sampling point data in the sampling point data sequence, and the N consecutive sampling point data contained in each frame form a time domain interval. Subsequently, the single frame, that is, each continuous sampling point data contained in a single time domain interval will be used as the minimum unit of time-frequency domain transformation processing, and the time-frequency domain transformation processing will be performed, that is, each continuous sampling point data contained in a single frame / single time domain interval corresponds to a frequency domain result (the output result of the time-frequency domain transformation), and each sampling point data in the same frame / time domain interval participates in the time-frequency domain transformation processing, and corresponds to a frequency domain data component in the frequency domain data finally transformed from the frame / time domain interval. In order to avoid truncation of data features (such as speech features) at both ends of the time domain interval, in the embodiment of the present application, a part of overlapping intervals will be added between each two adjacent time domain intervals, that is, there are overlapping sampling point data in each two adjacent time domain intervals.

[0064] Based on the above processing, when locating the first time domain interval to be processed on the time domain data to be processed, there are two specific cases. One case is that when the sampling point data sequence contained in the time domain data to be processed is initially processed, a time domain interval of a predetermined interval length can be directly located from the head of the sampling point data sequence as the first time domain interval to be processed. The predetermined interval length is used to characterize the number of sampling point data contained in a time domain interval, that is, a frame. For example, 8 consecutive sampling point data are directly located from the head of the sampling point data sequence to obtain the first time domain interval containing the 8 consecutive sampling point data.

[0065] Another case is that when the sampling point data sequence included in the time domain data to be processed is processed at a non-starting time, that is, the first time domain interval to be processed is not the first time domain interval in the sampling point data sequence. In this case, the time domain interval that has been processed most recently can be logically regarded as a sliding window, and a predetermined step length is slid in time sequence on the sampling point data sequence (that is, the left and right boundaries of the time domain interval that has been processed most recently are increased by a predetermined step length at the same time) to obtain the first time domain interval to be processed currently. In this case, the first time domain interval also includes a plurality of continuous sampling point data corresponding to the predetermined interval length.

[0066] The predetermined step length is smaller than the predetermined interval length, and by limiting the predetermined step length to be smaller than the predetermined interval length, there are overlapping sampling point data between adjacent time domain intervals. For example, assuming that the predetermined interval length is 8, the predetermined step length can be set to any integer smaller than 8 and greater than or equal to 1. For example, when it is set to 1, the length of each frame shift (i.e., time domain interval sliding) is 1 sampling point data, and there are 7 overlapping sampling point data between the new time domain interval obtained after the frame shift and the previous adjacent time domain interval.

[0067] Step 103: Perform time-domain to frequency-domain processing on the data of each sampling point in the first time-domain interval to obtain first frequency-domain data corresponding to the data of each sampling point in the first time-domain interval.

[0068] Among them, if the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, there are partially overlapping sampling point data between the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

[0069] In an embodiment of the present application, different time-frequency transformation processing will be performed on the sampling point data contained in the first time domain interval according to different situations of whether the first time domain interval corresponds to a previous adjacent second time domain interval in the time domain data to be processed, that is, different situations of whether the first time domain interval is the first time domain interval in the sampling point data sequence of the time domain data to be processed.

[0070] Among them, if the first time domain interval does not correspond to a previous adjacent second time domain interval in the time domain data, that is, the first time domain interval is the first time domain interval in the sampling point data sequence of the time domain data to be processed, for this case, the existing time-frequency domain transformation algorithm can be directly used to perform time-frequency domain transformation operations on each sampling point data contained in the first time domain interval, and convert each sampling point data in the first time domain interval to one or more frequency domains to obtain the first frequency domain data in one or more frequency domains.

[0071] Among them, when the time domain is converted to a single frequency domain point (i.e., single frequency), the data of each sampling point in the first time domain interval can be converted to a single frequency domain by using, but not limited to, DFT (Discrete Fourier Transform) or Geortzel and other algorithms, so as to obtain the first frequency domain data corresponding to the data of each sampling point in the first time domain interval.

[0072] In the case of converting the time domain to multiple frequency domain points (i.e., multiple frequencies), the FFT (fast Fourier transform) algorithm can be used, but is not limited to, to convert the data of each sampling point in the first time domain interval to multiple frequency domains to obtain the first frequency domain data corresponding to each sampling point data in the first time domain interval in each frequency domain. However, it is not limited to this implementation. In other implementations, the multiple frequency domains can also be directly regarded as K single frequencies. For each sampling point data in the first time domain interval, a single-frequency time-frequency domain transformation operation is performed on each frequency domain in the multiple frequency domains, such as performing a DFT transformation, and a total of K single-frequency time-frequency domain transformations are performed for the first time domain interval to obtain the first frequency domain data corresponding to each sampling point data in the first time domain interval in each frequency domain. Wherein, K is an integer greater than 1.

[0073] If the first time domain interval corresponds to a second time domain interval adjacent to the time domain data to be processed, that is, the first time domain interval is not the first time domain interval in the sampling point data sequence of the time domain data to be processed, for this situation, the embodiment of the present application no longer uses the existing time-frequency transformation algorithm to perform a complete time-frequency domain transformation operation on it, but determines the target increment of the first frequency domain data corresponding to the sampling point data in the first time domain interval relative to the second frequency domain data corresponding to the second time domain interval adjacent to it, and obtains the first frequency domain data by applying the target increment to the second frequency domain data.

[0074] Among them, since the second time domain interval is the previous adjacent time domain interval of the first time domain interval in the sampling point data sequence, the timing of the time-frequency transformation processing of the second time domain interval is earlier than that of the first time domain interval. When the sampling point data in the first time domain interval is subjected to time-frequency transformation processing, the second frequency domain data corresponding to the sampling point data in the second time domain interval is known information and can be used directly.

[0075] Compared with the second time domain interval adjacent to it, the first time domain interval has some overlapping sampling point data, while the other sampling point data are non-overlapping sampling points, and the target increment at least includes the first difference data between the frequency domain data components corresponding to the non-overlapping sampling point data between the first time domain interval and the second time domain interval adjacent to it. In the single-frequency operation, the frequency domain data component corresponding to each sampling point data can be obtained by multiplying the time domain data of the sampling point data with the rotation factor and converting the operation result to the frequency domain.

[0076] For example, assuming that the length of the time domain interval is 8, the sliding step of the time domain interval is 1, that is, the frame shift is 1, there are 7 overlapping sampling point data between the first time domain interval and the second time domain interval adjacent to it, assuming that the sampling point data contained in the second time domain interval are sampling point data 0 to 7, and the sampling point data contained in the first time domain interval are sampling point data 1 to 8, then the target increment includes at least the first difference data between the frequency domain data components corresponding to the sampling point data 8 and the sampling point data 0.

[0077] On the basis of obtaining the target increment, by applying the target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval, the first frequency domain data corresponding to each sampling point data in the first time domain interval can be obtained, without having to perform a complete time-frequency domain transformation operation on each sampling point data in the first time domain interval based on a time-frequency transformation algorithm such as DFT or FFT.

[0078] To summarize, the time-frequency transformation method provided in the embodiment of the present application locates the first time domain interval currently to be processed on the time domain data to be processed, and performs time domain to frequency domain conversion processing on each sampling point data in the first time domain interval to obtain first frequency domain data corresponding to each sampling point data in the first time domain interval. If the first time domain interval corresponds to a previously adjacent second time domain interval on the time domain data, there is partially overlapping sampling point data in the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

[0079] That is to say, for the case where the first time domain interval to be processed is not the first time domain interval in the sampling point data sequence, based on the embodiment of the present application, it is not necessary to perform a complete time-frequency domain transformation operation on each sampling point data in the first time domain interval based on a time-frequency transformation algorithm such as DFT or FFT, but through incremental operation, directly apply the target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval adjacent to the first time domain interval, and the first frequency domain data corresponding to each sampling point data in the first time domain interval can be obtained. Compared with the known technology for time-frequency transformation, when performing a complete time-frequency domain transformation operation on each sampling point data in the time domain interval (each frame) indiscriminately based on a time-frequency transformation algorithm such as DFT or FFT, the embodiment of the present application can effectively reduce the time complexity of the time-frequency transformation through the above-mentioned incremental operation method, while improving the efficiency of the time-frequency transformation.

[0080] In an optional embodiment, a process for determining the target increment (i.e., the increment of the first frequency domain data corresponding to each sampling point data in the first time domain interval to be processed, relative to the second frequency domain data corresponding to each sampling point data in the second time domain interval adjacent to the first time domain interval) is further provided. The process for determining the target increment is divided into two different situations: not applying a window function to the sampling point data and applying a window function to the sampling point data. The following describes each in detail.

[0081] 1. No window function is applied to the sampling point data

[0082] In the case where the window function is not applied to the sampling point data, the first difference data between the frequency domain data components respectively corresponding to the non-overlapping sampling point data of the first time domain interval and the second time domain interval adjacent thereto can be determined, and the first difference data is used as the target increment. That is, in the case where the window function is not applied, the target increment only includes the first difference data between the frequency domain data components respectively corresponding to the non-overlapping sampling point data of the first time domain interval and the second time domain interval adjacent thereto, and the frequency domain components respectively corresponding to the sampling point data of the overlapping part between adjacent time domain intervals in the two adjacent time domain intervals are the same.

[0083] The time-frequency domain transformation without applying a window function to the sampling point data is divided into single-frequency and multi-frequency cases.

[0084] Among them, if the frequency domain in the time-frequency transform is a single frequency domain, it can be specifically based on a single frequency transform method such as discrete Fourier transform or Geortzel, first determine the first frequency domain data component corresponding to the newly added sampling point data in the first time domain interval compared to the second time domain interval in the single frequency domain, and the second frequency domain data component corresponding to the missing sampling point data in the single frequency domain, and on this basis, further determine the difference data between the first frequency domain data component and the second frequency domain data component to obtain the first difference data, that is, the target increment.

[0085] For example, assuming that the length of the time domain interval is 8, the sliding step length of the time domain interval is 1, that is, the frame shift is 1, there are 7 overlapping sampling point data between the first time domain interval and the second time domain interval adjacent to it, and assuming that the sampling point data contained in the second time domain interval are sampling point data 0 to 7, and the sampling point data contained in the first time domain interval are sampling point data 1 to 8, then specifically based on single-frequency transformation methods such as DFT or Geortzel, the first frequency domain data component corresponding to the newly added sampling point data 8 of the first time domain interval relative to the second time domain interval can be determined, and the second frequency domain data component corresponding to the missing sampling point data 0 can be determined, and then the difference data between the two can be determined as the target increment. It should be noted that for the second frequency domain data component of the missing sampling point data, such as the missing sampling point data 0, the frequency domain data component of the missing sampling point data (such as sampling point data 0) when the time-frequency domain transformation is performed on the preceding time domain interval of the first time domain interval can be directly reused without recalculating.

[0086] In the single-frequency operation, the frequency domain data component corresponding to each sampling point data can be obtained by multiplying the time domain data of the sampling point data with the rotation factor and converting the operation result to the frequency domain.

[0087] When converting the time domain to the frequency domain, in the known technology, for the frame-by-frame time-frequency transformation of the data to be processed with a length of L (such as audio data containing L sampling point data), assuming that the length of a frame is N (that is, a time domain interval contains N sampling point data), when there is no overlapping interval between adjacent frames / time domain intervals, combined with reference to Figure 2 , the time complexity of the single-frequency time-frequency transform operation is L / N*O(N)=O(L).

[0088] However, the start and end of data features such as speech features are often random, and it is difficult to completely include a complete feature in a certain time domain interval. The features may be truncated at both ends of the interval, resulting in the inability to periodically connect the features, which will cause errors between the subsequent feature calculation results and the actual feature situation. In order to avoid truncation of data features (such as speech features) at both ends of the time domain interval as much as possible, a part of the overlapping interval will be added between each two adjacent time domain intervals to reduce the probability of feature truncation. However, this will lead to a significant increase in time complexity and calculation amount. For example, if the frame shift of a time domain interval of length N is changed from N sampling points to N / 2 sampling points (that is, the overlapping interval contains N / 2 sampling points), the calculation amount will become twice that of the original frame shift of N. If the frame shift is 1 sampling point (that is, the overlapping interval contains N-1 sampling points), the calculation amount will become N times that of the original frame shift of N, and the time complexity will also increase accordingly. When the frame shift is 1, the single-frequency time complexity will become N*O(L), which seriously affects the processing efficiency of time-frequency transformation.

[0089] The advantages of the embodiments of the present application compared with the known technology under the single-frequency condition are analyzed below.

[0090] Assuming a single frequency, a time-frequency transformation is performed on the N sampling points in the first time domain interval in the sampling point data sequence, which is represented by X([0:N-1], f) = Function(x[0:N-1]), which specifically represents the transformation of the sampling point data 0 to N-1 in the interval [0:N-1] of the time domain x to the frequency f of the frequency domain X. Function represents the time-frequency transformation in a single frequency, which is a formula with accumulation and averaging properties.

[0091] In the time-frequency domain transformation process, when the time domain interval slides to the right based on the processing progress, for example, sliding to the right with a step size of 1 frame shift, transforming from X([L:R], f) to X([L+1:R+1], f) (that is, the time domain interval slides from [L:R] to [L+1:R+1]), the new interval will cause the information of the sampling point x[R+1] to be added in the time domain, and the information of the sampling point x[L] to be reduced. For this situation, based on the embodiment of the present application, according to the accumulation and averaging properties of Function, by applying the target increment to the frequency domain information of X([L:R], the frequency domain information of X([L+1:R+1] is obtained, that is, the frequency domain data component of the newly added sampling point x[R+1] and the frequency domain data component of the reduced sampling point x[L] are determined, and based on the accumulation and averaging properties of Function, the difference data between the two frequency domain data components is applied to the frequency domain data of X([L:R], and the frequency domain information corresponding to X([L+1:R+1] can be obtained.

[0092] Among them, the length of the time domain interval is fixed to N, and the time complexity of transforming from X([L:R], f) to X([L+1:R+1], f) is O(1). Then, for the data to be processed with a length of L (such as voice data), the overall time complexity of its time-frequency transform is O(L), which is much smaller than N*O(L) in the known technology, thereby reducing the time complexity.

[0093] If the frequency domain in the time-frequency transform is a multi-frequency domain, the multi-frequency domain can be regarded as K single frequencies, and for each frequency domain in the K single frequencies, the third frequency domain data component corresponding to the newly added sampling point data in the first time domain interval compared to the second time domain interval, and the fourth frequency domain data component corresponding to the missing sampling point data in each frequency domain are determined, and the difference data between the third frequency domain data component and the fourth frequency domain data component corresponding to each frequency domain are determined, so as to obtain the difference data between the third frequency domain data component and the fourth frequency domain data component on each frequency domain (K frequency domains), as the first difference data on each frequency domain, that is, as the target increment. Subsequently, for each frequency domain, the first frequency domain data corresponding to each sampling point data in the first time domain interval on the frequency domain can be obtained by applying the difference data corresponding to the second frequency domain data on the frequency domain to the sampling point data in the second time domain interval.

[0094] In addition, for the multi-frequency domain situation, it is also possible but not limited to determining the third frequency domain data component corresponding to the newly added sampling point data in each frequency domain compared to the second time domain interval in the first time domain interval, and the fourth frequency domain data component corresponding to the missing sampling point data in each frequency domain based on the fast Fourier transform method, and determining the difference data between the third frequency domain data component and the fourth frequency domain data component corresponding to each frequency domain as the first difference data on each frequency domain.

[0095] In the fast Fourier transform, N points are regarded as a time domain interval (i.e., one frame) for a time-frequency transform, and specifically the N points are converted from the time domain to multiple frequency domains (i.e., multi-frequency). Assuming a multi-frequency scenario, a time-frequency transform is performed on the N points in the first time domain interval in the sampling point data sequence, expressed as X([0:N-1], f0:fN-1)=Function(x[0:N-1]), which specifically represents the transformation of the sampling point data 0~N-1 in the interval [0:N-1] of the time domain x to multiple frequencies f0:fN-1 (i.e., multi-frequency domain) of the frequency domain X, and Function represents the time-frequency transform in multi-frequency.

[0096] The function of fast Fourier transform is as follows Figure 3 As shown, it can be realized by butterfly operation, where W represents a fixed rotation factor, and the appearance of * indicates multiplication operation on the corresponding data, such as Figure 3 The first * that appears from left to right in the second row indicates that the time domain data of the sampling point x(4) is rotated by the rotation factor w 0 NFor multiplication, there are intermediate variables at the nodes from one to many (corresponding to the position indicated by the tail of the arrow in the figure) and from many to one (corresponding to the position indicated by the head of the arrow in the figure), and the operation of the intermediate variable from many to one is to calculate the addition of each variable. For a more detailed implementation process of fast Fourier transform, please refer to the relevant content of fast Fourier transform in the known technology.

[0097] In the case of multiple frequencies, if adjacent time domain intervals do not overlap, in the known technology, if K single frequencies are considered, the multi-frequency domain is regarded as K single frequencies for time-frequency domain transformation, and its time complexity is L / N*O(KN)=O(KL). If the fast Fourier transform is used, combined with reference to Figure 4 The schematic diagram of the time complexity of the fast Fourier transform in the case of multiple frequencies and no overlap of adjacent time domain intervals is shown. The time complexity of calculating N frequencies at a time is L / N*O(NlogN)=O(LlogN). In practical applications, the method with the smaller time complexity in these two implementation methods can be adopted, so that the time complexity in this case in practical applications is min(O(LlogN), O(KL)).

[0098] In order to reduce the probability of feature truncation, when adjacent time domain intervals (i.e., adjacent frames) are overlapped, in the known technology, combined with reference to Figure 5 The schematic diagram of the time complexity of the fast Fourier transform in the case of multiple frequencies and overlapping adjacent time domain intervals is shown. The overlapping of adjacent time domain intervals will further increase the time complexity. If the frame shift of the N-length time domain interval is changed from N to N / 2, the amount of calculation will be doubled. If the frame shift is 1, the amount of calculation will be N times the original. Considering the transformation method of K single frequencies, when adjacent time domain intervals overlap, when the frame shift is 1, the time complexity in the known technology will become N*min(O(LlogN), O(KL)), which is unacceptable.

[0099] Need to explain, Figure 5 Specifically, it is a schematic diagram of the time complexity of the fast Fourier transform when there is overlap between multiple frequencies and adjacent time domain intervals without applying a window function to the sampling point data.

[0100] In the implementation method based on incremental operation provided in the embodiment of the present application, for the time-frequency transformation of the data to be processed with a length of L (such as audio data containing L sampling point data), if the time-frequency transformation method of treating the multi-frequency domain as K single frequencies is adopted, its time complexity is K*O(L)=O(KL). If the fast Fourier transform is adopted, its time complexity is O(LlogN). For example, for Figure 5In the example of , the length of the time domain interval N = 8, assuming that the frame shift is 1, during the time-frequency transformation process, it is necessary to first slide 1 sampling point from the time domain interval [0:7] to [1:8], and then slide with a frame shift of 1 in sequence, and determine the time domain data corresponding to the sampling point data in each non-starting time domain interval based on an incremental method, until sliding to the last time domain interval in the sampling point data sequence, for example, until sliding over N sampling points to the time domain interval [8:15]. In this process, a total of NlogN times of multiplication of the rotation factor need to be calculated, and on average each time from X([L:R], f) to X([L+1:R+1], f), the time complexity is O(logN), and the overall time complexity is O(LlogN). Therefore, using the implementation method based on incremental operation provided by the embodiment of the present application, the time complexity of multi-frequency transformation when the frame shift is 1 is min(O(LlogN), O(KL)), which is significantly reduced compared to N*min(O(LlogN), O(KL)) in the known technology.

[0101] 2. Applying a window function to the sampling point data

[0102] In order to further reduce the leakage of spectrum information, a window function may be applied to the sampling point data when performing the time-frequency transform. Optionally, the window function may be a window function of a polynomial window or a cosine window.

[0103] The polynomial window may be, but is not limited to, a rectangular window, a Bartley window or a triangular window, and the cosine window may be, but is not limited to, a Hann window, a Hamming window, a Blackman window or a flat-top window.

[0104] In the case of applying a window function to the sampling point data, the first difference data between the frequency domain data components corresponding to the non-overlapping sampling point data of the first time domain interval and the second time domain interval adjacent thereto, and the second difference data between the frequency domain data components corresponding to the overlapping sampling point data can be determined, and the target increment includes the first difference data and the second difference data. That is, in the case of applying a window function, the target increment includes not only the first difference data between the frequency domain data components corresponding to the non-overlapping sampling point data of the first time domain interval and the second time domain interval adjacent thereto, but also the second difference data between the frequency domain data components corresponding to the overlapping sampling point data in the first and second time domain intervals, and the frequency domain data components corresponding to the sampling point data of the overlapping part between adjacent time domain intervals are different in the two adjacent time domain intervals.

[0105] See also Figure 6Schematic diagram of the time complexity of the fast Fourier transform of the known technology in the case of windowing in a multi-frequency and adjacent time domain interval overlapping scenario shown in FIG. , where w(i) represents the function value of the window function, and the number of function values ​​of the window function is consistent with the number of sampling point data in a single time domain interval. The value of i is different, and the value of w(i) is usually different. In the time-frequency transform, see Figure 6 , each sampling point data needs to be multiplied point by point with its corresponding window function value, such as Figure 6 In the example, x(0) needs to be multiplied by w(0), x(1) needs to be multiplied by w(1), and so on, until the next time domain interval, then point-to-point multiplication is performed according to the corresponding relationship between the sampling point x(i) and the window function value w(i) in the next time domain interval, and on this basis, the sampling point data of the time domain interval is converted from the time domain to the frequency domain by performing operations between the point-to-point multiplication results and the rotation factors and time-frequency domain transformation. Due to the application of the window function, the frequency domain data components corresponding to the sampling point data of the overlapping parts between adjacent time domain intervals in the two adjacent time domain intervals are different. Therefore, the target increment includes not only the first difference data between the frequency domain data components corresponding to the non-overlapping sampling point data of the first time domain interval and the second time domain interval adjacent to it, but also the second difference data between the frequency domain data components corresponding to the overlapping sampling point data in the first and second time domain intervals.

[0106] Among them, when a window function is applied to the sampling point data, the first difference data between the frequency domain data components corresponding to the non-overlapping sampling point data of the first time domain interval and the second time domain interval adjacent to it can also be determined by applying corresponding time-frequency domain transformation algorithms such as DFT, Geortzel, FFT, etc., to determine the frequency domain data components corresponding to the newly added sampling point data in the first time domain interval compared to the second time domain interval adjacent to it, and reuse the frequency domain data components corresponding to the missing sampling point data for calculation.

[0107] When a window function is applied to the sampling point data, the second difference data between the frequency domain data components corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval adjacent thereto can be determined by the following processing procedure: performing a difference operation on the frequency domain data components corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval; if the result of the difference operation is not 0, recursively performing a frequency domain data difference operation on the overlapping interval formed by the overlapping sampling point data and the preceding adjacent interval of the overlapping interval until the end condition is met, and determining the difference operation result of the frequency domain data components corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval when the end condition is met as the second difference data.

[0108] Specifically, when determining the frequency domain data corresponding to each sampling point data in the first time domain interval based on the incremental update method provided in the embodiment of the present application, it is necessary to differenciate the frequency domain data corresponding to the first time domain interval and the second time domain interval adjacent to it to obtain the target increment. The result of the difference will be composed of the differences in the frequency domain data components corresponding to the head and tail (i.e., non-overlapping areas) and the middle overlapping parts. Taking the frame shift of 1 as an example, the calculation amount of the difference data of the head and tail (non-overlapping area) parts is O(1). If the difference of the middle overlapping part cannot be offset to 0, it is further processed according to the recursive idea, wherein the middle overlapping part is specifically incrementally updated for the second time until the D-th incremental update, when the middle overlapping part can be offset to 0 (corresponding to the case where the applied window function is a polynomial window) or is a part of the D-th overlapping part (corresponding to the case where the applied window function is a cosine window), the end condition can be met. At this time, the superposition of the D-times head and tail calculation amount O(1) is O(D), and the difference operation result of the frequency domain data components corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval can be determined as the second difference data.

[0109] Through the above recursive processing, the differences in the frequency domain data components of the overlapping areas in the windowed case can be continuously tracked / iterated to a complexity of O(1), that is, the complexity of the incremental update = O(1) + the incremental update complexity of the next layer. Finally, through the above processing, the complexity of the difference operation of the middle overlapping part is optimized to O(D) (that is, the superposition of D head and tail calculations O(1)).

[0110] Among them, the second incremental update of the middle overlapping part specifically refers to taking the "overlapping part" (such as sliding the time domain interval 0-3 to 1-4, the first time domain interval is 1-4, and the second time domain interval adjacent to it is 0-3, and the corresponding overlapping part is 1-3) as the interval after sliding, and comparing the "overlapping part" (i.e. 1-3) with the interval before sliding (0-2) corresponding to the interval after sliding, to continue to incrementally update the overlapping part (i.e. 1-3) according to the above-mentioned incremental update idea, until the recursion meets the above-mentioned end condition, and the same applies to other rounds of recursion in the entire recursive process.

[0111] When the window function applied to the sampling point data is a polynomial window, D represents the degree of the highest order term in the polynomial; when the window function applied to the sampling point data is a cosine window, D=2 (cos performs a second derivative operation, which is still cos).

[0112] Based on the above processing, when a window function (polynomial window or cosine window) is applied to the sampling point data, the time complexity of time-frequency domain transformation is improved compared with the processing method of the known technology, whether it is single-frequency or multi-frequency.

[0113] For example, when the frame shift is 1, the time-frequency transformation complexity of single-frequency windowing will be improved from O(NL) in the known technology to O(DL). Usually, D will be less than or much less than N. For multi-frequency windowing, if the time-frequency transformation method that regards the multi-frequency domain as K single frequencies is adopted, the time complexity can be K times the time complexity of a single frequency, that is, K*O(DL)=O(KDL). If a fast Fourier transform method is adopted, such as butterfly operation, the time complexity can be improved from O(LNlogN) to O(DLlogN). Therefore, in the case of multi-frequency windowing, the time complexity can be improved to min(O(DLlogN), O(KDL)), which is also improved compared with the time complexity N*min(O(LlogN), O(KL)) in the known technology (in the known technology, when the frame shift is 1 and windowing is performed, the time complexity is also N*min(O(LlogN), O(KL))).

[0114] For windowed or unwindowed and single-frequency or multi-frequency situations, after obtaining the frequency domain data corresponding to each time domain interval (frame) through the time-frequency domain transformation processing scheme of the embodiment of the present application, the frequency domain data of each time domain interval can be output optionally to provide frequency domain data support for subsequent processing. For example, in a speech processing scenario, speech features (such as phonemes) can be extracted based on the output frequency domain data of each frame, and semantic recognition can be performed based on the extracted speech features.

[0115] To summarize, in single-frequency or multi-frequency time-frequency transformation processing, for non-first time domain intervals on a sampling point data sequence, the embodiment of the present application determines the difference in frequency domain data between the current time domain interval and the previous adjacent time domain interval through an incremental operation, and by applying the increment formed by the difference on the basis of the frequency domain data corresponding to the previous adjacent time domain interval, the frequency domain data corresponding to the current time domain interval can be obtained, without having to perform a complete time-frequency domain transformation operation on each sampling point data in the current time domain interval based on a time-frequency transformation algorithm such as DFT or FFT, thereby effectively reducing the time complexity of the time-frequency transformation and improving the efficiency of the time-frequency transformation.

[0116] In an alternative embodiment, see Figure 7 As shown in the flowchart of the time-frequency transformation method, the time-frequency transformation method provided by the present application, for a single frequency domain, after obtaining the first frequency domain data corresponding to each sampling point data in the first time domain interval, may further include the following processing:

[0117] Step 104: Determine whether the frequency domain characteristics of the first frequency domain data meet a first characteristic condition.

[0118] The first characteristic condition may be, but is not limited to, set as the amplitude / energy of the frequency domain characteristic reaching a threshold.

[0119] This step makes a preliminary judgment on the first frequency domain data corresponding to each sampling point data in the first time domain interval by determining whether the frequency domain characteristics of the first frequency domain data meet the first characteristic condition. If the frequency domain characteristics of the first frequency domain data meet the first characteristic condition, such as reaching the set threshold, the processing of the next step 105 is continued; otherwise, step 105 is skipped, and the first time domain interval is directly slid to the right by the set step size (such as 1 sampling point) to determine the next first time domain interval to be processed.

[0120] Step 105: If the first characteristic condition is met, increase the right boundary of the first time domain interval, and determine whether there is target sampling point data whose corresponding frequency domain characteristics meet the second characteristic condition in the time domain interval obtained after increasing the right boundary; if so, update the right boundary of the first time domain interval to the target sampling point data, and determine the first frequency domain data corresponding to each sampling point data in the first time domain interval after the update; and when determining the first time domain interval next time, determine the left boundary of the first time domain interval based on the target sampling point data, and determine the right boundary of the first time domain interval based on the predetermined interval length.

[0121] The second characteristic condition may be set as follows: the frequency domain characteristics of the sampling point data on the left side of the target sampling point data continuously become stronger in time sequence, and the frequency domain characteristics of the sampling point data on the right side of the target sampling point data gradually become weaker in time sequence.

[0122] When the frequency domain characteristics of the first frequency domain data meet the first characteristic condition, such as the frequency domain characteristic amplitude / energy reaches the threshold, the right boundary of the first time domain interval can be increased with a certain amplification step (such as 1 sampling point) while the left boundary of the first time domain interval is fixed, and it is determined whether there is target sampling point data whose corresponding frequency domain characteristics meet the second characteristic condition in the time domain interval obtained after the right boundary is increased, for example, whether there is a certain target sampling point data whose frequency domain characteristics of the left sampling points continuously become stronger in time sequence, and the frequency domain characteristics of the right sampling point data gradually become weaker in time sequence. If not, the right boundary of the first time domain interval is continuously increased based on the amplification step of the right boundary until the target sampling point data meeting the second characteristic condition is detected, and the right boundary of the first time domain interval is updated to the target sampling point data, and then the frequency domain data corresponding to each sampling point data in the currently obtained first time domain interval (that is, the right boundary is the target sampling point data) is determined based on the incremental update method.

[0123] Afterwards, the current first time interval (ie, the right boundary is the target sampling point data) may be further slid based on the window sliding concept to determine the next first time domain interval to be processed.

[0124] Among them, when determining the first time domain interval next time, that is, determining the next first time domain interval to be processed, the left boundary of the first time domain interval can be determined based on the target sampling point data, and the right boundary of the first time domain interval can be determined based on the set predetermined interval length (such as 8).

[0125] It is easy to understand that in this implementation, the set predetermined interval length (such as 8) is specifically the minimum interval length of the first time domain interval, and its length can be changed subsequently through steps 104-105.

[0126] When determining the left boundary of the first time domain interval based on the target sampling point data, the left boundary can be determined for the next first time domain interval to be processed based on the target sampling point data and the preset maximum overlapping length of the sampling points. For example, assuming that the target sampling point data is sampling point data M and the maximum overlapping length is MR, the left boundary of the next first time domain interval to be processed can be determined as M-MR, and the left boundary plus the minimum interval length of the first time domain interval is used as the right boundary of the next first time domain interval to be processed, thereby obtaining the next first time domain interval to be processed. Subsequently, for the next first time domain interval to be processed, steps 104-105 can be looped until the processing of the sampling point data sequence is completed.

[0127] The maximum overlapping length of sampling points is used to represent the maximum overlapping number of sampling point data between the first time domain interval and the time domain interval that is adjacent to the first time domain interval in the time domain data.

[0128] In practical applications, optionally, when determining the left boundary of the next first time domain interval to be processed based on the target sampling point data, multiple left boundaries can be determined for the next first time domain interval to be processed based on the target sampling point data and the maximum overlapping length of the sampling points. For example, multiple overlapping interval lengths are selected between the overlapping interval lengths 1 and MR, and the left boundary of the next first time domain interval to be processed is determined in combination with the target sampling point data, thereby obtaining multiple left boundaries of the next first time domain interval to be processed, and for each left boundary, the right boundary of the next first time domain interval to be processed is determined based on the minimum interval length, and the corresponding A plurality of next first time domain intervals to be processed, and for each next first time domain interval to be processed, loop to steps 104-105 of this embodiment, and try a plurality of left boundaries to finally obtain a suitable first time domain interval that just covers the complete data features as much as possible through feature detection and interval lengthening (such as feature detection and interval lengthening based on the first feature condition and the second feature condition) processing (that is, the first time domain interval is neither too large to include additional featureless intervals nor too small to fail to include them completely) relative to the feature interval of the complete data features, so as to perform effective time-frequency domain transformation processing on the data of each sampling point contained therein.

[0129] In this embodiment, optionally, only the frequency domain transformation results corresponding to each time domain interval with the corresponding target sampling point data as the right boundary after the interval is lengthened can be output in the end. Since each time domain interval with the corresponding target sampling point data as the right boundary is the interval with the strongest frequency domain feature signal obtained after feature detection (the strongest frequency domain feature signal indicates that the frequency domain features are complete and do not include additional featureless intervals), the corresponding frequency domain transformation results can be output in a targeted manner, which can provide effective frequency domain data support for subsequent processing based on frequency domain data (such as frequency domain feature extraction and related application processing such as speech semantic recognition based on this), and can also improve the subsequent data processing efficiency and reduce processing time.

[0130] Corresponding to the above-mentioned time-frequency conversion method, the embodiment of the present application also provides a time-frequency conversion device, whose composition structure is as follows: Figure 8 As shown, including:

[0131] The acquisition module 801 is used to obtain the time domain data to be processed; the time domain data includes a sampling point data sequence formed by a plurality of sampling point data in the time domain;

[0132] A positioning module 802, configured to locate a first time domain interval currently to be processed on the time domain data;

[0133] The processing module 803 is used to perform time-domain to frequency-domain processing on each sampling point data in the first time domain interval to obtain first frequency domain data corresponding to each sampling point data in the first time domain interval;

[0134] Among them, if the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, there are partially overlapping sampling point data between the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

[0135] In an optional implementation, the positioning module 802 is configured to perform one of the following:

[0136] Locating a time domain interval of a predetermined interval length from the head of the sampling point data sequence to obtain the first time domain interval;

[0137] The most recently processed time domain interval is slid on the sampling point data sequence in a time sequence by a predetermined step length to obtain the first time domain interval; the first time domain interval includes a plurality of continuous sampling point data corresponding to the predetermined interval length, and the predetermined step length is less than the predetermined interval length.

[0138] In an optional implementation manner, the processing module 803 is specifically configured to:

[0139] If the first time domain interval does not correspond to a second time domain interval adjacent to the first time domain interval in the time domain data, converting each sampling point data in the first time domain interval into one or more frequency domains to obtain first frequency domain data in the one or more frequency domains;

[0140] If the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, determine a target increment of frequency domain data corresponding to the sampling point data in the first time domain interval relative to the second frequency domain data, and apply the target increment to the second frequency domain data to obtain the first frequency domain data;

[0141] The target increment includes at least first difference data between frequency domain data components corresponding to non-overlapping sampling point data between the first time domain interval and the second time domain interval.

[0142] In an optional implementation manner, the processing module 803, when determining a target increment of the frequency domain data corresponding to the sampling point data in the first time domain interval relative to the second frequency domain data, is specifically configured to:

[0143] Without applying a window function to the sampling point data, determining first difference data between frequency domain data components corresponding to non-overlapping sampling point data of the first time domain interval and the second time domain interval as the target increment;

[0144] When a window function is applied to the sampling point data, first difference data between frequency domain data components corresponding to non-overlapping sampling point data of the first time domain interval and the second time domain interval, and second difference data between frequency domain data components corresponding to overlapping sampling point data are determined, and the target increment includes the first difference data and the second difference data.

[0145] In an optional implementation manner, the processing module 803, when applying a window function to the sampling point data, determines the second difference data between the frequency domain data components respectively corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval, is specifically used to:

[0146] Performing a difference operation on frequency domain data components respectively corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval;

[0147] If the result of the difference operation is not 0, recursively perform frequency domain data difference operation on the overlapping interval formed by the overlapping sampling point data and the previous adjacent interval of the overlapping interval until the end condition is met, and the difference operation result of the frequency domain data components corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval respectively when the end condition is met is determined as the second difference data.

[0148] In an optional implementation manner, if the frequency domain in the time-frequency transform is a single frequency domain, after obtaining the first frequency domain data corresponding to each sampling point data in the first time domain interval, the processing module 803 is further used to:

[0149] Determining whether the frequency domain characteristics of the first frequency domain data meet a first characteristic condition;

[0150] If the first characteristic condition is met, the right boundary of the first time domain interval is increased, and it is determined whether there is target sampling point data with corresponding frequency domain characteristics that meet the second characteristic condition in the time domain interval obtained after increasing the right boundary; if so, the right boundary of the first time domain interval is updated to the target sampling point data, and the first frequency domain data corresponding to each sampling point data in the first time domain interval after the update is determined; and when the first time domain interval is determined next time, the left boundary of the first time domain interval is determined based on the target sampling point data, and the right boundary of the first time domain interval is determined based on the predetermined interval length.

[0151] In an optional implementation manner, the processing module 803, when determining the left boundary of the first time domain interval based on the target sampling point data, is specifically configured to:

[0152] Determining a plurality of left boundaries for the first time domain interval based on the target sampling point data and the maximum overlapping length of the sampling points;

[0153] The maximum overlapping length of sampling points is used to represent the maximum overlapping number of sampling point data between the first time domain interval and the time domain interval adjacent to the first time domain interval in the time domain data.

[0154] In an optional implementation manner, the acquisition module 801 is specifically configured to:

[0155] A sampling point data sequence formed by a plurality of sampling point data of the audio data in the time domain is obtained by sampling the audio data in the time domain.

[0156] The present application also discloses an electronic device, the composition structure of the electronic device, such as Fig. 9 As shown, including at least:

[0157] The collection device 10 is used to collect sampling point data.

[0158] Among them, the acquisition device 10 can specifically but not limited to sample voice data, image data or video data in the time domain for processing scenarios such as voice, image or video, so as to obtain the time domain data to be processed formed by the sampling point data sequence of the voice data, image data or video data waiting to be processed in the time domain.

[0159] The processor 20 is configured to perform the following processing:

[0160] Obtaining time domain data to be processed; the time domain data includes a sampling point data sequence formed by a plurality of sampling point data in the time domain;

[0161] Locating a first time domain interval currently to be processed on the time domain data;

[0162] Performing time-domain-to-frequency-domain processing on each sampling point data in the first time domain interval to obtain first frequency-domain data corresponding to each sampling point data in the first time domain interval;

[0163] Among them, if the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, there are partially overlapping sampling point data between the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

[0164] In actual applications, the processor 20 is not limited to the above functions, and can specifically execute the time-frequency conversion method provided by any method embodiment of the present application.

[0165] The processor 20 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU) or other programmable logic devices, etc.

[0166] The electronic device may also include a display device and / or a display interface and may be connected to an external display device.

[0167] Optionally, the electronic device may further include a camera assembly, and / or be connected to an external camera assembly.

[0168] In addition, the electronic device may also include components such as a memory, a communication interface, and a communication bus. The acquisition device, the memory, the processor, and the communication interface communicate with each other through the communication bus.

[0169] The communication interface is used for communication between electronic devices and other devices. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0170] An embodiment of the present application also provides a storage medium, which carries one or more computer instruction sets. When the one or more computer instruction sets are executed by an electronic device, the electronic device can implement the time-frequency conversion method as described in any one of the above items.

[0171] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0172] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0173] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or the part that makes a creative contribution can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0174] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0175] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A time-frequency conversion method, comprising: Obtaining time domain data to be processed; the time domain data includes a sampling point data sequence formed by a plurality of sampling point data in the time domain; Locating a first time domain interval currently to be processed on the time domain data; Performing time-domain-to-frequency-domain processing on each sampling point data in the first time domain interval to obtain first frequency-domain data corresponding to each sampling point data in the first time domain interval; Among them, if the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, there are partially overlapping sampling point data between the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

2. The time-frequency transformation method according to claim 1, wherein the locating the first time domain interval currently to be processed on the time domain data comprises one of the following: Locating a time domain interval of a predetermined interval length from the head of the sampling point data sequence to obtain the first time domain interval; The most recently processed time domain interval is slid on the sampling point data sequence in a time sequence by a predetermined step length to obtain the first time domain interval; the first time domain interval includes a plurality of continuous sampling point data corresponding to the predetermined interval length, and the predetermined step length is less than the predetermined interval length.

3. The time-frequency conversion method according to claim 1, wherein the step of performing time-domain-to-frequency-domain processing on each sampling point data in the first time domain interval to obtain first frequency domain data corresponding to each sampling point data in the first time domain interval comprises: If the first time domain interval does not correspond to a second time domain interval adjacent to the first time domain interval in the time domain data, converting each sampling point data in the first time domain interval into one or more frequency domains to obtain first frequency domain data in the one or more frequency domains; If the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, determine a target increment of frequency domain data corresponding to the sampling point data in the first time domain interval relative to the second frequency domain data, and apply the target increment to the second frequency domain data to obtain the first frequency domain data; The target increment includes at least first difference data between frequency domain data components corresponding to non-overlapping sampling point data between the first time domain interval and the second time domain interval.

4. The time-frequency transformation method according to claim 3, wherein determining a target increment of the frequency domain data corresponding to the sampling point data in the first time domain interval relative to the second frequency domain data comprises: Without applying a window function to the sampling point data, determining first difference data between frequency domain data components corresponding to non-overlapping sampling point data of the first time domain interval and the second time domain interval as the target increment; When a window function is applied to the sampling point data, first difference data between frequency domain data components corresponding to non-overlapping sampling point data of the first time domain interval and the second time domain interval, and second difference data between frequency domain data components corresponding to overlapping sampling point data are determined, and the target increment includes the first difference data and the second difference data.

5. The time-frequency transformation method according to claim 4, wherein, when a window function is applied to the sampling point data, determining second difference data between frequency domain data components respectively corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval comprises: Performing a difference operation on frequency domain data components respectively corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval; If the result of the difference operation is not 0, recursively perform frequency domain data difference operation on the overlapping interval formed by the overlapping sampling point data and the previous adjacent interval of the overlapping interval until the end condition is met, and the difference operation result of the frequency domain data components corresponding to the overlapping sampling point data of the first time domain interval and the second time domain interval respectively when the end condition is met is determined as the second difference data.

6. The time-frequency transformation method according to claim 2, if the frequency domain in the time-frequency transformation is a single frequency domain, after obtaining the first frequency domain data corresponding to each sampling point data in the first time domain interval, further comprising: Determining whether the frequency domain characteristics of the first frequency domain data meet a first characteristic condition; If the first characteristic condition is met, increase the right boundary of the first time domain interval, and determine whether there is target sampling point data whose corresponding frequency domain characteristics meet the second characteristic condition in the time domain interval obtained after increasing the right boundary; if so, update the right boundary of the first time domain interval to the target sampling point data, and determine the first frequency domain data corresponding to each sampling point data in the first time domain interval after the update; When the first time domain interval is determined next time, the left boundary of the first time domain interval is determined based on the target sampling point data, and the right boundary of the first time domain interval is determined based on the predetermined interval length.

7. The time-frequency transformation method according to claim 6, wherein determining the left boundary of the first time domain interval based on the target sampling point data comprises: Determining a plurality of left boundaries for the first time domain interval based on the target sampling point data and the maximum overlapping length of the sampling points; The maximum overlapping length of sampling points is used to represent the maximum overlapping number of sampling point data between the first time domain interval and the time domain interval adjacent to the first time domain interval in the time domain data.

8. The time-frequency transformation method according to claim 1, wherein obtaining the time domain data to be processed comprises: A sampling point data sequence formed by a plurality of sampling point data of the audio data in the time domain is obtained by sampling the audio data in the time domain.

9. A time-frequency conversion device, comprising: An acquisition module, used to obtain time domain data to be processed; the time domain data includes a sampling point data sequence formed by a plurality of sampling point data in the time domain; A positioning module, used for positioning a first time domain interval to be processed on the time domain data; A processing module, configured to perform time-domain to frequency-domain processing on each sampling point data within the first time domain interval to obtain first frequency domain data corresponding to each sampling point data within the first time domain interval; Among them, if the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, there are partially overlapping sampling point data between the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

10. An electronic device comprising: A collection device, used for collecting sampling point data; A processor for performing the following processing: Obtaining time domain data to be processed; the time domain data includes a sampling point data sequence formed by a plurality of sampling point data in the time domain; Locating a first time domain interval currently to be processed on the time domain data; Performing time-domain-to-frequency-domain processing on each sampling point data in the first time domain interval to obtain first frequency-domain data corresponding to each sampling point data in the first time domain interval; Among them, if the first time domain interval corresponds to a second time domain interval adjacent to the time domain data, there are partially overlapping sampling point data between the first time domain interval and the second time domain interval, and the first frequency domain data is the frequency domain data obtained by applying a target increment to the second frequency domain data corresponding to each sampling point data in the second time domain interval.

Images