Data extraction method and device and electronic equipment
By using simple and complex recognition model combinations in speech separation technology, the problem of high resource consumption is solved, and the resource optimization and processing efficiency improvement of the speech separation process is achieved.
Patent Information
- Application Number
- CN202411046599.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-20
AI Technical Summary
The existing speech separation technology consumes a high resource in speech processing with different separation difficulties, resulting in unnecessary waste of resources.
Two recognition models are adopted: a simple first recognition model for speech feature extraction with less resource occupancy, and a complex second recognition model for dealing with mismatched features by combining features to reduce resource consumption.
It effectively reduces unnecessary resource consumption during speech separation, improves processing time and accuracy, and realizes rapid processing of simple problems and fine processing of complex problems.
Smart Images
Figure CN119170036A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech recognition technology, and in particular to a data extraction method, device and electronic device. Background Art
[0002] At present, speech separation technology uses the same recognition model to perform multiple downsampling and upsampling of mixed audio to separate the target speaker's speech for speech with different separation difficulties. However, this method of speech separation consumes a lot of resources. How to reduce unnecessary resource consumption in the speech separation process is a technical problem that technicians in this field need to solve. Summary of the invention
[0003] In view of this, embodiments of the present application provide a data extraction method, device, and electronic device.
[0004] According to the first aspect of the present application, an embodiment of the present application provides a data extraction method, including:
[0005] Acquire the sound data to be extracted and the reference sound data; extract the first sound feature and other sound features corresponding to the reference sound data in the sound data to be extracted based on the first recognition model, and determine the target information corresponding to the first sound feature; the other sound features are the sound features other than the first sound feature in the sound data to be extracted; if the target information indicates that the first sound feature matches the reference sound data, form the target sound data based on the first sound feature;
[0006] If the first sound feature representing the target information does not match the reference sound data, a second sound feature corresponding to the reference sound data is extracted from the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on a second recognition model; based on the second sound feature, the target sound data is formed; wherein the resources occupied by the second recognition model when working are higher than the resources occupied by the first recognition model when working.
[0007] Optionally, the data extraction method includes:
[0008] If the similarity between the first sound feature represented by the target information and the sound feature included in the reference sound data is greater than or equal to a first preset threshold, determining that the first sound feature represented by the target information matches the reference sound data;
[0009] If the similarity between the first sound data representing the target information and the sound feature included in the reference sound data is less than a first preset threshold, it is determined that the first sound feature representing the target information does not match the reference sound data.
[0010] Optionally, the data extraction method includes:
[0011] If the difference between the first sound feature represented by the target information and the other sound features is greater than or equal to a second preset threshold, determining that the first sound feature represented by the target information matches the reference sound data;
[0012] If the difference between the first sound feature represented by the target information and the other sound features is less than a second preset threshold, it is determined that the first sound feature represented by the target information does not match the reference sound data.
[0013] Optionally, extracting the second sound feature corresponding to the reference sound data from the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the second recognition model includes:
[0014] Determine a third sound feature corresponding to the reference sound data based on the second recognition model, wherein the third sound feature is a sound feature that satisfies a similarity condition with the sound feature included in the reference sound data, and the speech duration of the sound data corresponding to the third sound feature is greater than the speech duration corresponding to the reference sound data;
[0015] Based on the second recognition model, the sound feature corresponding to the third sound feature is extracted from the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features to obtain the second sound feature corresponding to the reference sound data in the first sound feature.
[0016] Optionally, extracting the second sound feature corresponding to the reference sound data from the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the second recognition model includes:
[0017] At least a portion of the processing moment when the second recognition model processes the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features, the sound feature extracted at the previous processing moment is used as the reference sound feature, and the second recognition model is used to extract the sound feature of the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features;
[0018] Based on the sound feature extracted at each processing time, a second sound feature corresponding to the reference sound data is formed among the first sound feature or the sound feature combined with the first sound feature and other sound features.
[0019] Optionally, determining a third sound feature corresponding to the reference sound data based on the second recognition model includes:
[0020] determining sound features included in the reference sound data based on the second recognition model;
[0021] A third sound feature that satisfies a similarity condition with the sound feature included in the reference sound data is determined based on the second recognition model.
[0022] Optionally, determining, based on the second recognition model, a third sound feature that satisfies a similarity condition with a sound feature included in the reference sound data includes:
[0023] A third sound feature that satisfies a similarity condition with the sound feature included in the reference sound data is searched from the second recognition model.
[0024] Optionally, the energy consumption when the second recognition model is working is higher than the energy consumption when the first recognition model is working.
[0025] According to a second aspect of the present application, an embodiment of the present application provides an electronic device, including:
[0026] A first encoder, a second encoder and a processor;
[0027] The first encoder is used to obtain the sound data to be extracted and the reference sound data; extract the first sound feature corresponding to the reference sound data and other sound features in the sound data to be extracted, and determine the target information corresponding to the first sound feature; the other sound features are the sound features in the sound data to be extracted except the first sound feature;
[0028] The processor is used for, if the first sound feature represented by the target information matches the reference sound data, forming the target sound data based on the first sound feature; if the first sound feature represented by the target information does not match the reference sound data, extracting, based on the second encoder, a second sound feature corresponding to the reference sound data from the first sound feature or the sound feature of the sound data to be extracted or the combination of the first sound feature and other sound features; forming the target sound data based on the second sound feature;
[0029] The resources occupied by the second encoder when it is working are higher than the resources occupied by the first encoder when it is working.
[0030] According to a third aspect of the present application, an embodiment of the present application provides a data extraction device, including:
[0031] An acquisition module, used to obtain the sound data to be extracted and the reference sound data;
[0032] A first extraction module is used to extract a first sound feature corresponding to the reference sound data and other sound features from the sound data to be extracted based on the first recognition model, and to determine target information corresponding to the first sound feature; the other sound features are sound features other than the first sound feature from the sound data to be extracted;
[0033] A forming module, configured to form target sound data based on the first sound feature if the first sound feature representing the target information matches the reference sound data;
[0034] The second extraction module is used to extract the second sound feature corresponding to the reference sound data from the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the second recognition model if the first sound feature representing the target information does not match the reference sound data; based on the second sound feature, form the target sound data; wherein the resources occupied by the second recognition model when working are higher than the resources occupied by the first recognition model when working.
[0035] Another aspect of an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned data extraction method.
[0036] Another aspect of the embodiments of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned audio information recognition method.
[0037] Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program, which implements the above-mentioned data extraction method when executed by a processor.
[0038] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A schematic diagram of a data extraction method in an embodiment of the present application;
[0040] Figure 2 A schematic diagram of another data extraction method in an embodiment of the present application;
[0041] Figure 3 A schematic diagram of another data extraction method in an embodiment of the present application;
[0042] Figure 4 This is a schematic diagram of the structure of a data extraction device in an embodiment of the present application;
[0043] Figure 5 A schematic diagram of the hardware structure of an electronic device in an embodiment of the present application;
[0044] Figure 6 Schematic diagram of the hardware structure of another electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0046] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments described are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.
[0047] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.
[0048] At present, speech separation technology uses the same recognition model to perform multiple downsampling and upsampling of mixed audio to separate the target speaker's speech for speech with different separation difficulties. Therefore, regardless of the difficulty of speech separation, a recognition model with a more complex algorithm and high resource consumption is used for speech separation, which easily causes unnecessary resource consumption for easier-to-separate speech.
[0049] To this end, the present application embodiment provides a data extraction method, such as Figure 1 As shown, including:
[0050] S101, obtaining sound data to be extracted and reference sound data.
[0051] In this embodiment, the sound data to be extracted may be mixed sound data including the sound data of the target object. For example, if the electronic device is a terminal device such as a mobile phone or a smart speaker, the electronic device may collect sound data outside the electronic device and use the collected sound data as the sound data to be extracted.
[0052] The reference sound data is the sound data of the target object used as a reference for speech separation. Accordingly, the goal of performing speech separation on the sound data to be extracted is to separate the sound data of the target object from the sound data to be extracted.
[0053] There are several possibilities for obtaining reference sound data:
[0054] For example, in a possible implementation, the reference sound data may be pre-stored in an electronic device. For example, taking a smart speaker or a terminal device with a voice recognition application installed as an example, the electronic device may prompt the user to record the voice of the target object and save it as the reference sound data.
[0055] In another possible implementation, for some special application scenarios, the target speaker whose corresponding audio needs to be separated may be temporarily determined, and on this basis, the sound data of the target object collected by the electronic device may be obtained as reference sound data.
[0056] It should be noted that the present application does not specifically limit the method for obtaining the reference sound data, that is, those skilled in the art can adjust its type according to actual conditions. The above situation is only an exemplary description of the methods in which the reference sound data in the present application can be obtained, but is not limited to the situations described in the above embodiments.
[0057] S102, extracting a first sound feature corresponding to the reference sound data and other sound features from the sound data to be extracted based on the first recognition model, and determining target information corresponding to the first sound feature; the other sound features are sound features other than the first sound feature from the sound data to be extracted.
[0058] In this embodiment, the first sound feature corresponding to the reference sound data may be a sound feature extracted from the sound data to be extracted by taking the sound feature included in the reference sound data as the reference sound feature. The first sound feature corresponding to the reference sound data may be a sound feature similar to the sound feature included in the reference sound data. For example, the first sound feature corresponding to the reference sound data may have a similarity of 80% to the sound feature included in the reference sound data; for another example, the first sound feature corresponding to the reference sound data may have a similarity of 90% to the sound feature included in the reference sound data.
[0059] In this embodiment, the first recognition model can use the sound features contained in the reference sound data as a reference to extract the first sound features corresponding to the reference sound data, as well as other sound features, from the sound data to be extracted, and can determine the target information corresponding to the first sound features. For example, the first recognition model can use the sound features contained in the reference sound data as a reference to extract the first sound features corresponding to the reference sound data, as well as other sound features, from the sound data to be extracted by performing processes such as multiple downsampling. Among them, the target information can characterize the matching degree information between the first sound feature and the reference sound data. The sound features, the first sound features or other sound features contained in the reference sound data can be represented as at least one of voiceprint features, pitch features, timbre features, frequency domain features (such as Mel frequency cepstral coefficients, MFCC; spectrum; spectrogram, etc.), time domain features, loudness features, rhythm features, and pitch features.
[0060] In some embodiments, the target information may be the first sound feature and similarity information with the sound feature included in the reference sound data.
[0061] In some embodiments, the target information may be information about the difference between the first sound feature and other sound features.
[0062] In this embodiment, the first recognition model may be a relatively simple sound feature extraction model compared to the second recognition model. It can also be understood that the computing resources occupied by the first recognition model when it is working are lower than the resources occupied by the second recognition model when it is working.
[0063] In some embodiments, when the first recognition model extracts sound features, a fuzzy algorithm can be used, that is, the algorithm is relatively simple and fuzzy matching is performed, so the amount of calculation is much smaller than precise calculation and precise matching, and the first recognition model occupies fewer resources when working.
[0064] In some embodiments, the first recognition model is a simple feature extraction model, that is, only one or two sound features in the sound data to be extracted are recognized, and then the recognition results of the one or two sound features are used as the first sound feature and other sound features corresponding to the reference sound data. Because only one or two sound features are recognized, the amount of calculation is relatively small.
[0065] It should be noted that, in other embodiments, the first recognition model may also use other algorithms, as long as the amount of calculation is smaller than the amount of calculation of performing only one recognition in the entire recognition process in the prior art, and this application does not impose any limitation.
[0066] In some embodiments, Figure 2As shown, the first recognition model may include a first feature extraction network and a second feature extraction network. The first feature extraction network may process the reference sound data to obtain the sound features contained in the reference sound data. The second feature extraction network may extract the first sound feature and other sound features from the sound data to be extracted with reference to the sound features contained in the reference sound data. And the similarity information between the first sound feature and the other sound features may be calculated to obtain the target information corresponding to the first sound feature. Or the difference information between the first sound feature and the sound features contained in the reference sound data may be calculated to obtain the target information corresponding to the first sound feature.
[0067] S103: If the first sound feature represented by the target information matches the reference sound data, target sound data is generated based on the first sound feature.
[0068] In some embodiments, if the target information includes the similarity between the first sound feature and the sound feature contained in the reference sound data, and the similarity is greater than or equal to a first preset threshold, then it can be said that the target information represents that the first sound feature matches the reference sound data; for example, if the similarity displayed in the target information is 98%, which exceeds the first preset threshold of 90%, then the target information represents that the first sound feature matches the reference sound data.
[0069] In some embodiments, if the target information includes the difference between the first sound feature and other sound features, and the difference is greater than or equal to a second preset threshold, it can be said that the target information represents that the first sound feature matches the reference sound data; for example, if the difference displayed in the target information is 90%, which exceeds the second preset threshold of 80%, then the target information represents that the first sound feature matches the reference sound data.
[0070] In some embodiments, if the target information includes the similarity between the first sound feature and the sound feature contained in the reference sound data, and the difference between the first sound feature and other sound features, and the similarity is greater than or equal to the first preset threshold, and the difference is greater than or equal to the second preset threshold, then it can be said that the target information represents that the first sound feature matches the reference sound data; for example, if the similarity displayed in the target information is 98%, which exceeds the first preset threshold of 90%; if the difference displayed in the target information is 90%, which exceeds the second preset threshold of 80%, then the target information represents that the first sound feature matches the reference sound data.
[0071] In this embodiment, if the target information characterizes that the first sound feature matches the reference sound data, it means that the sound feature that is relatively matched with the reference sound data can be extracted from the sound data to be extracted through the first recognition model, so that the target sound data can be directly formed through the first sound feature. For example, the sound data to be extracted is mixed sound data of users of different genders. Since the sound features of users of different genders are quite different, the sound features of users of different genders are better separated. The first recognition model can better extract the sound features of the target user from the mixed sound data based on the sound features contained in the reference sound data, so that the target sound data can be directly formed through the extracted sound features of the target user, thereby realizing the separation of the sound data of the target user from the sound data to be extracted.
[0072] In some embodiments, forming the target sound data based on the first sound feature may include: decoding the first sound feature to obtain the target sound data. Figure 2 As shown, a decoder is used to decode the first sound feature, such as performing multiple upsampling processes, to obtain target sound data.
[0073] S104, if the first sound feature represented by the target information does not match the reference sound data, extract the second sound feature corresponding to the reference sound data from the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the second recognition model; based on the second sound feature, form the target sound data; wherein the resources occupied by the second recognition model when working are higher than the resources occupied by the first recognition model when working.
[0074] In this embodiment, if the first sound feature represented by the target information does not match the reference sound data, it means that the first recognition model cannot extract the sound feature that matches the reference sound data well from the sound data to be extracted, so that the second recognition model can be used to continue processing and extract the second sound feature corresponding to the reference sound data. The second recognition model can be a relatively complex sound feature extraction model compared to the first recognition model, or in other words, the resources occupied by the second recognition model when working are larger than the resources occupied by the first recognition model when working. The resources occupied by the first recognition model and the second recognition model when working can be queried through the resource manager.
[0075] In some embodiments, when the second recognition model extracts sound features, it can use a precise algorithm, that is, the algorithm is relatively complex and performs precise matching, so the amount of calculation is much larger than fuzzy calculation and fuzzy matching. The resources occupied by the second recognition model when it is working are greater than the resources occupied by the first recognition model when it is working.
[0076] In some embodiments, the second recognition model is a complex recognition model, that is, the first recognition model can not only recognize one or two sound features that the first recognition model can recognize, but also recognize other sound features. For example, the second recognition model can recognize three, five, or even more sound features, or all sound features.
[0077] It should be noted that, in other embodiments, the second recognition model may also use other algorithms, which is not limited in this application.
[0078] In some embodiments, since the first sound feature is extracted with reference to the sound feature contained in the reference sound data, although the first sound feature does not match the reference sound data, the first sound feature still contains the sound feature of the target object. In order to reduce the computational complexity of the second recognition model, the second recognition model can process the first sound feature recognized by the first recognition model, and extract the second sound feature corresponding to the reference sound data from the first sound feature.
[0079] In some embodiments, since the first sound feature does not match the reference sound data, the first sound feature and other sound features may both contain the sound features of the target object. In order to more accurately extract the sound features of the target object and reduce the computational complexity of the second recognition model, all the sound features extracted by the first recognition model can be used as input to the second recognition model. That is, the second recognition model can process the sound features of the combination of the first sound feature and other sound features identified by the first recognition model, and extract the second sound feature corresponding to the reference sound data from the sound features of the combination of the first sound feature and other sound features.
[0080] In some embodiments, since the second recognition model is a complex feature extraction model, the feature extraction network is more complex, and the sound features that can be recognized may include the sound features that can be recognized by the first recognition model, so that the second recognition model can also directly recognize the second sound features corresponding to the reference sound data from the sound data to be extracted, that is, the second recognition model can also process the sound data to be extracted again, and extract the second sound features corresponding to the reference sound data from the sound data to be extracted.
[0081] In some embodiments, forming target sound data based on the second sound feature includes: decoding the second sound feature to obtain the target sound data. Figure 2 As shown, a decoder is used to decode the second sound feature, such as performing multiple upsampling processes, to obtain target sound data.
[0082] The data extraction method provided in the embodiment of the present application sets a first recognition model and a second recognition model with different resource occupancy, and makes the sound data to be extracted be first processed by the first recognition model with less resource occupancy. When the first recognition model cannot process well, the sound data is processed by the second recognition model with more resource occupancy. In this way, since the first recognition model is a simple sound feature extraction model and the second recognition model is a complex sound feature extraction model, the extraction of simple sound data can be completed by a simple sound feature extraction model, and the extraction of complex sound data can be completed by a complex sound feature extraction model, thereby making simple processing faster and complex processing more refined, effectively reducing the processing of simple problems and having more resources to process complex problems, reducing unnecessary resource consumption in the process of speech separation, improving performance, and effectively accelerating processing time and accuracy.
[0083] In an optional embodiment, the data extraction method includes:
[0084] If the similarity between the first sound feature represented by the target information and the sound feature contained in the reference sound data is greater than or equal to a first preset threshold, it is determined that the first sound feature represented by the target information matches the reference sound data; if the similarity between the first sound data represented by the target information and the sound feature contained in the reference sound data is less than the first preset threshold, it is determined that the first sound feature represented by the target information does not match the reference sound data.
[0085] In this embodiment, the target information may be the first sound feature and similarity information between the first sound feature and the sound feature included in the reference sound data. The similarity information may be, for example, a similarity probability.
[0086] In this embodiment, a first preset threshold value, such as 90%, can be preset. Then, the similarity between the first sound feature and the sound feature included in the reference sound data can be determined through the target information, and then compared with the first preset threshold value. If the similarity is greater than or equal to the first preset threshold value, it is determined that the first sound feature represented by the target information matches the reference sound data; if the similarity is less than the first preset threshold value, it is determined that the first sound feature represented by the target information does not match the reference sound data.
[0087] In this embodiment, by determining whether the similarity between the first sound feature and the sound feature included in the reference sound data is greater than or equal to a first preset threshold, it can be determined relatively simply and intuitively whether the first sound feature matches the reference sound data.
[0088] In an optional embodiment, the data extraction method includes:
[0089] If the difference between the first sound feature represented by the target information and the other sound features is greater than or equal to the second preset threshold, it is determined that the first sound feature represented by the target information matches the reference sound data; if the difference between the first sound feature represented by the target information and the other sound features is less than the second preset threshold, it is determined that the first sound feature represented by the target information does not match the reference sound data.
[0090] In this embodiment, the target information may be the first sound feature and the difference information between the first sound feature and other sound features. The difference information may be, for example, a difference probability.
[0091] In this embodiment, a second preset threshold value, such as 80%, can be preset. Then, the difference between the first sound feature and the other sound features can be determined through the target information, and then compared with the second preset threshold value. If the difference is greater than or equal to the second preset threshold value, it is determined that the first sound feature is greatly different from the other sound features, so that the sound feature matching the reference sound data can be extracted well, and the target information indicates that the first sound feature matches the reference sound data; if the difference is less than the second preset threshold value, it is determined that the difference between the first sound feature and the other sound features is small, so that the sound feature matching the reference sound data cannot be extracted well, and the target information indicates that the first sound feature does not match the reference sound data.
[0092] In this embodiment, by determining whether the phase difference between the first sound feature and other sound features is greater than or equal to a second preset threshold, the difference between multiple sound features corresponding to the sound data to be extracted can be directly determined, and then it can be determined whether the first sound feature matches the reference sound data.
[0093] In an optional embodiment, if Figure 3 As shown, in step S104, if the first sound feature represented by the target information does not match the reference sound data, extracting the second sound feature corresponding to the reference sound data from the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the second recognition model includes:
[0094] S1041, if the first sound feature represented by the target information does not match the reference sound data, determine a third sound feature corresponding to the reference sound data based on the second recognition model, wherein the third sound feature is a sound feature that satisfies a similarity condition with the sound feature included in the reference sound data, and the speech duration of the sound data corresponding to the third sound feature is greater than the speech duration corresponding to the reference sound data.
[0095] S1042, extracting the sound feature corresponding to the third sound feature from the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the second recognition model, and obtaining the second sound feature corresponding to the reference sound data from the first sound feature.
[0096] With respect to step S1041, in this embodiment, the speech duration corresponding to the reference sound data is relatively short, for example, 30 seconds, and fewer sound features can be extracted from the reference sound data, so that when the sound features contained in the reference sound data are used as a reference, the sound features of the target object in the sound data to be extracted may not be accurate enough; and the second recognition model is a complex sound feature extraction model, and its corresponding feature extraction network can be more complex, so that the third sound feature that meets the similarity condition with the sound feature contained in the reference sound data can be determined by the second recognition model. Among them, the speech duration of the sound data corresponding to the third sound feature can be much longer than the speech duration corresponding to the reference sound data, for example, the speech duration corresponding to the reference sound data can be 30 seconds, and the speech duration of the sound data corresponding to the third sound feature can be 2 hours. In this way, the third sound feature is similar to the sound feature contained in the reference sound data, and can include more sound features.
[0097] In some embodiments, Figure 2 As shown, the second recognition model may include a third feature extraction network and a fourth feature extraction network. When training the third feature extraction network and the fourth feature extraction network of the second recognition model, a large amount of sound data with a long speech duration may be used for training, for example, a large amount of sound data with a speech duration of 2 hours may be used for training, so that the second recognition model can extract sound features in more complex situations, such as between users of the same gender. Then, the third sound feature corresponding to the reference sound data is determined based on the trained second recognition model.
[0098] In some embodiments, determining a third sound feature corresponding to reference sound data based on a second recognition model includes: determining a sound feature contained in the reference sound data based on the second recognition model; and determining a third sound feature that satisfies a similarity condition with the sound feature contained in the reference sound data based on the second recognition model.
[0099] In some embodiments, Figure 2As shown, the sound features contained in the reference sound data can be extracted by the third feature extraction network of the second recognition model. In a specific implementation process, the sound features contained in the reference sound data extracted by the first feature extraction network of the first recognition model can be input into the third feature extraction network of the second recognition model to obtain the sound features contained in the reference sound data; the reference sound data can also be directly input into the second recognition model, and the sound features contained in the reference sound data can be extracted by the third feature extraction network of the second recognition model.
[0100] In this embodiment, after the sound features included in the reference sound data are determined, a third sound feature that meets a similarity condition with the sound features included in the reference sound data can be acquired through a second recognition model based on the sound features included in the reference sound data.
[0101] In this embodiment, by first determining the sound features included in the reference sound data, and then determining the third sound features that meet the similarity condition based on the sound features included in the reference sound data, the third sound features corresponding to the reference sound data can be accurately and quickly determined.
[0102] In some embodiments, determining a third sound feature that satisfies a similarity condition with a sound feature included in reference sound data based on the second recognition model includes: searching the second recognition model for a third sound feature that satisfies a similarity condition with a sound feature included in the reference sound data.
[0103] In this embodiment, since the second recognition model is a complex sound feature extraction model, a large number of sound features can be stored in the second recognition model, and the speech duration of the speech data corresponding to the sound features is longer than the speech duration corresponding to the reference sound data. Figure 2 As shown, during the training of the second recognition model, the second recognition model can store a large number of sound features. Therefore, after the second recognition model determines the sound features included in the reference sound data, the third sound features that meet the similarity condition with the sound features included in the reference sound data can be searched from the second recognition model.
[0104] In this embodiment, by presetting a large number of sound features in the second recognition model, the third sound features that meet the similarity condition with the sound features included in the reference sound data can be quickly determined by referring to the sound features included in the reference sound data.
[0105] For step S1042, after the third sound feature corresponding to the reference sound data is determined, the third sound feature can be used as a reference to extract the sound feature in the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features to obtain the second sound feature in the first sound feature corresponding to the reference sound data.
[0106] In some embodiments, Figure 2 As shown, the fourth feature extraction network of the second recognition model can extract the sound feature corresponding to the third sound feature from the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the third sound feature.
[0107] In this embodiment, by determining the third sound feature corresponding to the reference sound data, since the speech duration of the sound data corresponding to the third sound feature is greater than the speech duration corresponding to the reference sound data, the third sound feature can have more detailed information. Based on the third sound feature, the second sound feature corresponding to the reference sound data is extracted from the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features. This can make the second sound feature more consistent with the sound feature of the target object, and the accuracy of sound feature extraction is greatly improved.
[0108] In an optional embodiment, in step S104, extracting the second sound feature corresponding to the reference sound data from the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the second recognition model includes:
[0109] At at least a portion of the processing moments when the second recognition model processes the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features, the sound feature extracted at the previous processing moment is used as the reference sound feature, and the second recognition model is used to extract the sound feature of the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features; based on the sound features extracted at each processing moment, a second sound feature corresponding to the reference sound data is formed in the sound feature of the first sound feature or the sound feature of the combination of the first sound feature and other sound features.
[0110] In specific implementation, the second feature extraction network of the second recognition model can be an attention network, which can extract the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the sound features contained in the reference sound data or the third sound feature corresponding to the reference sound data through the attention network.
[0111] In this embodiment, at least a portion of the processing moments when the second recognition model processes the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features, the sound feature extracted at the previous processing moment is used as the reference sound feature, and the second recognition model is used to extract the sound feature of the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features, so that at at least a portion of the processing moments, the attention mechanism is used to extract the sound features, so that the extracted sound features are more accurate.
[0112] In an optional embodiment, the energy consumption when the second recognition model is working is higher than the energy consumption when the first recognition model is working.
[0113] In this embodiment, the resources and energy consumption occupied by the second recognition model when it is working are higher than the resources and energy consumption occupied by the first recognition model when it is working.
[0114] In this embodiment, since the energy consumption of the second recognition model is higher than that of the first recognition model, the extraction of simple sound data can be achieved through the first recognition model with lower energy consumption, and the extraction of complex sound data can be achieved through the second recognition model with higher energy consumption. This can effectively reduce the energy consumption when processing simple problems and have more energy to process complex problems, thereby reducing unnecessary energy consumption in the speech separation process and improving performance.
[0115] The present application also provides a data extraction device, such as Figure 4 As shown, including:
[0116] An acquisition module 41 is used to obtain the sound data to be extracted and the reference sound data;
[0117] A first extraction module 42 is used to extract a first sound feature corresponding to the reference sound data and other sound features from the sound data to be extracted based on the first recognition model, and to determine target information corresponding to the first sound feature; the other sound features are sound features other than the first sound feature from the sound data to be extracted;
[0118] A forming module 43, configured to form target sound data based on the first sound feature if the first sound feature representing the target information matches the reference sound data;
[0119] The second extraction module 44 is used to extract the second sound feature corresponding to the reference sound data from the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and other sound features based on the second recognition model if the first sound feature representing the target information does not match the reference sound data; based on the second sound feature, form the target sound data; wherein the resources occupied by the second recognition model when working are higher than the resources occupied by the first recognition model when working.
[0120] The data extraction device provided in the embodiment of the present application sets a first recognition model and a second recognition model with different resource occupancy, and makes the sound data to be extracted be first processed by the first recognition model with less resource occupancy. When the first recognition model cannot process well, the sound data is processed by the second recognition model with more resource occupancy. In this way, since the first recognition model is a simple sound feature extraction model and the second recognition model is a complex sound feature extraction model, the extraction of simple sound data can be completed by a simple sound feature extraction model, and the extraction of complex sound data can be completed by a complex sound feature extraction model, thereby making simple processing faster and complex processing more refined, effectively reducing the processing of simple problems and having more resources to process complex problems, reducing unnecessary resource consumption in the process of speech separation, improving performance, and effectively accelerating processing time and accuracy.
[0121] The present application also provides an electronic device, such as Figure 5 As shown, including:
[0122] A first encoder 51 , a second encoder 52 and a processor 53 .
[0123] The first encoder 51 is used to obtain the sound data to be extracted and the reference sound data; extract the first sound feature corresponding to the reference sound data in the sound data to be extracted, and other sound features, and determine the target information corresponding to the first sound feature; the other sound features are the sound features in the sound data to be extracted except the first sound feature.
[0124] The processor 53 is used to form target sound data based on the first sound feature if the first sound feature represented by the target information matches the reference sound data; if the first sound feature represented by the target information does not match the reference sound data, extract the second sound feature corresponding to the reference sound data from the first sound feature or the sound feature of the sound data to be extracted or the combination of the first sound feature and other sound features based on the second encoder 52; and form the target sound data based on the second sound feature.
[0125] The resources occupied by the second encoder 52 when in operation are higher than the resources occupied by the first encoder 51 when in operation.
[0126] In some embodiments, Figure 6 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0127] like Figure 6 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0128] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0129] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Various computing units running machine learning model algorithms include, but are not limited to, a first encoder and a second encoder. The computing unit 801 performs the various methods and processes described above, such as a data extraction method. For example, in some embodiments, the data extraction method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data extraction method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the data extraction method in any other appropriate manner (eg, by means of firmware).
[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0131] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0132] The present application also provides a computer-readable storage medium. In the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the above. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0134] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0135] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0137] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0138] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A data extraction method, comprising: Obtaining sound data to be extracted and reference sound data; Extracting a first sound feature and other sound features corresponding to the reference sound data from the sound data to be extracted based on the first recognition model, and determining target information corresponding to the first sound feature; the other sound features are sound features other than the first sound feature from the sound data to be extracted; If the target information indicates that the first sound feature matches the reference sound data, forming target sound data based on the first sound feature; If the target information represents that the first sound feature does not match the reference sound data, extracting, based on a second recognition model, a second sound feature corresponding to the reference sound data from the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and the other sound features; Based on the second sound feature, forming target sound data; The resources occupied by the second recognition model when it is working are higher than the resources occupied by the first recognition model when it is working.
2. The data extraction method according to claim 1, comprising: If the similarity between the first sound feature represented by the target information and the sound feature included in the reference sound data is greater than or equal to a first preset threshold, determining that the first sound feature represented by the target information matches the reference sound data; If the similarity between the sound feature included in the first sound data represented by the target information and the reference sound data is less than a first preset threshold, it is determined that the first sound feature represented by the target information does not match the reference sound data.
3. The data extraction method according to claim 1, comprising: If the difference between the first sound feature represented by the target information and the other sound features is greater than or equal to a second preset threshold, determining that the first sound feature represented by the target information matches the reference sound data; If the difference between the first sound feature represented by the target information and the other sound features is less than a second preset threshold, it is determined that the first sound feature represented by the target information does not match the reference sound data.
4. The data extraction method according to claim 1, extracting the second sound feature corresponding to the reference sound data from the first sound feature or the sound feature of the sound data to be extracted or the sound feature of the combination of the first sound feature and the other sound features based on the second recognition model, comprises: Determine, based on the second recognition model, a third sound feature corresponding to the reference sound data, wherein the third sound feature is a sound feature that satisfies a similarity condition with a sound feature included in the reference sound data, and the speech duration of the sound data corresponding to the third sound feature is greater than the speech duration corresponding to the reference sound data; The sound feature corresponding to the third sound feature is extracted from the sound feature of the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and the other sound features based on the second recognition model, and the second sound feature corresponding to the reference sound data is obtained from the first sound feature.
5. The data extraction method according to any one of claims 1 to 4, extracting, based on a second recognition model, a second sound feature corresponding to the reference sound data from the first sound feature or the sound data to be extracted or the sound feature of a combination of the first sound feature and the other sound features, comprising: At least a portion of the processing moment when the second recognition model processes the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and the other sound features, the sound feature extracted at the previous processing moment is used as the reference sound feature, and the second recognition model is used to extract the sound feature of the first sound feature or the sound data to be extracted or the sound feature of the combination of the first sound feature and the other sound features; Based on the sound feature extracted at each processing time, a second sound feature corresponding to the reference sound data is formed among the first sound feature or the sound feature combined with the first sound feature and the other sound features.
6. The data extraction method according to claim 4, determining the third sound feature corresponding to the reference sound data based on the second recognition model, comprising: Determining sound features included in the reference sound data based on a second recognition model; A third sound feature that satisfies a similarity condition with the sound feature included in the reference sound data is determined based on the second recognition model.
7. The data extraction method according to claim 6, determining, based on the second recognition model, a third sound feature that satisfies a similarity condition with the sound feature included in the reference sound data, comprises: A third sound feature that satisfies a similarity condition with the sound feature included in the reference sound data is searched from the second recognition model.
8. The data extraction method according to claim 1, The energy consumption of the second recognition model when it is working is higher than the energy consumption of the first recognition model when it is working.
9. An electronic device, comprising: A first encoder, a second encoder and a processor; The first encoder is used to obtain the sound data to be extracted and the reference sound data; Extracting a first sound feature and other sound features corresponding to the reference sound data from the sound data to be extracted, and determining target information corresponding to the first sound feature; the other sound features are sound features other than the first sound feature from the sound data to be extracted; The processor is configured to form target sound data based on the first sound feature if the target information represents that the first sound feature matches the reference sound data; If the target information represents that the first sound feature does not match the reference sound data, extracting, based on a second encoder, a second sound feature corresponding to the reference sound data from the first sound feature or the sound features of the sound data to be extracted or the combination of the first sound feature and the other sound features; Based on the second sound feature, forming target sound data; The resources occupied by the second encoder when in operation are higher than the resources occupied by the first encoder when in operation.
10. A data extraction device, comprising: An acquisition module, used to obtain the sound data to be extracted and the reference sound data; A first extraction module is used to extract a first sound feature and other sound features corresponding to the reference sound data from the sound data to be extracted based on a first recognition model, and determine target information corresponding to the first sound feature; the other sound features are sound features other than the first sound feature from the sound data to be extracted; a forming module, configured to form target sound data based on the first sound feature if the target information represents that the first sound feature matches the reference sound data; A second extraction module is used to extract, based on a second recognition model, a second sound feature corresponding to the reference sound data from the first sound feature or the sound features of the sound data to be extracted or the combination of the first sound feature and the other sound features if the target information characterizes that the first sound feature does not match the reference sound data; Based on the second sound feature, target sound data is formed; wherein the resources occupied by the second recognition model when working are higher than the resources occupied by the first recognition model when working.