Audio analysis method and device, storage medium, electronic equipment and product
By extracting and concatenating audio features and speech-text features using multiple audio encoders, this approach addresses the inefficiency of existing large audio models when processing different types of audio tasks, achieving more efficient audio parsing.
Patent Information
- Application Number
- CN202511704190.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-03-03
AI Technical Summary
Existing large audio models use a single audio encoder, which means that only a single audio information can be extracted when processing different types of audio tasks, resulting in low processing efficiency in sub-speech information recognition and non-human voice tasks.
Multiple audio encoders are used to extract various audio features. The feature dimensions are unified to the target dimension of speech and text features through a linear layer. The multiple audio features are concatenated with the speech and text features and then input into a large language model for parsing.
It improves the processing efficiency and recognition accuracy of large audio models when handling different types of audio tasks, especially significantly improving performance in subspeech information recognition and non-human voice tasks.
Smart Images
Figure CN121600903A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computers, and more specifically, to an audio parsing method and apparatus, storage medium, electronic device and product. Background Technology
[0002] Most existing large-scale audio models, such as Whisper, Qwen-Audio, and LTU-AS, use a single audio encoder as the basic component for processing audio signals. The design of a single audio encoder means that it can only extract specific audio information when processing different types of audio tasks. When it comes to paraspeech information recognition (such as the speaker's gender and age) and non-human voice tasks (such as environmental sound classification and music analysis), its processing efficiency is low and its recognition accuracy is not high.
[0003] In related technologies, large audio models typically use a single audio encoder, which can only extract single audio information when processing different types of audio tasks, resulting in low processing efficiency in secondary speech information recognition and non-human voice tasks. No effective solution has yet been proposed. Summary of the Invention
[0004] This application provides an audio parsing method and apparatus, storage medium, electronic device and product to at least solve the problem in the related art that large audio models usually use a single audio encoder, which can only extract single audio information when processing different types of audio tasks, resulting in low processing efficiency in sub-speech information recognition and non-human voice tasks.
[0005] According to one embodiment of this application, an audio parsing method is provided, comprising: based on a first audio emitted by a first target object and a first question, obtaining multiple audio features in the first audio through an audio encoder, and extracting speech-text features in the first question through a text encoder, wherein the multiple audio features include at least one of the following: speech content features corresponding to the first audio, voiceprint features of a second target object in the first audio, and non-human voice features in the first audio; unifying the feature dimensions of the multiple audio features to the target feature dimension of the speech-text features through a linear layer, and concatenating the multiple audio features of the target feature dimension to obtain fused audio features; concatenating the fused audio features with the speech-text features to obtain input features of a first large language model; and inputting the input features into the first large language model to obtain the speech parsing result corresponding to the first audio.
[0006] In an optional embodiment, unifying the feature dimensions of the multiple audio features to the target feature dimension of the speech-text feature through a linear layer includes: determining the target feature dimension and setting corresponding first linear layers for each of the multiple audio features to obtain the linear layers, wherein the input dimension of the first linear layer is consistent with the feature dimension of the first audio feature corresponding to the first linear layer, and the output dimension of the first linear layer is consistent with the target feature dimension, and the first audio feature is the speech content feature, the voiceprint feature, or the non-human voice feature; for any second audio feature among the multiple audio features, the second audio feature is input to a second linear layer to adjust the feature dimension of the second audio feature to the target feature dimension through the second linear layer, wherein the multiple first linear layers include the second linear layer.
[0007] In an optional embodiment, before acquiring multiple audio features in the first audio based on the first audio and the first question emitted by the first target object through an audio encoder, the method further includes: collecting a first voice command emitted by the first target object, wherein the first voice command is used to train a first large language model; generating question-and-answer data corresponding to the first voice command, wherein the question-and-answer data includes: first question-and-answer data of a first type and second question-and-answer data of a second type, the first question-and-answer data including: a second question generated by a second large language model and a fixed answer corresponding to the second question, the second question-and-answer data including the second question and a first answer to the second question, the first answer being generated by the second large language model; and training the first large language model using the first voice command and the question-and-answer data.
[0008] In one optional embodiment, based on a first audio message and a first question emitted by a first target object, multiple audio features in the first audio message are obtained through an audio encoder, including: preprocessing the first audio message to obtain a second audio message; extracting content information of the second audio message through a first audio encoder to obtain the speech content features; extracting voiceprint information of the second target object through a second audio encoder to obtain the voiceprint features; and extracting non-human voice audio information in the second audio message through a third audio encoder to obtain the non-human voice features, wherein the audio encoder includes the first audio encoder, the second audio encoder, and the third audio encoder.
[0009] In one optional embodiment, preprocessing the first audio to obtain the second audio includes: performing noise reduction processing on the first audio to obtain the third audio; determining whether the first energy of multiple audio segments in the third audio is lower than a preset energy; if it is determined that there is a second energy among the multiple first energies that is lower than the preset energy, determining the first audio segment corresponding to the second energy, wherein the multiple audio segments correspond one-to-one with the multiple first energies; deleting the first audio segment from the third audio to obtain the second audio.
[0010] In an optional embodiment, extracting speech-text features from the first question using a text encoder includes: obtaining first text content of the first question; determining whether there are consecutive repeating fields among multiple fields of the first text content; deleting either the first field or the second field if the first field and the second field are consecutive and identical, to obtain second speech content, wherein the multiple fields include the first field and the second field; and inputting the second speech content into the text encoder to obtain the speech-text features.
[0011] According to another aspect of the present invention, an audio parsing apparatus is also provided, comprising: an extraction module, configured to, based on a first audio emitted by a first target object and a first question, acquire multiple audio features in the first audio through an audio encoder, and extract speech-text features in the first question through a text encoder, wherein the multiple audio features include at least one of the following: speech content features corresponding to the first audio, voiceprint features of a second target object in the first audio, and non-human voice features in the first audio; a unification module, configured to, through a linear layer, unify the feature dimensions of the multiple audio features to the target feature dimension of the speech-text features, and concatenate the multiple audio features of the target feature dimension to obtain fused audio features; a concatenation module, configured to, concatenate the fused audio features with the speech-text features to obtain input features of a first large language model; and a parsing module, configured to, input the input features into the first large language model to obtain a speech parsing result corresponding to the first audio.
[0012] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described audio parsing method when running.
[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the audio parsing method through the computer program.
[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described audio parsing method.
[0015] In this embodiment, based on a first audio message and a first question emitted by a first target object, multiple audio features in the first audio message are obtained through an audio encoder, and speech-text features in the first question are extracted through a text encoder. The multiple audio features include at least one of the following: speech content features corresponding to the first audio message, voiceprint features of the second target object in the first audio message, and non-human voice features in the first audio message. A linear layer unifies the feature dimensions of the multiple audio features to the target feature dimension of the speech-text features, and the multiple audio features of the target feature dimension are concatenated to obtain fused audio features. The fused audio features are concatenated with the speech-text features to obtain the input features of the first large language model. The input features are input into the first large language model to obtain the speech parsing result corresponding to the first audio message. This technical solution solves the problem in related technologies where large audio models typically use a single audio encoder, which can only extract single audio information when processing different types of audio tasks, resulting in low processing efficiency in secondary speech information recognition and non-human voice tasks. Attached Figure Description
[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a schematic diagram of the hardware environment for an audio parsing method according to an embodiment of this application;
[0019] Figure 2 This is a flowchart of an audio parsing method according to an embodiment of this application;
[0020] Figure 3 This is a schematic diagram of an audio parsing method according to an optional embodiment of this application;
[0021] Figure 4 This is a structural block diagram of an audio parsing device according to an embodiment of this application. Detailed Implementation
[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0024] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0025] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.
[0026] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for an audio parsing method according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a central processing unit (CPU) or a field-programmable gate array (FPGA)) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0027] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the audio parsing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0028] The computer terminal uses a wireless network provided by a communications provider. In one example, transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.
[0029] This embodiment provides an audio parsing method, applied to the aforementioned computer terminal. Figure 2 This is a flowchart of an audio parsing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:
[0030] Step S202: Based on the first audio and the first question emitted by the first target object, multiple audio features in the first audio are obtained by an audio encoder, and speech text features in the first question are extracted by a text encoder. The multiple audio features include at least one of the following: speech content features corresponding to the first audio, voiceprint features of the second target object in the first audio, and non-human voice features in the first audio.
[0031] Step S204: The feature dimensions of the various audio features are unified to the target feature dimension of the speech-text features through a linear layer, and the various audio features of the target feature dimension are concatenated to obtain the fused audio features.
[0032] Step S206: Concatenate the fused audio features with the speech-text features to obtain the input features of the first language model;
[0033] Step S208: Input the input features into the first large language model to obtain the speech parsing result corresponding to the first audio.
[0034] In this embodiment, based on a first audio message and a first question emitted by a first target object, multiple audio features in the first audio message are obtained through an audio encoder, and speech-text features in the first question are extracted through a text encoder. The multiple audio features include at least one of the following: speech content features corresponding to the first audio message, voiceprint features of the second target object in the first audio message, and non-human voice features in the first audio message. A linear layer unifies the feature dimensions of the multiple audio features to the target feature dimension of the speech-text features, and the multiple audio features of the target feature dimension are concatenated to obtain fused audio features. The fused audio features are concatenated with the speech-text features to obtain the input features of the first large language model. The input features are input into the first large language model to obtain the speech parsing result corresponding to the first audio message. This technical solution solves the problem in related technologies where large audio models typically use a single audio encoder, which can only extract single audio information when processing different types of audio tasks, resulting in low processing efficiency in secondary speech information recognition and non-human voice tasks.
[0035] like Figure 3As shown, an audio encoder performs deep analysis on the first audio segment (the input audio segment in the diagram) to obtain various audio features. Specifically, the first audio encoder (equivalent to the Whisper encoder in the diagram) extracts speech content features from the first audio; the second audio encoder (equivalent to the voiceprint encoder in the diagram) extracts the voiceprint features of the second target object from the first audio, including the speaker's gender, age, and personal timbre; and the third audio encoder (equivalent to the BEATs encoder in the diagram) identifies and extracts non-human voice features from the first audio, covering non-verbal audio elements such as background music and environmental noise. Next, a linear layer (equivalent to the mapping layer in the diagram) unifies the dimensions of the various audio features to the target feature dimension that matches the text features. The various audio features unified to the target feature dimension are then concatenated to generate fused audio features. This fusion operation can be performed in various ways, such as concatenation or weighted averaging, with the goal of integrating different types of audio information into a comprehensive vector to fully reflect the multidimensional characteristics of the instruction. In parallel, a text encoder (equivalent to the text segmenter and text embedding in the diagram) extracts speech-text features from the first problem. Finally, the audio features and speech-text features are concatenated to form the input features for the first language model. The constructed input features are then fed into the first language model for deep processing. The model will predict and parse the first audio based on the input features, ultimately generating the speech parsing result, such as... Figure 3 The output results.
[0036] In an optional embodiment, unifying the feature dimensions of the multiple audio features to the target feature dimension of the speech-text feature through a linear layer includes: determining the target feature dimension and setting corresponding first linear layers for each of the multiple audio features to obtain the linear layers, wherein the input dimension of the first linear layer is consistent with the feature dimension of the first audio feature corresponding to the first linear layer, and the output dimension of the first linear layer is consistent with the target feature dimension, and the first audio feature is the speech content feature, the voiceprint feature, or the non-human voice feature; for any second audio feature among the multiple audio features, the second audio feature is input to a second linear layer to adjust the feature dimension of the second audio feature to the target feature dimension through the second linear layer, wherein the multiple first linear layers include the second linear layer.
[0037] Optionally, firstly, the target feature dimension D of the speech-text features output by the text encoder needs to be defined. Assume the target feature dimension D is 1024. For each audio feature extractor (such as Whisper Encoder, speaker encoder, and beats encoder), a linear layer is created to adjust the feature dimension. The input dimension of these linear layers should match the original dimension of the corresponding audio features, and the output dimension is set to D (i.e., 1024). The feature vectors of various audio features are input into their respective linear layers. Assume the feature dimension of the speech content features is (1, 512), the feature dimension of the speaker features is (1, 256), and the feature dimension of the non-human voice features is (1, 1024). A linear layer (Linear(512, 1024)) is created for the speech content features, and a linear layer (Linear(256, 1024)) is created for the voiceprint features. Since the original dimension of the non-human voice features is 1024, they can be used directly or a linear layer (Linear(1024, 1024)) can be used for possible weight adjustments or transformations, but usually, direct use is sufficient. After the linear layer mapping, the dimension of each audio feature is adjusted to D (1024 in this example), allowing it to be concatenated with the text feature vector in the same dimensional space. After all the audio features are adjusted to match the target feature dimension of the speech-text features, the feature fusion operation can be performed. The audio features are combined with the speech-text features using the Concatenate or Add operation.
[0038] In an optional embodiment, before acquiring multiple audio features in the first audio based on the first audio and the first question emitted by the first target object through an audio encoder, the method further includes: collecting a first voice command emitted by the first target object, wherein the first voice command is used to train a first large language model; generating question-and-answer data corresponding to the first voice command, wherein the question-and-answer data includes: first question-and-answer data of a first type and second question-and-answer data of a second type, the first question-and-answer data including: a second question generated by a second large language model and a fixed answer corresponding to the second question, the second question-and-answer data including the second question and a first answer to the second question, the first answer being generated by the second large language model; and training the first large language model using the first voice command and the question-and-answer data.
[0039] A key step in designing and optimizing large audio models for specific target audiences is to customize the model's training to enhance its ability to understand and respond to specific types of speech commands. This process involves collecting speech command samples from the first target audience, creating a related question-and-answer dataset, and then using this data to fine-tune the model. First, a series of actual speech commands issued by the first target audience are collected. These commands form the data foundation for training and optimizing the model, ensuring that the model learns speech patterns and needs specific to the first target audience. After obtaining the speech commands, two types of question-and-answer datasets (first question-and-answer data and second question-and-answer data) are constructed to meet different training needs. The first question-and-answer data uses a pre-defined question template, from which a second large language model generates diverse questions. While the question format varies, the answer remains fixed, meaning that regardless of the question's variation, the model should be able to arrive at the same definitive answer. This dataset is primarily used to reinforce the model's ability to consistently provide correct responses to different expressions of the same command. Unlike the first question-and-answer data, each question in the second question-and-answer data is dynamically generated by the second large language model, and the answer to each question varies depending on the specific content of the question. This question-and-answer pairing is more flexible and realistic, aiming to teach the model to understand the true intent of a question and generate personalized answers closely related to it. The first large language model was trained using collected first-class voice commands and two types of question-and-answer data. This training process not only included the recognition and understanding of the voice commands themselves, but more importantly, the model learned how to generate appropriate responses based on the content of the question.
[0040] In one optional embodiment, based on a first audio message and a first question emitted by a first target object, multiple audio features in the first audio message are obtained through an audio encoder, including: preprocessing the first audio message to obtain a second audio message; extracting content information of the second audio message through a first audio encoder to obtain the speech content features; extracting voiceprint information of the second target object through a second audio encoder to obtain the voiceprint features; and extracting non-human voice audio information in the second audio message through a third audio encoder to obtain the non-human voice features, wherein the audio encoder includes the first audio encoder, the second audio encoder, and the third audio encoder.
[0041] In the training and optimization of large audio models, deep preprocessing and feature extraction of the input audio commands are crucial steps to improve the model's understanding and response accuracy. First, the collected first audio is preprocessed to remove noise and silence, standardize the audio format and sampling rate, and perform other necessary audio signal optimizations to ensure high-quality audio for the model's input. The preprocessed audio is labeled as the second audio. A specially designed first audio encoder (such as Whisper-large-v1) processes the third speech command, extracting semantic and textual content information from the second audio. This encoder focuses on identifying linguistic components in the audio and converting them into understandable text, thus obtaining speech content features. The second audio is then analyzed using a second audio encoder (such as a ResNet34-based voiceprint model) to obtain unique voiceprint information of the second target. Voiceprint features include the speaker's personal attributes, such as gender, age, and emotional state, which are crucial for identifying and verifying the speaker's identity. A third audio encoder (such as the BEATs model) is used to parse non-human audio components in the second audio, such as background music and ambient sounds. This encoder can identify and distinguish non-human voice elements, extract their features, and provide the model with a more comprehensive audio understanding capability.
[0042] Through the above process, three different dimensions of audio features are extracted from a single audio instruction: speech content features, voiceprint features, and non-human voice features. The fusion of these features enables the final large-scale audio model to comprehensively understand and respond to complex and diverse audio environments, achieving a more refined and accurate level in terms of language content comprehension, voice identity recognition, and audio scene perception. This parallel processing and feature fusion strategy using multiple encoders effectively overcomes the limitations of a single encoder in handling multiple types of audio tasks, significantly improving the generalization ability and application value of the large-scale audio model.
[0043] In one optional embodiment, preprocessing the first audio to obtain the second audio includes: performing noise reduction processing on the first audio to obtain the third audio; determining whether the first energy of multiple audio segments in the third audio is lower than a preset energy; if it is determined that there is a second energy among the multiple first energies that is lower than the preset energy, determining the first audio segment corresponding to the second energy, wherein the multiple audio segments correspond one-to-one with the multiple first energies; deleting the first audio segment from the third audio to obtain the second audio.
[0044] Before processing audio instructions to input highly complex audio into large-scale audio understanding models, denoising and silence detection and removal in the preprocessing stage are crucial steps to ensure audio quality and improve the model's analytical capabilities. First, the original first audio is denoised to remove environmental noise, background interference, and other interfering factors, resulting in a clearer third audio. The denoising technique aims to preserve the core components of the speech signal while reducing unnecessary noise and improving the signal-to-noise ratio. Further, by analyzing the energy levels of each audio segment in the third audio, the presence of low-energy silence segments is determined. The energy of each audio segment is related to its amplitude and duration; low energy typically indicates little or no sound. A preset energy level is used to distinguish between silent and non-silent segments. Once an audio segment is detected with energy below the preset energy level, it is identified as a silence segment and removed from the third audio, resulting in a second audio containing only valid speech signals.
[0045] In an optional embodiment, extracting speech-text features from the first question using a text encoder includes: obtaining first text content of the first question; determining whether there are consecutive repeating fields among multiple fields of the first text content; deleting either the first field or the second field if the first field and the second field are consecutive and identical, to obtain second speech content, wherein the multiple fields include the first field and the second field; and inputting the second speech content into the text encoder to obtain the speech-text features.
[0046] After obtaining the first text content of the first question, the preprocessing module checks for consecutive duplicate fields. Duplicate fields refer to adjacent text paragraphs or sentences in the first question that are identical. If a first field and a second field are detected to be adjacent and identical, the preprocessing module will delete one of the fields (either the first or the second field). This deduplication operation not only reduces redundancy in the model input but also optimizes the data flow, avoiding repeated processing of the same information, thereby improving the model's processing speed and resource utilization. The processed speech content, i.e., the second speech content, is fed into the text encoder. The text encoder (such as the text encoding part of LLaMA) transforms this speech content into a series of text feature vectors. This transformation process is based on a deep learning model with a Transformer architecture, which can capture semantic and contextual information in the speech content, preparing it for the model's subsequent understanding and generation work.
[0047] The above steps ensure that the question input to the text encoder is free of redundancy and repetitive information. This helps the large audio model understand and process the first question more efficiently and accurately. Removing duplicate fields is particularly important when dealing with long speech problems or continuous speech streams. It not only reduces unnecessary computation but also provides the model with clearer and more focused input, thereby improving user experience while enhancing model performance and response speed.
[0048] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the above audio parsing method, the process is described below with reference to embodiments, but this is not intended to limit the technical solutions of the embodiments of this application. Specifically:
[0049] Figure 4 This is a structural block diagram of an audio parsing device according to an embodiment of this application; as shown below. Figure 4 As shown, it includes:
[0050] The extraction module 42 is used to obtain multiple audio features in the first audio based on the first audio emitted by the first target object and the first question, and to extract speech text features in the first question through an audio encoder, wherein the multiple audio features include at least one of the following: speech content features corresponding to the first audio, voiceprint features of the second target object in the first audio, and non-human voice features in the first audio.
[0051] The unification module 44 is used to unify the feature dimensions of the multiple audio features to the target feature dimension of the speech-text features through a linear layer, and to concatenate the multiple audio features of the target feature dimension to obtain the fused audio features.
[0052] The splicing module 46 is used to splice the fused audio features with the speech-text features to obtain the input features of the first language model;
[0053] The parsing module 48 is used to input the input features into the first large language model to obtain the speech parsing result corresponding to the first audio.
[0054] In this embodiment, based on a first audio message and a first question emitted by a first target object, multiple audio features in the first audio message are obtained through an audio encoder, and speech-text features in the first question are extracted through a text encoder. The multiple audio features include at least one of the following: speech content features corresponding to the first audio message, voiceprint features of the second target object in the first audio message, and non-human voice features in the first audio message. A linear layer unifies the feature dimensions of the multiple audio features to the target feature dimension of the speech-text features, and the multiple audio features of the target feature dimension are concatenated to obtain fused audio features. The fused audio features are concatenated with the speech-text features to obtain the input features of the first large language model. The input features are input into the first large language model to obtain the speech parsing result corresponding to the first audio message. This technical solution solves the problem in related technologies where large audio models typically use a single audio encoder, which can only extract single audio information when processing different types of audio tasks, resulting in low processing efficiency in secondary speech information recognition and non-human voice tasks.
[0055] In an optional embodiment, the unification module 44 is further configured to determine the target feature dimension and set corresponding first linear layers for each of the multiple audio features to obtain the linear layers, wherein the input dimension of the first linear layer is consistent with the feature dimension of the first audio feature corresponding to the first linear layer, and the output dimension of the first linear layer is consistent with the target feature dimension, wherein the first audio feature is the speech content feature, the voiceprint feature, or the non-human voice feature; for any second audio feature among the multiple audio features, the second audio feature is input to the second linear layer so that the feature dimension of the second audio feature is adjusted to the target feature dimension through the second linear layer, wherein the multiple first linear layers include the second linear layer.
[0056] In an optional embodiment, the extraction module 42 is further configured to collect a first voice command issued by the first target object, wherein the first voice command is used to train the first large language model; generate question-and-answer data corresponding to the first voice command, wherein the question-and-answer data includes: a first type of first question-and-answer data and a second type of second question-and-answer data, the first question-and-answer data including: a second question generated by the second large language model and a fixed answer corresponding to the second question, the second question-and-answer data including the second question and a first answer to the second question, the first answer being generated by the second large language model; and train the first large language model using the first voice command and the question-and-answer data.
[0057] In an optional embodiment, the extraction module 42 is further configured to preprocess the first audio to obtain a second audio; extract the content information of the second audio through a first audio encoder to obtain the speech content features; extract the voiceprint information of the second target object through a second audio encoder to obtain the voiceprint features; and extract the non-human voice audio information in the second audio through a third audio encoder to obtain the non-human voice features, wherein the audio encoder includes the first audio encoder, the second audio encoder, and the third audio encoder.
[0058] In an optional embodiment, the extraction module 42 is further configured to perform noise reduction processing on the first audio to obtain a third audio; determine whether the first energy of multiple audio segments in the third audio is lower than a preset energy; if it is determined that there is a second energy among the multiple first energies that is lower than the preset energy, determine the first audio segment corresponding to the second energy, wherein the multiple audio segments correspond one-to-one with the multiple first energies; delete the first audio segment from the third audio to obtain the second audio.
[0059] In an optional embodiment, the extraction module 42 is further configured to obtain the first text content of the first question; determine whether there are consecutive repeating fields in the multiple fields of the first text content; if it is determined that the first field and the second field are consecutive and the first field and the second field are the same, delete the first field or the second field to obtain the second speech content, wherein the multiple fields include the first field and the second field; input the second speech content to the text encoder to obtain the speech text features.
[0060] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.
[0061] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:
[0062] S1, based on the first audio and the first question emitted by the first target object, multiple audio features in the first audio are obtained by an audio encoder, and speech text features in the first question are extracted by a text encoder, wherein the multiple audio features include at least one of the following: speech content features corresponding to the first audio, voiceprint features of the second target object in the first audio, and non-human voice features in the first audio.
[0063] S2, the feature dimensions of the various audio features are unified to the target feature dimension of the speech-text features through a linear layer, and the various audio features of the target feature dimension are concatenated to obtain the fused audio features;
[0064] S3, the fused audio features and the speech-text features are concatenated to obtain the input features of the first language model;
[0065] S4, input the input features into the first large language model to obtain the speech parsing result corresponding to the first audio.
[0066] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0067] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0068] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:
[0069] S1, based on the first audio and the first question emitted by the first target object, multiple audio features in the first audio are obtained by an audio encoder, and speech text features in the first question are extracted by a text encoder, wherein the multiple audio features include at least one of the following: speech content features corresponding to the first audio, voiceprint features of the second target object in the first audio, and non-human voice features in the first audio.
[0070] S2, the feature dimensions of the various audio features are unified to the target feature dimension of the speech-text features through a linear layer, and the various audio features of the target feature dimension are concatenated to obtain the fused audio features;
[0071] S3, the fused audio features and the speech-text features are concatenated to obtain the input features of the first language model;
[0072] S4, input the input features into the first large language model to obtain the speech parsing result corresponding to the first audio.
[0073] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0074] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.
[0075] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.
[0076] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.
[0077] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.
[0078] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.
[0079] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. An audio parsing method, characterized in that, include: Based on the first audio and the first question emitted by the first target object, multiple audio features in the first audio are obtained by an audio encoder, and speech text features in the first question are extracted by a text encoder. The multiple audio features include at least one of the following: speech content features corresponding to the first audio, voiceprint features of the second target object in the first audio, and non-human voice features in the first audio. The feature dimensions of the various audio features are unified to the target feature dimension of the speech-text feature through a linear layer, and the various audio features of the target feature dimension are concatenated to obtain the fused audio feature. The fused audio features are concatenated with the speech-text features to obtain the input features of the first language model; The input features are input into the first large language model to obtain the speech parsing result corresponding to the first audio.
2. The audio parsing method according to claim 1, characterized in that, A linear layer is used to unify the feature dimensions of the various audio features to the target feature dimension of the speech-text features, including: The target feature dimension is determined, and a corresponding first linear layer is set for each of the multiple audio features to obtain the linear layer. The input dimension of the first linear layer is consistent with the feature dimension of the first audio feature corresponding to the first linear layer, and the output dimension of the first linear layer is consistent with the target feature dimension. The first audio feature is the speech content feature, the voiceprint feature, or the non-human voice feature. For any second audio feature among the multiple audio features, the second audio feature is input into a second linear layer to adjust the feature dimension of the second audio feature to the target feature dimension, wherein the multiple first linear layers include the second linear layer.
3. The audio parsing method according to claim 1, characterized in that, Before obtaining multiple audio features from the first audio based on the first audio emitted by the first target object and the first question through an audio encoder, the method further includes: Collect the first voice command issued by the first target object, wherein the first voice command is used to train the first large language model; Generate question-and-answer data corresponding to the first voice command, wherein the question-and-answer data includes: first question-and-answer data of a first type and second question-and-answer data of a second type, the first question-and-answer data includes: a second question generated by a second language model and a fixed answer corresponding to the second question, the second question-and-answer data includes the second question and a first answer to the second question, the first answer being generated by the second language model; The first large language model is trained using the first voice command and the question-and-answer data.
4. The audio parsing method according to claim 1, characterized in that, Based on the first audio emitted by the first target object and the first question, multiple audio features in the first audio are obtained through an audio encoder, including: The first audio is preprocessed to obtain the second audio. The content information of the second audio is extracted by the first audio encoder to obtain the speech content features; The voiceprint information of the second target object is extracted by the second audio encoder to obtain the voiceprint features; The non-human voice audio information in the second audio is extracted by the third audio encoder to obtain the non-human voice features, wherein the audio encoder includes the first audio encoder, the second audio encoder, and the third audio encoder.
5. The audio parsing method according to claim 4, characterized in that, The first audio is preprocessed to obtain the second audio, including: The first audio is subjected to noise reduction processing to obtain the third audio; Determine whether the first energy of multiple audio segments in the third audio is lower than a preset energy; If it is determined that there is a second energy lower than the preset energy among the multiple first energies, a first audio segment corresponding to the second energy is determined, wherein the multiple audio segments correspond one-to-one with the multiple first energies; The first audio segment is deleted from the third audio segment to obtain the second audio segment.
6. The audio parsing method according to claim 1, characterized in that, Extracting speech-text features from the first question using a text encoder includes: Obtain the first text content of the first question; Determine whether there are consecutive duplicate fields among multiple fields of the first text content; If it is determined that the first field and the second field are consecutive and the first field and the second field are the same, delete the first field or the second field to obtain the second voice content, wherein the plurality of fields include the first field and the second field; The second speech content is input into the text encoder to obtain the speech-text features.
7. An audio parsing device, characterized in that, include: The extraction module is used to obtain multiple audio features in the first audio based on the first audio emitted by the first target object and the first question, and to extract speech text features in the first question through an audio encoder, wherein the multiple audio features include at least one of the following: speech content features corresponding to the first audio, voiceprint features of the second target object in the first audio, and non-human voice features in the first audio. The unification module is used to unify the feature dimensions of the multiple audio features to the target feature dimension of the speech-text features through a linear layer, and to concatenate the multiple audio features of the target feature dimension to obtain the fused audio features. The splicing module is used to splice the fused audio features with the speech-text features to obtain the input features of the first language model; The parsing module is used to input the input features into the first large language model to obtain the speech parsing result corresponding to the first audio.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method according to any one of claims 1 to 6.
9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 6 through the computer program.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.