Speech processing method and apparatus, device, medium, and program product

Abstract

A speech processing method and apparatus, a computer device, a storage medium, and a program product. The method comprises: acquiring overlapping speech data (S301); acquiring reference speech data of a specified object (S302); extracting a voiceprint representation vector of the specified object from the reference speech data, wherein the voiceprint representation vector is used for representing a voiceprint characteristic of the specified object, and inputting the overlapping speech data and the voiceprint representation vector into a preset speech segmentation model, wherein the speech segmentation model is used for segmenting the overlapping speech data on the basis of an attention mechanism to obtain a target speech signal matched with the voiceprint characteristic (S303); and generating a speech file of the specified object on the basis of the speech signal obtained by segmentation (S304). According to the method, the overlapping speech data can be segmented to obtain a clearer speech signal of any specified object.

Description

A voice processing method, device, equipment, medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 13, 2023, with application number: 202310699993.5 and application name: "A speech processing method, device, equipment, medium and program product", all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, in particular to the field of artificial intelligence, and specifically to a speech processing method, a speech processing device, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0003] Aliased speech data (or simply aliased voice) is speech data that contains speech signals from multiple sound sources (i.e., sound-generating objects). For example, in a conference scenario, the aliased speech data recorded by a recording device in the physical environment may include speech signals from multiple participants, as well as speech signals from certain devices in the physical environment (such as the device playing the conference video).

[0004] Currently, the source separation methods for aliased speech data include: 1. Separating aliased speech data by human ears. This artificial listening method results in a long segmentation process and low efficiency. 2. Relying on timbre frequency to separate aliased speech data. When there are multiple objects with similar timbre frequencies, accurate segmentation cannot be achieved. 3. Separating aliased speech data based on the distance of the sound source will limit the speech segmentation to the different distances between the sound sources. 4. Using a dedicated speech segmentation model for a specific object to separate aliased speech data. This method is not portable and cannot be universal.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a speech processing method, apparatus, device, medium, and program product, which can separate the pure speech signal of any specified object from mixed speech data and are universal.

[0007] In one aspect, an embodiment of the present application provides a speech processing method, which is executed by a computer device and includes:

[0008] Acquire aliased speech data, where the aliased speech data includes speech signals generated by each of at least two objects;

[0009] Acquire reference speech data of a designated object; the designated object is any one of at least two objects; the reference speech data includes a reference speech signal of the designated object;

[0010] extracting a voiceprint representation vector of a specified object from the reference speech data, wherein the voiceprint representation vector is used to represent the voiceprint characteristics of the specified object;

[0011] Inputting the aliased speech data and the voiceprint representation vector into a preset speech segmentation model, wherein the speech segmentation model is used to: segment the target speech signal that matches the voiceprint characteristics from the aliased speech data based on the attention mechanism;

[0012] Based on the segmented target speech signal, a speech file of a designated object is generated.

[0013] On the other hand, an embodiment of the present application provides a speech processing device, the device comprising:

[0014] an acquiring unit, configured to acquire aliased speech data, where the aliased speech data includes speech signals generated by each of at least two objects;

[0015] The acquisition unit is further configured to acquire reference speech data of a designated object; the designated object refers to any one of at least two objects; and the reference speech data includes a reference speech signal of the designated object;

[0016] A processing unit, configured to extract a voiceprint representation vector of a specified subject from the reference speech data, wherein the voiceprint representation vector is used to represent a voiceprint characteristic of the specified subject;

[0017] The processing unit is further configured to input the aliased speech data and the voiceprint representation vector into a preset speech segmentation model, wherein the speech segmentation model is configured to: segment a target speech signal that matches the voiceprint characteristics from the aliased speech data based on an attention mechanism;

[0018] The processing unit is further configured to generate a voice file of a designated object based on the segmented target voice signal.

[0019] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising:

[0020] a processor for loading and executing computer programs;

[0021] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned speech processing method is implemented.

[0022] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned speech processing method.

[0023] On the other hand, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned speech processing method.

[0024] In an embodiment of the present application, aliased speech data to be segmented is obtained, and the aliased speech data contains speech signals generated by each of at least two objects. If there is a need to segment the speech signal generated by a specified object among the at least two objects, a section of reference speech data of the specified object (such as a few seconds of speech generated by the specified object) can be obtained. The specified object can be any one of the at least two objects. A voiceprint representation vector of the specified object is extracted from the reference speech data. The voiceprint representation vector can represent the voiceprint characteristics of the specified object. The voiceprint characteristics are unique and can represent the identity of the specified object. In this way, the voiceprint representation vector that can uniquely represent the identity of the specified object and the aliased speech data to be segmented can be input into a preset speech segmentation model, so that the speech segmentation model can segment the target speech signal that matches the voiceprint characteristics of the specified object from the aliased speech data based on the attention mechanism, and thus generate a separate speech file for the specified object based on the segmented target speech signal. It can be seen that, on the one hand, the embodiment of the present application supports extracting a voiceprint representation vector representing the voiceprint characteristics of a specified object from the pure reference voice data of the specified object, and using the voiceprint representation vector as a reference, utilizing the attention mechanism provided by the voice segmentation model to clearly and accurately calculate and extract the target voice signal of the specified object from the aliased voice data, thereby improving the extraction purity of the target voice signal and achieving a more accurate voice separation effect. On the other hand, the embodiment of the present application only needs to obtain the reference voice data of the specified object to be able to segment the target voice signal of the specified object from the aliased voice data; if you want to obtain the voice data of other objects, you only need to replace the voiceprint representation vector of the object to be segmented, and there is no need to train a dedicated network for each object, which greatly improves convenience and portability, and enhances the versatility of this solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG1 is a schematic diagram of the architecture of a speech processing system provided by an exemplary embodiment of the present application;

[0026] FIG2 is a schematic diagram of the architecture of a speech processing scenario provided by an exemplary embodiment of the present application;

[0027] FIG3 is a flow chart of a speech processing method provided by an exemplary embodiment of the present application;

[0028] FIG4 is a schematic diagram of an interface for a user to input reference voice data of a specified object, provided by an exemplary embodiment of the present application;

[0029] FIG5 is a schematic diagram of the structure of an existing Unet network;

[0030] FIG6 is a schematic diagram of the structure of a speech segmentation model constructed by adding an Attention mechanism to each network layer in a Unet network, provided by an exemplary embodiment of the present application;

[0031] FIG7a is a schematic diagram of speech segmentation when a target network layer is a convolutional layer or a convolutional connection layer, provided by an exemplary embodiment of the present application;

[0032] FIG7 b is a schematic diagram of speech segmentation when the target network layer is an upsampling layer, provided by an exemplary embodiment of the present application;

[0033] FIG8 is a flow chart of another speech processing method provided by an exemplary embodiment of the present application;

[0034] FIG9 is a flow chart of a voiceprint vector extraction process provided by an exemplary embodiment of the present application;

[0035] FIG10 is a schematic structural diagram of an improved PANNS provided by an exemplary embodiment of the present application;

[0036] FIG11 is a schematic diagram of the structure of a transformer network provided by an exemplary embodiment of the present application;

[0037] FIG12 is a schematic structural diagram of a speech processing device provided by an exemplary embodiment of the present application;

[0038] FIG13 is a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0039] In an embodiment of the present application, a speech processing solution is provided, specifically a speech separation solution for performing signal source separation on aliased speech data. The aliased speech data can be referred to as aliased speech or mixed audio signal, which is an audio mixed with multiple speech signals (or audio signals); that is, the so-called "aliasing" can be understood as multiple speech signals mixed / blended together. In actual application scenarios, the aliased speech data can be understood as speech data containing speech signals generated by multiple sound sources, which is collected directly from the environment by a sound receiving device (such as a microphone). The multiple speech signals can be generated by different objects (or sound sources), and the objects here can include but are not limited to: humans, animals, or physical devices (such as cars), etc. The embodiment of the present application does not limit the sources of the multiple speech signals contained in the aliased speech data. For example, in a conference scenario where multiple people participate in a discussion, the collected voice data usually includes voice signals generated by different participants; of course, if the conference scenario also includes a device that plays audio and video, then the collected voice data also includes the voice signal emitted by the device; in this way, the voice data collected in the conference scenario can be called aliased voice data, which includes voice signals generated by multiple objects in the conversation scenario.

[0040] Furthermore, source separation refers to the process of separating the voice signal of a specified object from the aliased voice data. In other words, source separation can be simply understood as a technology that separates the aliased voice data through signal processing or other algorithms to separate the target voice signal of the specified object from the aliased voice data, and finally generates a separate audio file (or voice file) of the specified object. For example, after collecting a piece of aliased voice data in a noisy outdoor scene, the target voice signal generated by a specified object can be extracted from the aliased voice data through the source separation technology to generate a voice file of the specified object; in this way, when the voice file is played, only the voice generated by the specified object exists, thereby achieving the purpose of identifying the voice generated by a specified object.

[0041] Based on a brief introduction to the concepts of aliased speech data and source separation, embodiments of the present application provide a new speech processing solution. This solution primarily includes: obtaining aliased speech data to be segmented, wherein the aliased speech data includes speech signals generated by each of at least two objects, such as the speech signals generated by object 1 and object 2. If a user wishes to extract a target speech signal generated by a specified object (e.g., any of the at least two objects) from the aliased speech data, a segment of reference speech data containing a reference speech signal of the specified object can be obtained. Thus, a voiceprint representation vector for the specified object can be extracted based on the reference speech data. The voiceprint representation vector can characterize the voiceprint characteristics of the specified object, which can be understood as the object's unique vocal characteristics, such as its unique pitch or timbre. The voiceprint representation vector of the specified object and the aliased speech data are then input into a speech segmentation model. The attention mechanism within the speech segmentation model can then be used to segment and extract a target speech signal that matches the voiceprint characteristics of the specified object from the aliased speech data, thereby generating a separate speech file for the specified object based on the target speech signal.

[0042] It can be seen that, on the one hand, the embodiment of the present application relies on the uniqueness of the voiceprint characteristics of each user. It only needs to provide a section of reference voice data of any specified object to extract the voiceprint feature vector that characterizes the voiceprint characteristics of the specified object. Then, based on the voiceprint characterization vector, the target voice signal of any specified object can be separated and extracted from the aliased voice data. Not only does it achieve the purpose of accurately separating the target voice signal of the specified object from the aliased voice data, but it can also achieve source separation for any object to which any voice signal contained in the aliased voice data belongs, achieving a high degree of reusability and portability, reducing the complexity of user input operations, and making the entire system more universal. On the other hand, the embodiment of the present application calculates the voiceprint characteristics and aliased voice data of the specified object based on the attention mechanism, greatly improving the clarity and accuracy of extracting the target voice signal of the specified object from the aliased voice data, avoiding too much noise in the extracted target voice signal, and achieving a more accurate and pure voice separation effect.

[0043] The embodiment of the present application mainly implements a speech processing solution through a reusable speaker-specified speech segmentation system based on voiceprint vector embedding. That is, the system deploys the speech processing solution provided by the embodiment of the present application; thus, when any user has a need to separate the speech signal of aliased speech data, the system can be called to automatically separate and extract the speech file corresponding to the specified object from the aliased speech data. The exemplary architecture diagram of the system can be seen in Figure 1; as shown in Figure 1, the system mainly includes two modules, namely: a voiceprint vector extraction model and a speech segmentation model; the following is a brief introduction to these two modules, wherein:

[0044] (1) Voiceprint vector extraction model, which can be called a voiceprint vector extractor or voiceprint recognition network. The voiceprint vector extraction model is mainly used to identify the identity of the specified object to be segmented and extract the identity semantic vector of the specified object; the identity semantic vector here is called a voiceprint representation vector (or simply a voiceprint vector) in the embodiment of this application, which is used to represent the voiceprint characteristics of the specified object.

[0045] As shown in Figure 1, the voiceprint vector extraction model is built based on a modified Pretrained Audio Neural Networks (PANNS) network and a Transformer network. The voiceprint vector extraction model is fully trained using an open-source, large-scale speaker dataset (a dataset containing rich speech data). The trained voiceprint vector extraction model is capable of fully representing the voiceprint characteristics of an object. Thus, it serves as the voiceprint vector extractor for the entire system. During the inference phase, after loading the model parameters pre-trained using the large-scale speaker dataset, the trained voiceprint vector extraction model is used to calculate the voiceprint representation vector of a specified object using reference speech data (e.g., a short segment (e.g., a few seconds or more) of speech). This voiceprint representation vector is used to characterize the voiceprint characteristics of the specified object. Thus, training the voiceprint vector extraction model using a large-scale speaker dataset eliminates the need to specifically collect relevant training data for each object in the aliased speech data, eliminating the reliance on extracting data from the aliased speech data to build a model specific to that object.

[0046] Among them: ① The improved PANNS in the voiceprint vector extraction model is an improvement on the traditional PANNS; this improvement is mainly reflected in: designing a link for information exchange between the time domain link and the frequency domain link, so that there are multiple time domain and frequency domain information exchanges in the process of voiceprint representation vector extraction, thereby achieving information complementarity between the time domain and the frequency domain, enabling the high-level network to fully perceive the underlying network information and improve the accuracy of voiceprint vector extraction. Among them, PANNS is an audio neural network trained based on a large audio dataset (including speech data of a large number of speakers); it is usually used for audio pattern recognition or audio frame-level vectorization (embedding) as the encoding network at the front end of the model. ② The transformer network is a conversion model that relies on the attention mechanism to calculate input and output; the transformer network abandons the convolutional model structure and achieves good performance only through the attention mechanism and the feed forward neural network (Feed Forward Neural Network), without the need to use a sequence-aligned recurrent architecture.

[0047] (2) Speech segmentation model, which can be called semantic segmentation network or segmentation network. The speech segmentation network is mainly used to receive the voiceprint characteristics of the specified object (specifically, the voiceprint representation vector) input by the voiceprint vector extraction model, and extract the speech signal that matches the voiceprint characteristics of the specified object from the aliased speech data using the attention mechanism based on the voiceprint representation vector.

[0048] As shown in Figure 1, the speech segmentation model incorporates an attention mechanism. By introducing the attention mechanism into the segmentation network, the attention mechanism can be used to calculate the target speech signal associated with the voiceprint features of the designated subject during the feature processing of the aliased speech data. This allows the target speech signal of the designated subject to be segmented from the aliased speech data. This allows for clearer and more accurate calculation and extraction of the target speech signal of the designated subject from the aliased speech data, while also separating the pure target speech signal of the designated subject, achieving more accurate and pure speech separation. The attention mechanism is a solution that mimics human attention; simply put, it mimics human attention to quickly filter out desired information from a large amount of information. It is primarily used to address the difficulty of obtaining a reasonable vector representation for long input sequences in time series models. This approach involves retaining the intermediate results of the time series model, learning them with a new model, and correlating them with the output to achieve information filtering.

[0049] In summary, the system provided by the embodiment of the present application includes two modules. After the voiceprint vector extraction model extracts the voiceprint representation vector that can characterize the voiceprint characteristics of the specified object from the reference voice data of the specified object, the voiceprint representation vector can be embedded into the voice segmentation model; in this way, the voice segmentation model can extract and separate the voice signal that matches the voiceprint characteristics from the aliased voice data based on the attention mechanism, thereby achieving a better signal separation effect. On the one hand, referring to Figure 1, it can be seen that the system provided by the embodiment of the present application is a fully automatic segmentation system constructed based on multiple deep learning neural networks (such as an improved PANNS network, a conversion network, and a segmentation network integrated with an attention mechanism, etc.); for the fully automatic segmentation system, the user only needs to input the reference voice data of the specified object and the aliased voice data to be segmented into the fully automatic segmentation system, and the fully automatic segmentation system can automatically and quickly extract the voice signal of the specified object from the aliased voice data, greatly improving the efficiency of voice segmentation, completely getting rid of manual participation, and forming rapid standardization. On the other hand, by innovatively embedding the voiceprint characteristics extracted by the voiceprint vector extraction model into the model architecture of the speech segmentation model, the speech separation model in the system can be reused; where reusability means that each time the system performs source separation, it only needs to replace the voiceprint characteristics of the extracted object, and there is no need to train a separate segmentation network for each object. The network can be easily migrated, making the entire system extremely versatile.

[0050] The system shown in Figure 1 can be deployed on a computer device, specifically within an application running on the computer device (e.g., deployed as a plug-in within the application); that is, the solution is provided by the application running on the computer device. Specifically: ① An application can refer to a computer program designed to perform one or more specific tasks. By categorizing applications according to different dimensions (e.g., their operating mode, functionality, etc.), the types of the same application under different dimensions can be obtained. For example, categorized by their operating mode, applications can include, but are not limited to: clients installed on a terminal, mini-programs (subprograms of the client) that can be used without downloading or installing, web (World Wide Web) applications opened through a browser, and so on. Another example, categorized by their functional type, applications can include, but are not limited to, IM (Instant Messaging) applications, content interaction applications, audio applications, or video applications. Instant messaging applications refer to internet-based applications for instant messaging and social interaction. Instant messaging applications may include, but are not limited to, social applications with communication functions, map applications with social interaction functions, and game applications. Content interaction applications refer to applications that enable content interaction, such as online banking, sharing platforms, personal spaces, and news applications. Audio applications refer to applications that implement audio functions over the Internet. Audio applications may include, but are not limited to, music applications with music playback and editing capabilities, radio applications with radio playback capabilities, or live streaming applications with live streaming capabilities. Video applications refer to applications that can play images. Video applications may include, but are not limited to, applications that can play short videos (often short in length, such as a few seconds or minutes), long videos (such as movies or TV series, which often play for a longer duration), etc.

[0051] ② The computer device may include a terminal and / or a server. Among them, the terminal may include but is not limited to: a smart phone (such as a smart phone deploying the Android system, or a smart phone deploying the Internetworking Operating System (IOS)), a tablet computer, a portable personal computer, a mobile Internet device (Mobile Internet Devices, MID), a vehicle-mounted device, a head-mounted device, a smart TV or a smart home device. The embodiment of the present application does not limit the type of terminal, which is explained here. The terminal is deployed with the system shown in Figure 1 or an application (or plug-in) providing the system, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0052] It can be seen that the embodiments of the present application can be executed by a terminal or a server, or jointly by a terminal and a server. A schematic diagram of the system architecture of an exemplary voice processing solution jointly executed by a terminal and a server can be seen in Figure 2; as shown in Figure 2, terminal 201 is a device held by a user with a voice separation requirement. When a user has a requirement to separate a voice file of a specified object from aliased voice data, the user can send the aliased voice data to be segmented and the reference voice data of the specified object to server 202 through terminal 201. In this way, after receiving the reference voice data of the specified object and the aliased voice data to be segmented, server 202 can first perform identity recognition on the reference voice data through the voiceprint vector extraction model in the system to obtain a voiceprint representation vector for representing the identity of the specified object, and then embed the voiceprint representation vector into the voice segmentation model in the system; after receiving the voiceprint representation vector of the specified object and the aliased voice data to be segmented, the voice segmentation model can extract a pure target voice signal that matches the voiceprint characteristics represented by the voiceprint representation vector from the aliased voice data based on the attention mechanism, thereby generating a voice file of the specified object based on the target voice signal. In this way, the server 202 returns the voice file of the designated object to the terminal 201 , so that the user can play the voice file containing only the voice data of the designated object through the terminal 201 .

[0053] It should be understood that the above description of the voice processing solution process is briefly described using computer devices as terminals and servers as examples. However, when the computer device is a terminal or server, the concept of executing the voice processing solution by the computer device is similar to the process described above, with only the execution subject being different, and is not further described here. In addition, the connection between terminal 201 and server 202 shown in Figure 2 can be direct or indirect via wired or wireless communication, which is not limited in this application.

[0054] Furthermore, the speech processing solution provided in the embodiments of this application can be applied to any application scenario requiring speech separation. The computer device providing this solution may vary depending on the application scenario, and this is not a limitation. Application scenarios may include, but are not limited to, at least one of the following: film and television drama scenarios, audio and video creation scenarios, and conversation scenarios.

[0055] Optionally, the application scenario is a film and television drama scenario. For example, the film and television drama scenario is a dubbing scenario for a character in a film and television drama. Specifically: During the film and television drama production stage, a voice actor is often required to dub for a certain character in the film and television drama (for example, after the aliased voice data recorded by the radio is submitted for review, some of the lines do not comply with the regulations and need to be re-recorded). However, the voice data obtained by the radio recording during the filming process or post-production process of the film and television drama is usually aliased voice data containing multiple voice signals. Therefore, before dubbing, it is necessary to extract the voice signals of other actors in the aliased voice data except for the voice signal of the designated actor to be re-dubbed, so as to mix the voice signal of the designated actor after dubbing with the extracted pure voice signals of other actors to generate new aliased voice data and add it to the film and television drama. It can be seen that in the film and television drama scenario, the embodiment of the present application can provide accurate voice segmentation, and there is no need to use a large amount of data of all actors to train a dedicated segmentation network, and the pure actor voice can be extracted and segmented quickly and efficiently.

[0056] Optionally, the application scenario is an audio and video creation scenario. Exemplarily, the audio and video creation scenario is a secondary creation scenario for audio and video (i.e., re-creation for existing audio and video). Specifically: in the secondary creation scenario, users like to extract part of the lines of a specified actor in multiple audio and video clips for line dialogue editing, that is, to clip the voice data of the specified actor in different audio and video clips into the same audio and video. This will involve extracting the pure voice signal of the specified actor in multiple audio and video clips; considering that the lines in each audio and video contain background music or voice signals of other objects, it is necessary to remove the background music from the audio and video to obtain the pure voice signal of the specified actor, so as to facilitate the fusion of the multiple extracted pure voice signals to generate the edited voice file corresponding to the specified actor.

[0057] Optionally, the application scenario is a conversation scenario. Exemplarily, the conversation scenario is an online conference scenario. Specifically: in online conference scenarios, there is often a need to transcribe speech into text, that is, convert the recorded speech data into text form; however, in online conference scenarios with the participation of multiple people, transcribing mixed speech data containing speech signals of multiple people has always been a difficult problem. Transcription refers to the process of converting the speech signal of a specific person among multiple people into text. By using the embodiment of the present application, the speech signal of each object participating in the online conference can be segmented and extracted from the mixed speech data, and then the speech signal of each object can be input into the speech recognition system to realize text transcription, which can greatly improve the accuracy of the mixed speech transcription.

[0058] It is worth noting that the application scenarios to which the voice processing solution provided in the embodiments of the present application is applicable are not limited to the above-mentioned ones; and as the application scenarios vary, the applications or platforms that carry the voice processing solution may vary.

[0059] It should also be noted that the collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations. The acquisition of personal information requires the knowledge or consent of the individual subject (or the presence of a legal basis for information acquisition), and subsequent data use and processing shall be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as when obtaining reference voice data of a specified object, the permission or consent of the specified object shall be obtained, and the collection, use and processing of relevant data (such as the collection and release of barrages posted by the object, etc.) shall comply with the relevant laws, regulations and standards of the relevant region.

[0060] Based on the voice processing solution described above, the embodiment of the present application proposes a more detailed voice processing method. The voice processing method proposed in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0061] FIG3 shows a flow chart of a speech processing method provided by an exemplary embodiment of the present application. The speech processing method may be executed by a computer device in the aforementioned system, such as a terminal and / or a server. The speech processing method may include but is not limited to steps S301-S304:

[0062] S301: Acquire aliased speech data to be segmented.

[0063] S302: Acquire reference speech data of a designated object among at least two objects.

[0064] In steps S301-S302, the aliased speech data to be segmented includes speech signals generated by each of at least two objects. For example, a piece of heavy metal music includes a "lyrics" speech signal generated by a "singer," a "melody" speech signal generated by a "guitar," and a "melody" speech signal generated by a "drum set." Therefore, the heavy metal music is determined to be aliased speech data because the objects included in the aliased speech data are: a singer, a guitar, and a drum set, and the speech signals included in the aliased speech data are: a speech signal generated by the singer, a speech signal generated by the guitar, and a speech signal generated by the drum set.

[0065] Furthermore, if the user has a need to separate and extract the voice signal of a certain object from the aliased voice data, the reference voice data of the certain object can be obtained. At this time, the certain object is called a designated object. The reference voice data of the designated object is different from the aliased voice data, but the reference voice data includes the pure reference voice signal of the designated object. In this way, the reference voice signal contained in the reference voice data of the designated object can be used as a reference signal to separate the target voice signal of the designated object from at least two voice signals corresponding to at least two objects contained in the aliased voice data. The following is a brief introduction to the designated object and the reference voice data, wherein: ① The designated object can be any object from the at least two objects contained in the aliased voice data whose voice signal the user wants to extract; as can be seen from the above description, the object can refer to a human, an animal or a physical device; for the sake of convenience, the object type of the designated object is human as an example for introduction, which is specially explained here. For example, in the above-mentioned heavy metal music example, if the user wants to extract the "lyrics" produced by the "singer" from the noisy heavy metal music, then the "singer" is determined as the specified object. At this time, it is necessary to separate the voice signals produced by each instrument in the heavy metal music and the voice signals produced by the "singer", and extract the pure voice signals of the "singer".

[0066] ② Reference voice data refers to a segment of voice data containing the reference voice signal of the designated subject. To ensure that relatively pure voiceprint characteristics of the designated subject can be extracted from the reference voice data, the reference voice data should be a relatively pure segment of voice data containing the designated subject. For example, the reference voice data may contain only the reference voice signal of the designated subject. Alternatively, the reference voice data may contain both the reference voice signal of the designated subject and other voice signals, but it should be easy to extract the reference voice signal of the designated subject from the reference voice data mixed with other voice signals (e.g., the other voice signals have a lower signal frequency, while the reference voice signal of the designated subject has a relatively higher signal frequency). This facilitates analysis of the pure reference voice data to extract relatively accurate voiceprint characteristics of the designated subject. This embodiment of the application does not limit the type, duration, or source of the reference voice data. For example, the types of reference voice data may include, but are not limited to, an audio segment produced by the designated subject reading a passage, an audio segment produced by the designated subject speaking, or an audio segment produced by the designated subject singing. The duration of the reference voice data can range from a few seconds to more than ten seconds. The source of the reference voice data may include but is not limited to: when the designated object and the user with voice separation requirements are different users, the reference voice data may be sent to the user by the designated object, or downloaded or recorded by the user through certain channels (such as historical voice information); when the designated object and the user with voice separation requirements are the same user, the reference voice data may be entered in real time by the designated object, that is, collected in real time through a microphone deployed in the terminal held by the user.

[0067] FIG4 is a schematic diagram of an exemplary interface for a user to input reference voice data for a specified subject. As shown in FIG4 , a voice acquisition interface 401 is displayed on the terminal screen of a user's terminal. This voice acquisition interface 401 includes a reference voice data acquisition area 402. Specifically, this acquisition area 402 may display at least two voice acquisition portals, such as a collection portal 4021 and an upload portal 4022. When collection portal 4021 is triggered, indicating that the user wishes to input reference voice data for a specified subject (the specified subject is the user, or the specified subject and the user are in the same physical environment) through real-time collection, the terminal's microphone is turned on to collect the reference voice signal in the user's physical environment in real time to generate the reference voice data. When upload portal 4022 is triggered, indicating that the user wishes to input reference voice data for a specified subject by uploading a file, the user can upload the reference voice data for the specified subject from a storage space (such as the terminal's local storage space, cloud storage space, or server storage space).

[0068] It should be understood that the interface elements (such as the interface content contained in the interface) and interface style of the voice acquisition interface are not limited to those shown in Figure 4. For example, an upload entry for aliased voice data can also be displayed in the voice acquisition interface, through which the user can replace the aliased voice data to be segmented. For another example, a text conversion control (or component, button, option, etc.) can be added to the voice acquisition interface, so that the user can trigger the text conversion control before or after voice separation to convert the separated voice signal into text form with one click, shortening the text conversion path to a certain extent, thereby improving the text conversion efficiency.

[0069] S303: Extracting a voiceprint representation vector of a specified object from the reference speech data, and inputting the aliased speech data and the voiceprint representation vector into a preset speech segmentation model. The speech segmentation model is used to segment a speech signal that matches the voiceprint characteristics of the specified object from the aliased speech data based on an attention mechanism.

[0070] S304: Generate a voice file of a designated object based on the segmented voice signal.

[0071] In steps S303-S304, a voiceprint is a sound wave spectrum that carries voice information. It is a biometric characteristic composed of multiple characteristic dimensions such as wavelength, frequency, and intensity. Voiceprints are stable, measurable, and unique, and can be used to uniquely identify the voice characteristics of an object. In other words, voiceprints can be used to characterize the identity of an object. Therefore, after obtaining relatively pure reference voice data of a specified object, the embodiments of the present application support extracting the specified object's voiceprint characteristics from the reference voice data, so as to facilitate subsequent voice signal separation and extraction based on these unique voiceprint characteristics.

[0072] As shown in the system shown in Figure 1, by analyzing reference speech data using a voiceprint vector extraction model, voiceprint characteristics that characterize the identity of a specified subject can be extracted from the reference speech data. In practical applications, the voiceprint vector extraction model outputs a voiceprint representation vector (or simply a voiceprint vector) for the specified subject. Specifically, the voiceprint vector extraction model analyzes the reference speech data to obtain a voiceprint representation vector that can be used to characterize the voiceprint characteristics of the specified subject. Furthermore, after extracting the voiceprint representation vector for the specified subject, the voiceprint vector extraction model innovatively uses vector embedding to transfer the voiceprint information representation. This voiceprint representation vector is then input into the speech segmentation model to participate in the calculation of the attention mechanism, thereby segmenting the target speech signal that matches the voiceprint characteristics of the specified subject from the aliased speech data. This innovative vector embedding mechanism allows the speech segmentation model to be trained independently of any historical speech data of the subject. Instead, it only requires a small amount of reference speech data to extract the voiceprint representation vector, eliminating the reliance on large-scale speech data. This makes the system highly reusable and portable, improves overall system efficiency, reduces the complexity of user input operations, and enhances system versatility.

[0073] The speech segmentation model provided in the embodiments of the present application is obtained by improving the traditional speech segmentation network using the attention mechanism; specifically, it is obtained by integrating the attention mechanism into the traditional speech segmentation network. Among them, the traditional speech segmentation network involved in the embodiments of the present application is a Unet (or U-net, U-Net, etc.) network. Unet is one of the algorithms for semantic segmentation using a fully convolutional network; it mainly uses a symmetric U-shaped structure containing a compression path and an expansion path.

[0074] A schematic diagram of an exemplary network structure of a Unet network can be seen in Figure 5. As shown in Figure 5, the Unet network is a U-shaped symmetrical network structure, which includes a left-right symmetrical feature extraction subnetwork and an upsampling subnetwork, and the feature extraction subnetwork and the upsampling subnetwork are connected through a convolutional connection layer. Among them: ① The feature extraction subnetwork can be simply understood as a downsampling layer or an encoding network, which includes m (m=4 in Figure 5) convolutional layers (Convolutional layer) with a hierarchical distribution (the feature map output by the convolutional layer of the previous level is used as the input of the convolutional layer of the adjacent next level), where m is a positive integer; the hierarchical distribution of m convolutional layers means that m convolutional layers are connected in sequence, and the upper convolutional layer of any two adjacent convolutional layers in the m convolutional layers serves as the convolutional layer of the upper level, and the lower convolutional layer serves as the convolutional layer of the next level, and the feature map output by the convolutional layer of the previous level serves as the input of the convolutional layer of the adjacent next level. As shown in Figure 5, a pooling function can be deployed after each convolutional layer; by first using the convolutional network in the convolutional layer to perform feature extraction on the aliased speech data, and then using the pooling function (pool) to further extract higher-order features, the features that are desired to be highlighted in the aliased speech data are effectively retained; wherein, the embodiment of the present application does not limit the type of pooling function, such as if the pooling function is maximum pooling (max pool), which tends to be the largest feature within the pooling window (such as a window size of 2*2) in the feature map output by the convolutional layer. ② Accordingly, the feature extraction subnetwork and the upsampling subnetwork are symmetrical, and the upsampling subnetwork can be simply understood as a decoding network, which includes an upsampling layer (up sampling layer) corresponding to each convolutional layer in the feature extraction subnetwork. As shown in Figure 5, a transposed convolution (up-Conv) with a convolution kernel of 2*2 is also deployed after the convolution connection layer and each upsampling layer to achieve the upsampling function through transposed convolution. It can be seen from this that the symmetrical network structure of the Unet network can not only realize the network from scratch and initialize the weights, and then train the model; it can also borrow the convolutional layer structure of some existing networks (such as vgg (a convolutional network) in resnet (residual neural network)) and the corresponding trained weight files, and then add the subsequent upsampling layer for training calculations, etc.; in this way, using the existing weight model file in deep learning model training can greatly speed up the model training speed.

[0075] Furthermore, each convolutional layer, convolutional connection layer and upsampling layer includes multiple convolutional networks connected in sequence; as shown in Figure 5, the feature extraction subnetwork, convolutional connection layer and upsampling layer can each include three convolutional networks with a convolution kernel of 3*3. Among them, the convolutional network is also called a convolutional neural network (CNN); a convolutional neural network is a feedforward neural network, mainly composed of one or more convolutional layers and a fully connected layer at the top, and also includes associated weights and a pooling layer. As shown in Figure 5, an activation function can be deployed after each convolutional network to add nonlinear factors to the model through the activation function, so that the trained model can solve problems that the linear model cannot solve. The embodiment of the present application does not limit the type of activation function. For example, the activation function can be a ReLu function (ReLuSigmoidTanh, linear rectification function) and the like.

[0076] Furthermore, the Unet network can also effectively combine high-level feature maps and low-level feature maps through skip connections (also called copy and crop) to obtain the final feature map. The specific process of the skip connection may include: the feature map obtained by each convolutional layer in the feature extraction subnetwork will be concatenated to the corresponding upsampling layer in the upsampling subnetwork; thereby achieving the effective use of each layer of feature maps in subsequent calculations. Compared with the network structure that does not implement skip connections, this method of skip-connecting feature maps of different dimensions can effectively avoid direct supervision and loss calculation in high-level feature maps, effectively combine the features in low-level feature maps, so that the final feature map contains both high-dimensional features and many low-dimensional features, realizing the fusion of features at different scales and improving the accuracy of the model results.

[0077] FIG5 above gives a detailed introduction to the network structure of the traditional Unet network. The speech segmentation model provided in the embodiment of the present application is obtained by improving the network structure of the Unet network. The improvement of the network structure of the Unet network in the embodiment of the present application mainly includes: integrating the attention mechanism into all or part of the network layers (such as convolutional layers, convolutional connection layers and upsampling layers) in the network structure of the Unet network. In other words, the speech segmentation model obtained by improving the Unet network based on the attention mechanism is integrated with the attention mechanism in all or part of the network. For the sake of convenience, taking the example of adding the Attention mechanism to each network layer in the Unet network as a speech segmentation model, by adding the attention mechanism to each network layer, the voiceprint representation vector of the specified object can be embedded in each network layer in the Unet network, so that each network layer in the network can deeply feel the voiceprint information or voiceprint characteristics represented by the voiceprint representation vector, so that the final output voice signal is closer to the specified object, ensuring that the extracted voice signal is purer.

[0078] For example, a schematic diagram of the structure of a speech segmentation model constructed by incorporating an attention mechanism into each network layer of a Unet network can be seen in Figure 6. As shown in Figure 6, the improved speech segmentation model retains the same basic network architecture as the original Unet network, but incorporates an attention mechanism into each layer within the Unet network architecture. The input information for this attention mechanism is the voiceprint representation vector of the designated subject and the feature map output by the previous layer of the attention mechanism. By embedding the voiceprint representation vector of the designated subject into each network layer, primarily performing attention calculations with the feature maps of each network layer, the entire model can deeply perceive and learn the extracted voiceprint representation vector. This allows calculations at each layer to converge towards the voiceprint representation vector, ensuring that the ultimately extracted speech signal matches the voiceprint characteristics represented by the voiceprint representation vector. It is worth noting that the fusion position of the attention mechanism within the multiple convolutional networks corresponding to the network layer is not fixed; the fusion positions shown in Figure 6 are illustrative only.

[0079] Based on the relevant introduction of the network structure of the speech segmentation model in FIG6 above, the following takes the example of each network layer in the speech segmentation model being integrated with the attention mechanism to introduce the specific implementation process of the speech segmentation model based on the attention mechanism to segment the speech signal that matches the voiceprint characteristics of the specified object from the aliased speech data, and generate the speech file of the specified object based on the speech signal; the process may include but is not limited to steps (1)-(4), wherein:

[0080] (1) Convert the aliased speech data from the time domain to the frequency domain to obtain the speech spectrum features corresponding to the aliased speech data, that is, the speech spectrum features are the characteristic expressions of the aliased speech data in the frequency domain.

[0081] Considering that the input information of the Unet network belongs to the frequency domain, it is necessary to convert the aliased speech data to obtain the speech spectrum features corresponding to the aliased speech data; in this way, the speech spectrum features can be input into the speech segmentation network as an image. Among them, the time domain and frequency domain are two commonly used concepts in audio applications and are also two dimensional concepts for measuring audio features; the time domain is processed by displaying the sampling points of the speech signal in time, that is, binding it with time; the frequency domain is a characteristic representation of the energy distribution of the speech signal in each frequency band; the speech signal can be converted from the time domain to the frequency domain, or from the frequency domain to the time domain through conversion formulas (such as Fourier Transform, Laplace Transform, or ZTransform).

[0082] (2) Based on the attention mechanism, the correlation between the voiceprint representation vector and the speech spectrum features is calculated to obtain the speech spectrum feature segmentation that matches the voiceprint characteristics.

[0083] As shown in Figure 6, after obtaining the voiceprint representation vector of the specified object extracted by the voiceprint vector extraction model, the voiceprint representation vector is input into each network layer in the speech segmentation network, specifically the attention mechanism integrated in each network layer. In this way, according to the attention mechanism integrated in each network layer, the correlation between the voiceprint representation vector and the first feature map of the corresponding network layer can be calculated to obtain the second feature map output by the corresponding network layer; it is worth noting that the first feature map of the network layer is different depending on the position of the network layer in the speech segmentation network, which will be introduced in the subsequent embodiments. Then, the second feature map output by the 2m+1th network layer in the speech segmentation model (i.e., the last upsampling layer in the upsampling subnetwork) can be used as the speech spectrum feature segmentation that matches the voiceprint representation vector; the speech spectrum feature segmentation is specifically the segmentation in the speech spectrum feature that matches the voiceprint characteristics, that is, the segmentation in the speech spectrum feature that belongs to the specified object. It can be seen that by embedding the voiceprint representation vector used to characterize the voiceprint characteristics of the specified object into each network layer in the speech segmentation model to participate in the feature extraction process, each network layer in the speech segmentation network can deeply sense the voiceprint information of the specified object, so that the final segmented output voice signal is closer to the voiceprint characteristics of the specified object, ensuring that the extracted voice signal is purer and more accurate.

[0084] It is not difficult to understand that depending on the different fusion positions of the attention mechanism in the speech segmentation model, the specific implementation process of the network layers in the feature extraction subnetwork and the upsampling subnetwork in accordance with the fused attention mechanism to calculate the correlation between the voiceprint representation vector and the first feature map output by the previous layer of the corresponding network layer is different. The following takes any network layer in the speech segmentation model that is fused with the attention mechanism as an example, and illustrates the correlation calculation of the target network layer according to the attention mechanism, where:

[0085] In one implementation, as shown in Figure 7a, it is assumed that the target network layer is a convolutional layer or a convolutional connection layer in a speech segmentation model, and the fusion position of the attention mechanism in the multiple convolutional networks corresponding to the target network layer is: in the multiple sequentially connected convolutional networks included in the target network layer, the position between the first convolutional network and the second convolutional network adjacent to the first convolutional network; that is, the fusion position of the attention mechanism in the multiple convolutional networks corresponding to the target network layer is: in the multiple sequentially connected convolutional networks included in the target network layer, the position between the first convolutional network and the convolutional network adjacent to the first convolutional network and located after the first convolutional network in the multiple sequentially connected convolutional networks. Under this implementation, the target network layer calculates the correlation between the voiceprint representation vector and the first feature map of the target network layer to obtain the second feature map output by the target network layer. The specific implementation process may include: first, using the first convolutional network in the target network layer to perform feature extraction processing on the first feature map of the target network layer to obtain the third feature map of the target network layer; feature extraction processing here refers to the process of extracting useful information (i.e., features) from the first feature map for use in subsequent classification, clustering, and regression tasks; the feature extraction processing process may include: preprocessing the first feature map (such as denoising, normalization or standardization), and performing feature extraction on the preprocessed first feature map to extract useful features, and screening representative or discriminative features from the extracted features, and the screened features are used as features after feature extraction processing. Among them, when the target network layer is the first convolution layer 701 of the hierarchical distribution in the speech segmentation model (specifically, the feature extraction subnetwork), the first feature map of the target network layer is the speech spectrum feature obtained by performing frequency domain conversion on the aliased speech data; when the target network layer is other convolution layers (such as convolution layer 702) in the speech segmentation model except the first convolution layer 701, the first feature map of the target network layer is obtained by pooling the feature map output by the upper-level network layer adjacent to the target network layer (such as convolution layer 701); the pooling process is performed by the pooling layer in the target network layer, and the pooling process aims to reduce the size and parameter amount of the feature map output by the upper-level network layer through parallel processing or data compression, thereby reducing the amount of calculation. Then, according to the attention mechanism integrated in the target network layer, the correlation between the voiceprint representation vector and the third feature map of the target network layer is calculated to obtain the fourth feature map of the target network layer; the feature dimension of the third feature map of the target network layer is the same as the feature dimension of the fourth feature map; wherein, the feature map (such as the third feature map and the fourth feature map) can be expressed in the form of a vector, so the feature dimension of the feature map can be the dimension of the vector, and each dimension in the vector corresponds to a feature, that is, the feature dimension before and after the attention mechanism calculation is the same.Finally, the fourth feature map is extracted using the other convolutional networks except the first convolutional network in the target network layer to obtain the second feature map output by the target network layer.

[0086] It can be seen that when the target network layer is a convolutional layer or a convolutional connection layer in a speech segmentation model, the attention mechanism is integrated into the multiple convolutional networks contained in the convolutional layer or the convolutional connection layer. In this way, in the process of extracting features from the aliased speech data using the multiple convolutional networks contained in the convolutional layer or the convolutional connection layer, the features that match the voiceprint representation vector of the specified object can be focused on in the aliased speech data based on the attention mechanism. Thus, through the attention mechanism in the feature extraction process, a second feature map that matches the voiceprint representation vector is analyzed and obtained, and then based on the second feature map, the target speech signal of the specified object can be accurately segmented from the aliased speech data.

[0087] In another implementation, as shown in Figure 7b, assuming that the target network layer is an upsampling layer in a speech segmentation model, and the attention mechanism is integrated into the multiple convolutional networks corresponding to the target network layer at the position after the last convolutional network in the multiple sequentially connected convolutional networks, the target network layer can calculate the correlation between the voiceprint representation vector and the first feature map of the target network layer to obtain the second feature map output by the target network layer. The specific implementation process may include: first, performing feature extraction on the target feature map using the multiple sequentially connected convolutional networks in the target network layer to obtain the first feature map of the target network layer. The target feature map is obtained by concatenating the feature map output by the convolutional layer corresponding to the target network layer with the feature map output by the network layer above the target network layer. As shown in Figure 7b, the input information of the first upsampling layer 703 in the upsampling subnetwork is the target feature map, which is obtained by concatenating the feature map output by the convolutional connection layer 704 in the previous layer of the first upsampling layer 703 with the feature map output by the convolutional layer 705 corresponding to the first upsampling layer 703. Then, the attention mechanism integrated in the target network layer is used to calculate the correlation between the voiceprint representation vector and the first feature map of the target network layer to obtain the second feature map output by the target network layer; the feature dimension of the second feature map is the same as that of the first feature map.

[0088] It can be seen that when the target network layer is the upsampling layer in the speech segmentation model, the attention mechanism is integrated in the multiple convolutional networks contained in the upsampling layer. In this way, after the feature map output by the previous network layer and the feature map output by the corresponding convolutional layer are used to extract features, the attention mechanism can be used to focus on the features that match the voiceprint representation vector of the specified object in the first feature map after feature extraction, thereby analyzing and obtaining the second feature map that matches the voiceprint representation vector of the specified object, and then based on the second feature map, the target speech signal of the specified object can be accurately segmented from the aliased speech data.

[0089] It should be understood that the above Figures 7a and 7b are only exemplary processes for the target network layer to perform correlation calculation when the attention mechanism is respectively integrated into an exemplary fusion position in the feature extraction module, the convolution connection layer and the upsampling sub-network layer; when the attention mechanism is integrated into different fusion positions in the target network layer, the specific implementation process of the target network layer performing the correlation calculation is different.

[0090] (3) The speech spectrum feature segments that match the voiceprint characteristics are converted from the frequency domain to the time domain to obtain the target speech signal that matches the voiceprint characteristics.

[0091] After extracting a speech spectrum feature segment that matches the voiceprint characteristics of the specified subject from the speech spectrum features corresponding to the aliased speech data based on the aforementioned steps, it is necessary to convert the speech spectrum feature segment from the frequency domain to the time domain to obtain a target speech signal that conforms to the data transmission format. The method for converting the speech spectrum feature segment from the frequency domain to the time domain may include, but is not limited to, the aforementioned Fourier transform, Laplace transform, or Z transform, and is not limited thereto.

[0092] (4) Generate a voice file of a specified object based on a target voice signal that matches the voiceprint characteristics. The file format of the voice file can be set according to the personalized needs of the user, and the embodiment of the present application does not limit the file format of the voice file. For example, when the voice file is a text file, it supports the use of a voice recognition algorithm or tool to convert the voice signal that matches the voiceprint characteristics of the specified object into text, and perform text processing on the converted text (such as correcting spelling errors, adding conformances, and clarifying the text), and then save the text after text processing as a text file in a text format (such as .doc format). For another example, when the voice file is an audio file, the voice signal that matches the voiceprint characteristics of the specified object can be directly saved as an audio file in an audio format (such as .WAV format).

[0093] Based on the speech signal extraction process described in the above steps (1)-(4), it can be seen that the embodiment of the present application supports converting the aliased speech data from the time domain to the frequency domain to obtain the speech spectrum features of the aliased speech data in the frequency domain. After the aliased speech data is converted into speech spectrum features that can be correlated with the voiceprint representation vector and belong to the frequency domain, the attention mechanism of each network layer in the speech segmentation model can be used to calculate the correlation between the voiceprint representation vector and the speech spectrum features that belong to the frequency domain, thereby ensuring the feasibility of the correlation calculation. Considering that the final segmentation is the signal in the time domain, it is necessary to convert the speech spectrum features from the frequency domain to the time domain to obtain the target speech signal that matches the voiceprint characteristics, thereby ensuring that the final extracted signal is a time domain signal that can be understood and read by the device. Considering that the feature dimensions of each network layer in the speech segmentation network are different, specifically, the feature dimensions of each convolutional network in each network layer are different. Therefore, before inputting the voiceprint representation vector of the specified object into each network layer in the speech segmentation network, it is necessary to first use a network layer to transform the dimension of the voiceprint representation vector to obtain the voiceprint representation vector after the dimension transformation. Among them, dimensionality transformation refers to changing the feature dimension of the voiceprint representation vector so that the feature dimension of the voiceprint representation vector after dimension transformation is the same as the feature dimension of the feature map of the attention mechanism to be input into the corresponding network layer. Therefore, when the voiceprint representation vector after dimension transformation is input into the speech segmentation model, the speech segmentation model can effectively process the voiceprint representation vector and avoid the unavailability of the voiceprint representation vector caused by the different dimensions. Among them, the feature map of the attention mechanism to be input into the corresponding network layer can refer to the third feature map described above. In addition, the attention mechanism can be inserted between any two convolutional networks in a plurality of convolutional networks connected sequentially in the network layer; the first feature map of the network layer can refer to the feature map output by the convolutional network in the network layer that is adjacent to the attention mechanism and is located before the attention mechanism.

[0094] In summary, the embodiment of the present application innovatively constructs a fully automatic speech processing solution, which realizes the segmentation of speech signals based on the voiceprint vector embedding method. For users, they only need to input a small segment of reference speech data of the specified object and the aliased speech data to be segmented, so that the speech signal of the specified object can be automatically and quickly separated from the aliased speech data, which can greatly improve the efficiency of speech segmentation, completely get rid of manual participation, and form rapid standardization. In this solution, on the one hand, the voiceprint vector embedding method is adopted, and the voiceprint representation vector used to characterize the voiceprint characteristics of the specified object is input into the speech segmentation model to participate in the Attention calculation, which can make the speech segmentation model independent of any object's historical speech data for additional training, and can get rid of the dependence on the object's large-scale speech data, thereby making the system highly reusable and portable, making the entire system more universal. On the other hand, by improving the Unet network based on the attention mechanism, the input voiceprint representation vector can be calculated with the Attention mechanism at each network layer in the Unet network, thereby ensuring that the feature map output by the speech segmentation model is more consistent with the voiceprint characteristics represented by the voiceprint representation vector of the specified object, avoiding too much noise in the extracted speech signal of the specified object, improving the purity of the speech signal, and improving the segmentation accuracy of the speech segmentation model.

[0095] Please refer to FIG8 , which shows a flowchart of another voice processing method provided by an exemplary embodiment of the present application. The voice processing method may be executed by a computer device in the aforementioned system, such as a terminal and / or a server. The voice processing method may include but is not limited to steps S801-S806:

[0096] S801: Acquire aliased speech data to be segmented.

[0097] S802: Acquire reference speech data of a designated object among at least two objects.

[0098] It should be noted that, for the specific implementation process shown in steps S801-S802, reference can be made to the relevant description of the specific implementation process shown in steps S301-S302 in the embodiment shown in FIG3 , and no further details are given here.

[0099] S803: Perform short-term correlation analysis on the reference speech data of the designated object.

[0100] S804: Performing long-term correlation analysis on the reference speech data of the designated object to obtain a voiceprint representation vector of the designated object.

[0101] In steps S803-S804, in order to learn a clearer voiceprint characteristic of the specified subject from the reference voice data of the specified subject, the embodiment of the present application supports analyzing the reference voice data in combination with short-term correlation and long-term correlation to extract a voiceprint representation vector that can fully express the voiceprint characteristics of the specified subject. Among them, the short-term correlation analysis of the reference voice data can be simply understood as: the process of performing feature analysis on a relatively short segment (such as 20 milliseconds) of the voice signal in the reference voice data; considering that the voice signal in the reference voice data usually does not change in a short period of time, after discretizing the reference voice data, the information distribution of each short voice signal in the time domain and frequency domain can be used to extract the characteristics of the voice signal in the short period of time, thereby achieving feature analysis of each voice signal in the reference voice data. In short, short-term correlation analysis focuses on feature analysis of segmented voice signals in the reference voice data. In contrast, long-term correlation analysis of the reference voice data can be simply understood as: the process of performing feature analysis on the entire reference voice data; that is, long-term correlation analysis focuses on semantic expression of the entire signal sequence of the reference voice data.

[0102] The short-term correlation analysis and long-term correlation analysis mentioned above are mainly implemented by the voiceprint vector extraction model in the system shown in Figure 1. As described above, the voiceprint vector extraction model includes an improved audio neural network (PANNS) network and a transformer network. The improved audio neural network (PANNS) network can be simply referred to as the improved PANNS and is mainly used for short-term correlation analysis of reference speech data; the transformer network can be simply referred to as the transformer network and is mainly used for long-term correlation analysis of reference speech data.

[0103] A schematic diagram of an exemplary use of an improved PANNS and Transformer network to perform short-term correlation analysis and long-term correlation analysis on reference speech data can be seen in Figure 9. As shown in Figure 9, first, the reference speech data is converted from the time domain to the frequency domain to obtain the reference speech spectrum features corresponding to the reference speech data; wherein, the conversion formula for converting the time domain to the frequency domain can be found in the aforementioned related description and will not be repeated here. Then, in order to achieve short-term correlation analysis, the reference speech spectrum features can also be segmented to obtain reference speech spectrum feature segments corresponding to each speech data segment; the reference speech spectrum feature segments involved in the embodiment of the present application can be a logarithmic Mel spectrum (Log-mel or Logmel). wherein, the mel spectrum is a nonlinear frequency scale determined based on the human ear's sensory judgment of equidistant (i.e., frequency bands are equidistantly distributed on the Mel scale) pitch changes, and pitch refers to the height of the sound; when performing signal processing, it can be more easily set to cater to the changes in the human ear's auditory perception.

[0104] Then, using a segmented input method, the reference speech spectral feature segments corresponding to each speech data segment are input into the improved PANNS network, and the complete reference speech data is also input into the improved PANNS. In this way, the improved PANNS can segment the received reference speech data according to the segmentation rules followed when segmenting the reference speech spectral features, obtaining multiple speech data segments corresponding to the reference speech data. Furthermore, the improved PANNS performs short-term correlation analysis on each speech data segment based on the corresponding reference speech spectral feature segment, obtaining a voiceprint semantic feature vector corresponding to each speech data segment. This voiceprint semantic feature vector is used to characterize the semantic characteristics of the corresponding speech data segment.

[0105] Finally, the vector sequence composed of the voiceprint semantic feature vectors corresponding to each speech data segment (or called the voiceprint semantic feature vector sequence) is input into the Transformer network, so that the Transformer network can perform long-term correlation analysis on the voiceprint semantic feature vector sequence to obtain the voiceprint representation vector of the specified object. Here, the voiceprint semantic feature vector sequence includes the voiceprint semantic feature vector corresponding to each speech data segment. Among them, the Transformer network is a sequence network, and its input is an overall vector sequence, and its output is also a sequence (that is, the voiceprint representation vector of the specified object is a sequence); after the Transformer network outputs a sequence, the last vector in the sequence can be called state, which contains the semantic fusion of the entire sequence; in this way, the overall sequence output by the Transformer network can be averaged (mean), and the average result and the state can be superimposed to generate a voiceprint representation vector that integrates the semantic features expressed by the entire sequence. The above-mentioned method of obtaining the voiceprint representation vector by combining the speech features represented by the average result of the Transformer network output sequence and the speech features represented by the entire sequence can effectively ensure that the voiceprint representation vector can more fully reflect the voiceprint characteristics of the specified object and ensure the accuracy of voiceprint feature extraction.

[0106] Based on the above overall introduction to the voiceprint vector extraction model including the improved PANNS and Transformer network, the following describes the structure and process of the improved PANNS and Transformer network respectively:

[0107] (1) Improved PANNS.

[0108] Figure 10 shows an exemplary structural diagram of the improved PANNS. As shown in Figure 10, the input information of the improved PANNS is reference speech data, that is, the input of the improved PANNS uses the original speech sampling point sequence, that is, the original sequence of the audio signal. The improved PANNS can be divided into two branches: the time domain branch (also called the time domain processing branch) and the frequency domain branch (also called the frequency domain processing branch). The input information of the time domain branch is the reference speech data, and the input information of the frequency domain branch is the reference speech spectral features corresponding to the reference speech data; the reference speech spectral features are obtained by converting the time domain signal "reference speech data" from the time domain to the frequency domain. Furthermore, as can be seen from the above description, in order to fully extract and characterize the voiceprint characteristics of a specified subject, the improved PANNS focuses on short-term correlation analysis of the reference speech data, that is, it supports processing using the improved PANNS through segmented input. Specifically, the improved PANNS processes only a segment of the reference speech data and the reference speech spectral features corresponding to this speech data at a time.

[0109] That is, each time the input information of the time domain branch is input, it is a segment of speech data in the reference speech data; similarly, each time the input information of the frequency domain branch is input, it is a segment of reference speech spectrum features in the frequency domain corresponding to a segment of speech data in the reference speech data. After the reference speech data is input into the improved PANNS, the reference speech data can be converted from the time domain to the frequency domain to obtain the reference speech spectrum features corresponding to the reference speech data, and the reference speech spectrum features can be segmented to obtain a reference speech spectrum feature segment corresponding to each speech data segment; and the reference speech data can be segmented to obtain multiple speech data segments corresponding to the reference speech data. It is worth noting that the segmentation rules followed by the segmentation processing of the reference speech data and the segmentation rules followed by the segmentation processing of the reference speech spectrum features are the same; for example, the segmentation rules may include: when the reference speech data is segmented, speech data of 20 milliseconds in length is periodically collected as a segment, and when the reference speech spectrum features are segmented, each reference speech spectrum feature segment converted to the time domain corresponds to a speech data segment of 20 milliseconds in length. By using this correspondence between time domain segmentation and frequency domain segmentation, the accuracy of each characteristic analysis is ensured.

[0110] Continuing with Figure 10, in the case of segmented input, the improved PANNS can perform short-term correlation analysis on each speech data segment based on each speech data segment and the corresponding reference speech spectrum feature segment, thereby obtaining a voiceprint semantic feature vector corresponding to each speech data segment; this voiceprint semantic feature vector is used to characterize the semantic characteristics of the corresponding speech data segment. For ease of explanation, the specific process of the short-term correlation analysis described above is described by taking any speech data segment among multiple speech data segments as the target speech data segment as an example, wherein:

[0111] ① The time-domain branch contains multiple one-dimensional convolutional layers (Conv 1D) and maximum pooling layers (Max pooling); where Conv is the abbreviation for convolutional, D is the abbreviation for dimension, the dimension of the maximum pooling layer is 1, and the stride (abbreviated as s) is a one-dimensional vector with a length of 4. As shown in Figure 10, assuming that the time-domain branch includes the following sequence: one-dimensional convolutional layer → one-dimensional convolutional block (Conv 1Dblock, composed of one or more one-dimensional convolutional layers) → maximum pooling layer (dimension 1, stride s is 4) → one-dimensional convolutional block → maximum pooling layer → one-dimensional convolutional block → maximum pooling layer; then, after the target speech data is segmented and feature extracted by the convolutional layer, the adjacent maximum pooling layer after the convolutional layer performs feature selection on the feature map after feature extraction, which is conducive to further extracting the desired features in the feature map. By progressively layering multiple one-dimensional convolutional layers and maximum pooling layers, the reference speech data (specifically, the target speech data segments) can be subjected to multiple feature extractions to obtain a one-dimensional sequence (resize), and the one-dimensional sequence is transformed (reshaped) to obtain multiple two-dimensional time domain feature maps (or two-dimensional spectrum Wavegrams). The time domain feature map is a graphical representation in the time domain that describes the changes of the reference speech signal in the reference speech data over time, and can intuitively display the basic characteristics of the reference speech signal (such as the periodicity, frequency components, and phase relationships of the reference speech signal). The purpose of the dimensionality conversion here is to enable the converted time domain feature map to be fused with the frequency domain feature map output by the frequency domain branch. In the above process, by using a large number of one-dimensional convolutional layers in the time domain branch, the time domain characteristics of the speech signal in the reference speech data (such as information such as audio loudness and sampling point amplitude) can be directly learned when the reference speech data is subjected to feature extraction processing through the time domain branch.

[0112] ② The frequency domain branch includes multiple two-dimensional convolution layers (Conv 2D) and maximum pooling layers (Max pooling); wherein, the dimension of the maximum pooling layer of the frequency domain branch is 2. As shown in Figure 10, it is assumed that the frequency domain branch includes in sequence: a two-dimensional convolution block (composed of one or more two-dimensional convolution layers) → a maximum pooling layer (dimension is 2) → a two-dimensional convolution block → a maximum pooling layer → a two-dimensional convolution block; through these multiple two-dimensional convolution layers and maximum pooling layers, feature extraction processing can be performed on the reference speech spectrum feature segment (Logmel) corresponding to the target speech data segment to obtain multiple frequency domain feature maps (Feature maps). The frequency domain feature map is a graphical representation of the frequency components of the reference speech signal in the reference speech data in the frequency domain, which can display the various frequency components contained in the reference speech signal and their corresponding amplitudes or intensities. It is worth noting that the feature dimensions of the frequency domain feature map here are the same as the feature dimensions of the time domain feature map output by the time domain branch. In the above process, by using a large number of two-dimensional convolutional layers in the frequency domain branch, the frequency domain characteristics of the speech signal in the reference speech data can be directly learned when the frequency domain branch is used to perform feature extraction processing on the reference speech spectrum feature segments.

[0113] ③ After obtaining the time-domain feature map (Wavegram) output by the time-domain branch and the frequency-domain feature map (Feature map) output by the frequency-domain branch based on the aforementioned steps, the time-domain feature map and the frequency-domain feature map can be fused to generate the voiceprint semantic feature vector corresponding to the target speech data segment. Specifically, as shown in Figure 10, multiple information exchanges occur between the time-domain branch and the frequency-domain branch. The information features of the time-domain branch are reshaped and then fused (concat) with the features of the frequency-domain branch. The fused result is convolved with a 2D convolution block and then input into a higher-level fusion module for fusion. Considering that time-domain processing captures the temporal correlation of the reference speech data, while frequency-domain processing captures the correlation of the reference speech data at different frequencies, these two domains belong to different domains. Therefore, the information exchange between the two domains (time and frequency) described above ensures that the time and frequency domains maintain information complementarity, enabling the higher-level network to perceive the underlying network information and thus fully learn from the reference speech data.

[0114] Based on this, assuming that the number of feature extraction processes performed in the time-domain branch and the frequency-domain branch is k, where k is an integer greater than 1, and any feature extraction process is represented as the i-th feature extraction process, then the fusion between the time-domain feature map output by the time-domain branch and the frequency-domain feature map output by the frequency-domain branch mentioned above can specifically include: First, when i = 1 (i.e., the first feature extraction process), fuse the intermediate time-domain feature map obtained by the time-domain branch in the first feature extraction process and the intermediate frequency-domain feature map obtained by the frequency-domain branch in the first feature extraction process to generate the first intermediate feature vector after the first feature extraction process. Then, when 1 < i ≤ k, fuse the intermediate time-domain feature map obtained by the time-domain branch in the i-th feature extraction process, the intermediate frequency-domain feature map obtained by the frequency-domain branch in the i-th feature extraction process, and the (i - 1)-th intermediate feature vector obtained by the (i - 1)-th feature extraction process to generate the i-th intermediate feature vector after the i-th feature extraction process. Finally, based on the k-th intermediate feature vector after the k-th feature extraction process when i = k, generate the acoustic semantic feature vector corresponding to the target semantic data segment. Among them, the specific generation process here can include: input the k-th intermediate feature vector into a two-dimensional convolutional neural network (2D CNN layers) for feature extraction to obtain a vector sequence, and use this vector sequence to perform an average operation to obtain the average value (mean) and a maximum operation to obtain the maximum value (max). Then, add the obtained average value and maximum value (sum) and then pass through an activation function (Rule) to obtain a feature vector (vector), and use a normalization function (softmax) to normalize this feature vector to convert the feature vector into an acoustic semantic feature vector representing the probability distribution, that is, the acoustic semantic feature vector corresponding to the target speech data segment.

[0115] In summary, the entire reference speech data is framed and separately input into the improved PANNS. The improved PANNS selects to fuse the last vector of the entire sequence and the mean value of the entire sequence together, thereby generating the final acoustic signature vector, and can obtain a multi-band semantic feature vector sequence representing the entire reference speech data (including the acoustic semantic feature vector corresponding to each speech data segment). Through the improved PANNS, short-term correlation analysis of the reference speech data can be achieved, fully learning the acoustic characteristics of each segmented speech signal in the reference speech data, and enabling the extracted acoustic signature vector to fully express the acoustic information of the specified object.

[0116] (2) Transformer network.

[0117] Considering that the improved PANNS focuses on short-term correlations, that is, features are calculated on a segmented basis; therefore, in order to fully learn the voiceprint characteristics of a specific subject from the reference speech data, the embodiment of this application also introduces a Transformer network to learn long-term correlation information from the reference speech data. The attention mechanism of the Transformer network can better focus on the voiceprint characteristics of the reference speech data with respect to the specific subject, and better realize the extraction of global feature information for the reference speech data.

[0118] For example, the schematic diagram of the network structure of the Transformer network can be seen in Figure 11; as shown in Figure 11, the Transformer network adopts an encoder (encoding)-decoder (decoding) architecture. Among them, both the encoder side and the decoder side are formed by stacking N encoder layers (i.e., "N×" in Figure 11). The encoder layer mainly includes two sub-layers, among which: the first sub-layer contains a multi-head attention mechanism (Multi-Head Attention), residual and normalization (Add&Norm); the second sub-layer contains a feedforward neural network (Feed Forward), residual and normalization layer. Among them, the multi-head attention mechanism contained in the first sub-layer can help obtain the contextual semantics of the voiceprint semantic feature vector corresponding to the speech data segment. The decoder layer mainly consists of three sub-layers, among which: the first sub-layer contains the masked multi-head attention mechanism (Masked Multi-Head Attention), residual and normalization layers, the first sub-layer contains the multi-head attention mechanism (Multi-Head Attention), residual and normalization (Add&Norm) layers, and the first sub-layer contains the feedforward neural network (Feed Forward), residual and normalization layers; the decoding layer can help obtain the key content that needs attention through this three-layer structure.

[0119] In the specific implementation, the input information on the encoder side of the Transformer network is the voiceprint semantic feature vector sequence (Input Embedding) output by the improved PANNS. This voiceprint semantic feature vector sequence contains the voiceprint semantic feature vector corresponding to each semantic data segment. Then, positional encoding (Positional Encoding) is performed on the input voiceprint semantic feature vector sequence to achieve preprocessing of the voiceprint semantic feature vector sequence. Positional encoding is a method of using the word's position information to represent each word in the vector sequence secondary, allowing the data input to the encoder to carry the word's position information. Furthermore, the position-encoded data is input to the encoder side, where the various layers of the encoder side (as described above in the relevant description of the encoder side structure) encode the input data to obtain the encoding result. Then, after obtaining the encoding result output by the encoder side, the decoder side combines the encoding result with the output features of the decoder side at the previous time (shifted right) as the input data for the decoder side to decode the input data. Finally, the output features of the decoding side undergo linear transformation (Linear) and classification (softmax) to obtain the voiceprint representation vector (Output Probabilities) of the specified object. Through the above process, the voiceprint semantic feature vector sequence can be calculated through the Transformer network, which can make the entire voiceprint semantic feature vector sequence more clearly express the long-term semantics of the entire reference speech data. In other words, the final output voiceprint representation vector can fully and clearly express the voiceprint characteristics of the specified object.

[0120] S805: Inputting the aliased speech data and the voiceprint representation vector into a preset speech segmentation model, where the speech segmentation model is used to segment a target speech signal that matches the voiceprint characteristics of a specified object from the aliased speech data based on an attention mechanism.

[0121] It should be noted that the specific implementation process shown in step S805 can be referred to the embodiment shown in Figure 3 above. The relevant description of the specific implementation process of step S302 on segmenting the voice signal that matches the voiceprint characteristics of the specified object from the mixed voice data based on the attention mechanism will not be repeated here.

[0122] From the relevant description shown in the aforementioned step S302, it can be seen that the embodiment of the present application mainly adopts the improved Unet network (i.e., the speech segmentation network) to implement the segmentation of the aliased speech data based on the attention mechanism, so as to obtain a speech signal that matches the voiceprint characteristics of the specified object. Moreover, the speech segmentation network mainly integrates the attention mechanism into the traditional Unet network, such as adding the attention mechanism to each network layer in the traditional Unet network to achieve an improvement on the traditional Unet network. Taking into account that more Attention mechanisms may be introduced into the speech segmentation system, thereby causing the number of parameters of the entire model to be large, the embodiment of the present application also supports model distillation of the speech segmentation model that integrates the attention mechanism to obtain a speech segmentation model after model distillation; in this way, the aforementioned correlation calculation can be implemented by the speech segmentation model after model distillation, thereby reducing the scale of the entire system and reducing the overall parameter amount and time consumption. The voiceprint representation vector and the speech spectrum feature are both expressed in vector form, so the method for the speech segmentation model after model distillation to calculate the correlation between the voiceprint representation vector and the speech spectrum feature may include but is not limited to the dot product method; wherein, the correlation between the voiceprint representation vector and the speech spectrum feature calculated by the dot product method includes: the product of the modulus of the voiceprint representation vector and the modulus of the speech spectrum feature, and the product of the cosine value of the angle between the voiceprint representation vector and the speech spectrum feature; if the voiceprint representation vector is The speech spectrum characteristics are The angle between the voiceprint representation vector and the speech spectrum feature is θ, so the dot product of the voiceprint representation vector and the speech spectrum feature is The dot product result is used as the similarity between the voiceprint representation vector and the speech spectrum feature. Among them, model distillation is a method of learning a large model (teacher model) with a large number of parameters to obtain a more compact small model (student model) with a smaller number of parameters. Model distillation for speech segmentation models mainly implements model distillation for large models through technologies such as pruning and knowledge distillation; among them: pruning is called model pruning (Model Pruning), which is a model compression technology that aims to reduce the complexity of the speech segmentation model by deleting some unimportant parameters or structures in the speech segmentation model, improve the inference speed of the speech segmentation model, and reduce the storage requirements of the speech segmentation model. Knowledge distillation can be understood as taking a relatively complex speech segmentation model as a teacher model, training a student model with a small number of parameters and a simple structure, and during the training process, the student model learns and imitates the output of the teacher model, so that the trained student model not only has the advantage of small computational complexity, but also has the same model performance as the teacher model; in the embodiment of the present application, the trained learning model is the speech segmentation model after knowledge increment. The model pruning and knowledge distillation mentioned above are the main implementation technologies of model distillation, but model distillation can also include other technologies. The embodiments of this application do not limit the specific implementation methods of model distillation.

[0123] S806: Generate a voice file of a designated object based on the segmented target voice signal.

[0124] It should be noted that, for the specific implementation process shown in step S806, reference may be made to the relevant description of the specific implementation process shown in step S304 in the embodiment shown in FIG3 , and no further details will be given here.

[0125] In summary, the embodiment of the present application provides a reusable method for speech segmentation of a specified object with voiceprint vector embedding. The method uses a voiceprint vector extraction model composed of an improved PANNS and a Transformer network to extract the voiceprint of a small segment of reference speech data of the specified object. By combining the short-term correlation analysis of the improved PANNS and the long-term correlation analysis of the transformer network, the extracted voiceprint representation vector of the specified object can more fully and clearly express the voiceprint characteristics of the specified object. In addition, the embodiment of the present application also integrates the attention mechanism into the Unet network, so that the Unet network (i.e., the speech segmentation model) integrated with the attention mechanism can realize the segmentation of aliased speech data based on the attention mechanism, and can more clearly and accurately calculate and extract the speech signal of the specified object in the aliased speech data, ensure the purity of the extracted speech signal, and achieve a more accurate and pure speech separation effect. In addition, the embodiment of the present application only needs to obtain the reference voice data of the specified object to segment the voice signal of the specified object from the mixed voice data; if you want to obtain the voice data of other objects, you only need to change the voiceprint characteristics of the object to be segmented. There is no need to train a dedicated network for each object, which makes the solution convenient and portable, making this solution extremely versatile.

[0126] The above describes in detail the method of the embodiment of the present application. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0127] FIG12 shows a schematic diagram of the structure of a speech processing device provided by an exemplary embodiment of the present application; the speech processing device can be used to perform some or all of the steps in the method embodiments shown in FIG3 or FIG8. Referring to FIG12, the speech processing device includes the following units:

[0128] An acquiring unit 1201 is configured to acquire aliased speech data, where the aliased speech data includes speech signals generated by each of at least two objects;

[0129] The acquisition unit 1201 is further configured to acquire reference speech data of a designated object; the designated object refers to any one of at least two objects; the reference speech data includes a reference speech signal of the designated object;

[0130] The processing unit 1202 is configured to extract a voiceprint representation vector of a specified subject from the reference speech data, wherein the voiceprint representation vector is used to represent the voiceprint characteristics of the specified subject;

[0131] The processor 1202 is further configured to input the aliased speech data and the voiceprint representation vector into a preset speech segmentation model, wherein the speech segmentation model is configured to: segment a target speech signal that matches the voiceprint characteristics from the aliased speech data based on an attention mechanism;

[0132] The processing unit 1202 is further configured to generate a voice file of a designated object based on the segmented target voice signal.

[0133] In one implementation, the process of segmenting the target speech signal that matches the voiceprint characteristics from the aliased speech data based on the attention mechanism includes:

[0134] Convert the aliased speech data from the time domain to the frequency domain to obtain the speech spectrum features corresponding to the aliased speech data; the speech spectrum features are the characteristic performance of the aliased speech data in the frequency domain;

[0135] The voiceprint representation vector and the speech spectrum feature are correlated based on the attention mechanism to obtain a speech spectrum feature segment that matches the voiceprint feature; the speech spectrum feature segment is a segment of the speech spectrum feature that matches the voiceprint feature;

[0136] The speech spectrum feature segments are converted from the frequency domain to the time domain to obtain the target speech signal that matches the voiceprint characteristics.

[0137] In one implementation, the correlation calculation is implemented by a speech segmentation model; the speech segmentation model includes a feature extraction subnetwork and an upsampling subnetwork, and the feature extraction subnetwork and the upsampling subnetwork are connected by a convolutional connection layer;

[0138] The feature extraction subnetwork and the upsampling subnetwork are symmetrical; the feature extraction subnetwork contains m convolutional layers distributed in a hierarchical manner, and the upsampling subnetwork contains an upsampling layer corresponding to each of the m convolutional layers, where m is a positive integer; the convolutional layer, the convolutional connection layer, and the upsampling layer all include multiple convolutional networks connected sequentially;

[0139] Among them, the attention mechanism is integrated into all or part of the network layers in the speech segmentation model, and the fusion position of the attention mechanism in the multiple convolutional networks included in the network layer is not fixed; the network layer includes a convolutional layer, an upsampling layer and a convolutional connection layer.

[0140] In one implementation, each network layer in the speech segmentation model incorporates an attention mechanism; the processing unit 1202 is configured to calculate the correlation between the voiceprint representation vector and the speech spectrum features based on the attention mechanism to obtain speech spectrum feature segments that match the voiceprint characteristics, specifically for:

[0141] Input the voiceprint representation vector into each network layer in the speech segmentation model;

[0142] Based on the attention mechanism integrated in each network layer, the correlation between the voiceprint representation vector and the first feature map of the corresponding network layer is calculated to obtain the second feature map output by the corresponding network layer; the voiceprint characteristics represented by the second feature map match the voiceprint characteristics represented by the voiceprint representation vector;

[0143] The second feature map output by the 2m+1th network layer in the speech segmentation model is used as the speech spectrum feature segmentation that matches the voiceprint characteristics represented by the voiceprint representation vector; the 2m+1th network layer in the speech segmentation model is the last upsampling layer in the upsampling subnetwork.

[0144] In one implementation, any network layer in a speech segmentation model that incorporates an attention mechanism is represented as a target network layer; the target network layer is a convolutional layer or a convolutional-connected layer; the fusion position of the attention mechanism in the target network layer is: between the first convolutional network and the second convolutional network adjacent to the first convolutional network in the multiple sequentially connected convolutional networks included in the target network layer;

[0145] The processing unit 1202 is configured to calculate the correlation between the voiceprint representation vector and the first feature map of the corresponding network layer based on the attention mechanism integrated in each network layer to obtain the second feature map output by the corresponding network layer, specifically for:

[0146] The first convolutional network in the target network layer is used to perform feature extraction processing on the first feature map of the target network layer to obtain a third feature map of the target network layer; wherein, when the target network layer is the first convolutional layer of the hierarchical distribution in the feature extraction subnetwork, the first feature map of the target network layer is the speech spectrum feature; when the target network layer is a convolutional layer other than the first convolutional layer in the speech segmentation model, the first feature map of the target network layer is obtained by pooling the feature map output by the upper-level network layer adjacent to the target network layer;

[0147] According to the attention mechanism integrated in the target network layer, the correlation between the voiceprint representation vector and the third feature map of the target network layer is calculated to obtain the fourth feature map of the target network layer; the feature dimension of the third feature map is the same as that of the fourth feature map;

[0148] The fourth feature map is subjected to feature extraction processing by using other convolutional networks except the first convolutional network in the target network layer to obtain the second feature map output by the target network layer.

[0149] In one implementation, any network layer in a speech segmentation model that incorporates an attention mechanism is represented as a target network layer; the target network layer is an upsampling layer; and the fusion position of the attention mechanism in the multiple convolutional networks corresponding to the target network layer is the position after the last convolutional network in the multiple convolutional networks sequentially connected in the target network layer.

[0150] The processing unit 1202 is configured to calculate the correlation between the voiceprint representation vector and the first feature map of the corresponding network layer based on the attention mechanism integrated in each network layer to obtain the second feature map output by the corresponding network layer, specifically for:

[0151] Using multiple convolutional networks connected sequentially in the target network layer, feature extraction is performed on the target feature map to obtain the first feature map of the target network layer; the target feature map is obtained by concatenating the feature map output by the convolutional layer corresponding to the target network layer in the feature extraction subnetwork and the feature map output by the network layer above the target network layer;

[0152] The attention mechanism integrated in the target network layer is used to calculate the correlation between the voiceprint representation vector and the first feature map of the target network layer to obtain the second feature map output by the target network layer; the feature dimension of the second feature map is the same as that of the first feature map.

[0153] In one implementation, the processing unit 1202 is further configured to:

[0154] Performing dimension transformation on the voiceprint representation vector to obtain a dimension-transformed voiceprint representation vector;

[0155] Among them, the feature dimension of the voiceprint representation vector after dimension transformation is the same as the feature dimension of the feature map of the attention mechanism to be input into the corresponding network layer for fusion.

[0156] In one implementation, if the number of network layers integrated with the attention mechanism in the speech segmentation model is greater than a threshold, the processing unit 1202 is further configured to:

[0157] Perform model distillation on the speech segmentation model to obtain a speech segmentation model after model distillation;

[0158] Among them, the correlation calculation is realized by the speech segmentation model after model distillation.

[0159] In one implementation, the processing unit 1202 is configured to extract the voiceprint representation vector of the specified object from the reference speech data, specifically to:

[0160] Segmenting the reference voice data to obtain a plurality of voice data segments corresponding to the reference voice data;

[0161] Convert the reference speech data from the time domain to the frequency domain to obtain reference speech spectrum features corresponding to the reference speech data;

[0162] Segmenting the reference speech spectrum features to obtain reference speech spectrum feature segments corresponding to each speech data segment;

[0163] Based on each segment of speech data and the corresponding segment of reference speech spectral features respectively, perform short-term correlation analysis on each segment of speech data to obtain the voiceprint semantic feature vector corresponding to each segment of speech data; the voiceprint semantic feature vector is used to characterize the semantic characteristics of the segment of speech data.

[0164] Perform long-term correlation analysis on the sequence of voiceprint semantic feature vectors to obtain the voiceprint representation vector of the specified object; the sequence of voiceprint semantic feature vectors includes the voiceprint semantic feature vectors corresponding to each segment of speech data.

[0165] In one implementation, any segment of speech data among multiple segments of speech data is represented as the target segment of speech data; the processing unit 1202, when performing short-term correlation analysis on each segment of speech data based on each segment of speech data and the corresponding segment of reference speech spectral features respectively to obtain the voiceprint semantic feature vector corresponding to each segment of speech data, is specifically used for:

[0166] Perform feature extraction processing on the target segment of speech data to obtain a time-domain feature map.

[0167] Perform feature extraction processing on the segment of reference speech spectral features corresponding to the target segment of speech data to obtain a frequency-domain feature map.

[0168] Perform fusion processing on the time-domain feature map and the frequency-domain feature map to generate the voiceprint semantic feature vector corresponding to the target segment of speech data.

[0169] In one implementation, the number of times of feature extraction processing is k times, where k is an integer greater than 1; any one time of feature extraction processing is represented as the i-th time of feature extraction processing; the processing unit 1202, when performing fusion processing on the time-domain feature map and the frequency-domain feature map to generate the voiceprint semantic feature vector corresponding to the target segment of speech data, is specifically used for:

[0170] When i = 1, perform fusion processing on the intermediate time-domain feature map and the intermediate frequency-domain feature map obtained by the first time of feature extraction processing to generate the first intermediate feature vector after the first time of feature extraction processing.

[0171] When 1 < i ≤ k, perform fusion processing on the intermediate time-domain feature map and the intermediate frequency-domain feature map obtained by the i-th time of feature extraction processing, and the (i - 1)-th intermediate feature vector obtained by the (i - 1)-th time of feature extraction processing to generate the i-th intermediate feature vector after the i-th time of feature extraction processing.

[0172] Based on the k-th intermediate feature vector after the k-th time of feature extraction processing when i = k, generate the voiceprint semantic feature vector corresponding to the target segment of speech data.

[0173] According to one embodiment of the present application, the various units in the speech processing device shown in Figure 12 can be separately or completely merged into one or several other units to constitute, or one (some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the speech processing device can also include other units. In practical applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, a computer program (including program code) capable of executing the steps involved in the corresponding methods shown in Figures 3 and 8 can be run on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). To construct a speech processing device as shown in Figure 12, and to implement the speech processing method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the above-mentioned computing device via the computer-readable recording medium, and executed therein.

[0174] Based on the same inventive concept, the principles and beneficial effects of the speech processing device provided in the embodiment of the present application for solving the problem are similar to the principles and beneficial effects of the speech processing method in the method embodiment of the present application for solving the problem. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0175] Figure 13 shows a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application. Referring to Figure 13, the computer device includes a processor 1301, a communication interface 1302 and a computer-readable storage medium 1303. The processor 1301, the communication interface 1302 and the computer-readable storage medium 1303 can be connected via a bus or other means. The communication interface 1302 is used to receive and send data. The computer-readable storage medium 1303 can be stored in the memory of the computer device, the computer-readable storage medium 1303 is used to store computer programs, and the processor 1301 is used to execute the computer programs stored in the computer-readable storage medium 1303. The processor 1301 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more computer programs, and is specifically suitable for loading and executing one or more computer programs to implement the corresponding method flow or corresponding function.

[0176] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more computer programs suitable for being loaded and executed by the processor 1301 are also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0177] In one embodiment, the computer device may be the terminal or server mentioned in the aforementioned embodiment; one or more computer programs are stored in the computer-readable storage medium; the processor 1301 loads and executes the one or more computer programs stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned speech processing method embodiment; in a specific implementation, the one or more computer programs in the computer-readable storage medium are loaded and executed by the processor 1301 to execute the steps of each embodiment of the present application; wherein, the steps of each embodiment of the present application can refer to the relevant descriptions of the aforementioned embodiments, and will not be repeated here.

[0178] Based on the same inventive concept, the principles and beneficial effects of solving the problem by the computer device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problem by the speech processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0179] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned speech processing method.

[0180] Those skilled in the art will appreciate that the units and algorithmic steps of each example described in conjunction with the embodiments disclosed in this application can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes a computer program (one or more). When the computer program is loaded and executed on a computer device, the computer program executes the above-mentioned process or function of the embodiment of the present application. The computer device can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer program can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer program can be transmitted from a website site, computer device, server or data center to another website site, computer device, server or data center by wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer device can access or a data storage device such as a server or data center that includes one or more available media integrations. Available media can be magnetic media (for example, floppy disk, hard disk, tape), optical media (for example, DVD) or semiconductor media (for example, solid-state drive (SSD)) etc.

[0182] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A speech processing method, characterized in that: The method is performed by a computer device, and the method comprises: Acquire aliased speech data, where the aliased speech data includes a speech signal generated by each of at least two objects; Acquire reference speech data of a designated object, wherein the designated object refers to any one of the at least two objects; the reference speech data includes a reference speech signal of the designated object; Extracting a voiceprint representation vector of the designated object from the reference voice data, wherein the voiceprint representation vector is used to represent the voiceprint characteristics of the designated object; Inputting the aliased speech data and the voiceprint representation vector into a preset speech segmentation model, wherein the speech segmentation model is used to: segment a target speech signal matching the voiceprint characteristics from the aliased speech data based on an attention mechanism; A voice file of the designated object is generated based on the segmented target voice signal.

2. The method according to claim 1, characterized in that The process of segmenting a target speech signal matching the voiceprint characteristics from the aliased speech data based on an attention mechanism includes: Convert the aliased speech data from the time domain to the frequency domain to obtain speech spectrum features corresponding to the aliased speech data; the speech spectrum features are characteristic performances of the aliased speech data in the frequency domain; Based on the attention mechanism, the correlation between the voiceprint representation vector and the speech spectrum feature is calculated to obtain a speech spectrum feature segment matching the voiceprint feature; the speech spectrum feature segment is a segment in the speech spectrum feature that matches the voiceprint feature; The speech spectrum feature segments are converted from the frequency domain to the time domain to obtain the target speech signal matching the voiceprint characteristics.

3. The method according to claim 1 or 2, characterized in that The correlation calculation is implemented by the speech segmentation model; the speech segmentation model includes a feature extraction subnetwork and an upsampling subnetwork, and the feature extraction subnetwork and the upsampling subnetwork are connected via a convolutional connection layer; The feature extraction subnetwork and the upsampling subnetwork are symmetrical; the feature extraction subnetwork includes m convolutional layers distributed in a hierarchy, and the upsampling subnetwork includes an upsampling layer corresponding to each of the m convolutional layers, where m is a positive integer; the convolutional layer, the convolutional connection layer, and the upsampling layer all include a plurality of convolutional networks connected in sequence; Among them, the attention mechanism is integrated into all or part of the network layers in the speech segmentation model, and the fusion position of the attention mechanism in the multiple convolutional networks included in the network layer is not fixed; the network layer includes the convolutional layer, the upsampling layer and the convolutional connection layer.

4. The method according to any one of claims 1 to 3, characterized in that: Each network layer in the speech segmentation model is integrated with the attention mechanism; the correlation between the voiceprint representation vector and the speech spectrum feature is calculated based on the attention mechanism to obtain a speech spectrum feature segment matching the voiceprint characteristic, including: Inputting the voiceprint representation vector into each of the network layers in the speech segmentation model; Based on the attention mechanism integrated in each of the network layers, a correlation is calculated between the voiceprint representation vector and the first feature map of the corresponding network layer to obtain a second feature map output by the corresponding network layer; the voiceprint characteristics represented by the second feature map match the voiceprint characteristics represented by the voiceprint representation vector; The second feature map output by the 2m+1th network layer in the speech segmentation model is used as a speech spectrum feature segmentation that matches the voiceprint characteristics represented by the voiceprint representation vector; the 2m+1th network layer is the last upsampling layer in the upsampling subnetwork.

5. The method according to any one of claims 1 to 4, characterized in that: Any network layer in the speech segmentation model that is integrated with the attention mechanism is represented as a target network layer; the target network layer is the convolution layer or the convolution connection layer; the fusion position of the attention mechanism in the target network layer is: the position between the first convolution network and the second convolution network adjacent to the first convolution network among the multiple convolution networks connected sequentially included in the target network layer; The method of calculating the correlation between the voiceprint representation vector and the first feature map of the corresponding network layer based on the attention mechanism integrated in each network layer to obtain the second feature map output by the corresponding network layer includes: The first convolutional network in the target network layer is used to perform feature extraction processing on the first feature map of the target network layer to obtain a third feature map of the target network layer; wherein, when the target network layer is the first convolutional layer hierarchically distributed in the feature extraction subnetwork, the first feature map of the target network layer is the speech spectrum feature; when the target network layer is the other convolutional layer except the first convolutional layer in the speech segmentation model, the first feature map of the target network layer is obtained by pooling the feature map output by the network layer of the previous level adjacent to the target network layer; According to the attention mechanism integrated in the target network layer, the correlation between the voiceprint representation vector and the third feature map of the target network layer is calculated to obtain a fourth feature map of the target network layer; the feature dimension of the third feature map is the same as the feature dimension of the fourth feature map; The fourth feature map is subjected to feature extraction processing by using other convolutional networks in the target network layer except the first convolutional network to obtain a second feature map output by the target network layer.

6. The method according to any one of claims 1 to 5, characterized in that: Any network layer in the speech segmentation model that is integrated with the attention mechanism is represented as a target network layer; the target network layer is the upsampling layer; the fusion position of the attention mechanism in the target network layer is: the position after the last convolutional network in the multiple convolutional networks sequentially connected in the target network layer; The method of calculating the correlation between the voiceprint representation vector and the first feature map of the corresponding network layer based on the attention mechanism integrated in each network layer to obtain the second feature map output by the corresponding network layer includes: Using multiple convolutional networks connected sequentially in the target network layer, feature extraction processing is performed on the target feature map to obtain a first feature map of the target network layer; the target feature map is obtained by feature splicing the feature map output by the convolutional layer corresponding to the target network layer in the feature extraction subnetwork and the feature map output by the upper-level network layer of the target network layer; The attention mechanism integrated in the target network layer is used to calculate the correlation between the voiceprint representation vector and the first feature map of the target network layer to obtain a second feature map output by the target network layer; the feature dimension of the second feature map is the same as the feature dimension of the first feature map.

7. The method according to any one of claims 1 to 6, characterized in that: Before calculating the correlation between the voiceprint representation vector and the first feature map of the corresponding network layer based on the attention mechanism integrated in each network layer to obtain the second feature map output by the corresponding network layer, the method further includes: Performing dimension transformation on the voiceprint representation vector to obtain the voiceprint representation vector after dimension transformation; Among them, the feature dimension of the voiceprint representation vector after dimension transformation is the same as the feature dimension of the feature map of the attention mechanism to be input into the corresponding network layer for fusion.

8. The method according to any one of claims 1 to 7, characterized in that: If the number of network layers in the speech segmentation model that incorporate the attention mechanism is greater than a quantity threshold, the method further includes: Performing model distillation on the speech segmentation model to obtain a speech segmentation model after model distillation; The correlation calculation is implemented by the speech segmentation model after model distillation.

9. The method according to any one of claims 1 to 8, characterized in that The step of extracting the voiceprint representation vector of the designated object from the reference voice data comprises: Segmenting the reference voice data to obtain a plurality of voice data segments corresponding to the reference voice data; Converting the reference speech data from the time domain to the frequency domain to obtain reference speech spectrum features corresponding to the reference speech data; The reference speech spectrum feature is segmented to obtain a reference speech spectrum feature segment corresponding to each speech data segment; Based on each of the speech data segments and the corresponding reference speech spectrum feature segments, short-term correlation analysis is performed on each of the speech data segments to obtain a voiceprint semantic feature vector corresponding to each of the speech data segments; the voiceprint semantic feature vector is used to characterize the semantic characteristics of the speech data segment; A long-term correlation analysis is performed on the voiceprint semantic feature vector sequence to obtain a voiceprint representation vector of the specified object; the voiceprint semantic feature vector sequence includes a voiceprint semantic feature vector corresponding to each of the speech data segments.

10. The method according to any one of claims 1 to 9, characterized in that: Any one of the plurality of speech data segments is represented as a target speech data segment; and the short-term correlation analysis is performed on each of the speech data segments based on each of the speech data segments and the corresponding reference speech spectrum feature segment to obtain a voiceprint semantic feature vector corresponding to each of the speech data segments, including: Performing feature extraction processing on the target speech data segments to obtain a time domain feature map; Feature extraction processing is performed on the reference speech spectrum feature segments corresponding to the segmented target speech data to obtain a frequency-domain feature map; The time-domain feature map and the frequency-domain feature map are fused to generate a voiceprint semantic feature vector corresponding to the segmented target speech data.

11. The method according to any one of claims 1 to 10, characterized in that: The number of times of the feature extraction processing is k times, where k is an integer greater than 1; any one-time feature extraction processing is denoted as the i-th feature extraction processing; The fusing the time-domain feature map and the frequency-domain feature map to generate a voiceprint semantic feature vector corresponding to the segmented target speech data includes: When i = 1, the intermediate time-domain feature map and the intermediate frequency-domain feature map obtained by the first feature extraction processing are fused to generate a first intermediate feature vector after the first feature extraction processing; When 1 < i ≤ k, the intermediate time-domain feature map and the intermediate frequency-domain feature map obtained by the i-th feature extraction processing, and the (i - 1)-th intermediate feature vector obtained by the (i - 1)-th feature extraction processing are fused to generate the i-th intermediate feature vector after the i-th feature extraction processing; Based on the k-th intermediate feature vector after the k-th feature extraction processing when i = k, a voiceprint semantic feature vector corresponding to the segmented target speech data is generated.

12. A speech processing device, characterized in that: The voice processing device is mounted on a computer device, and the voice processing device includes: An acquisition unit for acquiring aliased speech data, where the aliased speech data contains speech signals generated by each of at least two objects; The acquisition unit is further configured to acquire reference speech data of a specified object, where the specified object refers to any one of the at least two objects; the reference speech data contains a reference speech signal of the specified object; A processing unit for extracting a voiceprint characterization vector of the specified object from the reference speech data, where the voiceprint characterization vector is used to characterize the voiceprint characteristics of the specified object; The processing unit is further configured to input the aliased speech data and the voiceprint characterization vector into a preset speech segmentation model, where the speech segmentation model is configured to: based on an attention mechanism, segment out a target speech signal matching the voiceprint characteristics from the aliased speech data; The processing unit is further configured to generate a voice file of the specified object based on the segmented target speech signal.

13. A computer device, characterized in that A processor adapted to execute a computer program; A computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the voice processing method according to any one of claims 1-11 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is adapted to be loaded and executed by the processor to implement the voice processing method according to any one of claims 1-11.

15. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by the processor, the voice processing method according to any one of claims 1-11 is implemented.

Images