Target speaker voice extraction result error correction method and related device

Abstract

The embodiment of the invention provides a target speaker voice extraction result error correction method and a related device, and relates to the technical field of voice signal processing. The method comprises the following steps: inputting registered voice and mixed voice into a double-branch voice extraction model for voice extraction to obtain interference voice and object voice; the object similarity and the interference similarity are calculated according to the voiceprint features of the registration voice, the interference voice and the object voice, and an error correction identifier is determined based on the object similarity and the interference similarity. And when error correction is carried out according to the error correction identifier, obtaining an error correction voice based on the interference voice or the object voice, and taking the error correction voice as a target voice. The auxiliary information is obtained through the association between the object voice and the interference voice to judge whether error correction is needed or not, possible error results are explicitly corrected into correct output, a model structure does not need to be greatly improved, a training mechanism does not need to be optimized, and a data set does not need to be expanded. And compared with a method for further improving the precision aiming at a correct result, the learning cost is lower, and the recognition accuracy is higher.

Description

Technical Field

[0001] The present application relates to the technical field of speech signal processing, and in particular to a method for correcting the results of speech extraction of a target speaker and a related device. Background Art

[0002] Target speaker extraction refers to extracting the clear target speaker's voice from the mixed voice signal by using pre-registered information representing the speaker's identity and voice separation. Compared with traditional voice separation technology, target speaker extraction technology focuses more on the specific speaker and is often more flexible.

[0003] In related technologies, the learning process of deep models used for target speaker speech extraction usually focuses on improving the direct extraction performance of the model, such as improving the recognition accuracy of the model by improving the model structure, optimizing the training mechanism, and expanding the data set. However, when the recognition accuracy reaches a certain level, the learning cost required to further improve the model accuracy becomes high, which makes it difficult to break through the current bottleneck stage of recognition accuracy. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and a related device for correcting the error of the target speaker's speech extraction result, so as to improve the recognition accuracy of the target speaker's speech extraction result without significantly increasing the learning cost.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for correcting a target speaker's speech extraction result, comprising:

[0006] Acquire a mixed voice and a registered voice of a target object, wherein the mixed voice at least includes an initial voice of the target object;

[0007] Inputting the registered speech and the mixed speech into a pre-trained dual-branch speech extraction model for speech extraction to obtain interference speech and target speech;

[0008] Calculating object similarity and interference similarity according to the voiceprint features of the registered voice, the interference voice and the object voice, and determining an error correction flag based on the object similarity and the interference similarity;

[0009] If the error correction flag indicates that the extraction result is directly output, the object speech is used as the target speech corresponding to the initial speech; if the error correction flag indicates that the extraction result is to be corrected, the error-corrected speech is obtained based on the interference speech or the object speech, and the error-corrected speech is used as the target speech corresponding to the initial speech.

[0010] In some embodiments, the calculating the object similarity and the interference similarity according to the voiceprint features of the registration voice, the interference voice and the object voice includes:

[0011] Acquire the registration voiceprint feature of the registration voice, the interference voiceprint feature of the interference voice, and the object voiceprint feature of the object voice;

[0012] The similarity between the registered voiceprint feature and the interference voiceprint feature is calculated to obtain the interference similarity, and the similarity between the registered voiceprint feature and the object voiceprint feature is calculated to obtain the object similarity.

[0013] In some embodiments, determining the error correction flag based on the object similarity and the interference similarity includes:

[0014] When the object similarity is greater than or equal to the interference similarity, the error correction flag is used to indicate to directly output the extraction result;

[0015] When the object similarity is less than the interference similarity, the error correction flag is used to indicate to correct the extraction result.

[0016] In some embodiments, obtaining the error correction speech based on the interfering speech or the target speech includes:

[0017] Select one of the interference speech and the target speech as a new registration speech, input the new registration speech and the mixed speech into the dual-branch speech extraction model for speech extraction, obtain a new target speech or a new interference speech, and select the error correction speech from the new target speech or the new interference speech;

[0018] Alternatively, the interfering speech is used as the error correction speech.

[0019] In some embodiments, the step of selecting the error correction speech from the new target speech or the new interfering speech includes:

[0020] When the interfering speech is used as a new registration speech, the new target speech is used as the error correction speech;

[0021] When the target speech is used as a new registration speech, the new interference speech is used as the error correction speech.

[0022] In some embodiments, the dual-branch speech extraction model includes a speaker encoder, a hybrid speech encoder, a fusion module, a target branch extraction module, and an interference branch extraction module; the step of inputting the registration speech and the hybrid speech into the pre-trained dual-branch speech extraction model for speech extraction to obtain the interference speech and the object speech includes:

[0023] Inputting the registered speech into the speaker encoder for feature extraction to obtain registered speech features, and inputting the mixed speech into the mixed speech encoder for feature extraction to obtain mixed speech features;

[0024] Inputting the registered speech feature and the mixed speech feature into the fusion module for feature fusion to obtain a fused speech feature;

[0025] The fused speech features and the mixed speech features are input into the target branch extraction module for feature extraction to obtain the object speech, and the fused speech features and the mixed speech features are input into the interference branch extraction module for feature extraction to obtain the interference speech.

[0026] In some embodiments, the target branch extraction module includes a target trunk network and a target mask extraction module; the step of inputting the fused speech feature and the mixed speech feature into the target branch extraction module for feature extraction to obtain the target speech includes:

[0027] Inputting the fused speech features into the target backbone network for feature extraction to obtain potential features;

[0028] Inputting the potential features into the target mask extraction module for feature extraction to obtain a speech mask;

[0029] The mixed speech feature is masked by using the speech mask to obtain the object speech feature, and the object speech feature is decoded to obtain the object speech.

[0030] To achieve the above-mentioned purpose, the second aspect of the embodiment of the present application proposes a target speaker speech extraction result correction device, comprising:

[0031] A voice acquisition module: used to acquire a mixed voice and a registered voice of a target object, wherein the mixed voice at least includes an initial voice of the target object;

[0032] Feature extraction module: used for inputting the registered speech and the mixed speech into a pre-trained dual-branch speech extraction model for speech extraction to obtain interference speech and target speech;

[0033] The error correction judgment module is used to calculate the object similarity and the interference similarity according to the voiceprint features of the registered voice, the interference voice and the object voice, and determine the error correction mark based on the object similarity and the interference similarity;

[0034] A speech error correction module: used to, if the error correction flag indicates that the extraction result is directly output, use the object speech as the target speech corresponding to the initial speech; if the error correction flag indicates that the extraction result is to be corrected, obtain the error-corrected speech based on the interfering speech or the object speech, and use the error-corrected speech as the target speech corresponding to the initial speech.

[0035] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0036] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0037] The target speaker speech extraction result correction method and related device proposed in the embodiment of the present application are obtained by obtaining a mixed speech and a registered speech of a target object, wherein the mixed speech at least contains the initial speech of the target object. The registered speech and the mixed speech are input into a pre-trained dual-branch speech extraction model for speech extraction to obtain an interference speech and an object speech. The object similarity and the interference similarity are calculated according to the voiceprint features of the registered speech, the interference speech and the object speech, and the error correction mark is determined based on the object similarity and the interference similarity. If the error correction mark indicates that the extraction result is directly output, the object speech is used as the target speech corresponding to the initial speech. If the error correction mark indicates that the extraction result is corrected, the error correction speech is obtained based on the interference speech or the object speech, and the error correction speech is used as the target speech corresponding to the initial speech. The embodiment of the present application extracts the speech of the target speaker through a dual-branch speech coding process, not only identifies the object speech from the mixed speech, but also identifies the interference speech at the same time, obtains auxiliary information through the association between the object speech and the interference speech, thereby judging whether the result error correction is needed, and when error correction is needed, the final target speech is obtained based on the error correction speech corresponding to the interference speech and the object speech. In this way, we can focus on possible extraction failures and correct possible erroneous results into correct outputs without significantly improving the model structure, optimizing the training mechanism, and expanding the data set. This is lower than the learning cost of further improving the accuracy of the correct results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of a method for correcting the target speaker's speech extraction results provided in an embodiment of the present application.

[0039] Figure 2 It is a flowchart of the target speaker extraction task provided in an embodiment of the present application.

[0040] Figure 3 It is a structural diagram of the dual-branch speech extraction model provided in the embodiment of the present application.

[0041] Figure 4 The embodiment of the present application provides a flowchart for inputting the registered speech and the mixed speech into a pre-trained dual-branch speech extraction model for speech extraction to obtain the interference speech and the target speech.

[0042] Figure 5 This is a flowchart provided by an embodiment of the present application for inputting fused speech features and mixed speech features into a target branch extraction module for feature extraction to obtain the target speech.

[0043] Figure 6 It is a flowchart provided in an embodiment of the present application for calculating object similarity and interference similarity based on the voiceprint features of the registered voice, the interference voice and the object voice.

[0044] Figure 7 It is a flowchart of obtaining error-corrected speech based on interfering speech or object speech provided in an embodiment of the present application.

[0045] Figure 8 It is a structural block diagram of a target speaker speech extraction result correction device provided by another embodiment of the present application.

[0046] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] First, some nouns involved in this application are analyzed:

[0051] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0052] In many application scenarios, such as teleconferencing, video conferencing, voice monitoring, and speech recognition, users only focus on the sound emitted by a specific object. However, in most real-world scenarios, pure speech is difficult to obtain, and users are often faced with speech signals mixed with the voices of multiple speakers and environmental noise. In other words, speech signal processing is needed to identify the target speaker.

[0053] Target speaker extraction refers to extracting the clear target speaker's voice from the mixed speech signal by using pre-registered information representing the speaker's identity and voice separation.

[0054] The target speaker extraction task has many application scenarios, such as improving speech recognition accuracy or enhancing audio conferencing. In speech recognition tasks, the purity of speech greatly affects the accuracy of recognition. As an upstream task of speech recognition, the target speaker extraction task can extract pure speech through preprocessing, significantly improve the accuracy and stability of the automatic speech recognition system, and effectively avoid interference from environmental noise and other speakers. Or in scenarios such as teleconferencing and video conferencing, target speaker extraction can improve communication quality by separating and amplifying the voices of individual participants, thereby achieving a smoother and more efficient meeting experience.

[0055] In related technologies, the learning process of deep models used for target speaker speech extraction usually focuses on improving the direct extraction performance of the model, such as improving the recognition accuracy of the model by improving the model structure, optimizing the training mechanism, and expanding the data set. However, when the recognition accuracy reaches a certain level, the learning cost required to further improve the model accuracy becomes high, which makes it difficult to break through the current bottleneck stage of recognition accuracy.

[0056] Based on this, the embodiment of the present application provides a method and related device for correcting the result of speech extraction of a target speaker, which extracts the speech of the target speaker through a dual-branch speech coding process, not only identifies the target speech from the mixed speech, but also identifies the interference speech at the same time, and obtains auxiliary information through the association between the target speech and the interference speech, so as to determine whether the result correction is needed. When correction is needed, the final target speech is obtained based on the correction speech corresponding to the interference speech and the target speech. In this way, attention is paid to possible extraction failures, and possible erroneous results are corrected to correct outputs, without significantly improving the model structure, optimizing the training mechanism, and expanding the data set. Compared with further improving the accuracy of the correct results, the learning cost is lower.

[0057] The embodiments of the present application provide a method for correcting the target speaker's speech extraction result and a related device, which are specifically described through the following embodiments. First, the method for correcting the target speaker's speech extraction result in the embodiments of the present application is described.

[0058] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0059] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0060] The target speaker speech extraction result correction method provided in the embodiment of the present application relates to the field of speech signal processing technology. The target speaker speech extraction result correction method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be a computer program running in a terminal or a server side. For example, the computer program can be a native program or software module in an operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports the target speaker speech extraction result correction, or it can be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form. Among them, the terminal communicates with the server via a network. The target speaker speech extraction result correction method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0061] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the target speaker voice extraction result correction method of this embodiment to provide related services and improve the driving experience. The server may be an independent server, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, and each service node in the blockchain system forms a peer-to-peer (P2P, Peer To Peer, P2P) network, and the P2P protocol is an application layer protocol running on the Transmission Control Protocol (Transmission Control Protocol, TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (Universal Serial Bus, USB) or network communication connection methods, and this embodiment is not limited here.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0064] The following describes a method for correcting the target speaker's speech extraction results in an embodiment of the present application.

[0065] Figure 1 is an optional flowchart of the target speaker speech extraction result correction method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps 110 to 140. It can also be understood that this embodiment Figure 1 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0066] Step 110: Acquire the mixed voice and the registered voice of the target object.

[0067] In one embodiment, the mixed speech at least includes the initial speech of the target object. The purpose of the target speaker extraction task is to extract the initial speech of the target object from the overall mixed speech when the initial speech of the target object is mixed with other sounds.

[0068] In one embodiment, referring to Figure 2 , Figure 2 It is a flowchart of the target speaker extraction task provided in an embodiment of the present application. Figure 2 There is at least one interfering object in the target object's speaking environment, so the target object's initial speech and the interfering initial speech of the interfering object are mixed, and then the environmental noise is superimposed to obtain the mixed speech in the environment. The interfering initial speech here can be the speech of the interfering object collected in real time, or it can be the relevant speech recorded in advance.

[0069] The mixed speech m is expressed as:

[0070]

[0071] Among them, x0 represents the initial speech of the target object, x i represents the initial interference speech of the ith interference object, and n represents the environmental noise.

[0072] Since the initial speech of the target object needs to be identified from the mixed speech, the speech of the target object needs to be obtained in advance as the evaluation criterion for matching verification. In this case, the speech of the target object is obtained in advance as the registration speech. The target speaker is extracted based on the registration speech, and the target speech related to the initial speech of the target object is obtained from the mixed speech using the trained two-branch speech extraction model. The target speech here has the highest similarity with the initial speech and contains all the basic information contained in the initial speech. In addition, the registration speech can be the speech recorded by the target object according to the speech template. The speech template can contain as many pronunciation details as possible to better learn the pronunciation of the target object, thereby improving the recognition accuracy of the target speech.

[0073] The target voice It is expressed as:

[0074]

[0075] Among them, Φ represents the dual-branch speech extraction model for extracting the target speech, θ represents the parameters of the dual-branch speech extraction model, and e represents the registration speech.

[0076] Step 120: Input the registered speech and the mixed speech into a pre-trained dual-branch speech extraction model for speech extraction to obtain the interference speech and the target speech.

[0077] In one embodiment, the corresponding interference information in the mixed speech is used to assist in the calibration of the recognition result, and an extraction error result is corrected to a correct output, which can not only improve the overall accuracy of the recognition result, enhance the model's modeling ability for mixed speech, improve the robustness and adaptability of the model, but also significantly reduce the learning cost. Therefore, the dual-branch speech extraction model of the embodiment of the present application not only needs to identify the target object-related object speech, but also needs to identify at least one interference object-related interference speech, where the interference speech can be the overall interference speech obtained by summarization.

[0078] Reference Figure 3 , Figure 3 It is a structural diagram of the dual-branch speech extraction model provided in the embodiment of the present application. Figure 3 The dual-branch speech extraction model 100 includes: a speaker encoder 200, a hybrid speech encoder 300, a fusion module 400, a target branch extraction module 500 and an interference branch extraction module 600, wherein the target branch extraction module 500 also includes: a target backbone network 510 and a target mask extraction module 520, and the interference branch extraction module 600 also includes: an interference backbone network 610 and an interference mask extraction module 620.

[0079] Among them, the registered speech is input into the speaker encoder 200, the mixed speech is input into the mixed speech encoder 300, and then the outputs of the speaker encoder 200 and the mixed speech encoder 300 are used as the inputs of the fusion module 400, and the outputs of the fusion module 400 enter the target branch extraction module 500 and the interference branch extraction module 600 respectively. In the target branch extraction module 500, the output of the fusion module 400 passes through the target backbone network 510 and the target mask extraction module 520 in sequence to obtain the object speech, and in the interference branch extraction module 600, the output of the fusion module 400 passes through the interference backbone network 610 and the interference mask extraction module 620 in sequence to obtain the interference speech.

[0080] In one embodiment, referring to Figure 4 , Figure 4 The flowchart of the embodiment of the present application is to input the registration speech and the mixed speech into the pre-trained dual-branch speech extraction model for speech extraction to obtain the interference speech and the target speech, which specifically includes the following steps:

[0081] Step 410: input the registered speech into the speaker encoder for feature extraction to obtain the registered speech features, and input the mixed speech into the mixed speech encoder for feature extraction to obtain the mixed speech features.

[0082] In one embodiment, in combination Figure 3, the speaker encoder encodes the target object's registered speech to obtain the target speaker's feature embedding, which is called the registered speech feature. For example, a deep learning model, such as a convolutional neural network or a recurrent neural network, can be used to extract spectral features, Mel-frequency cepstral coefficients, etc. from the registered speech, and then the registered speech features are obtained through these features. Among them, this feature embedding can be represented as a feature vector or a tensor containing time domain information, which is used to characterize the unique attributes of the target object.

[0083] In one embodiment, the mixed speech encoder encodes the mixed speech to obtain feature embedding, which is called mixed speech feature. For example, a long short-term memory network, a recursive neural network or a self-attention mechanism can be used to process mixed signals of multiple speech sources and perform effective feature extraction on them to obtain mixed speech features. Different from the registered speech feature, since the dual-branch speech extraction model needs to extract and restore the original speech from the mixed speech feature, the mixed speech is encoded in the form of a matrix tensor containing time domain information.

[0084] Step 420: Input the registered speech features and the mixed speech features into a fusion module for feature fusion to obtain fused speech features.

[0085] In one embodiment, the purpose of fusing the registered speech features and the mixed speech features is to enable the dual-branch speech extraction model to process the mixed speech features when the registered speech features are obtained. The methods of feature fusion in this embodiment include vector connection, vector dot product or cross attention mechanism. Among them, vector connection refers to: directly splicing the registered speech features and the mixed speech features together to form a new composite vector as a fused speech feature. Vector dot product refers to: performing element-level product operations on the registered speech features and the mixed speech features to fuse information and obtain fused speech features, thereby better capturing the correlation between the registered speech features and the mixed speech features. The cross attention mechanism refers to: by calculating the attention weights between the registered speech features and the mixed speech features, dynamically adjusting the proportion of the two in the fused speech features, the contribution of different features can be more accurately aligned and weighted.

[0086] Step 430: Input the fused speech features and the mixed speech features into the target branch extraction module for feature extraction to obtain the object speech, and input the fused speech features and the mixed speech features into the interference branch extraction module for feature extraction to obtain the interference speech.

[0087] In one embodiment, in combination Figure 3At the same time, the fused speech features and the mixed speech features are input into the target branch extraction module and the interference branch extraction module. The target branch extraction module outputs the target speech, and the interference branch extraction module outputs the interference speech. The processing of the two is basically the same, and the target branch extraction module is used as an example for explanation.

[0088] Reference Figure 5 , Figure 5 The flowchart of inputting the fused speech feature and the mixed speech feature into the target branch extraction module for feature extraction to obtain the target speech provided by the embodiment of the present application specifically includes the following steps:

[0089] Step 510: Input the fused speech features into the target backbone network for feature extraction to obtain latent features.

[0090] In one embodiment, the target backbone network is composed of a deep neural network, such as a convolutional neural network, a recurrent neural network, a time-domain convolutional network, etc. The target backbone network is used to process the fused speech features and extract the deep potential features contained therein. These potential features contain rich information and can be used in the subsequent recognition process of the target speaker's speech extraction.

[0091] Step 520: Input the potential features into the target mask extraction module for feature extraction to obtain a speech mask.

[0092] In one embodiment, the target mask extraction module is composed of a shallow fully connected network. The target mask extraction module receives the potential features mined from the target backbone network and generates a speech mask correspondingly, which is used to selectively extract information from the feature embedding corresponding to the mixed speech. The speech mask is a set of weights, each weight corresponding to a component (such as a time frame) in the mixed signal, and these weights are used to adjust the gain of each part of the mixed signal, thereby suppressing non-target signals, enhancing target signals, extracting the sound of the target object, and improving the accuracy of the speech extraction result of the target speaker.

[0093] Step 530: Use the speech mask to perform mask calculation on the mixed speech features to obtain the object speech features, and decode the object speech features to obtain the object speech.

[0094] In one embodiment, after obtaining the language mask, the voice mask is used to perform mask calculation on the mixed voice features, strengthen the voice of the target object, and weaken other voices. After the masking process, the non-target voice components in the mixed voice are attenuated, while the voice features of the target object are retained and highlighted, thereby obtaining the target voice features. Next, the target voice features are decoded, and the target voice features are converted back to an audio waveform in the time domain using a signal processing technology, such as a vocoder, to obtain the target voice.

[0095] It can be understood that the process of inputting the fused speech features and the mixed speech features into the interference branch extraction module for feature extraction to obtain the interference speech is basically the same as the process of inputting the fused speech features and the mixed speech features into the target branch extraction module for feature extraction to obtain the object speech. The difference between the two is that the training objectives of the target branch extraction module and the interference branch extraction module are different during training, so the model parameters of the two are different, so that they can complete different recognition tasks respectively.

[0096] Step 130: Calculate the object similarity and interference similarity based on the voiceprint features of the registration voice, the interference voice and the object voice, and determine the error correction mark based on the object similarity and the interference similarity.

[0097] In one embodiment, the premise for error correction is that the output target voice is wrong and is not the target voice of the desired target object, so it is necessary to first determine whether error correction is required. The embodiment of the present application sets an error correction flag, and the judgment is made based on the error correction flag, and the error correction flag is related to the voiceprint features of the registered voice, the interfering voice, and the target voice.

[0098] In one embodiment, referring to Figure 6 , Figure 6 The flowchart of calculating the object similarity and interference similarity according to the voiceprint features of the registered voice, the interference voice and the object voice provided by the embodiment of the present application specifically includes the following steps:

[0099] Step 610: Acquire the registration voiceprint features of the registration voice, the interference voiceprint features of the interference voice, and the target voiceprint features of the target voice.

[0100] In one embodiment, the voiceprint feature refers to a reference standard such as fundamental frequency, timbre and pronunciation speed that can be used to distinguish voices, wherein the voiceprint feature can be extracted using a pre-trained encoder. The extracted registration voiceprint feature of the registration voice is s enroll , the interference voiceprint feature of the interference speech is s itf , the object voiceprint feature of the object speech is s tgt .

[0101] Step 620: Calculate the similarity between the registered voiceprint feature and the interference voiceprint feature to obtain the interference similarity, and calculate the similarity between the registered voiceprint feature and the object voiceprint feature to obtain the object similarity.

[0102] In one embodiment, the registered voiceprint feature s is calculated enroll and interference voiceprint features itf The cosine similarity between them is used to obtain the interference similarity, which is expressed as:

[0103] Sim itf=Cos(s enroll ,s itf )

[0104] Among them, Sim itf represents interference similarity, and Cos represents the calculated cosine similarity.

[0105] Calculate the cosine similarity between the registered voiceprint features and the object voiceprint features to obtain the object similarity, which is expressed as:

[0106] Sim tgt =Cos(s enroll ,s tgt )

[0107] Among them, Sim tgt Indicates object similarity.

[0108] Next, the error correction flag is determined based on the object similarity and the interference similarity. When the object similarity is greater than or equal to the interference similarity, it means that the object voice is likely to be the registered voice, and no error correction is required. The error correction flag set at this time can be: directly output the extraction result. When the object similarity is less than the interference similarity, it means that the object voice is unlikely to be the registered voice, and error correction is required. The error correction flag set at this time can be: correct the extraction result.

[0109] Step 140: If the error correction flag indicates that the extraction result is to be directly output, the object speech is used as the target speech corresponding to the initial speech; if the error correction flag indicates that the extraction result is to be corrected, the error-corrected speech is obtained based on the interfering speech or the object speech, and the error-corrected speech is used as the target speech corresponding to the initial speech.

[0110] In one embodiment, when no error correction is required, the error correction indicator indicates that the extraction result is directly output, and the object speech is directly used as the target speech corresponding to the initial speech. When error correction is required, the error correction indicator indicates that the extraction result is corrected, and the error correction speech is obtained based on the interference speech or the object speech, and the error correction speech is used as the target speech corresponding to the initial speech.

[0111] Next, the specific error correction process is described. In one embodiment, refer to Figure 7 , Figure 7 This is a flow chart of obtaining error correction speech based on interfering speech or target speech provided by an embodiment of the present application, which specifically includes the following steps:

[0112] Step 710: Select one from the interference speech and the object speech as a new registration speech, and input the new registration speech and the mixed speech into the dual-branch speech extraction model for speech extraction to obtain a new object speech or a new interference speech, and select the correction speech from the new object speech or the new interference speech.

[0113] In one embodiment, in combination Figure 3 , the object speech and the interference speech are used to interact and provide auxiliary information to each other to achieve the purpose of error correction, so as to output the error correction speech that is closest to the initial speech. This step is divided into the following two cases.

[0114] The first case: when the interfering speech is used as the new registered speech, the new registered speech and the mixed speech are input into the dual-branch speech extraction model for speech extraction to obtain a new target speech or a new interfering speech, and then the new target speech is used as the error correction speech.

[0115] This is equivalent to extracting the interference speech for the first time. As the new registration speech, it is input into the dual-branch speech extraction model for re-extraction, and the new object speech extracted for the second time is obtained. And new interference voice Voice the new object As the final error correction speech.

[0116] In the above embodiment, if error correction is needed, it can be assumed that the target speech extracted for the first time is inaccurate, and the first interfering speech may actually contain the speech of the target object. Therefore, this interfering speech can be treated as a new registered speech. In this case, the double-branch speech extraction model will identify the first interfering speech from the mixed speech as a new recognition target, thereby embodying the previously recognized interference information in the new object speech as much as possible, similar to enhancing the interfering speech that may be correct for the first time. At this time, the new target speech is likely to represent the speech of a purer target object, so this error correction method can improve the recognizability and overall recognition performance of the target speech.

[0117] The second case: when the target speech is used as the new registration speech, the new registration speech and the mixed speech are input into the dual-branch speech extraction model for speech extraction to obtain a new target speech or a new interference speech, and then the new interference speech is used as the error correction speech.

[0118] This is equivalent to extracting the object speech obtained for the first time As the new registration speech, it is input into the dual-branch speech extraction model for re-extraction, and the new object speech extracted for the second time is obtained. And new interference voice The new interfering voice As the final error correction speech.

[0119] In the above embodiment, if error correction is required, the object speech extracted for the first time can be considered to be wrong. By treating the target speech as a new registered speech, the purpose is to hope that the dual-branch speech extraction model can re-analyze the erroneous object speech and find speech features that may not be fully utilized or mistakenly ignored, thereby strengthening this error bias in the object speech extracted for the second time, and performing secondary extraction and optimization on the speech features of the erroneous target object that may be missed in the initial extraction. At this time, the new object speech can be regarded as generated in the process of "making mistakes on top of mistakes", and the new interference speech that is opposite to the new object speech is closer to the real speech of the target object, so that the recognition accuracy of the new interference speech as the final target speech is higher.

[0120] Step 720: Alternatively, the interfering speech is used as the error correction speech.

[0121] In one embodiment, since the core design logic of the target branch extraction module and the interference branch extraction module of the dual-branch speech extraction model is to work in reverse, the task of the target branch extraction module is to extract and strengthen the signal features from the registered speech, while the interference branch extraction module focuses on identifying and strengthening the features of the non-target speech. Therefore, when there may be errors in the recognition of the object speech, the interference speech produced by the interference branch extraction module can be directly exchanged with the object speech produced by the target branch extraction module. This is because in some cases, the interference branch extraction module may accidentally capture more accurate speech features of the target object. The embodiment of the present application can achieve efficient speech error correction in this simple way, significantly improving the target identity accuracy and robustness of the dual-branch speech extraction model in the target speaker speech extraction process.

[0122] In one embodiment, three error correction methods are proposed in the two cases of step 710 and step 720, and one of the three error correction methods can be selected for error correction. In addition, the three methods can be executed simultaneously to obtain different error correction voices in the three cases, and then the error correction voiceprint features of the error correction voices are obtained, and the similarities between the three error correction voiceprint features and the registered voiceprint features are calculated, and the error correction voice with the greatest similarity is selected as the final error correction voice. The embodiment of the present application does not limit the selection, combination and execution times of the error correction methods.

[0123] In one embodiment, the error correction process may also be: inputting the mixed speech into a model structure capable of blind source separation, separating the speech signals of all speakers in the mixed speech, and then extracting the voiceprint features corresponding to all the separated speech signals. Next, the similarity between each voiceprint feature and the registered voiceprint feature is calculated, and the separated speech signal corresponding to the maximum similarity is selected as the error correction speech.

[0124] In one embodiment, the error correction process may also be to subtract the extracted target speech from the mixed speech.

[0125] The technical solution provided by the embodiment of the present application is achieved by obtaining a mixed voice and a registered voice of a target object, wherein the mixed voice at least contains the initial voice of the target object. The registered voice and the mixed voice are input into a pre-trained dual-branch voice extraction model for voice extraction to obtain interference voice and object voice. Object similarity and interference similarity are calculated based on the voiceprint features of the registered voice, the interference voice and the object voice, and an error correction flag is determined based on the object similarity and the interference similarity. If the error correction flag indicates that the extraction result is directly output, the object voice is used as the target voice corresponding to the initial voice; if the error correction flag indicates that the extraction result is to be corrected, the error correction voice is obtained based on the interference voice or the object voice, and the error correction voice is used as the target voice corresponding to the initial voice.

[0126] The embodiment of the present application extracts the speech of the target speaker through a dual-branch speech coding process, not only identifying the target speech from the mixed speech, but also identifying the interference speech at the same time, introducing an interactive link between the two recognition tasks, and providing auxiliary information to each other through the association between the target speech and the interference speech, so as to determine whether the result needs to be corrected. When error correction is required, the final target speech is obtained based on the error correction speech corresponding to the interference speech and the target speech. In this way, attention is paid to possible extraction failures, and possible erroneous results are explicitly corrected to correct outputs, without significantly improving the model structure, optimizing the training mechanism, and expanding the data set. Compared with the optimization method for further improving the accuracy of the correct result, its learning cost is lower, the optimization performance is higher, and the reasoning process is more controllable.

[0127] The embodiment of the present application also provides a target speaker voice extraction result correction device, which can implement the target speaker voice extraction result correction method, referring to Figure 8 , the device comprises:

[0128] The voice acquisition module 810 is used to acquire the mixed voice and the registered voice of the target object, where the mixed voice at least includes the initial voice of the target object.

[0129] Feature extraction module 820: used to input the registered speech and the mixed speech into a pre-trained dual-branch speech extraction model for speech extraction to obtain the interference speech and the target speech.

[0130] The error correction judgment module 830 is used to calculate the object similarity and the interference similarity according to the voiceprint features of the registered voice, the interference voice and the object voice, and determine the error correction mark based on the object similarity and the interference similarity.

[0131] Speech error correction module 840: used to directly output the extraction result if the error correction mark indicates that the extraction result should be directly output, and the object speech should be used as the target speech corresponding to the initial speech; if the error correction mark indicates that the extraction result should be corrected, the error-corrected speech is obtained based on the interfering speech or the object speech, and the error-corrected speech is used as the target speech corresponding to the initial speech.

[0132] The specific implementation of the target speaker speech extraction result correction device of this embodiment is basically the same as the specific implementation of the target speaker speech extraction result correction method described above, and will not be repeated here.

[0133] The present application also provides an electronic device, including:

[0134] at least one memory;

[0135] at least one processor;

[0136] at least one program;

[0137] The program is stored in the memory, and the processor executes the at least one program to implement the target speaker voice extraction result correction method implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0138] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0139] The processor 901 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0140] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the target speaker speech extraction result error correction method of the embodiment of this application;

[0141] Input / output interface 903, used to implement information input and output;

[0142] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0143] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0144] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0145] The embodiment of the present application also provides a storage medium, which is a storage medium that stores a computer program. When the computer program is executed by a processor, the above-mentioned target speaker speech extraction result error correction method is implemented.

[0146] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The target speaker speech extraction result correction method and related device proposed in the embodiment of the present application are obtained by obtaining a mixed speech and a registered speech of a target object, wherein the mixed speech at least contains the initial speech of the target object. The registered speech and the mixed speech are input into a pre-trained dual-branch speech extraction model for speech extraction to obtain an interference speech and an object speech. The object similarity and the interference similarity are calculated according to the voiceprint features of the registered speech, the interference speech and the object speech, and the error correction mark is determined based on the object similarity and the interference similarity. If the error correction mark indicates that the extraction result is directly output, the object speech is used as the target speech corresponding to the initial speech. If the error correction mark indicates that the extraction result is corrected, the error correction speech is obtained based on the interference speech or the object speech, and the error correction speech is used as the target speech corresponding to the initial speech. The embodiment of the present application extracts the speech of the target speaker through a dual-branch speech coding process, not only recognizes the object speech from the mixed speech, but also recognizes the interference speech at the same time, introduces an interactive link between the two recognition tasks, and provides auxiliary information to each other through the association between the object speech and the interference speech, so as to determine whether the result correction is needed. When error correction is needed, the final target speech is obtained based on the error correction speech corresponding to the interference speech and the object speech. In this way, we can focus on possible extraction failures and explicitly correct possible erroneous results to correct outputs without significantly improving the model structure, optimizing the training mechanism, and expanding the data set. Compared with the optimization method of further improving the accuracy of the correct results, it has lower learning costs, higher optimization performance, and a more controllable reasoning process.

[0148] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0149] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0150] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0152] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0153] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store programs.

[0158] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for correcting the result of speech extraction of a target speaker, characterized in that: include: Acquire a mixed voice and a registered voice of a target object, wherein the mixed voice at least includes an initial voice of the target object; Inputting the registered speech and the mixed speech into a pre-trained dual-branch speech extraction model for speech extraction to obtain interference speech and target speech; Calculating object similarity and interference similarity according to the voiceprint features of the registered voice, the interference voice and the object voice, and determining an error correction flag based on the object similarity and the interference similarity; If the error correction flag indicates that the extraction result is directly output, the object speech is used as the target speech corresponding to the initial speech; if the error correction flag indicates that the extraction result is to be corrected, the error-corrected speech is obtained based on the interference speech or the object speech, and the error-corrected speech is used as the target speech corresponding to the initial speech.

2. The target speaker speech extraction result error correction method according to claim 1, characterized in that: The calculating the object similarity and the interference similarity according to the voiceprint features of the registered voice, the interference voice and the object voice includes: Acquire the registration voiceprint feature of the registration voice, the interference voiceprint feature of the interference voice, and the object voiceprint feature of the object voice; The similarity between the registered voiceprint feature and the interference voiceprint feature is calculated to obtain the interference similarity, and the similarity between the registered voiceprint feature and the object voiceprint feature is calculated to obtain the object similarity.

3. The target speaker speech extraction result error correction method according to claim 1, characterized in that: The determining of the error correction flag based on the object similarity and the interference similarity includes: When the object similarity is greater than or equal to the interference similarity, the error correction flag is used to indicate to directly output the extraction result; When the object similarity is less than the interference similarity, the error correction flag is used to indicate to correct the extraction result.

4. The method for correcting the target speaker's speech extraction result according to claim 1, characterized in that: The step of obtaining the error correction speech based on the interfering speech or the target speech comprises: Select one of the interference speech and the target speech as a new registration speech, input the new registration speech and the mixed speech into the dual-branch speech extraction model for speech extraction, obtain a new target speech or a new interference speech, and select the error correction speech from the new target speech or the new interference speech; Alternatively, the interfering speech is used as the error correction speech.

5. The method for correcting the target speaker's speech extraction result according to claim 4, characterized in that: The step of selecting the error correction speech from the new target speech or the new interference speech comprises: When the interfering speech is used as a new registration speech, the new target speech is used as the error correction speech; When the target speech is used as a new registration speech, the new interference speech is used as the error correction speech.

6. The method for correcting the target speaker's speech extraction result according to claim 1, characterized in that: The dual-branch speech extraction model includes a speaker encoder, a hybrid speech encoder, a fusion module, a target branch extraction module and an interference branch extraction module; the registration speech and the hybrid speech are input into the pre-trained dual-branch speech extraction model for speech extraction to obtain interference speech and object speech, including: Inputting the registered speech into the speaker encoder for feature extraction to obtain registered speech features, and inputting the mixed speech into the mixed speech encoder for feature extraction to obtain mixed speech features; Inputting the registered speech feature and the mixed speech feature into the fusion module for feature fusion to obtain a fused speech feature; The fused speech features and the mixed speech features are input into the target branch extraction module for feature extraction to obtain the object speech, and the fused speech features and the mixed speech features are input into the interference branch extraction module for feature extraction to obtain the interference speech.

7. The method for correcting the target speaker's speech extraction result according to claim 6, characterized in that: The target branch extraction module includes a target trunk network and a target mask extraction module; The step of inputting the fused speech feature and the mixed speech feature into the target branch extraction module for feature extraction to obtain the object speech comprises: Inputting the fused speech features into the target backbone network for feature extraction to obtain potential features; Inputting the potential features into the target mask extraction module for feature extraction to obtain a speech mask; The mixed speech feature is masked by using the speech mask to obtain the object speech feature, and the object speech feature is decoded to obtain the object speech.

8. A device for correcting the result of speech extraction of a target speaker, characterized in that: include: A voice acquisition module: used to acquire a mixed voice and a registered voice of a target object, wherein the mixed voice at least includes an initial voice of the target object; Feature extraction module: used for inputting the registered speech and the mixed speech into a pre-trained dual-branch speech extraction model for speech extraction to obtain interference speech and target speech; The error correction judgment module is used to calculate the object similarity and the interference similarity according to the voiceprint features of the registered voice, the interference voice and the object voice, and determine the error correction mark based on the object similarity and the interference similarity; A speech error correction module: used to, if the error correction flag indicates that the extraction result is directly output, use the object speech as the target speech corresponding to the initial speech; if the error correction flag indicates that the extraction result is to be corrected, obtain the error-corrected speech based on the interfering speech or the object speech, and use the error-corrected speech as the target speech corresponding to the initial speech.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the target speaker speech extraction result error correction method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for correcting the target speaker speech extraction result according to any one of claims 1 to 7 is implemented.

Images