Audio acquisition method, device, electronic device and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for audio acquisition. The method comprises: acquiring target facial feature information of a target person; acquiring a target video clip containing the target person based on the target facial feature information; acquiring a target audio clip corresponding to the target video clip; and acquiring a speaking audio clip of the target person from the target audio clip. On the one hand, the method realizes automatic acquisition of the speaking audio clip of the target person without human intervention, thereby improving the efficiency of acquiring the speaking audio clip of the target person; on the other hand, the method makes full use of abundant Internet resources in the process of determining the target video clip containing the target person, thereby ensuring the richness and diversity of the acquired audio speaking audio clips of the target person.

Description

Technical Field

[0001] The present application relates to the technical field of audio signal processing, and more specifically, to an audio acquisition method, device, electronic device and storage medium. Background Art

[0002] With the development of human-computer voice interaction technology, the generation of personalized voice has become an important requirement in human-computer voice interaction scenarios, that is, in the voice interaction system, providing users with personalized customization services of voice timbre.

[0003] In related technologies, AI-based voice timbre conversion technology often requires pre-collection of audio clips of specified people, such as audio clips of celebrities, cartoon characters, etc. However, obtaining audio clips of specified people through manual recording and annotation is not only inefficient, but also requires a lot of manpower and financial resources. Summary of the invention

[0004] Embodiments of the present application provide a method, device, electronic device, and storage medium for audio acquisition.

[0005] In a first aspect, some embodiments of the present application provide a method for audio acquisition, the method comprising: acquiring target facial feature information of a target person; acquiring a target video clip containing the target person based on the target facial feature information; acquiring a target audio clip corresponding to the target video clip, and acquiring a speaking audio clip of the target person from the target audio clip.

[0006] In a second aspect, some embodiments of the present application also provide an audio acquisition device, which includes: a target facial feature information acquisition module, a target video clip acquisition module, a target audio clip acquisition module, and a speaking audio clip acquisition module. Among them, the target facial feature information acquisition module is used to acquire the target facial feature information of the target person. The target video clip acquisition module is used to acquire a target video clip containing the target person based on the target facial feature information. The target audio clip acquisition module is used to acquire a target audio clip corresponding to the target video clip. The speaking audio clip acquisition module is used to acquire the speaking audio clip of the target person from the target audio clip.

[0007] In a third aspect, some embodiments of the present application further provide an electronic device, the electronic device comprising: one or more processors, a memory, and one or more applications. The one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the above-mentioned audio acquisition method.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which computer program instructions are stored, and the computer program instructions can be called by a processor to execute the above-mentioned audio acquisition method.

[0009] In a fifth aspect, an embodiment of the present application further provides a computer program product, which, when executed, implements the above-mentioned audio acquisition method.

[0010] The embodiments of the present application propose a method, device, electronic device and storage medium for audio acquisition. In this method, a target video segment containing a target person is determined based on the target facial feature information of the target person, then a target audio segment corresponding to the target video segment is obtained, and finally a speaking audio segment of the target person is extracted from the target audio segment. On the one hand, this method realizes automatic acquisition of the speaking audio segment of the target person without human intervention, thereby improving the efficiency of acquiring the speaking audio segment of the target person; on the other hand, this method makes full use of abundant Internet resources in the process of determining the target video segment containing the target person, thereby ensuring the richness and diversity of the acquired audio speaking audio segments of the target person. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic flow chart of an audio acquisition method provided in the first embodiment of the present application is shown.

[0013] Figure 2 A flow chart of an audio acquisition method provided in the second embodiment of the present application is shown.

[0014] Figure 3 A flowchart of an audio acquisition method provided in the third embodiment of the present application is shown.

[0015] Figure 4 A flowchart of obtaining a target person's speaking audio segment based on a target audio segment provided by an embodiment of the present application is shown.

[0016] Figure 5 A schematic flow chart of an audio acquisition method provided in the fourth embodiment of the present application is shown.

[0017] Figure 6A flowchart of an audio acquisition method provided in the fifth embodiment of the present application is shown.

[0018] Figure 7 A schematic diagram of an audio database system provided in an embodiment of the present application is shown.

[0019] Figure 8 A module block diagram of an audio acquisition device provided in an embodiment of the present application is shown.

[0020] Fig. 9 A module block diagram of an electronic device provided in an embodiment of the present application is shown.

[0021] Fig.10 A module block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limitations on the present application. At the same time, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0023] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0024] In order to solve the problems existing in the above-mentioned background technology, the inventors have proposed a method, device, electronic device and storage medium for audio acquisition after long-term research. In this method, a target video segment containing a target person is determined based on the target facial feature information of the target person, then a target audio segment corresponding to the target video segment is obtained, and finally the target person's speaking audio segment is extracted from the target audio segment. On the one hand, this method realizes the automatic acquisition of the target person's speaking audio segment without human intervention, thereby improving the acquisition efficiency of the target person's speaking audio segment; on the other hand, this method fully utilizes the rich Internet resources in the process of determining the target video segment containing the target person, thereby ensuring the richness and diversity of the acquired target person's audio speaking audio segment.

[0025] The technical solution provided in the embodiment of the present application can be applied to application scenarios such as voice navigation, smart speakers, film and television dubbing, virtual idols, etc. Taking the application scenario of voice navigation as an example, the electronic device can obtain the target person's (such as a celebrity, a virtual idol) speaking audio clip based on the technical solution provided in the embodiment of the present application, and synthesize the voice navigation voice corresponding to the target person using the target person's speaking audio clip, thereby realizing personalized customization of voice navigation.

[0026] like Figure 1 As shown, Figure 1 A method for audio acquisition provided in the first embodiment of the present application is schematically shown. The method may include steps S110 to S140.

[0027] Step S110: Acquire target facial feature information of the target person.

[0028] The target facial feature information of the target person includes but is not limited to facial skin color information, gender information and facial feature information. In some embodiments, the target facial feature information of the target person is determined based on the facial image of the target person. The electronic device obtains the facial image of the target person, and obtains the target facial feature information of the target person based on the facial image. In an embodiment of the present application, the target facial feature information may be in vector form. Specifically, one or more dimensions of the feature vector representing the target facial feature information represent an item of feature information in the target facial feature information. For example, the 5th to 20th dimensions of the feature vector represent facial feature information in the target facial feature information.

[0029] As an implementation mode, a facial feature extraction algorithm is stored in the electronic device, and the facial feature extraction algorithm uses the facial image of the target person as the algorithm input and outputs the target facial feature information of the target person. The facial feature extraction algorithm can be a feature extraction algorithm based on HOG, a feature extraction algorithm based on LBP, a feature extraction algorithm based on a convolutional neural network, etc., which is not specifically limited in the embodiments of the present application. The facial image of the target person is an image pre-stored in the electronic device, or it can be obtained by the electronic device in real time from the cloud. In the case where the facial image of the target person is pre-stored in the electronic device, the electronic device obtains the target facial feature information of the target person in the following manner: the electronic device receives a facial feature extraction instruction for the facial image, receives the facial image input by the user through the image transmission interface, and then obtains the target facial feature information in the facial image based on the facial feature extraction algorithm.

[0030] As another implementation, a first mapping table of the correspondence between the name of the target person and the target facial feature information of the target person is stored in the electronic device. When the electronic device obtains the name of the target person, the target facial feature information corresponding to the target person can be determined through the first mapping table. Optionally, the first mapping table can also be stored in a server that establishes a communication connection with the electronic device. When the electronic device obtains the name of the target person, it sends a first query instruction to the server and receives the target facial feature information corresponding to the target person sent by the server based on the first query instruction.

[0031] Taking voice navigation as an example, if the user finds that there is no corresponding target person option in the voice navigation system, that is, the user cannot set the voice navigation voice to the target person's voice, the user can personalize the voice navigation voice by uploading the target person's face image or entering the target person's name.

[0032] Step S120: Based on the target person's facial feature information, a target video clip containing the target person is obtained.

[0033] In an embodiment of the present application, since the target video clip contains the target person, the audio clip corresponding to the target video clip may include the target person's speaking audio clip. Therefore, the target video clip containing the target person can be determined first, so that in the subsequent steps, the corresponding audio clip can be obtained from the target video clip to extract the target person's speaking audio clip.

[0034] Step S130: Acquire a target audio segment corresponding to the target video segment.

[0035] In the embodiment of the present application, the target audio segment and the target video segment are in one-to-one correspondence on the time axis, that is, the target audio segment and the target video segment have the same start time and the same end time. The electronic device obtains the time information corresponding to the target video segment and determines the corresponding target audio segment based on the time information. The time information corresponding to the target video segment includes at least two of the timestamp of the start frame image, the timestamp of the end frame image, and the duration of the video segment corresponding to the target video segment.

[0036] In some embodiments, before obtaining the target audio segment corresponding to the target video segment, first determine whether the target person is speaking through the lip posture information of the target person, and execute the subsequent steps if it is determined that the target person is speaking, so as to avoid the situation where the target person in the target audio segment does not speak in the target video segment, and then the speaking audio segment cannot be extracted. Therefore, after the step of obtaining the target video segment containing the target person based on the target facial feature information, and before the step of obtaining the target audio segment corresponding to the target video segment, the following step is also included: obtaining the lip posture information corresponding to the target person in the target video segment. If the lip posture information indicates that the target person is in a speaking state, the step of obtaining the target audio segment corresponding to the target video segment is executed.

[0037] As an implementation mode, the electronic device obtains the lip gesture information corresponding to the target person in the target video clip through an internally preset lip gesture recognition algorithm. Specifically, the lip gesture recognition algorithm is a gesture recognition algorithm based on deep learning, for example, a gesture recognition algorithm based on a convolutional neural network. This embodiment does not specifically limit the specific implementation method of the lip gesture recognition algorithm. If the lip gesture information indicates that the target person is in a speaking state, the electronic device executes the step of obtaining the target audio segment corresponding to the target video segment; conversely, if the lip gesture information indicates that the target person is not in a speaking state, the electronic device discards the corresponding target video segment. When it is determined that the target person in the target video segment is not in a speaking state, the target video segment is discarded in time. Since there is no need to execute subsequent steps, the computing resources of the electronic device can be saved.

[0038] Step S140: Acquire a target person's speaking audio segment from the target audio segment.

[0039] As an implementation manner, the electronic device may directly use the target audio segment as the target person's speaking audio segment.

[0040] As another implementation, the target audio segment includes a background audio segment and a target person's speech audio segment, wherein the background audio segment includes but is not limited to environmental background sounds (e.g., vehicle horns, animal calls, etc.) and non-target person's speech sounds. The electronic device removes the background audio segment from the target audio segment through an internally preset audio processing algorithm, thereby obtaining the target person's speech audio segment. Specifically, the specific implementation of the electronic device obtaining the target person's speech audio segment through the audio processing algorithm is described in detail in the following embodiments.

[0041] The embodiment of the present application provides a method for audio acquisition, in which a target video segment containing a target person is determined based on the target person's target facial feature information, then a target audio segment corresponding to the target video segment is acquired, and finally a target person's speech audio segment is extracted from the target audio segment. On the one hand, the method realizes automatic acquisition of the target person's speech audio segment without human intervention, thereby improving the acquisition efficiency of the target person's speech audio segment; on the other hand, the method makes full use of abundant Internet resources in the process of determining the target video segment containing the target person, thereby ensuring the richness and diversity of the acquired target person's audio speech audio segment.

[0042] like Figure 2 As shown, Figure 2 A method for audio acquisition provided by the second embodiment of the present application is schematically shown. In this method, an implementation scheme for determining a target video segment containing a target person based on target facial feature information of the target person is specifically introduced. This method may include steps S210 to S270.

[0043] Step S210: Acquire target facial feature information of the target person.

[0044] The specific implementation of step S210 can refer to the detailed description of step S110, which will not be described in detail here.

[0045] Step S220: Acquire candidate facial feature information from frame images included in at least one video clip.

[0046] In the embodiment of the present application, the target video segment is determined based on at least one video segment, and the at least one video segment can be stored in a video library to be processed. The video segments in the video library to be processed can be video segments on the Internet obtained based on web crawler technology, video segments downloaded through applications in electronic devices, and video segments recorded by researchers or users through video recording devices in electronic devices. In the embodiment of the present application, the acquisition source of the video segments and the specific types of the video segments are not specifically limited.

[0047] As an implementation mode, the electronic device extracts at least one frame of images from at least one video clip through an internally preset frame image extraction algorithm, wherein the frame image extraction algorithm can be obtained through OpenCV, a computer vision and machine learning software library. When the electronic device obtains at least one frame of images, it obtains candidate facial feature information corresponding to the person in the frame image based on the facial feature extraction algorithm. In the embodiment of the present application, the candidate facial feature information is also in vector form, and the feature vector corresponding to the candidate facial feature information and the feature vector corresponding to the target facial feature information have the same vector size. For example, the feature vector corresponding to the candidate facial feature information is a 200-dimensional vector, and the feature vector corresponding to the target facial feature information is also a 200-dimensional vector.

[0048] Step S230: If the similarity between the candidate facial feature information and the target facial feature information is greater than or equal to a preset similarity, the frame image is determined as the target frame image.

[0049] As an implementation mode, the electronic device calculates the similarity between the candidate face feature information and the target face feature information by using an internally preset vector similarity calculation method. Specifically, the vector similarity calculation method may include a calculation method based on cosine similarity, Euclidean distance, Hamming distance, Pearson correlation coefficient, etc. This is not specifically limited in the application examples.

[0050] If the vector similarity between the candidate facial feature information and the target facial feature information is greater than or equal to the preset similarity, the frame image is determined as the target frame image; if the vector similarity between the candidate facial feature information and the target facial feature information is less than the preset similarity, the frame image is not the target frame image. The preset similarity can be set by default by the electronic device, or it can be dynamically adjusted by researchers based on the matching accuracy of the facial feature information. The higher the matching accuracy, the greater the corresponding preset similarity; conversely, the lower the matching accuracy, the smaller the corresponding preset similarity. Specifically, the preset similarity can be any percentage between greater than or equal to 60% and less than 100%.

[0051] Step S240: Determine the video segment including the target frame image as a candidate video segment.

[0052] When determining that at least one frame image in the video segment is a target frame image, the electronic device determines the video segment as a candidate video segment.

[0053] Step S250: If the number of consecutive target frame images included in the candidate video segment is greater than or equal to a preset number, the candidate video segment is determined as a target video segment containing the target person.

[0054] The continuous target frame images are composed of at least two target frame images, and the time interval between two adjacent target frame images in the at least two target frame images is a preset time interval. The preset time interval is determined based on the frame rate of the candidate video segment, and the frame rate represents the number of frames displayed per second in the candidate video segment. Specifically, the preset time interval is the reciprocal of the frame rate of the candidate video segment. Taking the frame rate as 10 as an example, the candidate video segment displays 10 frames per second, and the preset time interval between two adjacent frames is 0.1 seconds.

[0055] As an implementation mode, the electronic device determines whether the target frame image is a continuous target frame image based on the timestamp information of the target frame image. Specifically, the electronic device sequentially calculates the time interval between two adjacent target frame images, and if the time interval is greater than a preset time interval, determines that the target frame image is not a continuous target frame image; if the time interval is equal to the preset time interval, determines that the target frame image is a continuous target frame image.

[0056] When the electronic device determines that the target frame image in the candidate video segment is a continuous target frame image, it further determines the number of continuous target frame images. If the number of continuous target frame images is greater than or equal to the preset number, the candidate video segment is determined as a target video segment containing the target person; if the number of continuous target frame images is less than the preset number, the candidate video segment is discarded. In the above manner, video segments in which the speaker speaks for too little time can be screened out. The preset number is set by the electronic device by default, and can also be dynamically adjusted by researchers according to their own needs. Specifically, the preset number, that is, the number of frames of the target video segment, is directly proportional to the time length of the target video segment. If the researcher needs to increase the time length of the target video segment, the preset number is increased; if the researcher needs to shorten the time length of the target video segment, the preset number is reduced. Among them, the preset number can be any natural number greater than 30.

[0057] Step S260: Acquire a target audio segment corresponding to the target video segment.

[0058] In the embodiment of the present application, the target audio segment and the continuous target frame images are in one-to-one correspondence on the time axis, that is, the target audio segment and the continuous target frame images have the same start time and the same end time. The electronic device obtains the time information corresponding to the continuous target frame images and determines the corresponding target audio segment based on the time information. The time information corresponding to the continuous target frame images includes at least two of the timestamp of the start target frame image, the timestamp of the end target frame image, and the duration of the video segment corresponding to the continuous target frame images.

[0059] Step S270: Acquire the target person's speaking audio segment from the target audio segment.

[0060] The specific implementation of step S270 can refer to the detailed description of step S140, which will not be repeated here.

[0061] The embodiment of the present application provides a method for audio acquisition, in which a target video segment is determined based on at least one video segment, and when the similarity between candidate facial feature information corresponding to a frame image in at least one video segment and target facial feature information is greater than or equal to a preset similarity, the video segment containing the frame image is determined as a candidate video segment; when the number of continuous target frame images included in the candidate video segment is greater than or equal to a preset number, the candidate video segment is determined as a target video segment containing the target person, and the target video segment can be used to subsequently acquire a speaking audio segment of the target person. The embodiment of the present application specifically provides a method for acquiring a target video segment, which ensures the reliability of the subsequent acquisition of the speaking audio segment of the target person.

[0062] like Figure 3 As shown, Figure 3 A method for audio acquisition provided by the third embodiment of the present application is schematically shown. In this method, the background audio segment in the target audio segment is removed to obtain the target person's speech audio segment. Specifically, this method may include steps S310 to S340.

[0063] Step S310: Acquire target facial feature information of the target person.

[0064] Step S320: Based on the target person's facial feature information, a target video clip containing the target person is obtained.

[0065] Step S330: Obtain a target audio segment corresponding to the target video segment.

[0066] The specific implementation of steps S310 to S330 may refer to the detailed description of steps S110 to S130, which will not be described in detail here.

[0067] Step S340: remove the background audio segment from the target audio segment to obtain the target person's speaking audio segment.

[0068] In this embodiment, the target audio segment includes a background audio segment and a target person's speaking audio segment. The electronic device removes the background audio segment from the target audio segment through an internally preset audio processing algorithm to obtain the target person's speaking audio segment. As an implementation method, the electronic device obtains a spectrogram corresponding to the target audio segment through an audio processing algorithm, and removes the audio signal corresponding to the background audio segment through a frequency domain filter to obtain the target person's speaking audio segment. Specifically, the spectrogram can be obtained based on Fourier transform, and the frequency domain filter includes but is not limited to Gaussian filter, Wiener filter, Butterworth filter, etc.

[0069] As another implementation, the electronic device obtains a first spectrogram corresponding to the target audio segment through an audio processing algorithm, obtains a first time-frequency mask in the first spectrogram through a neural network model, and then determines a second spectrogram of the speaking audio segment, and obtains the speaking audio segment of the target person through the second spectrogram. Specifically, step S340 may include the following steps S342 to S348.

[0070] Step S342: Obtain a first spectrogram of the target audio segment.

[0071] The first spectrogram is used to characterize the correspondence between time and the frequency of the audio signal in the target audio segment. As an implementation mode, the electronic device obtains the first spectrogram corresponding to the target audio segment through Hilbert transform.

[0072] Optionally, before obtaining the first spectrogram, a preprocessing operation is also performed on the target audio segment, wherein the preprocessing operation includes but is not limited to a frame operation, a windowing operation, and the like, which are not specifically limited in the embodiments of the present application.

[0073] Step S344: Obtain a first time-frequency mask corresponding to the first spectrogram.

[0074] The first time-frequency mask is used to indicate the spectrogram area corresponding to the target person's speech audio segment. In an embodiment of the present application, the size of the first time-frequency mask is the same as the size of the first spectrogram. As an implementation method, each element in the first time-frequency mask is composed of 0 or 1, wherein the dot product result of the area composed of the elements with a value of 1 and the corresponding area in the first spectrogram represents the spectrogram area corresponding to the target person's speech audio segment, and the dot product result of the area composed of the elements with a value of 0 and the corresponding area in the first spectrogram represents the spectrogram area corresponding to the background audio segment.

[0075] In the embodiment of the present application, the first time-frequency mask is obtained by a neural network model in an electronic device, the input of the neural network model is a target audio segment, and the output of the model is a first spectrogram of the same size as the target audio segment. For example, the size of the target audio segment is 256*256, and the first time-frequency mask obtained by the neural network model is also 256*256. Specifically, the neural network model can be a neural network model based on an FCN structure and a neural network model based on a U-Net structure. The implementation method of the neural network model is not specifically limited in the embodiment of the present application.

[0076] Step S346: Obtain a second spectrogram based on the first time-frequency mask and the first spectrogram.

[0077] As an implementation mode, the electronic device performs a dot product operation on the first time-frequency mask and the first spectrogram, and uses the dot product result corresponding to the dot product operation as the second spectrogram.

[0078] For example, the first spectrogram is a 4*4 matrix A, and the first time-frequency mask is a 4*4 matrix B1. And the matrix

[0079] Then the second spectrogram is a matrix B2 of size 4*4, and the matrix

[0080] Step S348: Based on the second spectrogram, obtain the target person's speaking audio segment.

[0081] As an implementation mode, the electronic device obtains the target person's speech audio signal by performing Hilbert transform on the second spectrogram, reconstructs the speech audio signal by using an audio reconstruction algorithm, and then obtains the target person's speech audio segment. Specifically, the audio reconstruction algorithm includes but is not limited to the Wavenet algorithm, the Griffin-Lim algorithm, and other algorithms, which are not specifically limited in the embodiments of the present application.

[0082] In other possible implementations, the electronic device needs to obtain a background audio segment as training data. The method for obtaining the background audio segment may include the following steps: the electronic device obtains a second time-frequency mask by performing a first time-frequency mask corresponding to the first spectrogram, and the second time-frequency mask is used to indicate the spectrogram area corresponding to the background audio segment. Specifically, the second time-frequency mask can be obtained by performing a differential operation on the first time-frequency mask. Then, the electronic device obtains a third spectrogram based on the second time-frequency mask and the first spectrogram. Specifically, the electronic device performs a dot product operation on the second time-frequency mask and the first spectrogram, and uses the dot product result corresponding to the dot product operation as the third spectrogram. Finally, the electronic device obtains the background audio segment based on the third spectrogram. Specifically, the electronic device obtains the background audio signal by performing a Hilbert transform on the third spectrogram, reconstructs the background audio signal through an audio reconstruction algorithm, and then obtains the background audio segment.

[0083] See also Figure 4 , Figure 4 The flowchart of a method for obtaining a target person's speech audio segment based on a target audio segment provided by an embodiment of the present application is schematically shown. Figure 4 In step S342, the electronic device obtains a time domain mixed signal by performing a preprocessing operation on the target audio segment, and obtains a spectrogram by performing a Hilbert transform on the time domain mixed signal, i.e., the first spectrogram in step S342. The electronic device uses the first spectrogram as the input of the deep neural network, and obtains the time-frequency mask of the human voice component of the deep neural network, i.e., the first time-frequency mask in step S344. The electronic device obtains a human voice spectrogram through the first time-frequency mask, i.e., the second spectrogram in step S346. The electronic device obtains a human voice time domain signal by performing a Hilbert transform on the second spectrogram, and reconstructs the human voice time domain signal to obtain a speech signal to obtain a speaking audio segment of the target person.

[0084] The embodiment of the present application provides a method for audio acquisition, in which the background audio segment in the target audio segment is removed to obtain the target person's speaking audio segment. In addition, the method provides a method for acquiring the target person's speaking audio segment based on the first spectrogram corresponding to the target audio segment and the neural network model. The method automatically acquires the target person's speaking audio segment in a targeted manner, thereby improving the acquisition efficiency of the speaking audio segment.

[0085] like Figure 5 As shown, Figure 5The fourth embodiment of the present application provides a method for obtaining audio. When a speech audio segment is obtained, the method further determines whether the speech audio segment corresponds to the target person through the voiceprint similarity between the second voiceprint feature of the speech audio segment and the first voiceprint feature of the target person. Specifically, the method may include steps S510 to S570.

[0086] Step S510: Acquire target facial feature information of the target person.

[0087] Step S520: Based on the target facial feature information, a target video clip containing the target person is obtained.

[0088] Step S530: Acquire a target audio segment corresponding to the target video segment.

[0089] Step S540: Acquire the target person's speaking audio segment from the target audio segment.

[0090] The specific implementation of steps S510 to S540 can refer to the detailed description of steps S110 to S140, which will not be described in detail here.

[0091] Step S550: Acquire the first voiceprint feature of the target person.

[0092] The voiceprint feature characterizes a set of acoustic description parameters (vectors) extracted from an audio clip. As an implementation mode, a speech audio reference clip of a target person is stored in an electronic device, and a voiceprint feature is extracted from the speech audio reference clip to obtain a first voiceprint feature of the target person. Specifically, the electronic device obtains the first voiceprint feature of the target person from the speech audio reference clip through a preset voiceprint feature extraction algorithm. The voiceprint feature extraction algorithm may be a voiceprint feature extraction algorithm based on iVector or a voiceprint feature extraction algorithm based on a DNN network, which is not specifically limited in the embodiments of the present application.

[0093] As another implementation, a third mapping table of the correspondence between the name of the target person and the first voiceprint feature of the target person is stored in the electronic device. When the electronic device obtains the name of the target person, it can determine the first voiceprint feature corresponding to the target person through the third mapping table. Optionally, the third mapping table can also be stored in a server that establishes a communication connection with the electronic device. When the electronic device obtains the name of the target person, it sends a second query instruction to the server and receives the first voiceprint feature corresponding to the target person sent by the server based on the second query instruction.

[0094] Step S560: Obtain a second voiceprint feature of the speaking audio segment.

[0095] As an implementation manner, the electronic device extracts the voiceprint features of the speech audio segment through a voiceprint feature extraction algorithm, and then obtains the second voiceprint feature of the speech audio segment.

[0096] Step S570: If the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is less than a preset voiceprint similarity, the speech audio segment is discarded.

[0097] If the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is less than the preset voiceprint similarity, the electronic device discards the speaking audio segment; conversely, if the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is greater than or equal to the preset voiceprint similarity, the electronic device retains the speaking audio segment.

[0098] As an implementation method, the voiceprint similarity between the second voiceprint feature and the first voiceprint feature can be calculated by a vector similarity calculation method, which will not be described in detail here. The preset voiceprint similarity can be set by default by the electronic device, and researchers can also dynamically adjust it according to the actual acquisition results of the target person's speaking audio segment. Specifically, the preset voiceprint similarity can be any percentage greater than 70% and less than 100%. Taking the preset voiceprint similarity of 80% as an example, if the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is less than 80%, the speaking audio segment is discarded; conversely, if the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is greater than or equal to 80%, the speaking audio segment is retained.

[0099] In the embodiment of the present application, the voiceprint similarity between the second voiceprint feature of the speaking audio segment and the first voiceprint feature of the target person is used to determine whether the speaking audio segment corresponds to the target person. If the voiceprint similarity is less than the preset voiceprint similarity, it means that the speaking audio segment obtained by the audio acquisition method does not correspond to the target person. In this case, the speaking audio segment needs to be discarded. This method determines whether the speaking audio segment corresponds to the target person by the voiceprint feature of the speaking audio segment, which can further ensure the accuracy of the acquisition of the speaking audio segment.

[0100] Optionally, a standard voiceprint similarity is also provided in the electronic device, and the standard voiceprint similarity is used to judge the audio quality of the spoken audio segment, and the standard voiceprint similarity is greater than a preset voiceprint similarity. If the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is greater than or equal to the preset voiceprint similarity, and less than the standard voiceprint similarity, it means that there is still a background audio segment in the spoken audio segment, and the spoken audio segment needs to be optimized; if the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is greater than the standard voiceprint similarity, no processing is required. As an implementation method, the electronic device can optimize the spoken audio segment through a deep residual shrinkage network.

[0101] In an embodiment of the present application, when the speaking audio segment corresponds to the target person, the audio quality of the speaking audio segment is judged by the standard voiceprint similarity. When the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is less than the standard voiceprint similarity, the speaking audio segment is optimized to further ensure the quality of the speaking audio segment.

[0102] The embodiment of the present application provides a method for audio acquisition, in which, when a speaking audio segment is acquired, the voiceprint similarity between the second voiceprint feature of the speaking audio segment and the first voiceprint feature of the target person is further used to determine whether the speaking audio segment corresponds to the target person. If the voiceprint similarity is less than the preset voiceprint similarity, it means that the speaking audio segment acquired by the audio acquisition method does not correspond to the target person. In this case, the speaking audio segment needs to be discarded. This method can further ensure the accuracy of acquiring the speaking audio segment by determining whether the speaking audio segment corresponds to the target person through the voiceprint feature of the speaking audio segment.

[0103] like Figure 6 As shown, Figure 6 A method for audio acquisition provided by the fifth embodiment of the present application is schematically shown. After acquiring the target person's speech audio segment, the method further includes the step of acquiring the feature tag corresponding to the speech audio segment. After the speech audio segment is stored in the audio database system, the user can quickly find the speech audio segments corresponding to different types through the feature tags corresponding to the speech audio segment. Specifically, the method may include steps S610 to S680.

[0104] Step S610: Acquire target facial feature information of the target person.

[0105] Step S620: Based on the target facial feature information, a target video clip containing the target person is obtained.

[0106] Step S630: Acquire a target audio segment corresponding to the target video segment.

[0107] Step S640: Acquire the target person's speaking audio segment from the target audio segment.

[0108] The specific implementation of steps S610 to S640 may refer to the detailed description of steps S110 to S140, which will not be described in detail here.

[0109] Step S650: Perform audio recognition processing on the spoken audio segment to obtain text information corresponding to the spoken audio segment.

[0110] As an implementation method, an audio recognition algorithm is pre-installed in the electronic device. Based on the audio recognition algorithm, the electronic device performs audio recognition processing on the speech audio segment to obtain text information corresponding to the speech audio segment. Specifically, the audio recognition algorithm can be a dynamic time warping (DTW) method based on pattern matching, or a hidden Markov model (HMM) method based on a statistical model.

[0111] Step S660: Process the text information to obtain at least one keyword in the text information.

[0112] In some embodiments, the keyword is a word whose frequency of appearance in the text information is greater than or equal to a preset frequency. As an implementation mode, a text processing algorithm is pre-set in the electronic device, and based on the text processing algorithm, the electronic device processes the text information to obtain at least one keyword in the text information. Specifically, the text processing algorithm can be a keyword extraction algorithm based on a word graph model or a keyword extraction algorithm based on a topic model. In other embodiments, the keyword is determined by the TF-IDF algorithm. The TF-IDF algorithm is used to determine the TF index and IDF index of each word in the text information, wherein the TF index represents the number of times the specified word appears in the current text information, wherein the more times the specified word appears in the current text information, the larger the corresponding TF index. The IDF index represents the number of times the specified word appears in other text information, and the fewer times the specified word appears in other text information, the larger the corresponding IDF index. The TF-IDF algorithm determines the reference index value corresponding to each word based on the TF index and IDF index of each word and the first weight corresponding to the TF index and the second weight corresponding to the IDF index. The larger the reference index value, the more times the corresponding word appears in the current text information and the fewer times it appears in other text information. The electronic device sorts the reference index value of each word in descending order, and determines the words corresponding to the first n reference index values ​​in the sequence as keywords, where n is the number of keywords, and n can be any natural number greater than 0.

[0113] Step S670: Based on at least one keyword, obtain a feature tag of the spoken audio segment.

[0114] The feature tags of the spoken audio clips are used to describe the subject information corresponding to the text information in the spoken audio clips. For example, the subject information can be funny, food, fashion, travel, entertainment, life, information, parent-child, knowledge, games, cars, finance, sports, music, movies, variety shows, animation, technology, health and other information.

[0115] As an implementation mode, a fourth mapping table of correspondences between keywords and feature tags is stored in the electronic device. The electronic device determines the feature tag of the speech audio segment corresponding to at least one keyword through the fourth mapping table. Exemplarily, if the acquired keywords are "cake" and "low sugar", the feature tag of the corresponding speech audio segment is determined to be "food" through the fourth mapping table; if the acquired keywords are "four-wheel drive" and "engine", the feature tag of the corresponding speech audio segment is determined to be "car" through the fourth mapping table.

[0116] Step S680: store the spoken audio segment, and label the spoken audio segment based on the feature tag.

[0117] In some embodiments, when the target person's speech audio segment is obtained, the target person's speech audio segment is further stored in an audio database system, in which the user's ID and the speech audio segment stored by the user are stored. Figure 7 , Figure 7 A schematic diagram of an audio database system provided by an embodiment of the present application is schematically shown. In the audio database system, when the user's ID and stored speaking audio clips are obtained, the speaking audio clips stored by the user are searched and deduplicated. If there is a speaking audio clip identical to the speaking audio clip stored by the user in the audio database system, a reminder message is sent to the user, and the reminder message is used to remind the user that the same speaking audio clip already exists in the audio database system; if there is no speaking audio clip identical to the speaking audio clip stored by the user in the audio database system, the subject corresponding to the speaking audio clip stored by the user, that is, the feature tag in step S670, is obtained, and based on the subject corresponding to the speaking audio clip, the speaking audio clip is stored, and the speaking audio clip is marked based on the feature tag. As an implementation method, the electronic device can search and deduplicate the speaking audio clip stored by the user through an inverted index method.

[0118] The embodiment of the present application provides a method for audio acquisition, in which, after acquiring the target person's speech audio segment, the method further includes the step of acquiring the feature tag corresponding to the speech audio segment. After the speech audio segment is stored in the audio database system, the user can quickly find the speech audio segments corresponding to different types through the feature tags corresponding to the speech audio segment.

[0119] See also Figure 8, which shows an audio acquisition device provided by an embodiment of the present application, the device 800 includes: a target facial feature information acquisition module 810, a target video segment acquisition module 820, a target audio segment acquisition module 830 and a speaking audio segment acquisition module 840. Among them, the target facial feature information acquisition module 810 is used to obtain the target facial feature information of the target person. The target video segment acquisition module 820 is used to obtain the target video segment containing the target person based on the target facial feature information. The target audio segment acquisition module 830 is used to obtain the target audio segment corresponding to the target video segment. The speaking audio segment acquisition module 840 is used to obtain the speaking audio segment of the target person from the target audio segment.

[0120] In some embodiments, the target video segment acquisition module 820 is further used to acquire candidate facial feature information from a frame image included in at least one video segment; if the similarity between the candidate facial feature information and the target facial feature information is greater than or equal to a preset similarity, the frame image is determined as a target frame image. The video segment including the target frame image is determined as a candidate video segment; if the number of consecutive target frame images included in the candidate video segment is greater than or equal to a preset number, the candidate video segment is determined as a target video segment containing the target person.

[0121] In some embodiments, the speaking audio segment acquisition module 840 is further used to remove the background audio segment from the target audio segment to acquire the speaking audio segment of the target person.

[0122] In some embodiments, the speech audio segment acquisition module 840 is further used to acquire a first spectrogram of the target audio segment, the first spectrogram being used to characterize the correspondence between time and the frequency of the audio signal in the target audio segment. A first time-frequency mask corresponding to the first spectrogram is acquired, the first time-frequency mask being used to indicate the spectrogram region corresponding to the speech audio segment of the target person. Based on the first time-frequency mask and the first spectrogram, a second spectrogram is acquired. Based on the second spectrogram, the speech audio segment of the target person is acquired.

[0123] In some embodiments, the device 800 further includes a lip gesture information acquisition module (not shown in the figure). The lip gesture information acquisition module (not shown in the figure) is used to acquire the lip gesture information corresponding to the target person in the target video segment. The target audio segment acquisition module 830 is used to execute the step of acquiring the target audio segment corresponding to the target video segment if the lip gesture information indicates that the target person is in a speaking state.

[0124] In some embodiments, the device 800 further includes a first voiceprint feature acquisition module (not shown in the figure), a second voiceprint feature acquisition module (not shown in the figure), and a speech audio segment discarding module (not shown in the figure). The first voiceprint feature acquisition module (not shown in the figure) is used to acquire the first voiceprint feature of the target person. The second voiceprint feature acquisition module (not shown in the figure) is used to acquire the second voiceprint feature of the speech audio segment. The speech audio segment discarding module (not shown in the figure) is used to discard the speech audio segment if the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is less than a preset voiceprint similarity.

[0125] In some embodiments, the device 800 further includes a text information acquisition module (not shown in the figure), a keyword acquisition module (not shown in the figure), a feature tag acquisition module (not shown in the figure) and a speech audio segment storage module (not shown in the figure). Among them, the text information acquisition module (not shown in the figure) is used to perform audio recognition processing on the speech audio segment to obtain text information corresponding to the speech audio segment. The keyword acquisition module (not shown in the figure) is used to process the text information and obtain at least one keyword in the text information. The feature tag acquisition module (not shown in the figure) is used to obtain a feature tag of the speech audio segment based on at least one keyword. The speech audio segment storage module (not shown in the figure) is used to store the speech audio segment and mark the speech audio segment based on the feature tag.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0127] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0128] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0129] The present application provides a device for audio acquisition. In the device, a target video segment containing a target person is determined based on the target person's target facial feature information, then a target audio segment corresponding to the target video segment is acquired, and finally a speaking audio segment of the target person is extracted from the target audio segment. On the one hand, the device realizes automatic acquisition of the speaking audio segment of the target person without human intervention, thereby improving the acquisition efficiency of the speaking audio segment of the target person; on the other hand, the device makes full use of abundant Internet resources in the process of determining the target video segment containing the target person, thereby ensuring the richness and diversity of the acquired audio speaking audio segments of the target person.

[0130] See also Fig. 9 , which shows that the embodiment of the present application also provides an electronic device 900, which includes: one or more processors 910, a memory 920 and one or more applications. Among them, the one or more applications are stored in the memory 920 and are configured to be executed by the one or more processors 910, and the one or more applications are configured to execute the method described in the above embodiment.

[0131] The processor 910 may include one or more processing cores. The processor 910 uses various interfaces and lines to connect various parts of the entire battery management system, and executes various functions and processes data of the battery management system by running or executing instructions, programs, code sets or instruction sets stored in the memory 920, and calling data stored in the memory 920. Optionally, the processor 910 can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 910 can integrate one or more combinations of a central processing unit 910 (CPU), a graphics processing unit 910 (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 910, but may be implemented separately through a communication chip.

[0132] The memory 920 may include a random access memory 920 (Random Access Memory, RAM), and may also include a read-only memory 920 (Read-Only Memory). The memory 920 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created by the electronic device during use (such as a phone book, audio and video data, chat record data), etc.

[0133] See also Fig.10 , which shows that an embodiment of the present application also provides a computer-readable storage medium 1000, in which computer program instructions 1010 are stored. The computer program instructions 1010 can be called by a processor to execute the method described in the above embodiment.

[0134] The computer-readable storage medium 1000 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. Optionally, the computer-readable storage medium 1000 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1000 has a storage space for computer program instructions 1010 for executing any method steps of the above method. These computer program instructions 1010 can be read from or written to one or more computer program products, and the computer program instructions 1010 can be compressed in an appropriate form.

[0135] The above are only preferred embodiments of the present application, and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technical personnel in the field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present application. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A method for audio acquisition, characterized in that: The method comprises: Obtain target facial feature information of the target person; Based on the target person's facial feature information, obtaining a target video clip containing the target person; Obtaining a target audio segment corresponding to the target video segment; Acquire the target person's speech audio segment from the target audio segment; wherein, acquiring the target person's speech audio segment from the target audio segment specifically includes: Acquire a first spectrogram of the target audio segment, where the first spectrogram is used to represent a correspondence between time and frequency of an audio signal in the target audio segment; Obtaining a first time-frequency mask corresponding to the first spectrogram, where the first time-frequency mask is used to indicate a spectrogram region corresponding to the speech audio segment of the target person; Performing a dot product operation on the first time-frequency mask and the first spectrogram, and determining a dot product result corresponding to the dot product operation as a second spectrogram; Performing a Hilbert transform on the second spectrogram to obtain a speech audio signal of the target person; The speech audio signal is reconstructed by an audio reconstruction algorithm to obtain a speech audio segment of the target person.

2. The method according to claim 1, characterized in that The step of acquiring a target video clip containing the target person based on the target person's facial feature information includes: Acquire candidate facial feature information from a frame image included in at least one video clip; If the similarity between the candidate facial feature information and the target facial feature information is greater than or equal to a preset similarity, determining the frame image as a target frame image; Determine a video segment including the target frame image as a candidate video segment; If the number of continuous target frame images included in the candidate video segment is greater than or equal to a preset number, the candidate video segment is determined as a target video segment containing the target person.

3. The method according to claim 1 or 2, characterized in that: After acquiring the target video segment containing the target person based on the target facial feature information and before acquiring the target audio segment corresponding to the target video segment, the method further includes: Obtaining lip posture information corresponding to the target person in the target video clip; If the lip gesture information indicates that the target person is in a speaking state, the step of obtaining the target audio segment corresponding to the target video segment is performed.

4. The method according to claim 1 or 2, characterized in that: After acquiring the target person's speech audio segment from the target audio segment, the method further includes: Acquire a first voiceprint feature of the target person; Acquire a second voiceprint feature of the speech audio segment; If the voiceprint similarity between the second voiceprint feature and the first voiceprint feature is less than a preset voiceprint similarity, the speech audio segment is discarded.

5. The method according to claim 1 or 2, characterized in that: After acquiring the target person's speech audio segment from the target audio segment, the method further includes: Performing audio recognition processing on the speech audio segment to obtain text information corresponding to the speech audio segment; Processing the text information to obtain at least one keyword in the text information; Based on at least one of the keywords, obtaining a feature tag of the speech audio segment; The speech audio segment is stored, and the speech audio segment is labeled based on the feature tag.

6. An audio acquisition device, characterized in that: The device comprises: A target facial feature information acquisition module is used to acquire target facial feature information of a target person; A target video segment acquisition module, used for acquiring a target video segment containing the target person based on the target person's facial feature information; A target audio segment acquisition module is used to acquire a target audio segment corresponding to the target video segment; A speaking audio segment acquisition module is used to acquire the speaking audio segment of the target person from the target audio segment; wherein the speaking audio segment acquisition module is specifically used to acquire a first spectrogram of the target audio segment, the first spectrogram being used to characterize the correspondence between time and audio signal frequency in the target audio segment; acquire a first time-frequency mask corresponding to the first spectrogram, the first time-frequency mask being used to indicate the spectrogram region corresponding to the speaking audio segment of the target person; perform a dot product operation on the first time-frequency mask and the first spectrogram, and determine the dot product result corresponding to the dot product operation as a second spectrogram; perform a Hilbert transform on the second spectrogram to obtain the speaking audio signal of the target person; and perform signal reconstruction on the speaking audio signal through an audio reconstruction algorithm to acquire the speaking audio segment of the target person.

7. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, one or more of the applications are stored in the memory and configured to be executed by one or more of the processors, and one or more of the applications are configured to execute the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and the computer program instructions can be called by a processor to execute the method according to any one of claims 1 to 5.

Images