Speaker separation method, device, electronic device and storage medium

Abstract

The present invention provides a speaker separation method, device, electronic device and storage medium, the method comprising: performing personnel detection on a video frame to obtain the personnel position of the personnel in the video frame; performing sound source localization on an audio segment corresponding to the video frame to obtain the sound source position; and performing speaker separation on the audio segment based on the relative position relationship between the personnel position and the sound source position. The speaker separation method, device, electronic device and storage medium provided by the present invention can avoid the influence of environmental noise and the degree of difference in voiceprint features of speakers of different roles, and can further accurately perform speaker separation on the audio segment based on the relative position relationship between the personnel position and the sound source position.

Description

Technical Field

[0001] The present invention relates to the field of intelligent voice technology, and in particular to a speaker separation method, device, electronic equipment and storage medium. Background Art

[0002] Speaker separation refers to segmenting the audio data belonging to each speaker in an audio file, merging the audio data of the same speaker into one category, and separating the audio data of different speakers.

[0003] At present, speaker separation is mostly achieved by extracting the voiceprint features of each speaker in the audio file and comparing the voiceprint features of each speaker with the voiceprint features in the voiceprint feature library. However, an audio file may contain voiceprint features corresponding to multiple different speakers, and the voiceprint features of some speakers are not stored in the voiceprint feature library, resulting in low speaker separation accuracy. Summary of the invention

[0004] The present invention provides a speaker separation method, device, electronic equipment and storage medium, which are used to solve the defect of low speaker separation accuracy in the prior art.

[0005] The present invention provides a method for separating speakers, comprising:

[0006] Performing personnel detection on the video frame to obtain the personnel position of the personnel in the video frame;

[0007] Performing sound source localization on the audio segment corresponding to the video frame to obtain the sound source position;

[0008] Based on the relative position relationship between the person position and the sound source position, the audio segment is subjected to speaker separation.

[0009] According to a speaker separation method provided by the present invention, the relative position relationship between the person position and the sound source position is determined based on the following steps:

[0010] Based on the position of the person and the position of the microphone when collecting the audio segment, determine the person arrival phase, where the person arrival phase is used to characterize the arrival phase when the person position is a sound source;

[0011] Determine the sound source arrival phase based on the sound source position and the microphone position;

[0012] The relative position relationship is determined based on the personnel arrival phase and the sound source arrival phase.

[0013] According to a speaker separation method provided by the present invention, determining the arrival phase of a person based on the person's position and the microphone position when collecting the audio segment includes:

[0014] Based on the position of the person, determining a person ring angle and a person pitch angle of the person relative to an array center of a microphone array, wherein the microphone array is used to collect the audio segment;

[0015] Determining a relative center angle of each microphone relative to a center of the array based on a microphone position of each microphone in the microphone array;

[0016] Determine the personnel phase of each microphone based on the personnel ring angle, the personnel pitch angle and the relative central angle;

[0017] Based on the personnel phases of each two microphones, the personnel arrival phase is determined.

[0018] According to a speaker separation method provided by the present invention, determining the sound source arrival phase based on the sound source position and the microphone position includes:

[0019] Based on the sound source position, determining a sound source ring angle and a sound source pitch angle of the sound source relative to the array center of the microphone array;

[0020] Determining a relative center angle of each microphone relative to a center of the array based on a position of each microphone in the microphone array;

[0021] Determine the sound source phase of each microphone based on the sound source ring angle, the sound source pitch angle and the relative central angle;

[0022] Based on the sound source phases of each two microphones, the sound source arrival phase is determined.

[0023] According to a speaker separation method provided by the present invention, the speaker separation of the audio segment based on the relative position relationship between the person position and the sound source position includes:

[0024] When the audio segment is speech, speaker separation is performed on the audio segment based on the relative position relationship and the sound source arrival phase.

[0025] According to a method for speaker separation provided by the present invention, when the audio segment is speech, based on the relative position relationship and the sound source arrival phase, the speaker separation is performed on the audio segment, comprising:

[0026] Extracting audio features of the audio segment;

[0027] It is determined whether the audio segment is speech based on the audio feature, and if the audio segment is speech, the speaker separation is performed on the audio segment based on the relative position relationship and the sound source arrival phase.

[0028] According to a speaker separation method provided by the present invention, the speaker separation of the audio segment based on the relative position relationship between the person position and the sound source position includes:

[0029] Determine the distance between the person and the sound source based on the person arrival phase and the sound source arrival phase;

[0030] The audio segments are subjected to speaker separation based on the distance between the persons and the sound source.

[0031] According to a speaker separation method provided by the present invention, the step of performing personnel detection on a video frame to obtain the position of a person in the video frame includes:

[0032] Determine a target detection result based on a previous person target detection frame in a previous video frame and a person target detection frame in the video frame;

[0033] Determine a visual detection result based on the visual features of the previous person in the previous video and the visual features of the person in the video frame;

[0034] The person matching result is determined based on the target detection result and the visual detection result.

[0035] The present invention also provides a speaker separation device, comprising:

[0036] A detection unit, used to perform personnel detection on a video frame to obtain a position of a person in the video frame;

[0037] A positioning unit, used to perform sound source positioning on the audio segment corresponding to the video frame to obtain a sound source position;

[0038] The separation unit is used to separate the speakers of the audio segment based on the relative position relationship between the person position and the sound source position.

[0039] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned speaker separation methods are implemented.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned speaker separation methods are implemented.

[0041] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned speaker separation methods are implemented.

[0042] The speaker separation method, device, electronic device and storage medium provided by the present invention perform person detection on video frames to obtain the person positions of the persons in the video frames, and perform sound source positioning on the audio segments corresponding to the video frames to obtain the sound source positions. Since the person positions are determined based on the image information in the video frames, the influence of environmental noise and the degree of difference in voiceprint features of speakers with different roles can be avoided, and then the speaker separation of the audio segments can be accurately performed based on the relative position relationship between the person positions and the sound source positions. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is one of the flow charts of the speaker separation method provided by the present invention;

[0045] Figure 2 It is a flow chart of the step of determining the relative position relationship between the position of a person and the position of a sound source in the speaker separation method provided by the present invention;

[0046] Figure 3 is a flowchart of an implementation of step 210 in the speaker separation method provided by the present invention;

[0047] Figure 4 is a flow chart of an implementation of step 220 in the speaker separation method provided by the present invention;

[0048] Figure 5 It is a schematic diagram of determining the arrival phase of a sound source based on a microphone array in the speaker separation method provided by the present invention;

[0049] Figure 6 It is one of the flowcharts of the implementation of step 130 in the speaker separation method provided by the present invention;

[0050] Figure 7 This is the second flow chart of the implementation of step 130 in the speaker separation method provided by the present invention;

[0051] Figure 8 is a flow chart of an implementation of step 110 in the speaker separation method provided by the present invention;

[0052] Fig. 9 This is the second flow chart of the speaker separation method provided by the present invention;

[0053] Fig.10It is a structural schematic diagram of the speaker separation device provided by the present invention;

[0054] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] At present, traditional speaker separation methods mostly perform endpoint detection on audio files, retain the audio files corresponding to the voice clips, and then extract features from the audio files to obtain multiple voiceprint features. All voiceprint features are clustered, and then each type of voiceprint feature is compared with the voiceprint features in the voiceprint feature library to achieve speaker separation. However, an audio file may contain voiceprint features corresponding to multiple different speakers, and the voiceprint features of some speakers are not stored in the voiceprint feature library, resulting in the inability to accurately separate speakers. At the same time, for different speakers of the same gender, their voiceprint features have a high degree of similarity, so they may be mistakenly clustered into one category, which in turn leads to incorrect speaker separation.

[0057] In addition, since the movement of a person's mouth is closely related to pronunciation when they speak, traditional methods also achieve speaker separation based on the consistency between audio and lip shape. However, this method requires that the speaker's lip shape is completely visible. If the speaker's lip shape is blocked, accurate speaker separation cannot be achieved.

[0058] To this end, the present invention provides a speaker separation method. Figure 1 It is one of the flow charts of the speaker separation method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0059] Step 110: Perform personnel detection on the video frame to obtain the positions of the personnel in the video frame.

[0060] Here, the video frame is an image frame corresponding to the audio segment for which speaker separation is required, and the video frame may contain information of multiple persons. The person positions of persons in the video frame include person role information and position information of persons with different roles.

[0061] Optionally, the video frame may be subjected to personnel detection to extract key point features for characterizing the role information and location information of each person, wherein the key point features may be features corresponding to the face, eyes, ears, etc. For example, the video frame may be input into a target detection model to obtain the key point features of each person in the video frame output by the target detection model.

[0062] In addition, in the video frames corresponding to different moments, there may be people of different roles, or people of the same role may be located at different positions. For example, the video frame at moment T includes people A, B, and C, and the video frame at moment T+1 includes people A, B, and D, that is, in the video frame at moment T+1, person C is reduced and person D is added. For another example, in the video frame at moment T, person A is located at position 1, and in the video frame at moment T+1, person A is located at position 2, that is, person A moves from position 1 to position 2.

[0063] Based on this, the corresponding person roles and positions of each person role in the video frame at each moment can also be used as prior information for speaker separation to help improve the accuracy of speaker separation. For example, if it is determined that the video frame at time T includes person A, person B, and person C, then the corresponding prior information is that the speaker in the audio segment corresponding to the video frame at time T may be one or more of person A, person B, and person C, and cannot be other persons other than person A, person B, and person C. For another example, person A is located at position 1 in the video frame at time T, and if the position of the sound source in the audio segment corresponding to the video frame at time T is position 2, then the corresponding prior information is that person A is not the speaker corresponding to the sound source.

[0064] Step 120: localize the sound source of the audio segment corresponding to the video frame to obtain the sound source position.

[0065] Specifically, an audio segment refers to an audio file that needs to be speaker separated. An audio segment may contain multiple voice segments, each of which corresponds to a speaker. The sound source position is the position of the speaker, that is, after determining the sound source position corresponding to each voice segment, the speaker position corresponding to each voice segment can be obtained. Optionally, when locating the sound source of an audio segment, a microphone array or a microphone array may be used for sound source localization, or a sound intensity probe may be used for sound source localization, which is not specifically limited in the embodiment of the present invention.

[0066] Step 130: Based on the relative position relationship between the person's position and the sound source position, perform speaker separation on the audio segment.

[0067] Specifically, the relative position relationship between the person position and the sound source position is used to characterize the distance between the person position and the sound source position in the video frame. The smaller the distance between the two, the closer the distance between the corresponding person position and the sound source position, indicating that the probability that the corresponding person is the sound source is higher; the smaller the distance between the two, the larger the distance between the corresponding person position and the sound source position, indicating that the probability that the corresponding person is the sound source is lower.

[0068] Optionally, the distance between the person's position and the sound source position may be used to characterize the relative position relationship between the person's position and the sound source position, and the person corresponding to the minimum distance is taken as the speaker of the sound source to achieve speaker separation of the audio segment.

[0069] It can be seen that compared with the traditional method of extracting the voiceprint features of each speaker in the audio file and comparing the voiceprint features of each speaker with the voiceprint features in the voiceprint feature library to achieve speaker separation, the embodiment of the present invention determines the position of the person from the video frame without being affected by environmental noise and the degree of difference in the voiceprint features corresponding to speakers of different roles, so that even when the voiceprint features in the voiceprint feature library are incomplete or the similarity of the voiceprint features of different speakers of the same gender is high in the traditional method, the speaker separation of the audio segment can be accurately performed. Compared with the traditional method of achieving speaker separation based on the consistency between audio and lip shape, the embodiment of the present invention can also accurately determine the position of the person and the sound source position when the speaker's lip shape is blocked, so as to accurately perform speaker separation on the audio segment.

[0070] The speaker separation method provided by the embodiment of the present invention performs person detection on video frames to obtain the person positions of the persons in the video frames, and performs sound source positioning on the audio segments corresponding to the video frames to obtain the sound source positions. Since the person positions are determined based on the image information in the video frames, the influence of environmental noise and the degree of difference in voiceprint features of speakers with different roles can be avoided, and then the speaker separation of the audio segments can be accurately performed based on the relative position relationship between the person positions and the sound source positions.

[0071] Based on the above embodiments, Figure 2 FIG. 1 is a flow chart of the step of determining the relative position relationship between the position of a person and the position of a sound source in the speaker separation method provided by the present invention. Figure 2 As shown, the relative position relationship between the personnel position and the sound source position is determined based on the following steps:

[0072] Step 210: Determine the arrival phase of the person based on the person's position and the microphone position when collecting the audio segment, where the person's arrival phase is used to characterize the arrival phase when the person's position is the sound source;

[0073] Step 220: Determine the sound source arrival phase based on the sound source position and the microphone position;

[0074] Step 230: Determine the relative position relationship based on the arrival phase of the person and the arrival phase of the sound source.

[0075] Specifically, the person arrival phase is the phase when the person position arrives at the microphone when the person position is assumed to be the sound source. After determining the person position, combined with the microphone position, the angle of the person relative to the microphone can be determined, and then based on the angle of the person relative to the microphone, the person arrival phase can be obtained.

[0076] The sound source arrival phase refers to the phase when the actual sound source in the audio segment reaches the microphone. After determining the sound source position, combined with the microphone position, the angle of the sound source relative to the microphone can be determined, and then based on the angle of the sound source relative to the microphone, the sound source arrival phase can be obtained.

[0077] Since the arrival phase of the person is the arrival phase when the person is assumed to be the sound source, if you want to further determine whether the person is the sound source, you need to make a further judgment in combination with the sound source arrival phase. Therefore, the embodiment of the present invention determines the relative position relationship based on the person arrival phase and the sound source arrival phase. The relative position relationship is used to characterize the distance between the person position and the sound source position. The smaller the distance between the two, the higher the probability that the person is the sound source position, that is, the higher the probability that the person is the speaker.

[0078] It can be seen that the embodiment of the present invention can determine the relative position relationship used to characterize the distance between the position of the person and the position of the sound source based on the person arrival phase and the sound source arrival phase, so that the speakers of the audio segment can be accurately separated based on the relative position relationship.

[0079] Based on any of the above embodiments, Figure 3 is a flow chart of an implementation of step 210 in the speaker separation method provided by the present invention, such as Figure 3 As shown, step 210 specifically includes:

[0080] Step 211: Based on the position of the person, determine the person's ring angle and the person's pitch angle relative to the array center of the microphone array, where the microphone array is used to collect audio segments;

[0081] step 2 12. Determine a relative center angle of each microphone relative to the center of the array based on the microphone position of each microphone in the microphone array;

[0082] Step 213: determine the personnel phase of each microphone based on the personnel ring angle, the personnel pitch angle and the relative central angle;

[0083] Step 214: Determine the arrival phase of a person based on the person phases of every two microphones.

[0084] Specifically, the person's ring angle and the person's pitch angle are used to characterize the angle of the person relative to the array center of the microphone array, and the relative center angle is used to characterize the angle of each microphone relative to the array center of the microphone array. When the person's position is assumed to be the sound source position, since the signal at the same person's position arrives at each microphone at different times, resulting in differences in the phase of each microphone channel signal, based on the person's ring angle, the person's pitch angle and the relative center angle, the person's phase used to characterize the person's arrival at each microphone can be determined, and then based on the person's phase of each two microphones, the person's phase difference between the channels of each two microphones can be determined, and finally the person's arrival phase can be finally determined based on the phase difference of each person.

[0085] Based on any of the above embodiments, Figure 4 is a flow chart of an implementation of step 220 in the speaker separation method provided by the present invention, such as Figure 4 As shown, step 220 specifically includes:

[0086] Step 221: Based on the sound source position, determine the sound source ring angle and the sound source pitch angle of the sound source relative to the array center of the microphone array;

[0087] Step 222: determine a relative center angle of each microphone relative to the center of the array based on the position of each microphone in the microphone array;

[0088] Step 223, determining the sound source phase of each microphone based on the sound source ring angle, the sound source pitch angle and the relative center angle;

[0089] Step 224: Determine the sound source arrival phase based on the sound source phases of every two microphones.

[0090] Specifically, the sound source ring angle and the sound source pitch angle are used to characterize the angle of the sound source relative to the array center of the microphone array, and the relative center angle is used to characterize the angle of each microphone relative to the array center of the microphone array. Since the sound source signal arrives at each microphone at different times, resulting in differences in the phase of each microphone channel signal, the sound source phase used to characterize the sound source arriving at each microphone can be determined based on the sound source ring angle, the sound source pitch angle, and the relative center angle, and then based on the sound source phase of each two microphones, the sound source phase difference between the channels of each two microphones is determined, and finally the sound source arrival phase is finally determined based on the phase difference of each sound source.

[0091] Figure 5 Schematic diagram of determining the arrival phase of a sound source based on a microphone array in the speaker separation method provided by the present invention, such as Figure 5As shown in the figure, 1-8 represent the positions of the microphones in the microphone array, α represents the sound source ring angle relative to the array center of the microphone array, and β represents the sound source pitch angle relative to the array center of the microphone array. Since the time for the sound source signal to reach each microphone is different, the phase of each channel signal is different. Taking the array center of the microphone array as the reference point, the sound source signal received by each microphone is:

[0092]

[0093] Where s(t) represents the signal amplitude, f represents the frequency, R represents the array radius, and θ i represents the relative central angle of each microphone relative to the center of the array, c represents the speed of sound, Represents the phase of the array center. Based on short-time Fourier transform, the phase of each channel at each moment can be obtained as:

[0094]

[0095] Therefore, the inter-channel phase difference (IPD) between channels i and j is:

[0096]

[0097] By calculating the phase difference of the sound sources between multiple groups of channels, the phase of the sound sources can be obtained.

[0098] Similarly, based on the same method mentioned above, when calculating the arrival phase of a person, it is assumed that the person's position is the sound source position, and the person's ring angle relative to the array center of the microphone array is determined. and personnel pitch angle Then we can get the personnel phase difference between channels i and j:

[0099]

[0100] By calculating the personnel phase difference between multiple groups of channels, the personnel arrival phase can be obtained.

[0101] It can be seen that the embodiment of the present invention, combined with the positions of each microphone in the microphone array, can accurately determine the arrival phase of the person and the arrival phase of the sound source, and then can accurately obtain the relative position relationship between the person position and the sound source position based on the arrival phase of the person and the arrival phase of the sound source.

[0102] Based on any of the above embodiments, based on the relative position relationship between the person position and the sound source position, performing speaker separation on the audio segment includes:

[0103] When the audio segment is speech, the speakers of the audio segment are separated based on the relative position relationship and the sound source arrival phase.

[0104] Specifically, there may be speech and noise in the audio segment. In order to avoid mistaking noise as the sound source, the embodiment of the present invention performs speaker separation on the audio segment based on the phase position relationship and the sound source arrival phase when the audio segment is speech.

[0105] Optionally, after obtaining the audio segment corresponding to the video frame, voice endpoint detection can be performed on the audio segment to identify the voice in the audio segment and mark the voice part in the audio segment. Thus, if the audio segment is voice, the speakers of the audio segment can be separated based on the relative position relationship and the arrival phase of the sound source.

[0106] It can be seen that, in the embodiment of the present invention, when the audio segment is speech, speaker separation is performed on the audio segment, thereby avoiding the influence of noise carried in the audio segment on the speaker separation accuracy.

[0107] Based on any of the above embodiments, Figure 6 is one of the flow charts of the implementation of step 130 in the speaker separation method provided by the present invention, such as Figure 6 As shown, in step 130, when the audio segment is speech, based on the relative position relationship and the sound source arrival phase, the audio segment is separated into speakers, including:

[0108] Step 131a, extracting audio features of the audio segment;

[0109] Step 132a: determine whether the audio segment is speech based on the audio features, and if the audio segment is speech, separate the speakers of the audio segment based on the relative position relationship and the sound source arrival phase.

[0110] Specifically, after determining the audio segment for speaker separation, the audio features of the audio segment are extracted, so that the audio segment can be distinguished as speech or noise based on the audio features; if the audio segment is noise, it indicates that the sound source corresponding to the audio segment is not the speaker, that is, the audio segment does not need to be separated from the speaker; if the audio segment is speech, it indicates that the sound source corresponding to the audio segment is the speaker, that is, the audio segment needs to be separated from the speaker.

[0111] Optionally, after the audio features are extracted, the audio features, relative position relationship and sound source arrival phase can be input into a speaker separation model, and the speaker separation model determines whether the audio segment is speech based on the audio features, and if the audio segment is speech, the audio segment is speaker separated based on the relative position relationship and the sound source arrival phase, and the speaker separation result output by the speaker separation model is obtained. Among them, the relative position relationship contains the distance information between the person and the sound source. The smaller the distance, the higher the probability that the corresponding person is the speaker, that is, the relative position relationship can make a preliminary judgment on the probability that the person in the video frame is the speaker. The sound source arrival phase is used to characterize the arrival angle of the microphone receiving the sound source signal. Combined with the preliminary judgment result of the probability that the person is the speaker in the relative position relationship, the speaker of the sound source can be further determined, and then the audio segment is speaker separated to obtain the speaker separation result.

[0112] Among them, after the audio features, relative position relationships and sound source arrival phases are input into the speaker separation model, the audio features, relative position relationships and sound source arrival phases can be spliced ​​through the feature splicing layer of the speaker separation model to obtain splicing features, and then the speaker separation layer of the speaker separation model determines the speaker separation results based on the splicing features.

[0113] It is understandable that the audio features in the embodiment of the present invention can also be input into the speaker separation model in the form of a spectrogram, and then the speaker separation model fuses the spectrogram, relative position relationship and sound source arrival phase to obtain the speaker separation result.

[0114] In addition, before the audio features, relative position relationships, and sound source arrival phases are input into the speaker separation model, the speaker separation model can be pre-trained, which can be achieved by performing the following steps: First, collect sample audio features, sample relative position relationships, and sample sound source arrival phases in a large number of sample video frames, and determine the corresponding speaker labels of each person in the sample video frames through manual annotation. Then, the initial model is trained based on the sample audio features, sample relative position relationships, sample sound source arrival phases, and speaker labels of each person in the sample video frames, thereby obtaining a speaker separation model. The speaker separation model can be trained based on the loss function of the classification network, such as based on the cross entropy loss function.

[0115] Based on any of the above embodiments, Figure 7 FIG. 2 is a flow chart of the implementation of step 130 in the speaker separation method provided by the present invention. Figure 7 As shown, step 130 specifically includes:

[0116] Step 131b: determining the distance between the person and the sound source based on the person arrival phase and the sound source arrival phase;

[0117] Step 132b: Based on the distance between the person and the sound source, perform speaker separation on the audio segment.

[0118] Specifically, after determining the arrival phase of the person and the arrival phase of the sound source, the distance between the person and the sound source can be calculated. The larger the distance, the smaller the probability that the person is the sound source, that is, the smaller the probability that the person is the speaker; the larger the distance, the greater the probability that the person is the sound source, that is, the greater the probability that the person is the speaker.

[0119] Optionally, the distance between the person and the sound source in the embodiment of the present invention can be represented by the cosine distance between the two. The closer the cosine distance is to 1, the smaller the distance between the person and the sound source is, that is, the greater the probability that the person's position is the sound source position.

[0120] It can be seen that the embodiment of the present invention can accurately determine whether the position of the person is the position of the sound source based on the distance between the person and the sound source, and further can accurately separate the speakers of the audio segment.

[0121] Based on any of the above embodiments, Figure 8 is a flow chart of an implementation of step 110 in the speaker separation method provided by the present invention, such as Figure 8 As shown, step 110 specifically includes:

[0122] Step 111: determining a target detection result based on a previous person target detection frame in a previous video frame and a person target detection frame in a video frame;

[0123] Step 112: determining a visual detection result based on the visual features of the previous person in the previous video and the visual features of the person in the video frame;

[0124] Step 113: Determine a person matching result based on the target detection result and the visual detection result.

[0125] Specifically, in the video frames corresponding to different moments, there may be people with different roles, or people with the same role may be located in different positions, and the positions of people in the video frames at each moment can be used as prior information for speaker separation to help improve the accuracy of speaker separation.

[0126] For example, the video frame at time T detects three persons a, b, and c, and the video frame at time T+1 detects four persons a, b, d, and e. Since person c is reduced and person d and person e are increased in the video frame at time T+1 relative to the video frame at time T, the speaker in the audio segment corresponding to the video frame at time T+1 cannot be person c.

[0127] In addition, when determining the position of a person in a video frame, the embodiment of the present invention can determine the movement trajectory of the person in combination with the position of the person in the previous video frame, and can determine the person corresponding to the sound source position by combining the movement trajectory of the person and the position of the sound source. For example, in the video frame at time T, person A is located at position X, person B is located at position Y, and person A is the speaker. In the video frame at time T+1, person A is located at position Y, person B is located at position X, and person B is the speaker. If the speaker separation is performed only based on the sound source position, it can only be determined that someone is speaking at position X in the video frame at time T and the video frame at time T+1, but it cannot be determined who is speaking in each video frame. Therefore, the embodiment of the present invention can determine the movement trajectories of person A and person B in combination with the position of the person, A(T:X->T+1:Y), B(T+:Y->T+1:X). It can thus be determined that person A is the speaker in the video frame at time T, and person B is the speaker in the video frame at time T+1.

[0128] Among them, the specific method for determining the position of the person is: based on the previous person target detection frame in the previous video frame and the person target detection frame in the video frame, the intersection and union ratio between the two target detection frames can be determined, and the intersection and union ratio is used as the target detection result; wherein, the larger the intersection and union ratio, the higher the overlap between the two target detection frames, that is, the higher the probability that the two target detection frames are the same person. At the same time, based on the visual features of the previous person in the previous video and the visual features of the person in the video frame, the person matching can be further performed based on the visual feature information in the two video frames to determine the visual detection result. Finally, the target detection result and the visual detection result can be weighted and summed to determine the final person matching result.

[0129] It can be seen that the embodiment of the present invention can accurately match people based on the previous video frame and the person target detection frame in the video frame, as well as the previous video frame and the person visual features in the video frame, and thus obtain a person matching result.

[0130] Based on any of the above embodiments, Fig. 9 FIG. 2 is a flow chart of the speaker separation method provided by the present invention. Fig. 9 As shown, the method includes:

[0131] The object detection is performed on the video frame to obtain the object detection frame used to characterize the position of the person in the video frame, and then the ring angle and pitch angle of the person relative to the video device are determined by camera calibration based on the position information of the person in the object detection frame. It should be noted that the camera and the microphone in the video device are located in the same coordinate system. If the camera and the microphone are located in different coordinate systems, a mapping relationship between the camera coordinate system and the microphone coordinate system can be established.

[0132] Since the people in the video frames at different moments are moving, the multi-target tracking method can be used to track each person in the video frame, determine the movement trajectory of each person, obtain the position of the person in the video frame, and determine the arrival phase of the person based on the position of the person and the position of each microphone in the microphone array.

[0133] At the same time, the sound source is localized for the audio segment corresponding to the video frame to obtain the sound source position, and the arrival phase of the sound source is determined based on the sound source position and the positions of each microphone in the microphone array. The cosine distance between the arrival phase of the person and the arrival phase of the sound source is used as the spatial feature.

[0134] The audio features, spatial features and sound source arrival phase of the audio segment are input into the speaker separation model. When the speaker separation model determines that the audio feature is speech, it determines the speaking probability of each person in the video frame based on the spatial features and sound source features, thereby achieving speaker separation for the audio segment.

[0135] The speaker separation device provided by the present invention is described below. The speaker separation device described below and the speaker separation method described above can be referenced to each other.

[0136] Fig.10 Schematic diagram of the structure of the speaker separation device provided by the present invention. Fig.10 As shown, the device comprises:

[0137] A detection unit 1010 is used to perform personnel detection on a video frame to obtain a position of a person in the video frame;

[0138] A positioning unit 1020, configured to perform sound source positioning on the audio segment corresponding to the video frame to obtain a sound source position;

[0139] The separation unit 1030 is used to separate the speakers of the audio segment based on the relative position relationship between the person position and the sound source position.

[0140] Based on any of the above embodiments, the device further includes:

[0141] A person arrival phase determination unit, used to determine a person arrival phase based on the person position and the microphone position when collecting the audio segment, wherein the person arrival phase is used to characterize an arrival phase when the person position is a sound source;

[0142] A sound source arrival phase determination unit, configured to determine the sound source arrival phase based on the sound source position and the microphone position;

[0143] A relative position determination unit is used to determine the relative position relationship based on the arrival phase of the person and the arrival phase of the sound source.

[0144] Based on any of the above embodiments, the personnel arrival phase determination unit includes:

[0145] A person angle determination unit, used to determine a person ring angle and a person pitch angle of the person relative to an array center of a microphone array based on the person position, wherein the microphone array is used to collect the audio segment;

[0146] A first microphone angle determination unit, configured to determine a relative center angle of each microphone relative to the center of the array based on a microphone position of each microphone in the microphone array;

[0147] A personnel phase determination unit, configured to determine the personnel phase of each microphone based on the personnel ring angle, the personnel pitch angle, and the relative center angle;

[0148] The person arrival phase determination subunit is used to determine the person arrival phase based on the person phases of every two microphones.

[0149] Based on any of the above embodiments, the sound source arrival phase determination unit includes:

[0150] A sound source angle determination unit, used to determine a sound source ring angle and a sound source pitch angle of the sound source relative to the array center of the microphone array based on the sound source position;

[0151] A second microphone angle determination unit, configured to determine a relative center angle of each microphone relative to the center of the array based on the position of each microphone in the microphone array;

[0152] A sound source phase determination unit, configured to determine the sound source phase of each microphone based on the sound source ring angle, the sound source pitch angle, and the relative center angle;

[0153] The sound source arrival phase determination subunit is used to determine the sound source arrival phase based on the sound source phases of every two microphones.

[0154] Based on any of the above embodiments, the separation unit 1030 is used to:

[0155] When the audio segment is speech, speaker separation is performed on the audio segment based on the relative position relationship and the sound source arrival phase.

[0156] Based on any of the above embodiments, the separation unit 1030 includes:

[0157] An extraction unit, used to extract audio features of the audio segment;

[0158] The speaker separation unit is used to determine whether the audio segment is speech based on the audio feature, and if the audio segment is speech, perform speaker separation on the audio segment based on the relative position relationship and the sound source arrival phase.

[0159] Based on any of the above embodiments, the separation unit 1030 includes:

[0160] a distance determination unit, configured to determine the distance between the person and the sound source based on the person arrival phase and the sound source arrival phase;

[0161] The separation subunit is used to separate the speakers of the audio segment based on the distance between the person and the sound source.

[0162] Based on any of the above embodiments, the detection unit 1010 includes:

[0163] The target detection unit is used to determine the target detection result based on the previous person target detection frame in the previous video frame and the person target detection frame in the video frame;

[0164] A visual detection unit, configured to determine a visual detection result based on a visual feature of a previous person in the previous video and a visual feature of a person in the video frame;

[0165] A personnel matching unit is used to determine the personnel matching result based on the target detection result and the visual detection result.

[0166] Fig.11 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Fig.11 As shown, the electronic device may include: a processor 1110, a memory 1120, a communications interface 1130 and a communications bus 1140, wherein the processor 1110, the memory 1120 and the communications interface 1130 communicate with each other through the communications bus 1140. The processor 1110 may call the logic instructions in the memory 1120 to execute a speaker separation method, which includes: performing personnel detection on a video frame to obtain the personnel position of the personnel in the video frame; performing sound source localization on an audio segment corresponding to the video frame to obtain the sound source position; and performing speaker separation on the audio segment based on the relative position relationship between the personnel position and the sound source position.

[0167] In addition, the logic instructions in the above-mentioned memory 1120 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0168] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the speaker separation method provided by the above-mentioned methods, and the method includes: performing personnel detection on a video frame to obtain the personnel position of the personnel in the video frame; performing sound source localization on an audio segment corresponding to the video frame to obtain the sound source position; and performing speaker separation on the audio segment based on the relative position relationship between the personnel position and the sound source position.

[0169] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-mentioned speaker separation methods, the methods comprising: performing person detection on a video frame to obtain the person position of the person in the video frame; performing sound source localization on an audio segment corresponding to the video frame to obtain the sound source position; and performing speaker separation on the audio segment based on the relative position relationship between the person position and the sound source position.

[0170] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for separating speakers, characterized in that: include: Performing personnel detection on the video frame to obtain the personnel position of the personnel in the video frame; Performing sound source localization on the audio segment corresponding to the video frame to obtain the sound source position; Based on the relative position relationship between the position of the person and the position of the sound source, performing speaker separation on the audio segment; The performing of personnel detection on the video frame to obtain the position of the personnel in the video frame includes: Determine a target detection result based on a previous person target detection frame in a previous video frame and a person target detection frame in the video frame; Determine a visual detection result based on the visual features of the previous person in the previous video and the visual features of the person in the video frame; The person matching result is determined based on the target detection result and the visual detection result.

2. The speaker separation method according to claim 1, characterized in that: The relative position relationship between the position of the person and the position of the sound source is determined based on the following steps: Based on the position of the person and the position of the microphone when collecting the audio segment, determine the person arrival phase, where the person arrival phase is used to characterize the arrival phase when the person position is a sound source; Determine the sound source arrival phase based on the sound source position and the microphone position; The relative position relationship is determined based on the personnel arrival phase and the sound source arrival phase.

3. The speaker separation method according to claim 2, characterized in that: The determining the arrival phase of the person based on the person's position and the microphone position when collecting the audio segment includes: Based on the position of the person, determining a person ring angle and a person pitch angle of the person relative to an array center of a microphone array, wherein the microphone array is used to collect the audio segment; Determining a relative center angle of each microphone relative to a center of the array based on a microphone position of each microphone in the microphone array; Determine the personnel phase of each microphone based on the personnel ring angle, the personnel pitch angle and the relative central angle; Based on the personnel phases of each two microphones, the personnel arrival phase is determined.

4. The speaker separation method according to claim 2, characterized in that: The step of determining the sound source arrival phase based on the sound source position and the microphone position comprises: Based on the sound source position, determining a sound source ring angle and a sound source pitch angle of the sound source relative to the array center of the microphone array; Determining a relative center angle of each microphone relative to a center of the array based on a position of each microphone in the microphone array; Determine the sound source phase of each microphone based on the sound source ring angle, the sound source pitch angle and the relative central angle; Based on the sound source phases of each two microphones, the sound source arrival phase is determined.

5. The speaker separation method according to claim 2, characterized in that: The step of performing speaker separation on the audio segment based on the relative position relationship between the person position and the sound source position comprises: When the audio segment is speech, speaker separation is performed on the audio segment based on the relative position relationship and the sound source arrival phase.

6. The speaker separation method according to claim 5, characterized in that: When the audio segment is speech, based on the relative position relationship and the sound source arrival phase, performing speaker separation on the audio segment includes: Extracting audio features of the audio segment; It is determined whether the audio segment is speech based on the audio feature, and if the audio segment is speech, the speaker separation is performed on the audio segment based on the relative position relationship and the sound source arrival phase.

7. The speaker separation method according to claim 2, characterized in that: The step of performing speaker separation on the audio segment based on the relative position relationship between the person position and the sound source position comprises: Determine the distance between the person and the sound source based on the person arrival phase and the sound source arrival phase; The audio segments are subjected to speaker separation based on the distance between the persons and the sound source.

8. A speaker separation device, characterized in that: include: A detection unit, used to perform personnel detection on a video frame to obtain a position of a person in the video frame; A positioning unit, used to perform sound source positioning on the audio segment corresponding to the video frame to obtain a sound source position; A separation unit, configured to perform speaker separation on the audio segment based on a relative position relationship between the person position and the sound source position; The performing of personnel detection on the video frame to obtain the position of the personnel in the video frame includes: Determine a target detection result based on a previous person target detection frame in a previous video frame and a person target detection frame in the video frame; Determine a visual detection result based on the visual features of the previous person in the previous video and the visual features of the person in the video frame; The person matching result is determined based on the target detection result and the visual detection result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the speaker separation method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the speaker separation method according to any one of claims 1 to 7 are implemented.

Images