Audio signal processing method, device and electronic equipment

Abstract

The embodiment of the present application discloses an audio signal processing method, device and electronic device, the method comprising: performing speech recognition, sound source localization and speech endpoint detection on the audio signal collected in a multi-person speaking scene; determining the speaker change time point in the audio signal according to the sound source localization result, the speaker change time point comprising: the position of the speaker change event moment on the time axis in the audio signal; correcting the speaker change time point according to the position of the speech endpoint on the time axis in the speech endpoint detection result; separating the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments. Through the embodiment of the present application, the efficiency and accuracy of meeting records can be improved, and the workload of meeting record staff can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of audio signal processing, and in particular to an audio signal processing method, device and electronic device. Background Art

[0002] In multi-person speaking scenarios such as meetings and court trials, there is usually a need to record the content of the meeting. In the traditional way, a special clerk is usually required to record the specific speech content and the corresponding speaker on the spot. The recording method is usually that the clerk enters the speech content heard on the spot into a computer or other computer equipment by typing. However, this places high demands on the clerk's professional ability and concentration. Once the speaker changes or someone "interrupts the conversation" at a certain moment, the speaker information needs to be changed in time and the speech content needs to be recorded. Therefore, there may be recording errors or omissions of some content.

[0003] Although there are some speech recognition related products in the prior art, they can usually only convert the collected voice signals into text, and the clerks are required to segment the text strings, mark the specific speaker information, etc. Therefore, a lot of manual operations are still required, and a high error rate will still occur.

[0004] Therefore, how to further reduce the workload of secretaries and other personnel in scenarios where multiple people are speaking and improve the accuracy of the recorded content has become a technical problem that needs to be solved by technical personnel in this field. Summary of the invention

[0005] The present application provides an audio signal processing method, device and electronic device, which can improve the efficiency and accuracy of meeting recording and reduce the workload of meeting recording staff.

[0006] This application provides the following solutions:

[0007] An audio signal processing method, comprising:

[0008] Perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in multi-person speaking scenarios;

[0009] Determine a speaker change time point in the audio signal according to the sound source localization result, wherein the speaker change time point includes: a position of the speaker change event moment in the audio signal on the time axis;

[0010] According to the position of the voice endpoint on the time axis in the voice endpoint detection result, the speaker change time point is corrected;

[0011] According to the corrected speaker change time point, the text obtained by speech recognition is separated to obtain multiple text segments.

[0012] A method for generating meeting minutes, comprising:

[0013] Perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in conference scenarios where multiple people are speaking;

[0014] Determine a speaker change time point in the audio signal according to the sound source localization result, wherein the speaker change time point includes: a position of the speaker change event moment in the audio signal on the time axis;

[0015] According to the position of the voice endpoint on the time axis in the voice endpoint detection result, the speaker change time point is corrected;

[0016] According to the corrected speaker change time point, the text obtained by speech recognition is separated to obtain multiple text segments;

[0017] A meeting record of the meeting is generated according to the separated multiple text segments.

[0018] A live video processing method, comprising:

[0019] Perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a live video broadcast scenario where multiple speakers are located in the same space.

[0020] Determine a speaker change time point in the audio signal according to the sound source localization result, wherein the speaker change time point includes: a position of the speaker change event moment in the audio signal on the time axis;

[0021] According to the position of the voice endpoint on the time axis in the voice endpoint detection result, the speaker change time point is corrected;

[0022] According to the corrected speaker change time point, the text obtained by speech recognition is separated to obtain multiple text segments;

[0023] According to the timeline information corresponding to the separated multiple text segments, the text segments are added to the video images collected in the live video scene to generate a live video image with subtitles.

[0024] An audio signal processing device, comprising:

[0025] A signal processing unit, used to perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a multi-person speaking scenario;

[0026] a change time point determination unit, configured to determine a speaker change time point in the audio signal according to a sound source localization result, wherein the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis;

[0027] A correction unit, used for correcting the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result;

[0028] The speech recognition result segmentation unit is used to segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments.

[0029] A pickup comprises the above-mentioned audio signal processing device.

[0030] A device for generating meeting records, comprising:

[0031] A signal processing unit, used to perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a conference scene where multiple people are speaking;

[0032] a change time point determination unit, configured to determine a speaker change time point in the audio signal according to a sound source localization result, wherein the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis;

[0033] A correction unit, used for correcting the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result;

[0034] A speech recognition result segmentation unit, used to segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments;

[0035] A conference record generating unit is used to generate a conference record of the conference according to the separated multiple text segments.

[0036] A live video processing device, comprising:

[0037] A signal processing unit, for performing speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a live video broadcast scenario where multiple speakers are located in the same space;

[0038] a change time point determination unit, configured to determine a speaker change time point in the audio signal according to a sound source localization result, wherein the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis;

[0039] A correction unit, used for correcting the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result;

[0040] A speech recognition result segmentation unit, used to segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments;

[0041] The subtitle adding unit is used to add the text segments to the video image collected in the video live broadcast scene according to the time axis information corresponding to the separated multiple text segments, so as to generate a live video image with subtitles.

[0042] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0043] Through the embodiments of the present application, it is possible to perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a multi-person speaking scene. In this way, after determining the occurrence location of the speaker change event based on the sound source localization result, the occurrence location of the speaker change event can also be corrected based on the location of the voice endpoint in the speech endpoint detection result. Thereafter, the speech recognition result can be truncated according to the occurrence location of the corrected speaker change event to obtain multiple text segments. Then, speaker marking and other processing can be performed directly based on the separated multiple text segments, thereby reducing the workload of the meeting recorder and reducing the error rate. In addition, through the above method, since the occurrence location of the speaker change event can be determined more accurately, the speech recognition result can be truncated more accurately to avoid the phenomenon of missing the beginning or end of a word.

[0044] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0047] Figure 2 is a flow chart of the first method provided in an embodiment of the present application;

[0048] Figure 3 It is a schematic diagram of a correction method provided in an embodiment of the present application;

[0049] Figure 4-1 , 4-2 is a schematic diagram of a conference record generation interface provided in an embodiment of the present application;

[0050] Figure 5 is a flow chart of the second method provided in an embodiment of the present application;

[0051] Figure 6 is a flowchart of the third method provided in an embodiment of the present application;

[0052] Figure 7 is a schematic diagram of a first device provided in an embodiment of the present application;

[0053] Figure 8 is a schematic diagram of a second device provided in an embodiment of the present application;

[0054] Fig. 9 is a schematic diagram of a third device provided in an embodiment of the present application;

[0055] Fig.10 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0057] In an embodiment of the present application, for a multi-person speaking scenario, in addition to performing speech recognition on the collected audio signal, sound source localization can also be performed. That is to say, in a specific multi-person speaking scenario, such as a multi-person meeting, a specific speaker can usually speak at his or her seat, etc., and the speaker's position usually does not change during the meeting. Therefore, it is possible to determine whether there is an event of speaker change by identifying a sudden change in the direction of the sound source. If so, it is also possible to determine the location where the speaker change occurred, and truncate and separate the speech recognition results at the corresponding location. In this way, the speech recognition result is no longer a whole paragraph of text content, but can be separated into multiple segments according to the time point of the speaker change, so as to facilitate subsequent further speaker marking work.

[0058] Alternatively, in an optional manner, the separated text segments can also be automatically marked with speakers based on the sound source localization results, and text segments from the same sound source direction can also be automatically marked with the same mark, for example, they can be marked as 1, 2, 3, or ID-type identifiers such as A, B, C, etc. You can also add information such as the speaker's name later by manual modification, and the speaker's name corresponding to the same ID can all be automatically modified. In addition, if the correspondence between the position and the speaker's name in a specific scene can be obtained in advance, you can also directly add a specific speaker name identifier to the text segment corresponding to the specific sound source direction, and so on. In the above manner, the workload of manual marking can be reduced, and it is also beneficial to improve accuracy and reduce error rates.

[0059] However, the inventors of the present application found in the process of implementing the above scheme that in actual application, due to the complex environment and the presence of interference factors such as noise and reverberation, the sound source localization sometimes cannot respond in a timely and accurate manner, which may cause the phenomenon of losing the first word and the last word. For example, before, speaker A was speaking, and the actual time when a speaker B started speaking was 3 minutes 22 seconds 10, that is, the actual occurrence time of the speaker change event was 3 minutes 22 seconds 10; however, due to the existence of errors, the occurrence position of the speaker change event determined by the sound source localization result was 3 minutes 22 seconds 50, which was 40ms later than the actual occurrence time. In this 40ms, the pronunciation of the first word spoken by speaker B may have ended. Therefore, when the speech recognition result is truncated according to the speaker change position identified in the sound source localization result, it is possible to start truncating from the second word spoken by speaker B and lose the first word. The situation of losing the last word is similar.

[0060] In view of the above problems, corresponding solutions are also provided in the embodiments of the present application. Specifically, the sound source localization technology can be combined with the speech endpoint detection technology. The endpoint here refers to the critical point of change between silence and effective speech signals. The so-called speech endpoint detection specifically means that for any audio signal, the starting point and end point corresponding to each speech segment can be found, the speech period and the non-speech period can be distinguished, and silence, noise, etc. can also be removed.

[0061] It can be seen that through the voice endpoint detection technology, the starting point and the end point corresponding to each voice period in the audio signal can be accurately determined. Therefore, the occurrence position of the speaker change event can be corrected according to the position of the voice endpoint in the voice endpoint detection result. Then, according to the corrected occurrence position of the speaker change event, the text obtained by speech recognition is separated to obtain multiple text segments. For example, the sound source localization result and the voice endpoint detection result can be aligned on the time axis. After determining the occurrence position of the speaker change event according to the sound source localization result, it can be determined whether there is a detected voice endpoint within the target time before and after the occurrence position. If so, the position of the voice endpoint can be determined as the actual occurrence position of the speaker change event. In the above manner, the occurrence position of the speaker change event can be determined more accurately, and then the speech recognition result can be truncated more accurately to avoid the phenomenon of missing the beginning or the end of a word.

[0062] In specific implementation, from the perspective of system architecture, the embodiments of the present application may specifically correspond to products such as "intelligent sound pickup". Figure 1 , specifically including hardware and software. The hardware part mainly corresponds to microphone arrays, etc., which are used to collect audio signals in multi-person speaking scenarios. In specific implementation, since participants usually sit around the conference table during a meeting, in order to better perform speech recognition and sound source localization, devices such as smart microphones can be placed in the center of the conference table.

[0063] The software is mainly used to process the collected audio signals, which may include multiple modules such as speech recognition, sound source localization, and speech endpoint detection. Based on the results of speech endpoint detection and sound source localization, the location of the speaker change event can be accurately determined, and then the audio signal can be identified as multiple text segments, and the truncation position of each text segment corresponds to the location of the speaker change event.

[0064] Among them, part of the software can be run in the form of an application program on a terminal device such as a personal computer. For example, the above application program can be run on a user computer device responsible for recording the meeting, and a microphone array such as an intelligent microphone can be connected to the computer device. In this way, the audio signal collected by the intelligent microphone can be transmitted to the computer device, and the application program running in the computer device can perform specific data processing and provide a corresponding interface to display the speech recognition results and the text segments that are disassembled or annotated according to the sound source localization results, and finally generate corresponding meeting records.

[0065] The specific implementation method provided in the embodiments of the present application is described in detail below.

[0066] Embodiment 1

[0067] This embodiment is for Figure 1 The multi-person speaking scenario shown in FIG. 1 provides an audio signal processing method, see Figure 2 , the method may specifically include:

[0068] S210: Perform speech recognition, sound source localization, and speech endpoint detection on the audio signals collected in a multi-person speaking scenario.

[0069] Among them, regarding speech recognition, specifically, it is to let the machine convert the speech signal into corresponding text through the recognition and understanding process. Regarding its specific implementation method, it will not be described in detail in the embodiments of this application.

[0070] Regarding sound source localization, in the embodiments of the present application, it mainly refers to sound source localization based on a microphone array. Specifically, it can mainly determine the direction of the sound source (which can be represented by an angle). Since the positions of the microphone and each speaker are usually fixed in scenes such as meetings, when the speaker changes, the direction angle of the specific sound reaching the microphone will change. Therefore, it is possible to detect the direction of the sound source to determine whether an event of changing the speaker occurs during the speech. For example, the seat of a certain participant A is 45 degrees relative to the microphone reference angle (for example, 0 degrees to the north, etc.), and the seat of participant B is 90 degrees relative to the microphone reference angle. If at a certain moment t during the speech of participant A, participant B suddenly interrupts A and starts speaking. At this time, through sound source localization, it can be determined that the sound source localization result of a certain signal frame is 45 degrees, and the sound source localization result of the next signal frame suddenly changes to 90 degrees. Therefore, the position of the signal frame can be determined as the position where the speaker changes, and so on.

[0071] Among them, in order to be able to perform the above-mentioned sound source localization more accurately and timely, in a preferred embodiment of the present application, the wave direction spectrum information of the current signal frame and the target number of signal frames before and after it (for example, 4 before and after) can also be first obtained to form a matrix spectrogram, wherein, since the wave direction spectrum can include values ​​corresponding to multiple angles, more calculations and processing can be performed based on these data. Among them, it can include smoothing the matrix spectrogram. That is, by smoothing the information of the current signal frame and the target number of signal frames before and after it, some unimportant components (for example, non-human voices) are filtered out, and important components (for example, voices with higher human voice resolution) are more prominent. In this way, the result of sound source localization can be made more accurate.

[0072] There are many specific definitions of signal frames. For example, every 20ms of audio data can be determined as a signal frame, so that the real-time performance of sound source localization is guaranteed. The length of the window selected can also be set according to actual needs. However, it cannot be too long, otherwise it may affect the real-time performance of the sound source localization result, nor can it be too short, otherwise it may affect the effect of smoothing, etc. Therefore, in an optional implementation, the window length can be specifically set to 9. In other words, a window with a length of 9 can be selected by taking the current frame as the center, taking 4 frames forward and 4 frames backward. Each current frame can select a window in this way.

[0073] In addition, each signal frame in the window can obtain its arrival direction spectrogram. There are also multiple ways to obtain the arrival direction spectrogram. For example, a vector with a length of 360 can be calculated for a specific signal frame through technologies such as gcc-phat (generalized cross-correlation), representing 0-360 degrees. The vector represents the arrival direction spectrogram of the signal frame. In this way, each signal frame can obtain a 360-dimensional vector. When the window length is 9, a matrix spectrogram with a size of 360*9 can be obtained.

[0074] Specifically, there can be multiple ways to perform smoothing. For example, in one way, the corresponding kurtosis can be calculated for the current signal frame and the target number of signal frames before and after it, and then the matrix spectrogram can be smoothed using the target filter and the kurtosis information. Specifically, for each signal frame in the matrix spectrogram, the corresponding kurtosis (kurtosis, a characteristic number that characterizes the peak height of the probability density distribution curve at the average value, intuitively, the kurtosis reflects the sharpness of the peak, and high kurtosis means that the increase in variance is caused by the extreme difference of the low frequency greater than or less than the average value) is calculated, and 9 kurtosis values ​​are obtained respectively. There can also be multiple choices of filters. For example, an optional filter can be a Kalman filter, etc., and the Kalman filter and 9 kurtosis values ​​can be used as weights to smooth the 360*9 matrix spectrogram. Kalman filtering is an algorithm that uses a linear system state equation and the observation data of the system to optimally estimate the system state. Since the observation data is affected by noise and interference in the system, the estimation process of the system state can also be regarded as a filtering process.

[0075] After the smoothing process is completed, the angle corresponding to the value that meets the condition (such as the maximum value) can be taken according to the current frame of the smoothed matrix spectrogram, and the angle can be determined as the sound source localization result of the current frame.

[0076] In this way, since the sound source can be localized in units of signal frames, real-time performance can be guaranteed. In addition, since the direction of arrival spectrogram information of multiple signal frames can be involved in the calculation and processing, the accuracy of sound source localization is also higher. Based on this high-precision and high-real-time sound source localization, even if there is a situation such as "interrupting the conversation", the speaker change event and its corresponding occurrence location can be detected in a timely manner based on the sound source localization result.

[0077] Regarding speech endpoint detection, when it is specifically implemented, it can be performed synchronously with the sound source localization. There can be many specific implementation methods for speech endpoint detection, for example, it can include threshold-based speech endpoint detection, speech endpoint detection as a classifier, model speech endpoint detection, and so on. In a preferred embodiment of the present application, the speech endpoint detection can be performed using a neural network model obtained through pre-training. This type of speech endpoint detection based on a neural network model has higher accuracy and can not only filter silence, but also filter other non-human noises according to the composition of the training data.

[0078] S220: Determine a speaker change time point in the audio signal according to the sound source localization result, where the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis.

[0079] Through the method provided by the embodiment of the present application, a specific sound source localization result can be obtained for each signal frame. After obtaining the sound source localization result of each signal frame, the speaker change time point can also be determined according to the sound source localization result, that is, the position of the speaker change event in the audio signal on the time axis. There are many ways to specifically determine the speaker change time point. For example, in one way, the difference between the sound source localization result of the current signal frame and the sound source localization result of the previous signal frame can be first determined. If the difference is greater than the threshold, it can be determined that a speaker change event has occurred, and the position of the current signal frame can be determined as the speaker change time point. Among them, the so-called speaker change event refers to an event in which the speaker changes. For example, from t1 to t2, A is speaking, and from t2, B starts to speak, then a speaker change event occurs at t2, and correspondingly, the position at t2 on the time axis is the position where the speaker change event occurs. Among them, since the position of A is different from the position of B, the directions of the sound waves generated when they speak reaching the microphone are different. The embodiment of the present application analyzes the wave arrival direction spectrum to identify the position at time t2.

[0080] The setting of the above threshold can be determined according to the actual situation, and the threshold can be set dynamically. For example, an operation option for setting the threshold can be provided in a specific client interface, so that different thresholds can be set according to the size of the conference room, the density of the participants, etc. For example, if the density of the participants is larger, the phase difference of the adjacent participants relative to the microphone will be closer, and at this time, a smaller threshold can be set, and so on.

[0081] In addition, in the specific implementation, the occurrence position of the speaker change event determined by sound source positioning can also be verified by extracting the speaker's timbre features. In other words, different speakers often have different timbres. Therefore, in the process of detecting speaker change events by sound source positioning, this feature can also be used to verify the detection results of speaker change events. For example, the speaker change event is detected at a certain moment t2 by the sound source positioning method of the embodiment of the present application. After that, the timbre features in several signal frames before and after the moment t2 can be extracted and compared. If there are indeed obvious differences, the credibility of the detection results of the sound source positioning method can be further improved. If the timbre features are compared and it is found that the timbre features before and after the moment t2 have not changed significantly, then when outputting the separated text, a mark can be added at the position to remind subsequent editors to further confirm the position, and so on.

[0082] S230: Correcting the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result;

[0083] As mentioned above, since there may be errors in the process of sound source localization, even if more data for calculation is constructed by using the direction of arrival spectrogram and a window of a certain length, the errors may not be completely eliminated. Therefore, the speaker change time point can also be corrected according to the position of the voice endpoint in the voice endpoint detection result.

[0084] There may be multiple specific correction methods. For example, in one method, since speech endpoint detection and sound source localization can be performed simultaneously, the detection results of the two can be aligned through the time axis first; then, after determining the speaker change time point based on the sound source localization result, it can be determined whether there is a detected speech endpoint within the target time before and after the change point. If so, the position of the speech endpoint can be used to correct the speaker change time point.

[0085] Specifically, when the position of the voice endpoint is used to correct the speaker change time point, different processing can be performed according to the relative position relationship between the specific voice endpoint and the speaker change time point to avoid being cut off at the "word". Figure 3 As shown in the figure, assuming that for a certain audio signal, 10 speaker change time points (i.e., the locations where speaker change events occur) are determined through sound source localization, and 6 speech endpoints are determined through speech endpoint detection. Among them, there are speech endpoints near speaker change time points 1, 2, 3, 7, and 10. Therefore, the positions of the speech endpoints can be used to correct the positions of the speaker change time points.

[0086] Specifically, since the voice endpoints specifically include a voice start point and an end point, where a start point and an adjacent end point are the detected voice ends, and the end point and the next adjacent start point are non-voice segments (may be a silent period, or a noise period without voice, etc.). In specific implementation, if a voice endpoint is found near the speaker change time point, it is also possible to determine which point is used as the actual speaker change time point based on the category of the voice endpoint and the positional relationship with the speaker change time point. Among them, if the speaker change time point determined by the sound source localization result is after the voice start point or before the voice end point, the position of the voice endpoint can be determined as the actual speaker change time point. For example, if Figure 3 As shown, there are voice endpoints near the speaker change time points 1, 3, and 7, and they are all located after the voice start point (S). Therefore, in order to avoid losing the beginning of the word, the corresponding voice start point can be used as the actual speaker change time point. In addition, there is also a voice endpoint near the speaker change time point 2, and it is located before the voice end point (E). Therefore, in order to avoid losing the end of the word, the corresponding voice end point can be used as the actual speaker change time point.

[0087] If the speaker change time point determined by the sound source localization result is before the speech start point or after the speech end point, the actual speaker change time point can be determined based on the sound source localization result. Figure 3 In the example shown, there is a speech endpoint near the speaker change time point 10 determined by sound source localization, and it is the speech end point. However, since the speaker change time point 10 is located after the speech end point, the result of sound source localization can still be used as the actual speaker change time point.

[0088] If the same non-voice time period detected by the voice endpoint includes multiple speaker change time points, the other speaker change time points after the first speaker change time point in the non-voice time period can be deleted. Figure 3 In the example shown, speaker change time points 5 and 6 are determined based on the sound source localization results. However, these two speaker change time points are both located in the same non-speech time period (after the end point of the previous speech end and before the start point of the next speech end). In this case, speaker change time point 6 can be deleted. As can be seen from the figure, there are 10 speaker change time points in the sound source localization results, which become 9 after correction. That is, since this time period is a non-speech segment, there is either no sound or noise during the period, so it is not necessary to cut off the speech recognition result at this position.

[0089] In addition, if there is no detected voice endpoint within the target time before and after the occurrence position of the speaker change event, the occurrence position of the speaker change event can also be determined based on the sound source localization result. Figure 3 In the figure, the speaker change time points 4, 8, and 9 determined by sound source localization do not have any voice endpoints nearby. Therefore, the actual speaker change time point can be determined directly based on the sound source localization result.

[0090] It should be noted that the reason why the speaker change time points 4, 8, and 9 are detected, but the voice endpoints are not detected near the corresponding positions, is usually because the voice connection before and after the speaker change is relatively tight. For example, the next speaker immediately takes over as soon as the previous speaker finishes speaking. Or, there may be a phenomenon such as "interrupting the conversation", that is, someone starts speaking before the previous speaker has finished speaking, so that there is no non-voice period between the speeches of two people. It can be seen that in the embodiment of the present application, if only voice endpoint detection is used, the above-mentioned speaker change time points may not be detected; and if only sound source localization is used, the phenomenon of missing the beginning or the end of a word may occur. Therefore, in the embodiment of the present application, the combination of sound source localization and voice endpoint detection can detect the speaker change time points more timely, and at the same time can avoid the phenomenon of missing the beginning or the end of a word.

[0091] S240: Separate the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments.

[0092] Through the above method, the occurrence location corresponding to the speaker change event can be identified, and it can be corrected according to the voice endpoint detection result. Furthermore, on this basis, the specific speech recognition result can be truncated into multiple text segments, and then the specific clerk and other users can associate the specific speaker name and other information to the text segment. In other words, the result of speech recognition of the audio signal is the text content of the entire paragraph. Of course, these text contents can also be associated with the time axis of the specific audio signal. In the embodiment of the present application, by identifying the speaker change time point in the audio signal, it can be determined at which time points the speaker change occurred, and then according to the position of these time points on the time axis, the entire text content recognized by speech can be separated into multiple text segments, each of which can be a sentence, or multiple sentences, and so on.

[0093] Alternatively, in another way, in order to further reduce the workload of the clerk, a speaker identification can be automatically added to the separated text. As mentioned above, a label can be added to a specific text segment, and the specific label content can be the speaker's ID, etc. Among them, the speaker's ID can be determined in the specific process of audio signal processing, and the same label can also be added to different text segments corresponding to the same speaker. That is to say, in the process of multiple people speaking, the same speaker may speak multiple times in different time periods. Since the scene in the embodiment of the present application can usually be a scene such as a meeting, the speaker usually speaks at his seat. Therefore, when the same speaker speaks in different time periods, the sound source localization results of the signal frame generated in the specific time period should be the same or within the same range. Through this principle, not only can the location of the speaker change event be identified, but also the speakers corresponding to different time periods can be identified as the same person. Specifically, multiple time periods can be separated according to the location of the speaker change event, and the sound source localization results of multiple signal frames in the same time period can be counted respectively to determine the interval range of the sound source localization results of each time period. Then, according to the similarity of the interval range between different time periods, the speakers corresponding to the multiple time periods are judged as the same person. Afterwards, the same label is added to the text segments corresponding to different time segments of the same speaker.

[0094] Specifically from the perspective of program implementation, after a speaker starts speaking, the sound source localization result corresponding to the speaker can be identified (the volume can be a certain angle value), and it can be determined whether the user is speaking for the first time. If the speaker has not spoken before, a new ID is assigned to the speaker and added as the speaker tag of the currently separated text segment. If the speaker has spoken before, the ID that has been assigned to him before can be added as the tag of the currently separated text segment.

[0095] Specifically, there are multiple ways to determine whether a speaker is speaking for the first time. For example, after the speaker change is first identified, the correspondence between the ID assigned to the user and the sound source localization result can be saved; when the speaker change is identified again later, it is determined whether the sound source localization result corresponding to the signal frame at the location where the change occurred appears in the previously saved correspondence. If it does not appear, it proves that it is the first speech, and a new ID is assigned to it. Otherwise, it is not the first speech, and the ID corresponding to the sound source localization result can be used to mark the separated text segment.

[0096] Of course, since the time corresponding to a signal frame is in the millisecond level, a speaker will involve multiple signal frames during the speech process, and each signal frame can correspond to a sound source localization result; and it is almost impossible for the user to be completely still during the speech process, and the error of the sound source localization itself, etc., makes it possible that even if the speaker does not change, the sound source localization results corresponding to each frame may not be exactly the same. However, as long as the speaker does not change, the sound source localization results of multiple signal frames will be within a certain range (if the deviation is too much, or even exceeds the threshold, it will be determined that the speaker has changed), for example, all around 45 degrees, etc. Therefore, after identifying a speaker change event, before the speaker changes again, the sound source localization results corresponding to the signal frames in the time period can be counted, and the interval range of the sound source localization results corresponding to the corresponding speaker can be determined, and then the corresponding relationship between the ID corresponding to the speaker and the interval range information of the sound source localization result is saved. For example, the sound source localization result of a speaker may be between 43 degrees and 47 degrees, etc. In other words, the participants who have spoken will be assigned a speaker ID and correspond to the interval range of a sound source localization result. In this way, after detecting the new speaker change time, it can be determined whether the sound source localization result corresponding to the signal frame where the change occurs appears in the interval range of the sound source localization result corresponding to a certain speaker ID. If so, it is not the first time for the speaker to speak, and the newly separated text segment can be associated with the speaker ID. Otherwise, it is the first time for the speaker to speak, so a new ID can be reassigned to the speaker, and so on.

[0097] In this way, in addition to separating multiple text segments according to the position where the speaker change occurs, a speaker identification can also be added to the text segment. Although this identification can be marked in the form of an ID, since "same-person judgment" can also be performed, the text segments corresponding to the same speaker can also be marked as the same speaker ID. In this way, when editing the name of a specific speaker later, one of the speaker IDs is edited and modified to the name of the speaker, etc., and the labels of other text segments corresponding to the same speaker ID can also be automatically modified. Specifically, an operation option for editing the label of the text segment can be provided. After receiving the editing result of one of the text segments, other text segments with the same label corresponding to the text segment can be determined, and the labels of the other text segments can be modified to the editing result. In this way, batch editing of information such as speaker names can be achieved, thereby further improving efficiency and reducing the probability of errors.

[0098] For example, Figure 4-1 As shown in FIG. 1 , it is a recognition result interface diagram of the application under a specific implementation mode. It can be seen from the figure that the embodiment of the present application can recognize a specific audio signal as text, and cut the text at the speaker change position according to the sound source localization result to generate multiple text segments, and can add speaker labels to different text segments. When different text segments correspond to the same speaker, the same label is also added. For example, Figure 4-1 In the example shown, text segment 1 and text segment 3 both correspond to user 2, text segment 2 corresponds to user 1, and so on. In addition, operation options for editing labels may also be provided for specific text segments, for example, Figure 4-1 As shown at 41 in FIG. 1 , the user can modify the label of the text segment through this operation option, and can implement batch modification of multiple different text segments associated with the same label. Figure 4-2 As shown, assuming that the user edits the label of text segment 1 to "Zhang San", the label corresponding to text segment 3 can also be automatically modified to "Zhang San" at the same time, and so on.

[0099] In addition, in the specific implementation, recommended tags can also be provided based on the pre-acquired sound feature information of different speakers and their corresponding speaker identifiers. For example, specifically, the sound features of the participants of a certain meeting can be extracted in advance, and the sound features can be saved by corresponding to the specific speaker identifiers. In this way, assuming that it is determined through the sound source localization results that a speaker change has occurred at a certain moment t2, the sound features of the signal frame after this moment can be extracted, and the previously saved sound features can be retrieved based on the extracted sound features. If the sound features of one of them match, the speaker identifier corresponding to the sound feature can be used to provide recommended tags. For example, the speaker identifier saved in the specific index can be the speaker's name, work number and other identity information, and the name or work number information can be used to provide recommended tags. In this way, if the identification is correct, the workload of subsequent manual editing can be further reduced.

[0100] It should be noted that, in specific implementation, the audio signal processing method provided in the embodiment of the present application may have a variety of specific application scenarios, for example, it may be a conference scenario where multiple people speak (including seminars, court trial meetings, etc.). Or, it may also include a live video broadcast scenario where multiple people speak, etc. Regardless of the specific scenario, multiple speakers may be located in the same space, so that the speaker change event is detected by sound source localization.

[0101] Among them, for situations where there are associated video signals, such as live video scenes with multiple people speaking, in an optional implementation, after the text obtained by the voice recognition is separated, the time axis information corresponding to the separated multiple text segments (for a specific video, the video signal, audio signal and text information obtained by specific voice recognition can all correspond to the same time axis) can be added to the associated video image to generate a video image with subtitles.

[0102] The above-mentioned process of adding subtitles can be performed during the live broadcast process, based on the live broadcast data stream collected in real time. Therefore, specific speech recognition and sound source localization can be performed based on the live broadcast data stream. Similarly, in scenes such as meetings, signal processing can also be performed in real time during the meeting, and meeting records can be generated.

[0103] In addition, in addition to being applied in specific conference or video live broadcast scenarios, the embodiments of the present application can also be applied in other scenarios. For example, the segmentation results of the speech recognition results can be saved in the form of a document for use in query and other scenarios, etc.

[0104] In summary, through the embodiments of the present application, it is possible to perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a multi-person speaking scene. In this way, after determining the occurrence location of the speaker change event based on the sound source localization result, the occurrence location of the speaker change event can also be corrected based on the location of the voice endpoint in the speech endpoint detection result. Afterwards, the speech recognition result can be truncated according to the occurrence location of the corrected speaker change event to obtain multiple text segments. Then, speaker marking and other processing can be performed directly based on the separated multiple text segments, thereby reducing the workload of the meeting recorder and reducing the error rate. In addition, through the above method, since the occurrence location of the speaker change event can be determined more accurately, the speech recognition result can be truncated more accurately to avoid the phenomenon of missing the beginning or end of a word.

[0105] Embodiment 2

[0106] This second embodiment is aimed at the application in the conference scene and provides a method for generating conference records. Figure 5 , the method may include:

[0107] S510: performing speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a conference scene where multiple people are speaking;

[0108] S520: Determine a speaker change time point in the audio signal according to the sound source localization result, where the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis;

[0109] S530: Correcting the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result;

[0110] S540: Segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments;

[0111] S550: Generate a meeting record of the meeting according to the separated multiple text segments.

[0112] In a specific implementation, labels may also be added to the multiple separated text segments, and the labels are used to represent speaker identifications.

[0113] In addition, the same label can be added to different text segments corresponding to the same speaker.

[0114] Furthermore, an operation option for editing the label of the text segment can be provided; after receiving the editing result of one of the text segments, other text segments corresponding to the same label as the text segment are determined, and the labels of the other text segments are modified to the editing result, and the editing result corresponds to the identity information of the speaker.

[0115] Embodiment 3

[0116] This embodiment 3 is aimed at the application in the live video scene and provides a live video processing method. Figure 6 , the method may include:

[0117] S610: performing speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a live video broadcast scenario in which multiple speakers are speaking, wherein the multiple speakers are located in the same space;

[0118] S620: Determine a speaker change time point in the audio signal according to the sound source localization result, where the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis;

[0119] S630: Correcting the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result;

[0120] S640: Separate the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments;

[0121] S650: adding the text segments to the video image collected in the live video scene according to the timeline information corresponding to the separated multiple text segments, so as to generate a live video image with subtitles.

[0122] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0123] Corresponding to the first embodiment, the present application embodiment further provides an audio signal processing device, see Figure 7 , the device may include:

[0124] The signal processing unit 710 is used to perform speech recognition, sound source localization, and speech endpoint detection on the audio signals collected in a multi-person speaking scenario;

[0125] A change time point determination unit 720 is used to determine a speaker change time point in the audio signal according to the sound source localization result, wherein the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis;

[0126] A correction unit 730, configured to correct the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result;

[0127] The speech recognition result segmentation unit 740 is used to segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments.

[0128] Specifically, the correction unit may include:

[0129] An alignment subunit, used to align the sound source localization result with the speech endpoint detection result on the time axis;

[0130] A judgment subunit is used to judge whether there is a detected voice endpoint within a target time before and after the speaker change time point after determining the speaker change time point according to the sound source localization result;

[0131] The correction subunit is used to correct the speaker change time point by using the position of the voice endpoint on the time axis, if any.

[0132] In addition, the correction subunit can also be used to:

[0133] If there is no detected voice endpoint within the target time before and after the speaker change time point, the actual speaker change time point is determined based on the sound source localization result.

[0134] Wherein, the voice endpoint includes a voice starting point or an ending point;

[0135] The correction subunit can be specifically used for:

[0136] The speaker change time point is corrected according to the chronological relationship between the speaker change time point and the speech start point or end point appearing in the target time before and after the speaker change time point.

[0137] Specifically, the correction subunit can be used for:

[0138] If the speaker change time point determined by the sound source localization result is located after the speech start point or before the speech end point, the position of the speech endpoint is determined as the actual speaker change time point.

[0139] Alternatively, if the speaker change time point determined by the sound source localization result is before the speech start point or after the speech end point, the actual speaker change time point is determined based on the sound source localization result.

[0140] Alternatively, if the same non-voice time period detected by the voice endpoint includes multiple speaker change time points, the other speaker change time points after the first speaker change time point in the non-voice time period are deleted.

[0141] In a specific implementation, the device may further include:

[0142] A speech endpoint detection unit is used to perform the speech endpoint detection through a pre-trained neural network model.

[0143] The change time point determination unit may be specifically used for:

[0144] In the process of performing sound source localization on the audio signal in units of signal frames, determining a difference between a sound source localization result of a current signal frame and a sound source localization result of a previous signal frame;

[0145] If the difference is greater than the target threshold, it is determined that a speaker change event occurs, and the position of the current signal frame is determined as the speaker change time point.

[0146] Specifically, the change time point determination unit can be used to: in the process of performing sound source localization on the audio signal in units of signal frames, obtain the wave direction spectrum information of the current signal frame and the target number of signal frames before and after it to form a matrix spectrum, and smooth the matrix spectrum; according to the angle corresponding to the value that meets the target condition in the wave direction spectrum corresponding to the smoothed wave direction, determine the sound source localization result of the current signal frame,

[0147] In addition, the device may further include: a label adding unit, which is used to add labels to the separated text segments, and the labels are used to represent speaker identification.

[0148] Specifically, the label adding unit can be used for:

[0149] Add the same label to different text segments corresponding to the same speaker.

[0150] In a specific implementation, the device may further include:

[0151] A statistical unit, used to separate multiple time periods according to the occurrence position of the speaker change event, and respectively perform statistics on the sound source localization results of multiple signal frames in the same time period to determine the interval range of the sound source localization results of each time period;

[0152] A fan judgment unit, configured to judge the speakers corresponding to the multiple time periods according to the similarity of the interval ranges between the different time periods;

[0153] The label adding unit can be used for:

[0154] The same label is added to the text segments corresponding to different time segments of the same speaker.

[0155] In addition, the device may also include:

[0156] an operation option providing unit, used for providing an operation option for editing the label of the text segment;

[0157] The editing unit is used to determine other text segments corresponding to the same label as the text segment after receiving the editing result of one of the text segments, and modify the label of the other text segments to the editing result.

[0158] Furthermore, the device may further include:

[0159] The tag recommendation unit is used to provide recommended tags based on pre-acquired voice feature information of different speakers and their corresponding speaker identifiers.

[0160] The verification unit is used to verify the occurrence position of the speaker change event determined by sound source localization by extracting the speaker's timbre features.

[0161] The multi-speaking scene includes a multi-speaking conference scene, in which multiple speakers are located in the same space.

[0162] In addition, the audio signal may also be associated with a video signal;

[0163] At this time, the device may further include:

[0164] The subtitle adding unit is used to separate the text obtained by the speech recognition and add the subtitles to the associated video image according to the time axis information corresponding to the multiple separated text segments to generate a video image with subtitles.

[0165] The multi-speaking scene includes a live video broadcast scene of multi-speaking speakers, in which multiple speakers are located in the same space.

[0166] In addition, an embodiment of the present application further provides a pickup, which may include the aforementioned audio signal processing device.

[0167] Corresponding to the second embodiment, the present application embodiment also provides a device for generating meeting records, see Figure 8 , the device may include:

[0168] The signal processing unit 810 is used to perform speech recognition, sound source localization, and speech endpoint detection on the audio signals collected in a conference scene where multiple people are speaking;

[0169] A change time point determination unit 820 is used to determine a speaker change time point in the audio signal according to the sound source localization result, wherein the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis;

[0170] A correction unit 830, configured to correct the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result;

[0171] A speech recognition result segmentation unit 840 is used to segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments;

[0172] The meeting record generating unit 850 is used to generate the meeting record of the meeting according to the separated multiple text segments.

[0173] In a specific implementation, the device may further include:

[0174] The label adding unit is used to add labels to the separated text segments, wherein the labels are used to represent speaker identifications.

[0175] Specifically, the label adding unit can be used for:

[0176] Add the same label to different text segments corresponding to the same speaker.

[0177] In addition, the device may also include:

[0178] an operation option providing unit, used for providing an operation option for editing the label of the text segment;

[0179] The editing unit is used to determine other text segments corresponding to the same label as the text segment after receiving the editing result of one of the text segments, and modify the label of the other text segments to the editing result.

[0180] Corresponding to the third embodiment, the present application embodiment also provides a live video processing device, see Fig. 9 , the device may include:

[0181] A signal processing unit 910 is used to perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a live video broadcast scenario where multiple speakers are located in the same space;

[0182] A change time point determination unit 920 is used to determine a speaker change time point in the audio signal according to the sound source localization result, wherein the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis;

[0183] A correction unit 930, configured to correct the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result;

[0184] A speech recognition result segmentation unit 940 is used to segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments;

[0185] The subtitle adding unit 950 is used to add the text segments to the video image collected in the live video scene according to the timeline information corresponding to the separated multiple text segments, so as to generate a live video image with subtitles.

[0186] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0187] And an electronic device, comprising:

[0188] one or more processors; and

[0189] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0190] in, Fig.10 The electronic device architecture is shown as an example, which may include a processor 1010, a video display adapter 1011, a disk drive 1012, an input / output interface 1013, a network interface 1014, and a memory 1020. The processor 1010, the video display adapter 1011, the disk drive 1012, the input / output interface 1013, the network interface 1014, and the memory 1020 may be communicatively connected via a communication bus 1030.

[0191] Among them, the processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in this application.

[0192] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store an operating system 1021 for controlling the operation of the electronic device 1000, and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device 1000. In addition, a web browser 1023, a data storage management system 1024, and an icon font processing system 1025, etc. can also be stored. The above-mentioned icon font processing system 1025 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0193] The input / output interface 1013 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0194] The network interface 1014 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0195] The bus 1030 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1010, the video display adapter 1011, the disk drive 1012, the input / output interface 1013, the network interface 1014, and the memory 1020).

[0196] It should be noted that, although the above device only shows a processor 1010, a video display adapter 1011, a disk drive 1012, an input / output interface 1013, a network interface 1014, a memory 1020, a bus 1030, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0197] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0198] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0199] The audio signal processing method, device and electronic device provided by the present application are introduced in detail above. The principles and implementation methods of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An audio signal processing method, characterized in that: include: Perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in multi-person speaking scenarios; Determine a speaker change time point in the audio signal according to the sound source localization result, wherein the speaker change time point includes: a position of the speaker change event moment in the audio signal on the time axis; According to the position of the voice endpoint on the time axis in the voice endpoint detection result, the speaker change time point is corrected, including: aligning the sound source localization result with the voice endpoint detection result on the time axis; after determining the speaker change time point according to the sound source localization result, judging whether there is a detected voice endpoint within the target time before and after the speaker change time point; if there is, correcting the speaker change time point using the position of the voice endpoint on the time axis; According to the corrected speaker change time point, the text obtained by speech recognition is separated to obtain multiple text segments.

2. The method according to claim 1, characterized in that include: If there is no detected voice endpoint within the target time before and after the speaker change time point, the actual speaker change time point is determined based on the sound source localization result.

3. The method according to claim 1, characterized in that include: The voice endpoint includes a voice starting point or an ending point; The modifying the speaker change time point by using the position of the voice endpoint on the time axis includes: The speaker change time point is corrected according to the chronological relationship between the speaker change time point and the speech start point or end point appearing in the target time before and after the speaker change time point.

4. The method according to claim 3, characterized in that The modifying of the speaker change time point includes: If the speaker change time point determined by the sound source localization result is located after the speech start point or before the speech end point, the position of the speech endpoint is determined as the actual speaker change time point.

5. The method according to claim 3, characterized in that: The modifying of the speaker change time point includes: If the speaker change time point determined by the sound source localization result is before the speech start point or after the speech end point, the actual speaker change time point is determined based on the sound source localization result.

6. The method according to claim 3, characterized in that The modifying of the speaker change time point includes: If the same non-voice time period detected by the voice endpoint includes multiple speaker change time points, the other speaker change time points after the first speaker change time point in the non-voice time period are deleted.

7. The method according to claim 1, characterized in that The speech endpoint detection is performed using a pre-trained neural network model.

8. The method according to claim 1, characterized in that: The step of determining the speaker change time point in the audio signal according to the sound source localization result includes: In the process of performing sound source localization on the audio signal in units of signal frames, determining a difference between a sound source localization result of a current signal frame and a sound source localization result of a previous signal frame; If the difference is greater than the target threshold, it is determined that a speaker change event occurs, and the position of the current signal frame is determined as the speaker change time point.

9. The method according to claim 1, characterized in that: In the process of performing sound source localization on the audio signal in units of signal frames, obtaining the direction of arrival spectrogram information of the current signal frame and a target number of signal frames before and after the current signal frame to form a matrix spectrogram, and performing smoothing processing on the matrix spectrogram; The sound source localization result of the current signal frame is determined according to the angle corresponding to the value that meets the target condition in the smoothed direction of arrival spectrum corresponding to the current signal frame.

10. The method according to claim 1, characterized in that Also includes: A label is added to the separated text segment, wherein the label is used to represent the speaker identification.

11. The method according to claim 10, characterized in that The step of adding labels to the separated text segments includes: Add the same label to different text segments corresponding to the same speaker.

12. The method according to claim 11, characterized in that Also includes: Separating multiple time periods according to the occurrence location of the speaker change event, and performing statistics on the sound source localization results of multiple signal frames in the same time period to determine the interval range of the sound source localization results of each time period; According to the similarity of the interval ranges between different time periods, the speakers corresponding to the multiple time periods are judged as the same person; The adding of the same label to different text segments corresponding to the same speaker includes: Add the same label to different text segments corresponding to the same speaker in different time periods.

13. The method according to claim 12, characterized in that Also includes: Providing an operation option for editing the label of the text segment; After receiving the editing result of one of the text segments, other text segments corresponding to the same label as the text segment are determined, and the labels of the other text segments are modified to the editing result.

14. The method according to claim 11, characterized in that Also includes: Recommended tags are provided based on pre-acquired voice feature information of different speakers and their corresponding speaker identifiers.

15. The method according to any one of claims 1 to 14, characterized in that Also includes: By extracting the speaker's timbre features, the location of the speaker change event determined by sound source localization is verified.

16. The method according to any one of claims 1 to 14, characterized in that The multi-person speaking scenario includes a conference scenario where multiple speakers speak, wherein multiple speakers are located in the same space; The method further comprises: A meeting record of the meeting is generated according to the separated multiple text segments.

17. The method according to any one of claims 1 to 14, characterized in that: The audio signal is also associated with a video signal; The method further comprises: After the text obtained by the speech recognition is segmented, the time axis information corresponding to the multiple segmented text segments is added to the associated video image to generate a video image with subtitles.

18. The method according to claim 17, characterized in that The multi-speaking scene includes a live video broadcast scene of multi-speaking speakers, wherein the multiple speakers are located in the same space.

19. A method for generating meeting minutes, characterized in that: include: Perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in conference scenarios where multiple people are speaking; Determine a speaker change time point in the audio signal according to the sound source localization result, wherein the speaker change time point includes: a position of the speaker change event moment in the audio signal on the time axis; According to the position of the voice endpoint on the time axis in the voice endpoint detection result, the speaker change time point is corrected, including: aligning the sound source localization result with the voice endpoint detection result on the time axis; after determining the speaker change time point according to the sound source localization result, judging whether there is a detected voice endpoint within the target time before and after the speaker change time point; if there is, correcting the speaker change time point using the position of the voice endpoint on the time axis; According to the corrected speaker change time point, the text obtained by speech recognition is separated to obtain multiple text segments; A meeting record of the meeting is generated according to the separated multiple text segments.

20. The method according to claim 19, characterized in that Also includes: Labels are added to the separated multiple text segments, where the labels are used to represent speaker identifications.

21. The method according to claim 20, characterized in that Add the same label to different text segments corresponding to the same speaker.

22. The method according to claim 21, characterized in that Also includes: Providing an operation option for editing the label of the text segment; After receiving the editing result of one of the text segments, other text segments corresponding to the same label as the text segment are determined, and the labels of the other text segments are modified to the editing result, and the editing result corresponds to the identity information of the speaker.

23. A live video processing method, characterized in that: include: Perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a live video broadcast scenario where multiple speakers are located in the same space. Determine a speaker change time point in the audio signal according to the sound source localization result, wherein the speaker change time point includes: a position of the speaker change event moment in the audio signal on the time axis; According to the position of the voice endpoint on the time axis in the voice endpoint detection result, the speaker change time point is corrected, including: aligning the sound source localization result with the voice endpoint detection result on the time axis; after determining the speaker change time point according to the sound source localization result, judging whether there is a detected voice endpoint within the target time before and after the speaker change time point; if there is, correcting the speaker change time point using the position of the voice endpoint on the time axis; According to the corrected speaker change time point, the text obtained by speech recognition is separated to obtain multiple text segments; According to the timeline information corresponding to the separated multiple text segments, the text segments are added to the video images collected in the live video scene to generate a live video image with subtitles.

24. An audio signal processing device, characterized in that: include: A signal processing unit, used to perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a multi-person speaking scenario; a change time point determination unit, configured to determine a speaker change time point in the audio signal according to a sound source localization result, wherein the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis; A correction unit is used to correct the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result, including: aligning the sound source localization result with the voice endpoint detection result on the time axis; after determining the speaker change time point according to the sound source localization result, judging whether there is a detected voice endpoint within a target time before and after the speaker change time point; if so, correcting the speaker change time point using the position of the voice endpoint on the time axis; The speech recognition result segmentation unit is used to segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments.

25. A pickup, characterized in that: Comprising the audio signal processing device as claimed in claim 24.

26. A device for generating meeting minutes, characterized in that: include: A signal processing unit, used to perform speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a conference scene where multiple people are speaking; a change time point determination unit, configured to determine a speaker change time point in the audio signal according to a sound source localization result, wherein the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis; A correction unit is used to correct the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result, including: aligning the sound source localization result with the voice endpoint detection result on the time axis; after determining the speaker change time point according to the sound source localization result, judging whether there is a detected voice endpoint within a target time before and after the speaker change time point; if so, correcting the speaker change time point using the position of the voice endpoint on the time axis; A speech recognition result segmentation unit, used to segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments; A conference record generating unit is used to generate a conference record of the conference according to the separated multiple text segments.

27. A live video processing device, characterized in that: include: A signal processing unit, for performing speech recognition, sound source localization, and speech endpoint detection on audio signals collected in a live video broadcast scenario where multiple speakers are located in the same space; a change time point determination unit, configured to determine a speaker change time point in the audio signal according to a sound source localization result, wherein the speaker change time point includes: a position of a speaker change event in the audio signal on a time axis; A correction unit is used to correct the speaker change time point according to the position of the voice endpoint on the time axis in the voice endpoint detection result, including: aligning the sound source localization result with the voice endpoint detection result on the time axis; after determining the speaker change time point according to the sound source localization result, judging whether there is a detected voice endpoint within a target time before and after the speaker change time point; if so, correcting the speaker change time point using the position of the voice endpoint on the time axis; A speech recognition result segmentation unit, used to segment the text obtained by speech recognition according to the corrected speaker change time point to obtain multiple text segments; The subtitle adding unit is used to add the text segments to the video image collected in the live video scene according to the time axis information corresponding to the separated multiple text segments, so as to generate a live video image with subtitles.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 23 are implemented.

29. A computer system, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 23.

Images