Voice source responsibility judgment method and device, electronic equipment and storage medium

By processing target audio and using the classification model, the inefficient judgment caused by voice recognition in the online ride-hailing industry is solved, and automatic closed-loop processing is achieved, reducing costs and improving accuracy.

CN118942446APending Publication Date: 2024-11-12NANJING LINGXING TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410724295.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-05
Publication Date
2024-11-12

Smart Images

  • Figure CN118942446A_ABST
    Figure CN118942446A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a voice source responsibility judgment method and device, electronic equipment and a storage medium. The method comprises the following steps: processing a target audio to obtain a plurality of voice segments arranged according to a time sequence; wherein the voice segments carry first labels for identifying voice sources, and the first labels carried by every two voice segments with adjacent time sequences are different; according to the plurality of voice segments arranged according to the time sequence, generating a text set composed of text segments corresponding to the voice segments; wherein the text fragment carries a second label for identifying the voice source; and determining the responsibility judgment result of the voice source according to the text set based on the responsibility judgment classification model, so that the voice fragment and the voice source in the audio are automatically separated, the text fragment is generated, and the responsibility judgment of the voice source is completed by using the classification model, thereby realizing the automatic closed-loop processing of a full link. The labor cost is reduced, and the responsibility judgment efficiency and accuracy are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for determining a voice source, a device for determining a voice source, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the online car-hailing industry, the management of driver attitude issues and the control of malicious order cancellations are very important for providing a good user experience and improving the quality of driver service. In order to accurately determine responsibility, customer service judges can make judgments by analyzing driver-passenger conversations. Therefore, driver-passenger conversations are a very valuable source of data.

[0003] However, directly performing speech recognition on two-channel audio may result in the loss of character information, which makes it impossible for the user to accurately determine whether the current text content is spoken by the caller or the called party, which is not conducive to improving the efficiency of accountability and solving the accountability problem.

[0004] Using other deep learning-based models (such as speaker log models) to solve this problem has the following problems: the model parameters are large and the additional deployment cost is high; a single model cannot solve the problem and needs to be combined with other models; models such as speaker logs cannot guarantee high accuracy. Summary of the invention

[0005] In view of the above problems, embodiments of the present invention are proposed to provide a method for determining a speech source, a corresponding device for determining a speech source, an electronic device, and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0006] In order to solve the above problem, an embodiment of the present invention discloses a method for determining the source of speech, the method comprising:

[0007] Processing the target audio to obtain a plurality of voice segments arranged in time sequence; wherein the voice segments carry a first label identifying a voice source, and the first labels carried by two voice segments adjacent in time sequence are different;

[0008] According to the multiple voice segments arranged in time sequence, a text set consisting of text segments corresponding to the voice segments is generated; wherein the text segments carry a second tag identifying the source of the voice;

[0009] Based on the accountability classification model, the accountability result of the speech source is determined according to the text set.

[0010] Optionally, the audios of different channels in the target audio have different speech sources, and the target audio is processed to obtain a plurality of speech segments arranged in time sequence, including:

[0011] Determining, based on the audio of the different channels, the start and end times of a plurality of initial voice segments arranged in time sequence and a first tag identifying a voice source;

[0012] According to the start and end times of the multiple initial voice segments arranged in time sequence and the first label identifying the voice source, the start and end times of the initial voice segments that are adjacent in time sequence and have the same first label identifying the voice source are merged to obtain the multiple voice segments arranged in time sequence.

[0013] Optionally, determining the start and end times of a plurality of initial voice segments arranged in time sequence and a first tag identifying a voice source according to the audio of the different channels includes:

[0014] For the audio of each channel, the start and end times of multiple initial voice segments arranged in time sequence are separated respectively, and a first tag identifying the voice source is determined according to the voice source corresponding to the channel;

[0015] The start and end times of the multiple initial voice segments separated from different channels and arranged in time sequence are sorted together to obtain the start and end times of the multiple initial voice segments arranged in time sequence and a first label identifying a voice source.

[0016] Optionally, generating a text set consisting of text segments corresponding to the multiple voice segments arranged in time sequence includes:

[0017] According to the first tags identifying the speech sources of the plurality of speech segments arranged in time sequence, adding corresponding speech source marked speech to the speech segments;

[0018] splicing a plurality of speech segments to which speech source marks are added and which are arranged in time sequence to obtain spliced ​​speech audio;

[0019] After converting the concatenated speech audio into corresponding text, a text set consisting of text segments carrying a second tag identifying the source of the speech is generated according to the marked text corresponding to the identified speech source marked speech.

[0020] Optionally, the determining the result of the voice source based on the responsibility classification model according to the text set includes:

[0021] Extracting a dialogue text consisting of text segments from the text collection according to a second tag in the text collection that identifies the source of the speech and according to a preset rule;

[0022] Inputting the dialogue text into the responsibility classification model to obtain a responsibility indication value corresponding to the dialogue text;

[0023] The accountability result is determined by combining the accountability indication values ​​corresponding to the respective dialogue texts.

[0024] Optionally, extracting the dialogue text consisting of text segments from the text collection according to a second tag in the text collection that identifies the source of the speech and a preset rule includes:

[0025] Extracting the conversation text according to a second tag in the text set that identifies the source of the speech, using a sliding window with a step size of 2 text segments and a width of at least 3 text segments;

[0026] For dialogue texts that exceed the preset length limit, they are split into dialogue texts that meet the preset length limit according to text segments.

[0027] Optionally, inputting the dialogue text into the responsibility classification model to obtain a responsibility indication value corresponding to the dialogue text includes:

[0028] According to the dialogue text, a text tensor representing the text of the dialogue text, an attention tensor representing the valid positions in the text tensor, and a source segment tensor representing the speech source corresponding to the text in the dialogue text are generated;

[0029] Inputting the text tensor, the attention tensor, and the source segment tensor into the encoder of the judgment and classification model for encoding to obtain text feature data;

[0030] The text feature data is input into the classifier of the responsibility classification model for classification to obtain the responsibility indication value corresponding to the dialogue text.

[0031] Optionally, the step of inputting the text feature data into a classifier of the responsibility classification model for classification to obtain a responsibility indication value corresponding to the dialogue text includes:

[0032] Generating audio feature data according to the speech segment corresponding to the conversation text;

[0033] Fusion of the text feature data and the audio feature data to obtain fused feature data;

[0034] The fused feature data is input into the classifier of the responsibility classification model for classification to obtain the responsibility indication value corresponding to the dialogue text.

[0035] Optionally, before determining the result of the voice source identification based on the identification classification model according to the text set, the method further includes:

[0036] The text set that does not contain sensitive words in the sensitive word library is filtered out, and the text set that contains sensitive words in the sensitive word library is retained.

[0037] The embodiment of the present invention also discloses a device for determining the source of speech, the device comprising:

[0038] An audio processing module, used for processing the target audio to obtain a plurality of voice segments arranged in time sequence; wherein the voice segments carry a first label identifying a voice source, and the first labels carried by two voice segments adjacent in time sequence are different;

[0039] A set generation module, used to generate a text set consisting of text segments corresponding to the multiple voice segments arranged in time sequence; wherein the text segment carries a second tag identifying the source of the voice;

[0040] The result determination module is used to determine the result of the voice source based on the judgment classification model and the text set.

[0041] Optionally, the audios of different channels in the target audio have different speech sources, and the audio processing module includes:

[0042] A label determination submodule, used to determine the start and end times of a plurality of initial voice segments arranged in time sequence and a first label identifying a voice source according to the audio of the different channels;

[0043] The merging submodule is used to merge the start and end times of the initial voice segments that are adjacent in time sequence and have the same first label identifying the voice source according to the start and end times of the multiple initial voice segments arranged in time sequence and the first label identifying the voice source, so as to obtain the multiple voice segments arranged in time sequence.

[0044] Optionally, the label determination submodule includes:

[0045] A label determination unit, for separating the start and end times of a plurality of initial voice segments arranged in time sequence from the audio of each channel, and determining a first label for identifying a voice source according to a voice source corresponding to the channel;

[0046] The sorting unit is used to sort the start and end times of multiple initial voice segments separated from different channels and arranged in time sequence, and obtain the start and end times of the multiple initial voice segments arranged in time sequence and a first label identifying the voice source.

[0047] Optionally, the set generation module includes:

[0048] A speech adding submodule, used for adding a corresponding speech source marking speech to the speech segment according to the first label identifying the speech source of the plurality of speech segments arranged in time sequence;

[0049] A splicing submodule, used for splicing a plurality of speech segments with speech source marks added thereto arranged in time sequence to obtain spliced ​​speech audio;

[0050] The set generation submodule is used to generate a text set consisting of text segments carrying a second tag identifying the source of the speech according to the marked text corresponding to the identified speech source marked speech after converting the spliced ​​speech audio into corresponding text.

[0051] Optionally, the result determination module includes:

[0052] An extraction submodule, configured to extract a dialogue text consisting of text segments from the text collection according to a second tag in the text collection that identifies the source of the speech and a preset rule;

[0053] An indicator value determination submodule, used for inputting the dialogue text into the responsibility classification model to obtain a responsibility indicator value corresponding to the dialogue text;

[0054] The result determination submodule is used to determine the accountability result in combination with the accountability indication value corresponding to each dialogue text.

[0055] Optionally, the extraction submodule includes:

[0056] A text extraction unit, configured to extract the conversation text according to a second tag in the text set that identifies the source of the speech, using a sliding window with a step size of 2 text segments and a width of at least 3 text segments;

[0057] The splitting unit is used to split the dialogue text exceeding the preset length limit into dialogue text meeting the preset length limit according to the text segments.

[0058] Optionally, the indication value determination submodule includes:

[0059] A tensor generation unit is used to generate, according to the dialogue text, a text tensor representing the text of the dialogue text, an attention tensor representing the valid positions in the text tensor, and a source segment tensor representing the speech source corresponding to the text in the dialogue text;

[0060] An encoding unit, used for inputting the text tensor, the attention tensor, and the source segment tensor into the encoder of the judgment and classification model for encoding to obtain text feature data;

[0061] The classification unit is used to input the text feature data into the classifier of the responsibility classification model for classification, so as to obtain the responsibility indication value corresponding to the dialogue text.

[0062] Optionally, the classification unit includes:

[0063] A feature generation subunit, used to generate audio feature data according to the speech segment corresponding to the dialogue text;

[0064] A fusion subunit, used for fusing the text feature data and the audio feature data to obtain fused feature data;

[0065] The classification subunit is used to input the fused feature data into the classifier of the responsibility classification model for classification, so as to obtain the responsibility indication value corresponding to the dialogue text.

[0066] Optionally, the device further comprises:

[0067] The collection filtering module is used to filter out the text collection that does not contain sensitive words in the sensitive word library before determining the judgment result of the voice source based on the judgment classification model according to the text collection, and retain the text collection that contains sensitive words in the sensitive word library.

[0068] The embodiment of the present invention further discloses an electronic device, including:

[0069] one or more processors; and

[0070] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the device to perform any of the voice source determination methods described above.

[0071] The embodiment of the present invention further discloses a computer-readable storage medium, wherein the computer program stored in the storage medium enables a processor to execute any of the above-mentioned methods for determining the source of speech.

[0072] According to an embodiment of the present invention, by processing the target audio, a plurality of voice segments arranged in time sequence are obtained; wherein the voice segments carry a first label identifying the source of the voice, and the first labels carried by two voice segments adjacent in time sequence are different; based on the plurality of voice segments arranged in time sequence, a text set consisting of text segments corresponding to the voice segments is generated; wherein the text segment carries a second label identifying the source of the voice; based on a judgment classification model, a judgment result of the voice source is determined according to the text set, so that the voice segments and the voice sources in the audio are automatically separated, and a text segment is generated, and the classification model is used to complete the judgment of the voice source, thereby realizing automatic closed-loop processing of the entire link, reducing labor costs, and improving judgment efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a flow chart of steps of an embodiment of a method for determining the source of speech of the present invention;

[0074] Figure 2 is a flowchart of another embodiment of a method for determining a voice source of the present invention;

[0075] Figure 3 It is a schematic diagram of the two-channel audio preprocessing process;

[0076] Figure 4 is a schematic diagram of the result of preprocessing of two-channel audio;

[0077] Figure 5 It is a schematic diagram of the flow of speech recognition processing;

[0078] Figure 6 It is a schematic diagram of the result of direct recognition of the original audio;

[0079] Figure 7 It is a schematic diagram of the final text recognition result;

[0080] Figure 8 is a schematic diagram of a final text recognition result with a second label identifying the source of the speech;

[0081] Fig. 9 It is a schematic diagram of the dialogue text judgment process;

[0082] Fig.10 It is a structural block diagram of an embodiment of a speech source determination device of the present invention;

[0083] Fig.11 The present invention is a structural block diagram of an electronic device for voice source determination according to an exemplary embodiment. DETAILED DESCRIPTION

[0084] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0085] Reference Figure 1 , shows a flow chart of a method for determining the source of speech according to an embodiment of the present invention, the method may specifically include the following steps:

[0086] Step 101, process the target audio to obtain a plurality of voice segments arranged in time sequence; wherein the voice segments carry a first label identifying a voice source, and the first labels carried by two voice segments adjacent in time sequence are different.

[0087] In the embodiment of the present invention, the target audio refers to audio containing voices from multiple voice sources, where the voice source may refer to the speaker of the voice, or the source device of the voice, etc. For example, the conversation audio between the driver and the passenger, or any other applicable audio, which is not limited in the embodiment of the present invention. The voice segment refers to the audio segment of the human voice separated from the target audio.

[0088] In the embodiment of the present invention, the separation of the speech segments of the target audio may be implemented using the Voice Activity Detection (VAD) technology, or any other applicable implementation method, which is not limited in the embodiment of the present invention.

[0089] In the embodiment of the present invention, the process of separating the speech segments from the target audio is the process of determining the start and end time of the speech segments in the target audio. Therefore, a plurality of speech segments arranged in time sequence can be obtained.

[0090] In an embodiment of the present invention, the specific implementation methods for determining the voice source of a voice segment and obtaining the first tag identifying the voice source may include multiple methods, which are not limited by the embodiment of the present invention. For example, human voice recognition is performed on the voice segment to determine the voice source. For another example, audio from different voice sources is stored as audio of different channels of the target audio, and the voice source can be determined based on the channel of the voice segment source. Among them, the tag identifying the voice source carried by the voice segment is recorded as the first tag.

[0091] In the embodiment of the present invention, if the voice sources of the multiple adjacent voice segments obtained by separation are the same, the multiple adjacent voice segments are merged, so that the first labels identifying the voice sources carried by two voice segments adjacent in time sequence are different.

[0092] Step 102: Generate a text set consisting of text segments corresponding to the multiple voice segments arranged in time sequence, wherein the text segment carries a second tag identifying the source of the voice.

[0093] In an embodiment of the present invention, speech-to-text recognition is performed on multiple voice segments to generate corresponding text segments. The generated text segment carries a label identifying the source of the voice, which is recorded as a second label. Among them, for a text segment converted from a certain voice segment, the second label is the same as the voice source identified by the first label. The second label identifying the voice source can use the same label as the first label identifying the voice source, or can use a different label from the first label identifying the voice source.

[0094] In the embodiment of the present invention, a set of text segments formed by using text segments corresponding to multiple voice segments arranged in time sequence is recorded as a text set. The second tag identifying the voice source carried by the text segment in the text set can distinguish the voice source of the text.

[0095] Step 103, based on the responsibility classification model, determine the responsibility result of the speech source according to the text set.

[0096] In the embodiment of the present invention, the classification model for judging the source of speech refers to a classification model for judging the source of speech, which can be any classifier applicable to the present application, and the embodiment of the present invention does not limit this. The classification model for judging the source of speech can adopt a supervised learning method, input a set of sample texts marked with the judgment result label of the source of speech, and continuously adjust the parameters in the classification model for judging the source of speech according to the judgment result label of the marked source of speech through the classification model for judging the source of speech until the required performance is achieved, thereby obtaining a trained classification model for judging the source of speech.

[0097] In an embodiment of the present invention, the accountability classification model can take the text in the text collection as input and output the accountability result of the voice source. For example, when the conversation between the driver and the passenger is judged, the accountability classification mode is a binary classification model. The higher the probability value output by the binary classification model, the higher the probability of judging the driver, and the lower the probability of judging non-drivers (including passengers and other situations). A preset threshold is set. When the probability value is higher than the preset threshold, the driver is judged, otherwise the non-driver is judged. Because the text collection carries a second tag that identifies the source of the voice, the call data of the driver and the passenger is effectively processed, and the efficiency and accuracy of accountability are improved, thereby improving the service quality and user experience of the online car-hailing industry. By realizing automatic closed-loop processing of the entire link, labor costs can be reduced and processing efficiency and accuracy can be improved. This will help provide a better user experience and driver service quality.

[0098] According to an embodiment of the present invention, by processing the target audio, a plurality of voice segments arranged in time sequence are obtained; wherein the voice segments carry a first label identifying the source of the voice, and the first labels carried by two voice segments adjacent in time sequence are different; based on the plurality of voice segments arranged in time sequence, a text set consisting of text segments corresponding to the voice segments is generated; wherein the text segment carries a second label identifying the source of the voice; based on a judgment classification model, a judgment result of the voice source is determined according to the text set, so that the voice segments and the voice sources in the audio are automatically separated, and a text segment is generated, and the classification model is used to complete the judgment of the voice source, thereby realizing automatic closed-loop processing of the entire link, reducing labor costs, and improving judgment efficiency and accuracy.

[0099] Reference Figure 2, shows a flow chart of steps of another embodiment of a method for determining the source of speech of the present invention, the method may specifically include the following steps:

[0100] Step 201: Determine the start and end times of a plurality of initial speech segments arranged in time sequence and a first tag for identifying a speech source according to the audio of different channels.

[0101] In the embodiment of the present invention, the speech sources of the audios of different channels in the target audio are different. The speech source of the speech segment determined based on the audio of one channel is the speech source corresponding to the channel. Therefore, it is only necessary to determine a plurality of initial speech segments arranged in time sequence based on the audios of each channel, and then obtain the first label for identifying the speech source of each initial speech segment.

[0102] In the embodiment of the present invention, the process of determining multiple initial voice segments according to the audio of different channels is the process of determining the start and end time of each initial voice segment. Then, the multiple initial voice segments corresponding to different channels are arranged together in time sequence.

[0103] In an optional embodiment of the present invention, a specific implementation method of determining the start and end times of multiple initial voice segments arranged in time sequence and the first label identifying the source of the voice based on the audio of the different channels may include: separating the start and end times of multiple initial voice segments arranged in time sequence for the audio of each channel, and determining the first label identifying the source of the voice based on the voice source corresponding to the channel; sorting the start and end times of the multiple initial voice segments arranged in time sequence separated from different channels together to obtain the start and end times of the multiple initial voice segments arranged in time sequence and the first label identifying the source of the voice.

[0104] The audio of one channel is separated into speech segments to obtain the start and end times of multiple initial speech segments arranged in time sequence, and a first label identifying the speech source is determined according to the speech source corresponding to the channel. The start and end times of multiple initial speech segments separated from different channels and arranged in time sequence are sorted together to obtain the start and end times of multiple initial speech segments arranged in time sequence and a first label identifying the speech source.

[0105] For example, consider that the recording in the telephone scenario is two-channel audio. Generally, the left channel stores the voice of the caller and the right channel stores the voice of the called person. Based on the known role information of the caller and the called person, such as passenger and driver, the two-channel audio can be normalized into a format where the driver is in the left channel and the passenger is in the right channel. Figure 3The schematic diagram of the two-channel audio preprocessing process shown in the figure shows that for conversational two-channel audio, the left and right channels are separated and the channel positions are adjusted. For known speakers in each channel, the target audio is separated into speech segments. This step can be achieved using voice endpoint detection technology. The specific operations include:

[0106] The audio of the left channel is transcoded and resampled to adjust the audio to the audio format and sampling rate required by the following deep learning model, and the audio frames are divided according to the specified length (such as 1024 sampling points as one frame).

[0107] Use a deep learning model to determine whether the current audio frame contains human voice, and record the probability value in a list. After the model completes the reasoning of the audio frame, it processes the list of human voice probability values ​​to obtain the start and end time points of the effective human voice, that is, the start and end time of the voice segment. Among them, in the judgment logic, a probability threshold of human voice can be set. When the threshold is exceeded, the human voice is considered to have started, and when three consecutive silent frames are encountered, the human voice is considered to have ended. At the same time, for human voice segments that are too short, they can be considered invalid. The start and end time points of the effective human voice segment and the first label of the corresponding voice source are stored as a timestamp list of the left channel (such as left_st = [{"begin":0,"end":16000,"flag":"driver"}]).

[0108] Repeat the above steps to obtain the timestamp list of the right channel (such as right_st = [{"begin":16500, "end":34000, "flag":"passenger"}]).

[0109] The timestamp lists of the left channel and the right channel are merged as st=left_st+right_st, and sorted according to the start time point to obtain the final timestamp list, that is, the start and end time of multiple initial voice segments arranged in time sequence and the first label identifying the voice source, such as:

[0110] st=[

[0111] {'start':11808,'end':32736,'type':'customer','length':20928},

[0112] {'start':54816,'end':78816,'type':'driver','length':24000},

[0113] {'start':90144,'end':115680,'type':'customer','length':25536},

[0114] {'start':130080, 'end':154080, 'type': 'driver', 'length':24000},

[0115] {'start':185376,'end':229344,'type':'customer','length':43968},

[0116] {'start':257568,'end':326112,'type':'customer','length':68544},

[0117] {'start':329760,'end':361440,'type':'customer','length':31680},

[0118] {'start':388128,'end':419808,'type':'driver','length':31680}

[0119] ].

[0120] Step 202, based on the start and end times of the multiple initial voice segments arranged in time sequence and the first label identifying the voice source, the start and end times of the initial voice segments that are adjacent in time sequence and have the same first label identifying the voice source are merged to obtain the multiple voice segments arranged in time sequence.

[0121] In the embodiment of the present invention, among the multiple initial voice segments arranged in time sequence obtained through the above steps, there may be a situation where the first labels identifying the voice sources of the initial voice segments adjacent in time sequence are the same. In order to make the voice segments a conversation between different voice sources, it is necessary to merge the start and end times of the initial voice segments adjacent in time sequence and having the same first label identifying the voice sources, so as to obtain the above multiple voice segments arranged in time sequence.

[0122] For example, Figure 3 As shown, for the above-mentioned binaural timestamp list, adjacent initial voice segments with the same tag (type) are merged, and the start and end timestamps of the merged initial voice segments are recorded to realize the question-answering (QA) form. The above-mentioned timestamp list is organized in the form of multiple "customer+driver", that is, the passenger and the driver ask and answer each other. The merged audio here needs to be stored for use by the subsequent judgment and classification model.

[0123] Step 203: adding corresponding voice source marked voice to the voice segment according to the first tags for identifying the voice sources of the multiple voice segments arranged in time sequence.

[0124] In an embodiment of the present invention, for multiple voice segments arranged in time sequence, the voice segments can be recognized one by one for speech-to-text conversion, for example, sent to an ASR (Automatic Speech Recognition) model for speech-to-text conversion recognition. However, considering that the cost of calling speech-to-text recognition in this way will be very high, some target audio will produce dozens of voice segments, multiple calls will increase time and cost, and cannot meet the online quasi-real-time requirements. Therefore, it is necessary to recognize multiple voice segments arranged in time sequence for speech-to-text conversion at one time, while also being able to distinguish the corresponding voice sources.

[0125] In the embodiment of the present invention, the speech source marking speech is a speech used to mark the source of speech. The speech source marking speech is a piece of speech that is easy to identify and is unlikely to be the same as the content of the speech segment. Specifically, any applicable speech can be used as the speech source marking speech, and the embodiment of the present invention does not limit this. According to the first tag of the speech segment that identifies the speech source, the corresponding speech source marking speech is added to the speech segment. The speech source marking speech corresponds to the first tag that identifies the speech source. Specifically, the speech source marking speech can be added before or after the speech segment.

[0126] For example, Figure 3 As shown, a special voice segment flag (flag) corresponding to the first label identifying the voice source is generated by means of text-to-speech (TTS) technology, i.e., voice source marked voice. The generated voice source marked voice needs to have the following characteristics: easy to identify, the voice-to-text recognition system must be able to recognize it; specificity, the voice source is very unlikely to say related content; sufficient silence is left at the front and back ends of the voice source marked voice to prevent affecting the normal recognition of the voice segment of the voice source. Considering the commonly used Chinese scenarios, the TTS technology is used by default to generate the two voice source marked voices of customer and driver, and a certain length of silence segments are filled at the front and back ends.

[0127] Step 204, splicing the multiple voice segments with voice source marks added thereto arranged in time sequence to obtain spliced ​​voice audio.

[0128] In the embodiment of the present invention, multiple voice segments with voice source mark voices added are spliced ​​together, and the obtained audio is recorded as spliced ​​voice audio.

[0129] For example, Figure 3As shown in the figure, after the speech source marked speech is spliced ​​into each speech segment with the corresponding label, all processed audios are spliced ​​together to obtain a spliced ​​speech audio, which can be sent to ASR for recognition at one time. Figure 4 The schematic diagram of the preprocessing result of the two-channel audio is shown in FIG. For the two-channel target audio, the human voice segment is separated by VAD technology to obtain the voice segments of the driver and the passenger. Then, the A audio flag (i.e., the voice source marking voice) and the B audio flag are inserted after each voice segment.

[0130] Step 205, after converting the concatenated speech audio into corresponding text, generate a text set consisting of text segments carrying a second tag identifying the source of the speech according to the marked text corresponding to the identified speech source marked speech.

[0131] In an embodiment of the present invention, speech-to-text recognition is performed on the spliced ​​speech audio to obtain a corresponding text, which contains a marked text converted from the speech with the speech source marked. According to the marked text corresponding to the speech with the speech source marked, the corresponding text is divided into text segments, that is, text segments corresponding to the speech segments, and according to the marked text, a second tag identifying the speech source is added to the text segment. Finally, a text set is generated based on the text segments carrying the second tag identifying the speech source.

[0132] For example, Figure 5 The data in the merged speech segment list is mono audio, which is sent to ASR for recognition to obtain recognized text. The recognized text is parsed and compared with Figure 6 The result of direct recognition of the original audio is shown in the figure. After being processed by the present invention, the responsibility judgment of the text recognition result obtained will be more intuitive. The final text recognition result is as follows Figure 7 As shown in the figure, customer and driver are the terminators of the conversation between the driver and the passenger in this round of conversation. After simple text processing, the final text recognition result with the second speaker label can be obtained. Figure 8 shown.

[0133] In an embodiment of the present invention, a conversational text recognition solution that uses a large number of parameters (500M+), is complex and has high deployment costs, such as a speaker log model, is converted into a combined solution that only requires voice endpoint detection (less than 2M) and pre-generated text-to-speech (TTS). This solution has a very low deployment cost, simple logic and high accuracy.

[0134] In an optional embodiment of the present invention, before determining the result of the voice source based on the judgment classification model according to the text set, it can also include: filtering out the text set that does not contain sensitive words in the sensitive word library, and retaining the text set that contains sensitive words in the sensitive word library.

[0135] In order to further save costs and avoid unnecessary calls to the judgment and classification model, a pre-set sensitive word library can be used to filter the text collection, that is, the conversations that may trigger the judgment logic are sent to the next process, otherwise they are filtered out.

[0136] For example, Figure 5 As shown, given a sensitive word library D, which contains n sensitive words, it can be expressed as D = {w1, w2, ..., wn}, where wi represents the i-th sensitive word. Define a function f(T, D) to determine whether the text T contains an element in the sensitive word library D. The function returns a binary value, 1 means that T contains the sensitive words in D, and 0 means that it does not contain them. First, you need to use word segmentation tools such as jieba to split the text T into a sequence of words T = [t1, t2, ..., tm], where ti represents the i-th word. Then, traverse each word or character ti in T to check whether it exists in the sensitive word library D. Assume that the function g(t, D) represents the operation of determining whether the word or character t is in the sensitive word library D, and the return result is 1 or 0. Then the function f(T, D) can be expressed as: f(T, D) = max(g(t1, D), g(t2, D), ..., g(tm, D)). Among them, max() represents the operation of taking the maximum value. If any word or character ti exists in the sensitive word library D during the traversal process, the return result is 1; otherwise, the return result is 0.

[0137] Step 206: extracting a dialogue text consisting of text segments from the text collection according to a second tag in the text collection that identifies the source of the speech and a preset rule.

[0138] In the embodiment of the present invention, the text processed by the judgment and classification model at one time is limited, and the dialogue text of the speech source can be extracted from the text collection in advance. The dialogue text is composed of text fragments. Specifically, the dialogue text can be extracted according to the second tag in the text collection that identifies the speech source and the preset rules.

[0139] For example, two speech sources A and B, according to the time sequence, extract the dialogue text composed of A's text segment and B's text segment, and so on, and extract the dialogue text backward continuously. For another example, two speech sources A and B, according to the time sequence, extract the dialogue text composed of A's text segment, B's text segment and A's text segment, and then extract the dialogue text composed of B's ​​text segment, A's text segment and B's text segment, and so on, and extract the dialogue text backward continuously. Specifically, any applicable preset rule can be used to extract the dialogue text, and the embodiment of the present invention does not limit this.

[0140] In an optional embodiment of the present invention, according to the second tag in the text collection that identifies the source of the speech, a specific implementation method of extracting a dialogue text composed of text fragments from the text collection according to a preset rule may include: according to the second tag in the text collection that identifies the source of the speech, extracting the dialogue text with a sliding window with a step size of 2 text fragments and a width of at least 3 text fragments; for dialogue text that exceeds the preset length limit, splitting it into dialogue text that meets the preset length limit according to the text fragments.

[0141] A dynamic sliding window is used to extract the dialogue text from the text set according to the second tag in the text set that identifies the source of the speech, wherein the step length of the sliding window is 2 text segments and the width is at least 3 text segments.

[0142] Some extracted conversation texts may exceed the preset length limit, so the conversation texts need to be further processed. The conversation texts that do not exceed the preset length limit can be used directly, while the conversation texts that exceed the preset length limit are split to obtain the conversation texts that meet the preset length limit. The conversation texts obtained after splitting also need to include complete text segments.

[0143] For example, Fig. 9 The diagram of the dialogue text judgment process shown in FIG. 1 shows that for a text set T, the text set with f(T, D) = 1 is fed into the judgment classification model for processing. When processing a text set, two different forms of input may appear: CONV a and CONV b :

[0144]

[0145] For these two cases, a dynamic sliding window method is used to extract the conversation text in QAQ format, such as CONV a The content that can be extracted is as follows:

[0146] a)

[0147] b)

[0148] c)

[0149] At the same time, in order to prevent the QAQ dialogue text from being too long and unable to be fed into the judgment and classification model, the dialogue text can be split and extracted into the following forms:

[0150] a) That is, the length of the QAQ conversation text does not exceed the preset length limit,

[0151] b) That is, the length of the QAQ dialogue text exceeds the preset length limit, and the first text segment is treated as a separate dialogue text.

[0152] c) That is, the last two text fragments are used as a dialogue text.

[0153] d) That is, the length of the QAQ dialogue text does not exceed the preset length limit.

[0154] Step 207: input the dialogue text into the responsibility classification model to obtain the responsibility indication value corresponding to the dialogue text.

[0155] In the embodiment of the present invention, the dialogue texts are input into the responsibility classification model one by one to obtain the responsibility indication value corresponding to the dialogue text. The responsibility indication value refers to the probability value of assigning responsibility to a certain speech source, and the probability value indicates the possibility of assigning responsibility to the speech source. Each dialogue text can obtain a responsibility indication value. Specifically, any applicable model for classifying dialogue texts can be used as the responsibility classification model, and the embodiment of the present invention does not limit this.

[0156] In an embodiment of the present invention, after the dialogue text is input into the judgment classification model, text feature data of the dialogue text is first generated, and then the text feature data can be sent to the classifier to obtain a judgment indicator value. For example, the text feature data can be sent as input to LightGBM (a machine learning method based on a gradient boosting algorithm) for training. LightGBM can learn the nonlinear relationship between features and give a classification probability value of the dialogue text, that is, a judgment indicator value.

[0157] In an optional embodiment of the present invention, a specific implementation method of inputting the dialogue text into the responsibility classification model to obtain a responsibility indication value corresponding to the dialogue text may include: generating, based on the dialogue text, a text tensor representing the text of the dialogue text, an attention tensor representing the valid positions in the text tensor, and a source segment tensor representing the speech source corresponding to the text in the dialogue text; inputting the text tensor, attention tensor, and source segment tensor into the encoder of the responsibility classification model for encoding to obtain text feature data; inputting the text feature data into the classifier of the responsibility classification model for classification to obtain a responsibility indication value corresponding to the dialogue text.

[0158] The dialogue text is processed to generate corresponding text tensors, attention tensors, and source segment tensors. The text tensor is a tensor used to represent the text of the dialogue text, the attention tensor is a tensor used to represent the valid position in the text tensor, and the source segment tensor is a tensor used to represent the speech source corresponding to the text in the dialogue text.

[0159] For example, Fig. 9As shown in the figure, after processing the dialogue text, the corresponding bert_input (text tensor), segment_mask (source segment tensor), and attention_mask (attention tensor) are obtained, and then the encoder part of the deep learning network of deberta_v2 (Decoding-enhanced Bidirectional Encoder Representations from Transformerswith disentangled attention v2, the second generation of decoding-enhanced bidirectional encoder representations from Transformer with disentangled attention) is used for encoding. Among them, bert_input is a list of dialogue text windows after the sliding window operation, each window is a tensor of length SEQ_LEN, which is used to input the BERT model for processing. Attention_mask is a tensor of the same shape as bert_input, which is used to indicate which positions of the tokens are valid and which are invalid. Specifically, for valid tokens in the dialogue text window, the value of the corresponding position of attention_mask is 1, and for invalid tokens, the value is 0. In this way, by masking the attention of invalid positions, the BERT model can ignore these invalid positions when processing input. segment_mask is also a tensor of the same shape as bert_input, which is used to distinguish between tokens of different text fragments. In the task, the input may contain multiple text fragments, and the role of segment_mask is to distinguish between tokens of different text fragments. For tokens in the same text fragment, the value of the corresponding position of segment_mask is 0, and for tokens between different text fragments, the value of the corresponding position of segment_mask is 1. In this task, 0 is defined as the text fragment of speech source A, and 1 is defined as the text fragment of speech source B. In this way, the BERT model can obtain independent encoding of each text fragment and provide corresponding guidance information when processing the relationship between different text fragments.

[0160] The text tensor, attention tensor, and speech source segment tensor are input into the encoder of the classification model for encoding to obtain text feature data. The encoder models and represents the text to capture the semantics and contextual relationship of the text. The output encoding result can be input into the classifier as text feature data.

[0161] In an optional embodiment of the present invention, a specific implementation method of inputting the text feature data into the classifier of the responsibility classification model for classification to obtain the responsibility indication value corresponding to the dialogue text may include: generating audio feature data based on the speech segment corresponding to the dialogue text; fusing the text feature data and the audio feature data to obtain fused feature data; inputting the fused feature data into the classifier of the responsibility classification model for classification to obtain the responsibility indication value corresponding to the dialogue text.

[0162] like Fig. 9 As shown, the above method only considers the text feature data and ignores the information of the audio itself. In order to make full use of the audio information, the speech segment corresponding to the text segment corresponding to the dialogue text can be obtained, and then the audio feature data of one or more speech segments corresponding to the dialogue text can be extracted. For example, MFCC (Mel frequency cepstral coefficient) and tone features (Pitch), etc. These audio features can represent the spectrum, energy and time domain characteristics of the audio.

[0163] In order to fuse the text feature data and the audio feature data, the text feature data and the audio feature data can be merged. The merging can adopt simple concatenation, weighted summation or other fusion strategies. The merged feature data is recorded as fused feature data. Then the fused feature data can be sent to the classifier to obtain the judgment indicator value. For example, the fused feature data can be sent as input to LightGBM (a machine learning method based on the gradient boosting algorithm) for training. LightGBM can learn the nonlinear relationship between features and give the classification probability value of the dialogue text, that is, the judgment indicator value.

[0164] The present invention provides a solution for conversational text classification based on the BERT deep learning model, which takes into account the contextual relationship between the two voice sources and uses a data processing method for conversational text; at the same time, on the basis of text feature data, it further integrates the audio's own features to bring better classification results.

[0165] Step 208, determining the accountability result in combination with the accountability indicator values ​​corresponding to the respective dialogue texts.

[0166] In the embodiment of the present invention, the judgment result refers to the result of judging or not judging a certain voice source, or the result of judging which voice source is judged, and the embodiment of the present invention does not limit this. In combination with the judgment indication value corresponding to each dialogue text, the specific implementation method of determining the judgment result may include multiple methods, and the embodiment of the present invention does not limit this.

[0167] For example, Fig. 9As shown, the judgment indicator values ​​corresponding to all dialogue texts are weighted and summed to obtain the total judgment indicator value prop of the entire dialogue, and then according to the relationship between the total judgment indicator value prop and a preset threshold value (such as 0.85), it is determined whether to judge or not judge a certain voice source. For another example, according to the relationship between the judgment indicator value corresponding to each dialogue text and the preset judgment threshold value, the sub-judgment result corresponding to each dialogue text is obtained, and then the sub-judgment results are counted, and the final judgment result is determined according to the statistical results.

[0168] In a specific implementation of an embodiment of the present invention, in order to determine the final accountability indicator value, the output accountability indicator values ​​may be weighted and summed. This weighted summation process may involve some weight strategies for determining which dialogue texts have more authoritative and credible accountability indicators. Finally, the total accountability indicator value of the entire dialogue may be obtained based on the result of the weighted summation. This total accountability indicator value can determine whether there is a certain situation in the dialogue or the possibility of answering the question.

[0169] According to an embodiment of the present invention, the start and end times of a plurality of initial voice segments arranged in time sequence and a first tag identifying a voice source are determined according to the audio of different channels; the start and end times of the initial voice segments that are adjacent in time sequence and have the same first tag are merged according to the start and end times of the plurality of initial voice segments arranged in time sequence and the first tag to obtain the plurality of voice segments arranged in time sequence; corresponding voice source marking voice is added to the voice segment according to the first tag identifying the voice source of the plurality of voice segments arranged in time sequence; the plurality of voice segments arranged in time sequence to which the voice source marking voice is added are spliced ​​to obtain spliced ​​voice audio; after converting the spliced ​​voice audio into corresponding text, According to the marked text corresponding to the identified speech source marked speech, a text set consisting of text fragments carrying a second label identifying the speech source is generated; according to the second label identifying the speech source in the text set, according to preset rules, a dialogue text consisting of text fragments is extracted from the text set; the dialogue text is input into the judgment classification model to obtain the judgment indication value corresponding to the dialogue text; the judgment result is determined in combination with the judgment indication value corresponding to each dialogue text, so that the speech fragments and speech sources in the audio are automatically separated, and a text fragment is generated, and the classification model is used to complete the judgment of the speech source, thereby realizing automatic closed-loop processing of the entire link, reducing labor costs, and improving judgment efficiency and accuracy.

[0170] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0171] Reference Fig.10 The structure block diagram of an embodiment of a speech source determination device of the present invention is shown, which may specifically include the following modules:

[0172] The audio processing module 301 is used to process the target audio to obtain a plurality of voice segments arranged in time sequence; wherein the voice segments carry a first tag identifying a voice source, and the first tags carried by two voice segments adjacent in time sequence are different;

[0173] A set generation module 302 is used to generate a text set consisting of text segments corresponding to the multiple voice segments arranged in time sequence; wherein the text segment carries a second tag identifying the source of the voice;

[0174] The result determination module 303 is used to determine the result of the voice source based on the judgment classification model and the text set.

[0175] Optionally, the audios of different channels in the target audio have different speech sources, and the audio processing module includes:

[0176] A label determination submodule, used to determine the start and end times of a plurality of initial voice segments arranged in time sequence and a first label identifying a voice source according to the audio of the different channels;

[0177] The merging submodule is used to merge the start and end times of the initial voice segments that are adjacent in time sequence and have the same first label identifying the voice source according to the start and end times of the multiple initial voice segments arranged in time sequence and the first label identifying the voice source, so as to obtain the multiple voice segments arranged in time sequence.

[0178] Optionally, the label determination submodule includes:

[0179] A label determination unit, for separating the start and end times of a plurality of initial voice segments arranged in time sequence from the audio of each channel, and determining a first label for identifying a voice source according to a voice source corresponding to the channel;

[0180] The sorting unit is used to sort the start and end times of multiple initial voice segments separated from different channels and arranged in time sequence, and obtain the start and end times of the multiple initial voice segments arranged in time sequence and a first label identifying the voice source.

[0181] Optionally, the set generation module includes:

[0182] A speech adding submodule, used for adding a corresponding speech source marking speech to the speech segment according to the first label identifying the speech source of the plurality of speech segments arranged in time sequence;

[0183] A splicing submodule, used for splicing a plurality of speech segments with speech source marks added thereto arranged in time sequence to obtain spliced ​​speech audio;

[0184] The set generation submodule is used to generate a text set consisting of text segments carrying a second tag identifying the source of the speech according to the marked text corresponding to the identified speech source marked speech after converting the spliced ​​speech audio into corresponding text.

[0185] Optionally, the result determination module includes:

[0186] An extraction submodule, configured to extract a dialogue text consisting of text segments from the text collection according to a second tag in the text collection that identifies the source of the speech and a preset rule;

[0187] An indicator value determination submodule, used for inputting the dialogue text into the responsibility classification model to obtain a responsibility indicator value corresponding to the dialogue text;

[0188] The result determination submodule is used to determine the accountability result in combination with the accountability indication value corresponding to each dialogue text.

[0189] Optionally, the extraction submodule includes:

[0190] A text extraction unit, configured to extract the conversation text according to a second tag in the text set that identifies the source of the speech, using a sliding window with a step size of 2 text segments and a width of at least 3 text segments;

[0191] The splitting unit is used to split the dialogue text exceeding the preset length limit into dialogue text meeting the preset length limit according to the text segments.

[0192] Optionally, the indication value determination submodule includes:

[0193] A tensor generation unit is used to generate, according to the dialogue text, a text tensor representing the text of the dialogue text, an attention tensor representing the valid positions in the text tensor, and a source segment tensor representing the speech source corresponding to the text in the dialogue text;

[0194] An encoding unit, used for inputting the text tensor, the attention tensor, and the source segment tensor into the encoder of the judgment and classification model for encoding to obtain text feature data;

[0195] The classification unit is used to input the text feature data into the classifier of the responsibility classification model for classification, so as to obtain the responsibility indication value corresponding to the dialogue text.

[0196] Optionally, the classification unit includes:

[0197] A feature generation subunit, used to generate audio feature data according to the speech segment corresponding to the dialogue text;

[0198] A fusion subunit, used for fusing the text feature data and the audio feature data to obtain fused feature data;

[0199] The classification subunit is used to input the fused feature data into the classifier of the responsibility classification model for classification, so as to obtain the responsibility indication value corresponding to the dialogue text.

[0200] Optionally, the device further comprises:

[0201] The collection filtering module is used to filter out the text collection that does not contain sensitive words in the sensitive word library before determining the judgment result of the voice source based on the judgment classification model according to the text collection, and retain the text collection that contains sensitive words in the sensitive word library.

[0202] According to an embodiment of the present invention, by processing the target audio, a plurality of voice segments arranged in time sequence are obtained; wherein the voice segments carry a first label identifying the source of the voice, and the first labels carried by two voice segments adjacent in time sequence are different; based on the plurality of voice segments arranged in time sequence, a text set consisting of text segments corresponding to the voice segments is generated; wherein the text segment carries a second label identifying the source of the voice; based on a judgment classification model, a judgment result of the voice source is determined according to the text set, so that the voice segments and the voice sources in the audio are automatically separated, and a text segment is generated, and the classification model is used to complete the judgment of the voice source, thereby realizing automatic closed-loop processing of the entire link, reducing labor costs, and improving judgment efficiency and accuracy.

[0203] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0204] Fig.11 1 is a block diagram of an electronic device 600 for voice source determination according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0205] Reference Fig.11 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0206] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned voice source determination method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0207] The memory 604 is configured to store various types of data to support operations on the device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0208] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0209] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0210] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0211] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0212] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the device 600, the relative positioning of components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0213] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0214] In an exemplary embodiment, the electronic device 600 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned shutdown control method.

[0215] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to complete the above-mentioned shutdown control method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0216] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a server, enables the server to perform a method for determining a voice source, the method comprising:

[0217] Processing the target audio to obtain a plurality of voice segments arranged in time sequence; wherein the voice segments carry a first label identifying a voice source, and the first labels carried by two voice segments adjacent in time sequence are different;

[0218] According to the multiple voice segments arranged in time sequence, a text set consisting of text segments corresponding to the voice segments is generated; wherein the text segments carry a second tag identifying the source of the voice;

[0219] Based on the accountability classification model, the accountability result of the speech source is determined according to the text set.

[0220] Optionally, the audios of different channels in the target audio have different speech sources, and the target audio is processed to obtain a plurality of speech segments arranged in time sequence, including:

[0221] Determining, based on the audio of the different channels, the start and end times of a plurality of initial voice segments arranged in time sequence and a first tag identifying a voice source;

[0222] According to the start and end times of the multiple initial voice segments arranged in time sequence and the first label identifying the voice source, the start and end times of the initial voice segments that are adjacent in time sequence and have the same first label identifying the voice source are merged to obtain the multiple voice segments arranged in time sequence.

[0223] Optionally, determining the start and end times of a plurality of initial voice segments arranged in time sequence and a first tag identifying a voice source according to the audio of the different channels includes:

[0224] For the audio of each channel, the start and end times of multiple initial voice segments arranged in time sequence are separated respectively, and a first tag identifying the voice source is determined according to the voice source corresponding to the channel;

[0225] The start and end times of the multiple initial voice segments separated from different channels and arranged in time sequence are sorted together to obtain the start and end times of the multiple initial voice segments arranged in time sequence and a first label identifying a voice source.

[0226] Optionally, generating a text set consisting of text segments corresponding to the multiple voice segments arranged in time sequence includes:

[0227] According to the first tags identifying the speech sources of the plurality of speech segments arranged in time sequence, adding corresponding speech source marked speech to the speech segments;

[0228] splicing a plurality of speech segments to which speech source marks are added and which are arranged in time sequence to obtain spliced ​​speech audio;

[0229] After converting the concatenated speech audio into corresponding text, a text set consisting of text segments carrying a second tag identifying the source of the speech is generated according to the marked text corresponding to the identified speech source marked speech.

[0230] Optionally, the determining the result of the voice source based on the responsibility classification model according to the text set includes:

[0231] Extracting a dialogue text consisting of text segments from the text collection according to a second tag in the text collection that identifies the source of the speech and according to a preset rule;

[0232] Inputting the dialogue text into the responsibility classification model to obtain a responsibility indication value corresponding to the dialogue text;

[0233] The accountability result is determined by combining the accountability indication values ​​corresponding to the respective dialogue texts.

[0234] Optionally, extracting the dialogue text consisting of text segments from the text collection according to a second tag in the text collection that identifies the source of the speech and a preset rule includes:

[0235] Extracting the conversation text according to a second tag in the text set that identifies the source of the speech, using a sliding window with a step size of 2 text segments and a width of at least 3 text segments;

[0236] For dialogue texts that exceed the preset length limit, they are split into dialogue texts that meet the preset length limit according to text segments.

[0237] Optionally, inputting the dialogue text into the responsibility classification model to obtain a responsibility indication value corresponding to the dialogue text includes:

[0238] According to the dialogue text, a text tensor representing the text of the dialogue text, an attention tensor representing the valid positions in the text tensor, and a source segment tensor representing the speech source corresponding to the text in the dialogue text are generated;

[0239] Inputting the text tensor, the attention tensor, and the source segment tensor into the encoder of the judgment and classification model for encoding to obtain text feature data;

[0240] The text feature data is input into the classifier of the responsibility classification model for classification to obtain the responsibility indication value corresponding to the dialogue text.

[0241] Optionally, the step of inputting the text feature data into a classifier of the responsibility classification model for classification to obtain a responsibility indication value corresponding to the dialogue text includes:

[0242] Generating audio feature data according to the speech segment corresponding to the conversation text;

[0243] Fusion of the text feature data and the audio feature data to obtain fused feature data;

[0244] The fused feature data is input into the classifier of the responsibility classification model for classification to obtain the responsibility indication value corresponding to the dialogue text.

[0245] Optionally, before determining the result of the voice source identification based on the identification classification model according to the text set, the method further includes:

[0246] The text set that does not contain sensitive words in the sensitive word library is filtered out, and the text set that contains sensitive words in the sensitive word library is retained.

[0247] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0248] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0249] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0250] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0251] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0252] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0253] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0254] The above is a detailed introduction to a method for determining a speech source and a device for determining a speech source provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for determining the source of speech, characterized in that: The method comprises: Processing the target audio to obtain a plurality of voice segments arranged in time sequence; wherein the voice segments carry a first label identifying a voice source, and the first labels carried by two voice segments adjacent in time sequence are different; According to the multiple voice segments arranged in time sequence, a text set consisting of text segments corresponding to the voice segments is generated; wherein the text segments carry a second tag identifying the source of the voice; Based on the accountability classification model, the accountability result of the speech source is determined according to the text set.

2. The method according to claim 1, characterized in that The audio of different channels in the target audio have different speech sources, and the target audio is processed to obtain a plurality of speech segments arranged in time sequence, including: Determining, based on the audio of the different channels, the start and end times of a plurality of initial voice segments arranged in time sequence and a first tag identifying a voice source; According to the start and end times of the multiple initial voice segments arranged in time sequence and the first label identifying the voice source, the start and end times of the initial voice segments that are adjacent in time sequence and have the same first label identifying the voice source are merged to obtain the multiple voice segments arranged in time sequence.

3. The method according to claim 2, characterized in that The step of determining the start and end times of a plurality of initial voice segments arranged in time sequence and a first tag identifying a voice source according to the audio of the different channels includes: For the audio of each channel, the start and end times of multiple initial voice segments arranged in time sequence are separated respectively, and a first tag identifying the voice source is determined according to the voice source corresponding to the channel; The start and end times of the multiple initial voice segments separated from different channels and arranged in time sequence are sorted together to obtain the start and end times of the multiple initial voice segments arranged in time sequence and a first label identifying a voice source.

4. The method according to claim 1, characterized in that: The step of generating a text set consisting of text segments corresponding to the multiple voice segments arranged in time sequence comprises: According to the first tags identifying the speech sources of the plurality of speech segments arranged in time sequence, adding corresponding speech source marked speech to the speech segments; splicing a plurality of speech segments to which speech source marks are added and which are arranged in time sequence to obtain spliced ​​speech audio; After converting the concatenated speech audio into corresponding text, a text set consisting of text segments carrying a second tag identifying the source of the speech is generated according to the marked text corresponding to the identified speech source marked speech.

5. The method according to claim 1, characterized in that The method of determining the result of the voice source based on the voice classification model according to the text set includes: Extracting a dialogue text consisting of text segments from the text collection according to a second tag in the text collection that identifies the source of the speech and according to a preset rule; Inputting the dialogue text into the responsibility classification model to obtain a responsibility indication value corresponding to the dialogue text; The accountability result is determined by combining the accountability indication values ​​corresponding to the respective dialogue texts.

6. The method according to claim 5, characterized in that The extracting the dialogue text consisting of text segments from the text collection according to the second tag identifying the source of the speech in the text collection and according to a preset rule includes: Extracting the conversation text according to a second tag in the text set that identifies the source of the speech, using a sliding window with a step size of 2 text segments and a width of at least 3 text segments; For dialogue texts that exceed the preset length limit, they are split into dialogue texts that meet the preset length limit according to text segments.

7. The method according to claim 5, characterized in that The step of inputting the dialogue text into the responsibility classification model to obtain the responsibility indication value corresponding to the dialogue text includes: According to the dialogue text, a text tensor representing the text of the dialogue text, an attention tensor representing the valid positions in the text tensor, and a source segment tensor representing the speech source corresponding to the text in the dialogue text are generated; Inputting the text tensor, the attention tensor, and the source segment tensor into the encoder of the judgment and classification model for encoding to obtain text feature data; The text feature data is input into the classifier of the responsibility classification model for classification to obtain the responsibility indication value corresponding to the dialogue text.

8. The method according to claim 7, characterized in that The step of inputting the text feature data into the classifier of the responsibility classification model for classification to obtain the responsibility indication value corresponding to the dialogue text includes: Generating audio feature data according to the speech segment corresponding to the conversation text; Fusion of the text feature data and the audio feature data to obtain fused feature data; The fused feature data is input into the classifier of the responsibility classification model for classification to obtain the responsibility indication value corresponding to the dialogue text.

9. The method according to claim 1, characterized in that: Before determining the result of the voice source identification based on the text set based on the identification classification model, the method further includes: The text set that does not contain sensitive words in the sensitive word library is filtered out, and the text set that contains sensitive words in the sensitive word library is retained.

10. A device for determining the source of speech, characterized in that: The device comprises: An audio processing module, used for processing the target audio to obtain a plurality of voice segments arranged in time sequence; wherein the voice segments carry a first label identifying a voice source, and the first labels carried by two voice segments adjacent in time sequence are different; A set generation module, used to generate a text set consisting of text segments corresponding to the multiple voice segments arranged in time sequence; wherein the text segment carries a second tag identifying the source of the voice; The result determination module is used to determine the result of the voice source based on the judgment classification model and the text set.

11. An electronic device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the apparatus to perform the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer program stored therein enables the processor to execute the method according to any one of claims 1 to 9.