Sentence splitting method, apparatus, storage medium, and electronic device

Abstract

The present disclosure relates to a method, device, storage medium and electronic device for sentence segmentation. The method comprises: obtaining target audio data; extracting speech recognition text corresponding to the target audio data and a first time period corresponding to each recognized character in the speech recognition text in the target audio data; performing speaker segmentation on the target audio data to obtain a second time period corresponding to each speech segment in the target audio data; and performing speaker segmentation on the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment to obtain a sentence segmentation result. Thus, the speaker time period information and the time period corresponding to each character in the speech recognition text in the target audio data can be effectively utilized to perform speaker segmentation on the speech recognition text, to reasonably and effectively segment the speaker conversion, to avoid a single sentence segment containing speech content of multiple speakers, and to improve the sentence segmentation effect.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of speech recognition, in particular, to a sentence segmentation method and device, a storage medium and an electronic device. BACKGROUND

[0002] In the application of speech recognition in video caption scenarios, the recognized text needs to be segmented for split-screen display. Moreover, in order to ensure the readability of the captions, it is often required that a single segment contains only one speaker, avoiding the situation that a single screen caption contains the speech content of different speakers. The conventional segmentation method only combines the semantic information of the speech recognition text to cut at the semantic turning points. This method has good effect on videos with a single speaker, but for multi-speaker dialogue scene videos, the use of semantic information alone will lead to poor cutting effect at the speaker switching point, resulting in a single segment containing the speech content of multiple speakers. SUMMARY

[0003] This part is provided to briefly introduce the concepts, which will be described in detail in the following specific embodiments. This part is not intended to identify the key features or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] In a first aspect, the present disclosure provides a sentence segmentation method, comprising:

[0005] obtaining target audio data;

[0006] extracting speech recognition text corresponding to the target audio data and a first time period corresponding to each recognized character in the speech recognition text in the target audio data;

[0007] performing speaker segmentation on the target audio data to obtain a second time period corresponding to each speech segment in the target audio data;

[0008] performing speaker segmentation on the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment, to obtain a segmentation result.

[0009] In a second aspect, the present disclosure provides a sentence segmentation device, comprising:

[0010] an obtaining module configured to obtain target audio data;

[0011] an extracting module configured to extract speech recognition text corresponding to the target audio data obtained by the obtaining module and a first time period corresponding to each recognized character in the speech recognition text in the target audio data;

[0012] The first segmentation module is configured to perform speaker segmentation on the target audio data obtained by the acquisition module to obtain a second time period corresponding to each speech segment in the target audio data.

[0013] The second segmentation module is configured to perform speaker segmentation on the speech recognition text according to the first time period corresponding to each recognized character extracted by the extraction module and the second time period corresponding to each speech segment obtained by the first segmentation module to obtain a segmentation result.

[0014] In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program, which, when executed by a processing device, implements the steps of the method provided in the first aspect of the present disclosure.

[0015] In a fourth aspect, the present disclosure provides an electronic device comprising:

[0016] a storage device having stored thereon one or more computer programs;

[0017] one or more processing devices configured to execute the one or more computer programs stored in the storage device to implement the steps of the method provided in the first aspect of the present disclosure.

[0018] In the above technical solution, the target audio data is obtained, the speech recognition text corresponding to the target audio data and the first time period corresponding to each recognized character in the speech recognition text in the target audio data are extracted, the speaker segmentation is performed on the target audio data to obtain the second time period corresponding to each speech segment in the target audio data, and then the speaker segmentation is performed on the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment to obtain a segmentation result. Thus, the speaker time period information and the time period corresponding to each character in the speech recognition text in the target audio data can be effectively utilized to perform the speaker segmentation on the speech recognition text, to realize the reasonable and effective segmentation at the speaker conversion position, to avoid the situation that a single segmentation result contains speech content of multiple speakers, and to improve the segmentation effect.

[0019] Other features and advantages of the present disclosure will be described in detail in the following detailed description section. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other features and advantages of the embodiments of the present disclosure will become more apparent by describing in detail some embodiments thereof with reference to the attached drawings in which:

[0021] Figure 1is a flowchart of a sentence dividing method according to an exemplary embodiment.

[0022] Figure 2 is a flowchart of a sentence dividing method according to another exemplary embodiment.

[0023] Figure 3 is a flowchart of a sentence dividing method according to another exemplary embodiment.

[0024] Figure 4 is a block diagram of a sentence dividing apparatus according to an exemplary embodiment.

[0025] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the present disclosure are shown. Like reference numerals can be used to refer to like elements throughout. It is to be understood that other embodiments can be taken without departing from the scope of the present disclosure, and that the scope of the present disclosure should not be interpreted as being limited to the embodiments set forth herein but should be accorded the full scope consistent with the claims and any equivalents thereof.

[0027] It is understood that each of the steps recited in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit execution of the steps shown. The scope of the present disclosure is not limited in this regard.

[0028] As used herein, the term "includes" and its variants are to be read to be equivalent to "comprising" and its variants. The term "based on" is to be interpreted as "based, at least in part, on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related terms are to be interpreted in a like fashion.

[0029] It should be noted that the terms "first", "second", and the like, used in the description and in the claims of the present disclosure are used only to distinguish one of the different apparatuses, modules, or units from the other, and do not imply the order or the relative importance of the functions performed by the different apparatuses, modules, or units.

[0030] It should be noted that the terms "one", "multiple", and the like, used in the present disclosure are illustrative and not limiting, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0031] Names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0032] Figure 1 is a flowchart of a sentence segmentation method according to an example embodiment. As shown in Figure 1 , the method can include S101-S104.

[0033] In S101, target audio data is obtained.

[0034] In the present disclosure, the target audio data can include multiple speaker speech segments. For example, the target audio data can be a multi-speaker dialogue recording, or an audio segment in a multi-speaker dialogue scene video.

[0035] In S102, speech recognition text corresponding to the target audio data and a first time period corresponding to each recognized character in the speech recognition text in the target audio data are extracted.

[0036] In the present disclosure, the target audio data can be subjected to speech recognition using an automatic speech recognition (ASR) technology to obtain speech recognition text and a start time and an end time of each character (hereinafter referred to as a recognized character) in the speech recognition text corresponding to the target audio data, i.e., a first time period.

[0037] In S103, speaker segmentation is performed on the target audio data to obtain a second time period corresponding to each speaker segment in the target audio data.

[0038] In the present disclosure, speaker segmentation of the target audio refers to detecting speaker transition points in the target audio data, and regarding speech between adjacent two speaker transition points as a speaker segment.

[0039] In S104, speaker segmentation is performed on the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speaker segment to obtain a sentence segmentation result.

[0040] In the technical solution, target audio data is acquired, speech recognition text corresponding to the target audio data and a first time period corresponding to each recognized character in the speech recognition text in the target audio data are extracted, meanwhile, speaker segmentation is performed on the target audio data to obtain a second time period corresponding to each speech segment in the target audio data, then, speaker segmentation is performed on the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment to obtain a segmentation result. Thus, the speaker time period information and the time period corresponding to each character in the speech recognition text in the target audio data can be effectively utilized to perform speaker segmentation on the speech recognition text, reasonable and effective segmentation at a speaker conversion point is realized, the situation that a single segment contains speech content of multiple speakers is avoided, and the segmentation effect is improved.

[0041] The specific implementation of performing speaker segmentation on the target audio data to obtain the second time period corresponding to each speech segment in the target audio data in S103 is described in detail below. Specifically, speaker segmentation can be performed in various implementation manners. In one implementation manner, speaker segmentation can be performed on the target audio data by receiving manually input segmentation markers to determine the start time and the end time of each speech segment in the target audio data, that is, the second time period corresponding to each speech segment in the target audio data.

[0042] In another implementation manner, the target audio data can be input into a pre-trained speaker recognition model to perform speaker segmentation on the target audio data to obtain the second time period corresponding to each speech segment in the target audio data. In this way, the speech segments of different speakers can be automatically segmented, which is convenient and fast, thereby improving the segmentation efficiency of the subsequent speech recognition text.

[0043] The specific implementation of performing speaker segmentation on the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment in S104 is described in detail below. Specifically, the following steps (1) and (2) can be implemented:

[0044] (1) determining a speaker conversion point character from the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment.

[0045] (2) performing speaker segmentation on the speech recognition text according to the speaker conversion point character as a segmentation basis.

[0046] The speaker conversion point character belongs to a previous segment.

[0047] For example, the speech recognition text is: "Have you eaten?" "I have already eaten" "I haven't", wherein the speaker converts the characters to "ma" and "ne", and thus the resulting sub-sentence results are: "Have you eaten?" "I have already eaten" "I haven't".

[0048] The following detailed description is provided for the specific implementation of determining the speaker conversion point character from the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment in step (2) above. In one implementation, for each second time period, the recognized character in the speech recognition text corresponding to the first time period including the end time of the second time period can be determined as the speaker conversion point character.

[0049] In another implementation, the speaker conversion point character can be determined by the following steps (21) to (25):

[0050] (21) inputting the speech recognition text into a pre-trained semantic model to obtain the probability that each recognized character in the speech recognition text belongs to a semantic punctuation point, wherein the greater the probability that the recognized character belongs to the semantic punctuation point, the more likely the recognized character is a semantic punctuation point.

[0051] (22) for each second time period, extending the end time or the start time of the second time period forward and backward to obtain the speaker conversion interval corresponding to the second time period.

[0052] In one implementation, the end time of each second time period is extended forward and backward respectively. Specifically, for each second time period, the end time of the second time period is extended forward by N ms and backward by M ms to obtain the speaker conversion interval [end_time-N, end_time+M] corresponding to the second time period, wherein end_time is the end time of the second time period.

[0053] In another implementation, the start time of each second time period is extended forward and backward respectively. Specifically, for each second time period, the start time of the second time period is extended forward by N ms and backward by M ms to obtain the speaker conversion interval [start_time-N, start_time+M] corresponding to the second time period, wherein start_time is the start time of the second time period.

[0054] It should be noted that M and N can be equal or not equal, and the present disclosure does not make specific limitations.

[0055] (23) determining the character in each recognized character corresponding to the preset time located in the speaker conversion interval corresponding to the second time period as the conversion point candidate character.

[0056] In the present disclosure, the preset time is one of the start time of the first time period and the end time of the first time period. In an implementation manner, a character in each recognized character, for which the start time of the corresponding first time period is located in the speaker transition interval corresponding to the second time period, can be determined as a transition point candidate character.

[0057] In another implementation manner, a character in each recognized character, for which the end time of the corresponding first time period is located in the speaker transition interval corresponding to the second time period, can be determined as a transition point candidate character.

[0058] (24) determining the pause duration of each transition point candidate character corresponding to the second time period.

[0059] In the present disclosure, the pause duration of the transition point candidate character is equal to the time interval between the start time of the recognized character after the transition point candidate character and adjacent to the transition point candidate character and the end time of the transition point candidate character in the speech recognition text.

[0060] (25) determining a speaker transition point character from the transition point candidate characters corresponding to the second time period according to the pause duration of each transition point candidate character corresponding to the second time period and the probability that the transition point candidate character belongs to a semantic punctuation point.

[0061] Specifically, a weighted sum of the pause duration of the transition point candidate character and the probability that the transition point candidate character belongs to a semantic punctuation point can be determined as the probability that the transition point candidate character belongs to a speaker transition point for each transition point candidate character corresponding to the second time period; and then, a transition point candidate character with the largest probability that the transition point candidate character belongs to a speaker transition point from the transition point candidate characters corresponding to the second time period is determined as the speaker transition point character.

[0062] Figure 2 is a flowchart of a sentence splitting method according to another exemplary embodiment. As shown in Figure 2 The above method further includes the following S105.

[0063] In S105, for each sentence in the sentence splitting result, the sentence is cut according to semantics to obtain a plurality of clauses.

[0064] In this way, the speech recognition text can be accurately split according to speaker and semantic information.

[0065] Figure 3 is a flowchart of a sentence splitting method according to another exemplary embodiment. As shown in Figure 3 The above method further includes the following S106.

[0066] In S106, a subtitle text corresponding to the target audio data is generated according to the plurality of clauses.

[0067] Since each clause is obtained by accurately segmenting the speech recognition text according to the speaker and semantic information, the speaker conversion position can be reasonably and effectively segmented, so that the situation that a single subtitle contains speech content of different speakers is avoided, and the user experience is improved.

[0068] Figure 4 is a block diagram of a segmenting device according to an example embodiment. As shown, the device 400 includes: Figure 4

[0069] The acquisition module 401 is configured to acquire target audio data.

[0070] The extraction module 402 is configured to extract speech recognition text corresponding to the target audio data acquired by the acquisition module 401 and a first time period corresponding to each recognized character in the speech recognition text in the target audio data.

[0071] The first segmentation module 403 is configured to perform speaker segmentation on the target audio data acquired by the acquisition module 401 to obtain a second time period corresponding to each speech segment in the target audio data.

[0072] The second segmentation module 404 is configured to perform speaker segmentation on the speech recognition text according to the first time period corresponding to each recognized character extracted by the extraction module 402 and the second time period corresponding to each speech segment obtained by the first segmentation module 403 to obtain a segmenting result.

[0073] In the above technical solution, the target audio data is acquired, the speech recognition text corresponding to the target audio data and the first time period corresponding to each recognized character in the speech recognition text in the target audio data are extracted, and the target audio data is simultaneously subjected to speaker segmentation to obtain the second time period corresponding to each speech segment in the target audio data. Then, the speech recognition text is subjected to speaker segmentation according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment to obtain a segmenting result. Thus, the speaker time period information and the time period corresponding to each character in the speech recognition text in the target audio data can be effectively utilized to segment the speech recognition text according to the speaker, to reasonably and effectively segment the speaker conversion position, to avoid the situation that a single segment contains speech content of multiple speakers, and to improve the segmenting effect.

[0074] ​In an implementation, the second segmentation module 403 is configured to perform speaker segmentation on the target audio data by receiving manually input segmentation labels, so as to determine the start time and the end time of each speech segment in the target audio data, i.e., the second time period corresponding to each speech segment in the target audio data.

[0075] In another implementation, the second segmentation module 403 includes:

[0076] a first determination sub-module configured to determine, from the speech recognition text, a speaker transition point character according to the first time period corresponding to each of the recognized characters and the second time period corresponding to each of the speech segments;

[0077] a segmentation sub-module configured to perform speaker segmentation on the speech recognition text by taking the speaker transition point character as a segmentation basis.

[0078] In this way, the speech segments of different speakers can be automatically segmented, which is convenient and fast, thereby improving the efficiency of subsequent sentence segmentation of the speech recognition text.

[0079] Optionally, the first determination sub-module includes:

[0080] a second determination sub-module configured to input the speech recognition text into a pre-trained semantic model to obtain a probability that each recognized character in the speech recognition text belongs to a semantic sentence break point;

[0081] an extension sub-module configured to, for each second time period, extend the end time or the start time of the second time period to obtain a speaker transition interval corresponding to the second time period; a third determination sub-module configured to determine, as a transition point candidate character, a character in each recognized character corresponding to a preset time located in the speaker transition interval corresponding to the second time period, wherein the preset time is one of the start time of the first time period and the end time of the first time period; a fourth determination sub-module configured to determine a pause duration of each transition point candidate character corresponding to the second time period; and a fifth determination sub-module configured to determine, from the transition point candidate character corresponding to the second time period, a speaker transition point character according to the pause duration of each transition point candidate character corresponding to the second time period and the probability that the transition point candidate character belongs to a semantic sentence break point.

[0082] Optionally, the fifth determination sub-module includes:

[0083] a sixth determination sub-module configured to, for each transition point candidate character corresponding to the second time period, determine, as a probability that the transition point candidate character belongs to a speaker transition point, a weighted sum of the pause duration of the transition point candidate character and the probability that the transition point candidate character belongs to a semantic sentence break point.

[0084] The seventh determining sub-module is configured to determine, as the speaker conversion point character, a conversion point candidate character that has the largest probability of belonging to a speaker conversion point among the conversion point candidate characters corresponding to the second time period.

[0085] Optionally, the first segmenting module 402 is configured to input the target audio data into a pre-trained speaker recognition model to perform speaker segmentation on the target audio data, so as to obtain a second time period corresponding to each speech segment in the target audio data.

[0086] Optionally, the apparatus 400 further includes:

[0087] The third segmenting module is configured to, for each sentence in the sentence result, perform semantic segmentation on the sentence to obtain a plurality of clauses.

[0088] Optionally, the apparatus 400 further includes:

[0089] The generating module is configured to generate a subtitle text corresponding to the target audio data according to the plurality of clauses.

[0090] The present disclosure also provides a computer readable medium having stored thereon a computer program, which, when executed by a processing apparatus, implements the steps of the above-mentioned sentence segmentation method.

[0091] Reference is made below to Figure 5 which shows a structural schematic diagram of an electronic device (e.g., a terminal device or a server) 500 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle-mounted terminal (e.g., a vehicle-mounted navigation terminal), and the like, as well as a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0092] As shown in Figure 5 , the electronic device 500 can include a processing apparatus (e.g., a central processing unit, a graphics processing unit, or the like) 501, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or loaded into a random access memory (RAM) 503 from a storage apparatus 508. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing apparatus 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0093] In general, the following devices can be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 508 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 509. The communication devices 509 can allow the electronic device 500 to communicate wirelessly or wired with other devices to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it is understood that all of the shown devices are not required to be implemented or present. More or less devices can alternatively be implemented or present.

[0094] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 509, or installed from the storage devices 508, or installed from the ROM 502. When the computer program is executed by the processing devices 501, the above-described functions defined in the methods of the embodiments of the present disclosure are performed.

[0095] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, optical fiber, RF, etc., or any suitable combination of the above.

[0096] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0097] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and is not assembled into the electronic device.

[0098] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: acquire target audio data; extract speech recognition text corresponding to the target audio data and a first time period corresponding to each recognized character in the speech recognition text in the target audio data; perform speaker segmentation on the target audio data to obtain a second time period corresponding to each speech segment in the target audio data; and perform speaker segmentation on the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment, to obtain a segmentation result.

[0099] Computer program code for carrying out operations of the present disclosure can be written in any of one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0100] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0101] The modules described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself, for example, the acquisition module can also be described as a "module for acquiring target audio data".

[0102] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0103] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media can include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0104] According to one or more embodiments of the present disclosure, example 1 provides a sentence segmentation method, comprising: acquiring target audio data; extracting speech recognition text corresponding to the target audio data and a first time period corresponding to each recognized character in the speech recognition text in the target audio data; performing speaker segmentation on the target audio data to obtain a second time period corresponding to each speech segment in the target audio data; and performing speaker segmentation on the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment to obtain a sentence segmentation result.

[0105] According to one or more embodiments of the present disclosure, example 2 provides the method of example 1, wherein the speaker segmentation of the speech recognition text according to the first time period corresponding to each of the recognized characters and the second time period corresponding to each of the speaking segments comprises: determining a speaker transition point character from the speech recognition text according to the first time period corresponding to each of the recognized characters and the second time period corresponding to each of the speaking segments; and performing speaker segmentation on the speech recognition text according to the speaker transition point character.

[0106] According to one or more embodiments of the present disclosure, example 3 provides the method of example 2, wherein the determining a speaker transition point character from the speech recognition text according to the first time period corresponding to each of the recognized characters and the second time period corresponding to each of the speaking segments comprises: inputting the speech recognition text into a pre-trained semantic model to obtain a probability that each recognized character in the speech recognition text belongs to a semantic punctuation point; extending the end time or the start time of each second time period before and after to obtain a speaker transition interval corresponding to the second time period; determining a character in each recognized character corresponding to a preset time located in the speaker transition interval corresponding to the second time period as a transition point candidate character, wherein the preset time is one of the start time of the first time period and the end time of the first time period; determining a pause duration of each transition point candidate character corresponding to the second time period; and determining a speaker transition point character from the transition point candidate characters corresponding to the second time period according to the pause duration of each transition point candidate character corresponding to the second time period and the probability that the transition point candidate character belongs to a semantic punctuation point.

[0107] According to one or more embodiments of the present disclosure, example 4 provides the method of example 3, wherein the determining a speaker transition point character from the transition point candidate characters corresponding to the second time period according to the pause duration of each transition point candidate character corresponding to the second time period and the probability that the transition point candidate character belongs to a semantic punctuation point comprises: determining a weighted sum of the pause duration of each transition point candidate character corresponding to the second time period and the probability that the transition point candidate character belongs to a semantic punctuation point as a probability that the transition point candidate character belongs to a speaker transition point for each transition point candidate character corresponding to the second time period; and determining a transition point candidate character with the maximum probability that the transition point candidate character belongs to a speaker transition point from the transition point candidate characters corresponding to the second time period as the speaker transition point character.

[0108] According to one or more embodiments of the present disclosure, example 5 provides the method of example 1, wherein the speaker segmentation of the target audio data to obtain a second time period corresponding to each speaking segment in the target audio data comprises:

[0109] input the target audio data into a pre-trained speaker recognition model to perform speaker segmentation on the target audio data to obtain a second time period corresponding to each speech segment in the target audio data.

[0110] According to one or more embodiments of the present disclosure, example 6 provides the method of any one of examples 1-5, further comprising: for each sentence in the sentence segmentation result, performing semantic segmentation on the sentence to obtain a plurality of clauses.

[0111] According to one or more embodiments of the present disclosure, example 7 provides the method of example 6, further comprising: generating a subtitle text corresponding to the target audio data according to the plurality of clauses.

[0112] According to one or more embodiments of the present disclosure, example 8 provides a sentence segmentation device, comprising: an acquisition module configured to acquire target audio data; an extraction module configured to extract a speech recognition text corresponding to the target audio data acquired by the acquisition module and a first time period corresponding to each recognized character in the speech recognition text in the target audio data; a first segmentation module configured to perform speaker segmentation on the target audio data acquired by the acquisition module to obtain a second time period corresponding to each speech segment in the target audio data; and a second segmentation module configured to perform speaker segmentation on the speech recognition text according to the first time period corresponding to each recognized character extracted by the extraction module and the second time period corresponding to each speech segment obtained by the first segmentation module to obtain a sentence segmentation result.

[0113] According to one or more embodiments of the present disclosure, example 9 provides the device of example 8, wherein the second segmentation module comprises: a first determination submodule configured to determine a speaker transition point character from the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment; and a segmentation submodule configured to perform speaker segmentation on the speech recognition text according to the speaker transition point character as a segmentation basis.

[0114] According to one or more embodiments of the present disclosure, example 10 provides the apparatus of example 9, wherein the first determining sub-module comprises: a second determining sub-module configured to input the speech recognition text into a pre-trained semantic model to obtain a probability that each recognized character in the speech recognition text belongs to a semantic punctuation point; an extension sub-module configured to, for each second time period, extend the end time or the start time of the second time period to obtain a speaker transition interval corresponding to the second time period; a third determining sub-module configured to determine, as a transition point candidate character, a character in each recognized character corresponding to a preset time located in the speaker transition interval corresponding to the second time period, wherein the preset time is one of the start time of the first time period and the end time of the first time period; a fourth determining sub-module configured to determine a pause duration of each transition point candidate character corresponding to the second time period; and a fifth determining sub-module configured to determine, from the transition point candidate characters corresponding to the second time period, a speaker transition point character according to the pause duration of each transition point candidate character corresponding to the second time period and the probability that the transition point candidate character belongs to a semantic punctuation point.

[0115] According to one or more embodiments of the present disclosure, example 11 provides the apparatus of example 10, wherein the fifth determining sub-module comprises: a sixth determining sub-module configured to, for each transition point candidate character corresponding to the second time period, determine a weighted sum of the pause duration of the transition point candidate character and the probability that the transition point candidate character belongs to a semantic punctuation point as a probability that the transition point candidate character belongs to a speaker transition point; and a seventh determining sub-module configured to determine, as the speaker transition point character, a transition point candidate character in the transition point candidate characters corresponding to the second time period that has the greatest probability of belonging to a speaker transition point.

[0116] According to one or more embodiments of the present disclosure, example 12 provides the apparatus of example 8, wherein the first segmentation module is configured to input the target audio data into a pre-trained speaker recognition model to perform speaker segmentation on the target audio data to obtain a second time period corresponding to each speaker segment in the target audio data.

[0117] According to one or more embodiments of the present disclosure, example 13 provides the apparatus of any one of examples 8-12, further comprising: a third segmentation module configured to, for each sentence in the sentence result, perform semantic segmentation on the sentence to obtain a plurality of clauses.

[0118] According to one or more embodiments of the present disclosure, example 14 provides the apparatus of example 13, further comprising: a generation module configured to generate a subtitle text corresponding to the target audio data according to the plurality of clauses.

[0119] According to one or more embodiments of the present disclosure, example 15 provides a computer readable medium having stored thereon a computer program which, when executed by a processing apparatus, implements the steps of the method of any one of examples 1-7.

[0120] According to one or more embodiments of the present disclosure, example 16 provides an electronic device comprising: a storage device having stored thereon one or more computer programs; and one or more processing apparatuses configured to execute the one or more computer programs stored in the storage device to implement the steps of the method of any one of examples 1-7.

[0121] The above description merely provides preferred embodiments of the present disclosure and a principle of applied technology. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0122] In addition, although each operation is depicted in a particular order, this should not be understood as requiring the operations to be performed in the particular order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination.

[0123] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With respect to the devices in the above-described embodiments, the specific manner in which the various modules perform operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

1. A method of sentence segmentation, characterized by, The method comprises: obtaining target audio data; extracting speech recognition text corresponding to the target audio data and a first time period corresponding to each recognized character in the speech recognition text; performing speaker segmentation on the target audio data to obtain a second time period corresponding to each speech segment in the target audio data; determining a speaker transition point character from the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment; performing speaker segmentation on the speech recognition text according to the speaker transition point character to obtain a segmentation result; wherein the determining of the speaker transition point character from the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speech segment comprises: inputting the speech recognition text into a pre-trained semantic model to obtain a probability that each recognized character in the speech recognition text belongs to a semantic segmentation point; for each second time period, extending the end time or the start time of the second time period to obtain a speaker transition interval corresponding to the second time period; determining a transition point candidate character in each recognized character, wherein the corresponding preset time is located in the speaker transition interval corresponding to the second time period, wherein the preset time is one of the start time of the first time period and the end time of the first time period; determining a pause duration of each transition point candidate character corresponding to the second time period; determining a speaker transition point character from the transition point candidate characters corresponding to the second time period according to the pause duration of each transition point candidate character corresponding to the second time period and the probability that the transition point candidate character belongs to a semantic segmentation point.

2. The method of claim 1, wherein, The determining of the speaker transition point character from the transition point candidate characters corresponding to the second time period according to the pause duration of each transition point candidate character corresponding to the second time period and the probability that the transition point candidate character belongs to a semantic segmentation point comprises: for each transition point candidate character corresponding to the second time period, determining a weighted sum of the pause duration of the transition point candidate character and the probability that the transition point candidate character belongs to a semantic segmentation point as a probability that the transition point candidate character belongs to a speaker transition point; determining a transition point candidate character corresponding to the second time period as a speaker transition point character, wherein the transition point candidate character has the maximum probability that the transition point candidate character belongs to a speaker transition point.

3. The method of claim 1, wherein, The performing of the speaker segmentation on the target audio data to obtain the second time period corresponding to each speech segment in the target audio data comprises: inputting the target audio data into a pre-trained speaker recognition model to perform speaker segmentation on the target audio data to obtain the second time period corresponding to each speech segment in the target audio data.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: performing semantic segmentation on each segment in the segmentation result to obtain a plurality of clauses.

5. The method of claim 4, wherein, The method further comprises: generating a subtitle text corresponding to the target audio data according to the plurality of clauses.

6. A punctuation device characterized by The method comprises: an obtaining module configured to obtain target audio data; extracting a speech recognition text corresponding to the target audio data obtained by the obtaining module and a first time period corresponding to each recognized character in the speech recognition text in the target audio data; a first segmentation module configured to perform speaker segmentation on the target audio data obtained by the obtaining module to obtain a second time period corresponding to each speaker segment in the target audio data; a second segmentation module configured to perform speaker segmentation on the speech recognition text according to the first time period corresponding to each recognized character extracted by the extracting module and the second time period corresponding to each speaker segment obtained by the first segmentation module to obtain a segmentation result; The second segmentation module comprises: a first determining sub-module configured to determine a speaker transition point character from the speech recognition text according to the first time period corresponding to each recognized character and the second time period corresponding to each speaker segment; a segmentation sub-module configured to perform speaker segmentation on the speech recognition text according to the speaker transition point character; The first determining sub-module comprises: a second determining sub-module configured to input the speech recognition text into a pre-trained semantic model to obtain a probability that each recognized character in the speech recognition text belongs to a semantic punctuation point; an extension sub-module configured to perform forward and backward extension on an end time or a start time of each second time period to obtain a speaker transition interval corresponding to the second time period; a third determining sub-module configured to determine a transition point candidate character from each recognized character, wherein a preset time corresponding to the recognized character is located in the speaker transition interval corresponding to the second time period, and the preset time is one of a start time of the first time period and an end time of the first time period; a fourth determining sub-module configured to determine a pause duration of each transition point candidate character corresponding to the second time period; and a fifth determining sub-module configured to determine a speaker transition point character from each transition point candidate character corresponding to the second time period according to the pause duration of each transition point candidate character corresponding to the second time period and the probability that the transition point candidate character belongs to a semantic punctuation point.

7. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processing device to implement the steps of the method of any one of claims 1-5.

8. An electronic device, comprising: comprise: a storage device having one or more computer programs stored thereon; one or more processing devices configured to execute the one or more computer programs in the storage device to implement the steps of the method of any one of claims 1-5.

Images