Audio processing method, device, electronic equipment and medium

Abstract

The embodiments of the present application provide an audio processing method, device, electronic device and medium, which relate to the fields of cloud technology and artificial intelligence technology. The method includes: obtaining the audio to be processed; determining at least one initial segmentation time point of the audio to be processed; determining at least one silent segment corresponding to each of the initial segmentation time points in the audio to be processed; determining the target segmentation time point corresponding to each of the initial segmentation time points according to at least one silent segment corresponding to each of the initial segmentation time points; obtaining each audio segment of the audio to be processed according to each of the target segmentation time points; performing speech recognition on each of the audio segments, and obtaining text corresponding to the target language based on the speech recognition results of each of the audio segments. The embodiments of the present application improve the accuracy of speech recognition results.

Description

Technical Field

[0001] The present application relates to the fields of cloud technology and artificial intelligence technology. Specifically, the present application relates to an audio processing method, device, electronic device and medium. Background Art

[0002] Most media information is recorded, and the audio information in the media information needs to be processed into text information for forming a record file or making subtitles, etc. In the existing technology, when processing the audio information in the media information, the audio information in the media information is first divided into several balanced segments, and then each segment of the audio information is processed separately, which will lead to the problem of inaccurate content obtained by processing. Summary of the invention

[0003] The present application provides an audio processing method, device, electronic device and medium that can effectively improve the accuracy of audio recognition.

[0004] In a first aspect, an embodiment of the present application provides an audio processing method, the method comprising:

[0005] Get the audio to be processed;

[0006] Determine at least one initial segmentation time point of the audio to be processed;

[0007] Determine at least one silent segment corresponding to each of the initial segment time points in the audio to be processed;

[0008] Determine the target segmentation time point corresponding to each initial segmentation time point according to at least one silent segment corresponding to each initial segmentation time point;

[0009] According to each target segmentation time point, each audio segment of the audio to be processed is obtained;

[0010] Speech recognition is performed on each audio segment, and a text corresponding to the target language is obtained based on the speech recognition result of each audio segment.

[0011] In a second aspect, an embodiment of the present application provides an audio processing device, the device comprising:

[0012] An audio acquisition module, used to acquire audio to be processed;

[0013] A segmentation point determination module is used to determine at least one initial segmentation time point of the audio to be processed, determine at least one silent segment corresponding to each initial segmentation time point in the audio to be processed, and determine a target segmentation time point corresponding to each initial segmentation time point according to the at least one silent segment corresponding to each initial segmentation time point;

[0014] An audio segmentation module, used to obtain each audio segment of the audio to be processed according to each target segmentation time point;

[0015] The audio recognition module is used to perform speech recognition on each audio segment and obtain text corresponding to the target language based on the speech recognition result of each audio segment.

[0016] In another possible implementation of the embodiment of the present application, the audio recognition module is specifically used to:

[0017] Speech recognition is performed on each audio segment in parallel through multi-threading, and a text corresponding to the target language is obtained based on the speech recognition result of each audio segment.

[0018] Another possible implementation method of the embodiment of the present application is that, for any initial segmentation time point, at least one silent segment corresponding to the initial segmentation time point includes a set number of silent segments adjacent to the initial segmentation time point in the audio to be processed, or a silent segment within a preset time length in the audio to be processed with the initial segmentation time point as the base time point.

[0019] In another possible implementation of the embodiment of the present application, the segmentation point determination module is specifically used to determine the target segmentation time point corresponding to each initial segmentation time point according to at least one silent segment corresponding to each initial segmentation time point:

[0020] For any initial segmentation time point, the target segmentation time point corresponding to the initial segmentation time point is determined according to the longest silent segment in at least one silent segment corresponding to the initial segmentation time point.

[0021] In another possible implementation of the embodiment of the present application, the segmentation point determination module is specifically used to:

[0022] Determine at least one initial segmentation time point of the audio to be processed according to at least one of the following information:

[0023] Speech recognition speed, number of segments, and preset segment duration.

[0024] In another possible implementation of the embodiment of the present application, each initial segmentation time point is an average time point of the audio to be processed;

[0025] The segmentation point determination module is also used to determine the duration of the initial segment corresponding to each initial segmentation time point; if the duration of the initial segment is less than the preset duration, each initial segmentation time point is adjusted according to the preset duration.

[0026] In another possible implementation of the embodiment of the present application, the audio processing device further includes a text display module, which is used to display the text to the user through the client.

[0027] Another possible implementation of the embodiment of the present application is when the audio to be processed is the audio in the video;

[0028] The text display module is specifically used to display the text to the user in the form of subtitles synchronously with the video screen through the client.

[0029] In another possible implementation of the embodiment of the present application, the text display module is further used to:

[0030] When a modification operation of the user on the displayed text is obtained, the displayed text is modified accordingly according to the modification operation, and the modified text is displayed to the user.

[0031] In another possible implementation of the embodiment of the present application, the audio acquisition module is specifically used to:

[0032] Get the initial audio data to be processed;

[0033] Extracting audio track data of a target audio track from the initial audio data;

[0034] The audio track data of the target audio track is determined as audio to be processed.

[0035] In another possible implementation of the embodiment of the present application, when the audio acquisition module extracts the track data of the target track in the initial audio data, it is specifically used to:

[0036] Extracting the track data of the designated track from the track data of each track included in the initial audio data, and determining the track data of the designated track as the track data of the target track;

[0037] or,

[0038] The track data of each track in the initial audio data is extracted, and the track data of each track is displayed to the user, and the track data of the target track is determined according to the user's selection operation on the track data of each track.

[0039] Another possible implementation of the embodiment of the present application also includes:

[0040] A task generation module is used to generate speech recognition tasks using each audio clip and determine the computing power required for the speech recognition task;

[0041] The audio recognition module is specifically used for:

[0042] When the remaining computing power of the audio processing device is not less than the computing power required for the speech recognition task, speech recognition is performed on each audio segment, and a text corresponding to the target language is obtained based on the speech recognition results of each audio segment.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0044] one or more processors;

[0045] Memory;

[0046] One or more applications, wherein the one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: perform operations corresponding to the audio processing method shown in the first aspect.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio processing method shown in the first aspect.

[0048] The beneficial effects of the technical solution provided by this application are:

[0049] The present application provides an audio processing method, device, electronic device and medium. Compared with the prior art, the present application first determines each target segmentation time point according to at least one silent segment corresponding to each initial segmentation time point of the audio, then obtains each audio segment according to each target segmentation time point, and performs speech recognition on each audio segment. Since each silent segment in the audio is usually a sign of the end of a sentence and / or the beginning of the next sentence, each initial segmentation time point is corrected according to at least one silent segment corresponding to each initial segmentation time point to obtain each target segmentation time point. Based on this method, the target segmentation time point can be located at the position of the silent segment, so that each sentence in each audio segment of the audio to be processed obtained according to each target segmentation time point can be guaranteed to be a complete sentence, and then the speech recognition result of each audio segment can be guaranteed to be a complete speech recognition result of each sentence, thereby improving the accuracy of the speech recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0051] Figure 1 A flowchart of an audio processing method provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of a display interface for setting a transcription speed provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a display interface for uploading audio and video provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a display interface for audio and video transcription provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of a process for obtaining audio to be processed provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a process for segmenting audio provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of a flow chart of computing power determination provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of a multi-threaded processing flow provided for the implementation of this application;

[0059] Fig. 9 A schematic diagram of a display interface for text display provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of a display interface for exporting a file provided in an embodiment of the present application;

[0061] Fig.11 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application;

[0062] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0063] Fig.13 A schematic diagram of the relationship between an initial segmentation time point and a silent segment provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0065] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0066] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0067] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] In an embodiment of the present application, after the server obtains the audio to be processed and processes the audio to be processed, it can store the audio to be processed and the information obtained after the processing in a database, which can be a database based on cloud technology.

[0069] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0070] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, image websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0071] Among them, the database can be simply regarded as an electronic filing cabinet - a place to store electronic files, where users can add, query, update, delete, etc. The so-called "database" is a collection of data stored together in a certain way, which can be shared with multiple users, has as little redundancy as possible, and is independent of the application.

[0072] A database management system (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, retrieval, security, and backup. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language used, such as SQL (Structured Query Language), XQuery; or according to performance focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages ​​at the same time.

[0073] In the embodiment of the present application, the speech technology in artificial intelligence technology can be used to perform speech recognition on the audio (or audio segment) to be processed. The speech translation technology in artificial intelligence technology can be used to perform speech recognition and translation processing on the audio (or audio segment) to be processed.

[0074] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0075] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0076] The key technologies of speech technology include automatic speech recognition technology, speech synthesis technology and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the development direction of human-computer interaction in the future, among which speech has become one of the most promising human-computer interaction methods in the future.

[0077] In addition, the calculation of data in each optional solution provided in the embodiment of the present application can be performed in the form of cloud computing. Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0078] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices. According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0079] Cloud computing is the product of the development and integration of traditional computer and network technologies, including grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0080] With the development of the Internet, real-time data streams, and the diversification of connected devices, as well as the demand for search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Different from the previous parallel distributed computing, the emergence of cloud computing will promote revolutionary changes in the entire Internet model and enterprise management model from a conceptual perspective.

[0081] An audio processing method provided in an embodiment of the present application can be executed by any electronic device, such as a mobile phone, a desktop computer, a laptop computer, a tablet computer and other terminal devices, or by a server, wherein the server can be a physical server or a cloud server, and the server can be an independent server or a server cluster. As an optional solution, the audio processing method provided in an embodiment of the present application can be executed by a server. For the convenience of description, the following embodiment description will take the server as the execution subject as an example for explanation.

[0082] like Figure 1 As shown, Figure 1 A flowchart of an audio processing method provided in an embodiment of the present application is provided. The method may include steps S101 to S106, which are specifically as follows:

[0083] Step S101, obtaining audio to be processed.

[0084] In the embodiment of the present application, the source of the audio to be processed is not limited in the embodiment of the present application. For example, the audio to be processed can be a pure audio file or audio extracted from a video file, or it can be an audio file uploaded by a user or audio obtained through a recording file. The audio file or video file can be a local file or a file downloaded from the network, or it can be a file obtained from other devices.

[0085] The user can upload the audio file or video file that he wants to process through the client. After receiving the uploaded audio file or video file, the client transmits the received audio file or video file to the server. The server can use the received audio file as the audio to be processed, or the server can extract the audio to be processed from the received audio file or video file.

[0086] Step S102: determining at least one initial segmentation time point of the audio to be processed.

[0087] Among them, the specific method of determining the initial segmentation time point is not limited in the embodiment of the present application, and can be determined by using any existing audio segmentation technology, such as the initial segmentation time points of the audio to be processed determined according to a set duration, a set number of segments or other segmentation strategies, and the duration of the audio segments corresponding to each initial segmentation time point can be the same or different.

[0088] Optionally, the initial segmentation time points of the audio to be processed can be determined by evenly dividing the audio to be processed according to the duration to obtain several initial segmentation time points. For example, the audio to be processed can be evenly divided into five segments according to the duration, and the initial segmentation time points are 4 time points.

[0089] Step S103: determining at least one silent segment corresponding to each initial segment time point in the audio to be processed.

[0090] In the embodiment of the present application, the various silent segments can be detected from the audio to be processed by using the Voice Activity Detection (VAD) technology. The VAD technology can be used to detect whether there is a voice signal in the audio, and identify the voice part and the non-voice part (in the embodiment of the present application, the silent segment) from the audio.

[0091] After identifying each silent segment from the audio, for any initial segmentation time point, the silent segment corresponding to the initial segmentation time point can be determined. Specifically, the silent segment corresponding to the initial segmentation time point refers to the silent segment near the reference point with the time point as the reference point, such as the silent segment closest to the reference point in time, or the silent segment within a certain time range from the reference point. The silent segment can be the silent segment before the reference point in the audio to be processed, or the silent segment after the reference point, or both before and after. The specific confirmation method of the silent segment can be configured according to the actual application requirements, and is not limited to the embodiments of the present application, and can include but is not limited to the methods listed above.

[0092] like Fig.13 As shown, Fig.13 A schematic diagram of the relationship between an initial segmentation time point and a silent segment is provided for an optional embodiment of the present application. In this embodiment, the silent segment corresponding to the initial segmentation time point includes: taking the time point as a reference point, the previous silent segment adjacent to the reference point, and the next two silent segments adjacent to the reference point, then Fig.13 The silent segments corresponding to an initial segmentation time point shown in are silent segments 2-4, and silent segment 1 is not the silent segment corresponding to the initial segmentation time point.

[0093] Step S104: determining a target segmentation time point corresponding to each initial segmentation time point according to at least one silent segment corresponding to each initial segmentation time point.

[0094] Normally, each silent segment in the audio is a sign of the end of a sentence and / or the beginning of the next sentence. Therefore, each initial segment time point is corrected according to at least one silent segment corresponding to each initial segment time point to obtain each target segment time point, so that the target segment time point can be located at the position of the silent segment, that is, the target segment time point is located at the end of a sentence and / or the beginning of the next sentence, thereby ensuring that each sentence in each audio segment of the audio to be processed obtained according to each target segment time point is a complete sentence. When speech recognition is performed on each audio segment subsequently, it can be ensured that the speech recognition result of each audio segment is a complete speech recognition result of each sentence, thereby improving the accuracy of the speech recognition result.

[0095] In the embodiment of the present application, for each initial segmentation time point, the initial segmentation time point corresponds to at least one silent segment. When there are at least two silent segments corresponding to an initial segmentation time point, a target silent segment can be selected from the at least two silent segments, and the target segmentation time point corresponding to the initial segmentation time point can be determined based on the target silent segment. The specific method of determining the target silent segment from at least two silent segments is not limited in the embodiment of the present application, such as selecting one of the at least two silent segments, selecting the silent segment with the longest duration, or selecting the silent segment closest to the reference point.

[0096] As an optional scheme, corresponding to any initial segmentation time point, if there are 0 silent segments corresponding to the initial segmentation time point, such as there are no silent segments near the initial segmentation time point, or there are no silent segments in the audio segment formed by the initial segmentation time point and its adjacent initial segmentation time points, the initial segmentation time point can be deleted.

[0097] For example, the audio to be processed is divided into five segments according to the duration, and each initial segment time point is 4 time points. If no silent segment is found near the first time point, the first time point is deleted. At this time, each initial segment time point is 3 time points, that is, the audio to be processed is divided into four segments according to the duration.

[0098] As an optional scheme, when determining the target segmentation time point corresponding to each initial segmentation time point, the duration between any two target segmentation time points can be determined. If the duration is greater than the duration threshold, at least one silent segment can be selected from each silent segment between the two target segmentation time points. It should be noted that the duration between any selected silent segment and any target segmentation time point is not less than the preset segmentation duration, and / or the duration between any two selected silent segments is not less than the preset segmentation duration. According to the at least one selected silent segment, at least one target segmentation time point is determined, that is, at least one target segmentation time point is added between the two target segmentation time points. Among them, the duration between any two target segmentation time points is not less than the preset segmentation duration.

[0099] For example, the duration between the target segmentation time point 1 and the target segmentation time point 2 is 9 minutes, which is greater than the duration threshold of 6 minutes. In this embodiment, the preset segmentation duration is 4 minutes. Then, a silent segment can be selected between the target segmentation time point 1 and the target segmentation time point 2. The duration between the silent segment and the target segmentation time point 1 is not less than 4 minutes, and the duration between the silent segment and the target segmentation time point 2 is not less than 4 minutes. The target segmentation time point 3 is determined based on the silent segment. In this way, the duration between the target segmentation time point 3 and the target segmentation time point 1, and the duration between the target segmentation time point 3 and the target segmentation time point 2, are both not less than 4 minutes.

[0100] Among them, according to the target silent segment (see the description in the previous text, for any initial segment time period, a target silent segment can be selected from at least two silent segments corresponding to the initial segment time point), when determining the target segment time point corresponding to the initial segment time point, the starting time point of the target silent segment can be determined as the target segment time point corresponding to the initial segment time point, the ending time point of the target silent segment can be determined as the target segment time point corresponding to the initial segment time point, the middle time point of the target silent segment can also be determined as the target segment time point corresponding to the initial segment time point, and the like, without limitation here.

[0101] Step S105, obtaining each audio segment of the audio to be processed according to each target segment time point.

[0102] In an embodiment of the present application, the audio to be processed can be segmented according to each target segmentation time point to obtain each audio segment; or the audio to be processed is not segmented, but the starting time point (or ending time point) and segmentation duration corresponding to each audio segment of the audio to be processed are determined according to each target segmentation time point, or the starting time point and ending time point corresponding to each audio segment of the audio to be processed are determined.

[0103] Step S106: performing speech recognition on each audio segment, and obtaining text corresponding to the target language based on the speech recognition result of each audio segment.

[0104] In the embodiment of the present application, automatic speech recognition technology (Automatic Speech Recognition, ASR) can be used to perform speech recognition on each audio clip. Among them, speech recognition is also called transcription, that is, converting sound into text. When performing speech recognition on each audio clip, the server can send information such as the task status of the transcription task and the progress of the transcription task to the client in real time. When the client receives the task status of the transcription task, the progress of the transcription, etc., it can display the received information. The task status can include multiple stages such as data preparation, transcription, transcription completion, transcription failure, etc.

[0105] In the embodiment of the present application, the speech recognition results of each audio segment can be processed accordingly to obtain text corresponding to the target language. The target language can be one or more. For example, the target language can be Chinese, or Chinese and English.

[0106] Among them, the speech recognition results of each audio segment can be processed in sequence, and then the speech recognition results of each audio segment can be spliced ​​to obtain a text corresponding to the target language; or the speech recognition results of each audio segment can be spliced ​​first, and then the spliced ​​speech recognition results can be processed in sequence, and then the spliced ​​speech recognition results can be processed in sequence, and then the text corresponding to the target language can be obtained.

[0107] For example, the speech recognition results of the audio clips are spliced ​​to obtain Chinese text, and the Chinese text is segmented and translated into English to obtain Chinese and English texts.

[0108] The audio processing method provided in the embodiment of the present application, compared with the prior art, first determines each target segmentation time point according to at least one silent segment corresponding to each initial segmentation time point of the audio, then obtains each audio segment according to each target segmentation time point, and performs speech recognition on each audio segment. Since each silent segment in the audio is usually a sign of the end of a sentence and / or the beginning of the next sentence, therefore, each initial segmentation time point is corrected according to at least one silent segment corresponding to each initial segmentation time point to obtain each target segmentation time point. Based on this method, the target segmentation time point can be located at the position of the silent segment, so that each sentence in each audio segment of the audio to be processed obtained according to each target segmentation time point can be guaranteed to be a complete sentence, and then the speech recognition result of each audio segment can be guaranteed to be a complete speech recognition result of each sentence, thereby improving the accuracy of the speech recognition result and the coherence of the sentence segmentation.

[0109] Another possible implementation of the embodiment of the present application is step S106, that is, performing speech recognition on each audio segment, and obtaining text corresponding to the target language based on the speech recognition result of each audio segment, which may further include step S107.

[0110] Step S107: display the text to the user through the client.

[0111] In an embodiment of the present application, the server can send texts in various target languages ​​to the client, and the client receives the texts in various target languages ​​and displays them to the user, wherein the specific display method is not limited. For example, the display interface of the client may include two display areas, one display area displays Chinese text, and the other display area displays English text, or the display interface of the client includes only one display area, and the display area can synchronously display Chinese text and English text in the form of sentences, as well as the start and end time of each sentence, etc.

[0112] After obtaining the text corresponding to the target language, the server can automatically send the text to the client, which will display it to the user. Alternatively, when the client receives a click to view the text from the user, it can generate a text request instruction and send the text request instruction to the server. After receiving the text request instruction, the server sends the text to the client, which will display it to the user. There is no limitation here.

[0113] If the audio to be processed is audio in a video, in step S107, the text is displayed to the user via the client, which may specifically include: displaying the text to the user in the form of subtitles in synchronization with the video screen via the client.

[0114] Specifically, the client can display the text to the user in the form of subtitles in synchronization with the video screen. When displaying the text, in addition to displaying the text to the user in the form of subtitles in synchronization with the video screen, the client can also display the text in combination with other display methods.

[0115] For example, when the client displays text, on the one hand, it displays the text to the user in the form of subtitles in synchronization with the video screen and displays the timeline at the same time. On the other hand, it displays the text in the form of sentences and displays the start and end time points of each sentence.

[0116] The audio processing method further includes: when a modification operation of the user on the displayed text is obtained, the displayed text is modified accordingly according to the modification operation, and the modified text is displayed.

[0117] In the embodiment of the present application, the user can modify the text displayed by the client. When the client obtains the modification operation, it can send the modification operation to the server, and the server modifies the text accordingly according to the modification operation. Then the server sends the modified text to the client, and the client displays the modified text. The user can modify the words in the displayed text, or adjust the start time point and / or end time point of the sentence in the displayed text to make the subtitles and the picture more synchronized.

[0118] Furthermore, the user can also export the text displayed by the client, or the text in the form of subtitles and the video screen, and store them in the target storage space.

[0119] Another possible implementation of the embodiment of the present application, step S101, obtaining the audio to be processed, may specifically include: obtaining initial audio data to be processed; extracting the track data of the target audio track in the initial audio data; and determining the track data of the target audio track as the audio to be processed.

[0120] In the embodiment of the present application, the initial audio data to be processed may be original audio data uploaded by a user, recorded, or obtained from the network, or may be audio data extracted from a video. When the initial audio data is track data of a single audio track, the initial audio data may be determined as the audio to be processed; when the initial audio data includes track data of at least two audio tracks, the track data of the target audio track in the initial audio data may be extracted; and the track data of the target audio track may be determined as the audio to be processed.

[0121] The method of extracting the audio track data of the target audio track in the initial audio data may specifically include method A1 or method A2.

[0122] Method A1 extracts the track data of a designated track from the track data of each track included in the initial audio data, and determines the track data of the designated track as the track data of the target track.

[0123] When the initial audio data includes track data of at least two tracks, usually the first track mainly records audio data. Therefore, the designated track can be the first track of the at least two tracks. Of course, according to actual conditions, the designated track can also be a track other than the first track of the at least two tracks. This is not limited here. For example, the designated track can be the third track.

[0124] Method A2, extracting the track data of each track in the initial audio data, and displaying the track data of each track to the user, and determining the track data of the target track according to the user's selection operation on the track data of each track.

[0125] When the initial audio data includes track data of at least two tracks, in some special cases, the audio data may be in tracks other than the first track (which may be referred to as other tracks) of the at least two tracks. When it is uncertain in which track of the other tracks the audio data is located, the track data of each of the at least two tracks may be extracted by a multimedia video processing tool, and the track data of each track may be displayed to the user, and the user may select the track data of the target track from the track data of each track.

[0126] The multimedia video processing tool is not specifically limited, for example, it may be Fast Forward Mpeg (a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams. It can easily achieve mutual conversion between multiple video formats), referred to as FFMPEG.

[0127] It should be noted that the multimedia video processing tool can extract the audio track data of each audio track to obtain the format information of each audio track. For any audio track, the format information of the audio track includes file name, duration, bit rate, stream information, encoding format, sampling rate, channel information, etc. Usually, the audio track data of the audio track with the largest bit rate can be used as the audio track data of the target audio track.

[0128] Optionally, multiple audio tracks of the audio to be processed can be extracted using FFMPEG, where the audio information of any audio track can be expressed as follows:

[0129] {

[0130] Input #0, wav, from 'pony3.wav': / / means the audio track comes from the file pony3.wav

[0131] Duration: 00:19:07.43, bitrate: 256kb / s / / indicates that the duration of the audio track is 00:19:07.43 and the bit rate is 256 kilobit per second (kb / s)

[0132] Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 16000 Hz, 1channels, s16, 256 kb / s

[0133] / / Indicates that the audio track is an audio stream, and the encoding format is pcm_s16le ([1][0][0][0] / 0x0001), where pcm stands for pulse code modulation, s16le stands for signed 16-bit little-endian, ([1][0][0][0] / 0x0001) represents the encoding label string and encoding label, the sampling rate is 16000 Hz, mono, the sampling format is s16, and the bit rate is 256kb / s

[0134] "streams": [

[0135] {

[0136] "index": 0, / / indicates the stream number

[0137] "codec_name": "pcm_s16le", / / indicates the encoding name

[0138] "codec_long_name": "PCM signed 16-bit little-endian", / / indicates the full name of the codec

[0139] "codec_type": "audio", / / indicates the encoding type

[0140] "codec_time_base": "1 / 16000", / / indicates the encoding time base

[0141] "codec_tag_string": "[1][0][0][0]", / / indicates the encoding tag string

[0142] "codec_tag": "0x0001", / / indicates the coding tag

[0143] "sample_fmt": "s16", / / indicates the sampling format

[0144] "sample_rate": "16000", / / indicates the sampling rate

[0145] "channels": 1, / / indicates the channel

[0146] "bits_per_sample": 16, / / indicates the number of bits per sample

[0147] "r_frame_rate": "0 / 0", / / indicates the actual frame rate

[0148] "avg_frame_rate": "0 / 0", / / indicates the average frame rate

[0149] "time_base": "1 / 16000", / / indicates the time base

[0150] "duration_ts": 18358881, / / indicates the duration of the ts stream

[0151] "duration": "1147.430063", / / indicates duration

[0152] "bit_rate": "256000" / / indicates the bit rate

[0153] } ]

[0155] }

[0156] Another possible implementation of the embodiment of the present application, step S102, determining at least one initial segmentation time point of the audio to be processed, may specifically include:

[0157] Determine at least one initial segmentation time point of the audio to be processed according to at least one of the following information:

[0158] Speech recognition speed, number of segments, and preset segment duration.

[0159] In an embodiment of the present application, a user may set at least one speech recognition speed, or a user may set the speech recognition speed for each time period, wherein the speech recognition speed is the transcription speed.

[0160] For example, the user can set the speech recognition speed between 8:00-10:00 to 20 times, and can also set the speech recognition speed between 12:00-14:00 to 10 times.

[0161] It should be noted that, for any thread, the maximum speed of speech recognition of the thread is 2, that is, when the user sets the transcription speed to 20, the minimum number of threads is 10.

[0162] The user can set the speech recognition speed independently. After the user sets the speech recognition speed, in step S106, when performing speech recognition and subsequent processing on each audio clip, the processing can be performed according to the set speech recognition speed, that is, the speed of performing speech recognition and subsequent processing on each audio clip depends on the speech recognition speed independently set by the user. Since the maximum speed of performing speech recognition and subsequent processing using one thread is 2, the higher the speed used for performing speech recognition and subsequent processing, the higher the number of threads will be.

[0163] In the embodiment of the present application, the user can also set segmented data, for example, the user can choose to segment into 10 segments.

[0164] In the embodiment of the present application, each initial segmentation time point of the audio to be processed can be determined according to at least one of the speech recognition speed, the number of segments, and the preset segmentation duration. For example, each initial segmentation time point of the audio to be processed can be determined by an even division method based on the number of segments; the minimum number of threads can also be determined according to the speech recognition multiple, and the minimum number of threads can be used as the number of segments to determine each initial segmentation time point of the audio to be processed by an even division method; the preset segmentation duration can also be used to determine each initial segmentation time point of the audio to be processed, so that the duration of each audio segment after the audio to be processed is segmented according to each initial segmentation time point is not less than the preset segmentation duration.

[0165] Wherein, each initial segmentation time point may be an evenly divided time point of the audio to be processed. At this time, before determining at least one silent segment corresponding to each initial segmentation time point in the audio to be processed, the following may also be included:

[0166] Determine the duration of the initial segment corresponding to each initial segment time point; if the duration of the initial segment is less than the preset segment duration, adjust each initial segment time point according to the preset segment duration.

[0167] In the embodiment of the present application, each initial segmentation time point of the audio to be processed can be determined by an average division method according to the speech recognition speed or the number of segments. In this case, each initial segmentation time point can be an average division time point of the audio to be processed. On this basis, the duration of the initial segment corresponding to each initial segmentation time point can be determined.

[0168] Furthermore, if the duration of the initial segment is less than the preset segment duration, each initial segment time point is adjusted according to the preset segment duration, and each adjusted initial segment time point is used as each initial segment time point in step S102, that is, each initial segment time point in step S102 is adjusted. At this time, the duration of the initial segment corresponding to each adjusted initial segment time point is greater than or equal to the preset segment duration; if the duration of the initial segment is not less than the preset segment duration, then each initial segment time point is the initial segment time point in step S102, that is, each initial segment time point in step S102 is before adjustment.

[0169] In an embodiment of the present application, for any initial segmentation time point, at least one silent segment corresponding to the initial segmentation time point includes a set number of silent segments adjacent to the initial segmentation time point in the audio to be processed, or a silent segment within a preset time length in the audio to be processed with the initial segmentation time point as the base time point.

[0170] Among them, the set number of silent segments can be a set number of silent segments before the initial segmentation time point, a set number of silent segments after the initial segmentation time point, a set number of silent segments before and after the initial segmentation time point, etc.

[0171] For example, at least one silent segment corresponding to a certain initial segmentation time point includes five silent segments adjacent to and before the initial segmentation time point in the audio to be processed.

[0172] Among them, the silent segment within the preset duration can be a silent segment within the preset duration before the reference time point, a silent segment within the preset duration after the reference time point, or a silent segment within the preset duration centered on the reference time point, etc.

[0173] Among them, step S104, determining the target segmentation time point corresponding to each of the initial segmentation time points according to at least one silent segment corresponding to each of the initial segmentation time points, may specifically include:

[0174] For any initial segmentation time point, the target segmentation time point corresponding to the initial segmentation time point is determined according to the longest silent segment in at least one silent segment corresponding to the initial segmentation time point.

[0175] In the embodiment of the present application, the voice activity detection (VAD) technology can be used to detect each silent segment from the audio to be processed, and then determine at least one silent segment corresponding to each of the initial segmentation time points in the audio to be processed. Usually, there is a long pause at the end of a sentence, so the segmentation can be performed according to the duration of the silent segment. The longer the duration of the silent segment, the more accurate the sentence segmentation, resulting in more accurate segmentation.

[0176] It should be noted that, for any initial segmentation time point, the head of the longest silent segment can be used as the target segmentation time point corresponding to the initial segmentation time point, the tail of the longest silent segment can be used as the target segmentation time point corresponding to the initial segmentation time point, or the center point of the longest silent segment can be used as the target segmentation time point corresponding to the initial segmentation time point, without limitation here.

[0177] Another possible implementation of the embodiment of the present application, step S106, performs speech recognition on each audio segment, and obtains text corresponding to the target language based on the speech recognition result of each audio segment, which may specifically include:

[0178] Speech recognition is performed on each audio segment in parallel through multi-threading, and a text corresponding to the target language is obtained based on the speech recognition result of each audio segment.

[0179] In the embodiment of the present application, the number of threads is at least two, that is, the multithreading is at least two threads. For any thread, the thread can process at least one audio clip. That is, the number of audio clips is greater than or equal to the number of threads.

[0180] As an optional implementation method, one thread can be used to process one audio clip. Specifically, one thread can be assigned to each audio clip, and each audio clip can be speech recognized in parallel through the thread corresponding to each audio clip, and a text corresponding to the target language can be obtained based on the speech recognition result of each audio clip, which can greatly improve the speed of speech recognition and subsequent processing. For example, if it takes 1 hour to perform speech recognition and subsequent processing on the unsegmented audio to be processed, when the audio to be processed is divided into 10 audio clips, and 10 threads are used to perform speech recognition and subsequent processing on each audio clip at the same time, the processing time can be shortened to about 6 minutes, which greatly reduces the processing time and improves the processing speed.

[0181] Among them, the executor of the audio processing method of the embodiment of the present application is an electronic device, and the method may also include: generating a speech recognition task using each audio segment, and determining the computing power required for the speech recognition task.

[0182] Step S106, performing speech recognition on each audio segment, and obtaining text corresponding to the target language based on the speech recognition result of each audio segment, may specifically include:

[0183] If the remaining computing power of the electronic device is not less than the computing power required for the speech recognition task, speech recognition is performed on each audio segment, and a text corresponding to the target language is obtained based on the speech recognition results of each audio segment.

[0184] In the embodiment of the present application, computing power is a unit of measurement for network processing capabilities. A speech recognition task can be generated using each audio clip, and the computing power required for the speech recognition task can be estimated. When the remaining computing power of the electronic device is not less than the estimated computing power, it means that the electronic device has the ability to process the speech recognition task at one time. At this time, the speech recognition task can be processed at one time, that is, speech recognition and subsequent processing are performed on each audio clip; when the remaining computing power of the electronic device is less than the estimated computing power, it means that the electronic device does not have the ability to process the speech recognition task at one time. The remaining computing power can be used to process part of the speech recognition task first, and then process another part of the speech recognition task. Or, in order to prevent the occurrence of server operation errors, crashes, etc., the speech recognition task can be processed at one time until the remaining computing power of the electronic device is not less than the estimated computing power.

[0185] The audio processing method of the present application is applicable to any speech recognition scenario. In any speech recognition scenario, the audio processing method of the present application can be used to segment the audio to be processed, perform speech recognition and subsequent processing, such as sentence segmentation processing, translation processing, etc. Among them, the audio processing method of the present application can be applied to software with audio processing function, and the software with audio processing function can be an independent audio processing software or a software including audio processing function and other functions, which is not limited here.

[0186] In order to better understand and illustrate the solution provided by the embodiment of the present application, an audio processing method provided by the embodiment of the present application will be described in detail below in combination with an actual application scenario.

[0187] The audio processing method in the embodiment of the present application can be applied to an audio and video transcription platform, which includes a client and a server. The user can log in to the client corresponding to the audio and video transcription platform and set the transcription speed in the client, where the transcription speed is the speech recognition speed. Figure 2 As shown, Figure 2 A schematic diagram of a display interface for setting a transcription speed provided in an embodiment of the present application, in which a user can set at least one time period and a transcription speed corresponding to each time period, wherein any time period includes a start time point and an end time point. Figure 2 The display interface includes a time period consisting of "time point 1-time point 2", and the transcription speed "speed a" corresponding to the time period; a time period consisting of "time point 3-time point 4", and the transcription speed "speed b" corresponding to the time period. Figure 2 The display interface also includes submit controls and cancel controls.

[0188] The user can also upload at least one audio file and / or at least one video file in the client. For each file uploaded by the user, the user can select the language of the file, that is, select the language of the uploaded file. If the user selects Chinese as the language of the audio file, it means that the audio file is a Chinese file. Figure 3 As shown, Figure 3 A schematic diagram of a display interface for uploading audio and video provided in an embodiment of the present application. A user can upload audio and video files in the display interface. When uploading an audio file or a video file, the user can select the language of the currently uploaded audio file or video file, for example, Figure 3 In the display interface, each time a user adds an audio file or a video file, the file size, file adding progress, etc. are displayed. For example, Figure 3The file size of the video file is 2.67 megabytes (MB), and the file adding progress is 100%, indicating that the addition is complete; the file size of the audio file is 15.98MB, and the file adding progress is 100%. In addition, the display interface also includes a submit control and a cancel control.

[0189] Furthermore, after the user uploads a file in the client, the client can send the file to the server, and the server transcribes the file. During the process of the server transcribing the file, the server can send the transcription status of the file to the client in real time, and the client can display the transcription status of the file in real time, such as Figure 4 As shown, Figure 4 A schematic diagram of a display interface for audio and video transcription provided in an embodiment of the present application. In the display interface for audio and video transcription, the file being transcribed and the status of the file can be viewed, for example Figure 4 The transcription status of the video file is 57%, which means that the video file has been transcribed 57%. The display interface can also display other information of the file, such as language, creator, creation time, etc. For example Figure 4 The language of the video file is Chinese, the creator is A, and the creation time is T.

[0190] Among them, the method used by the server to transfer files is as follows: Figure 5-Figure 8 As shown, Figure 5 A schematic diagram of a process for obtaining audio to be processed provided in an embodiment of the present application. Specifically:

[0191] The server can obtain the initial audio data to be processed, which can be an audio file sent by the client, and use the audio file as the audio data, or it can be audio data extracted from an audio file or video file sent by the client; extract the track data of the specified track from the track data of each track contained in the initial audio data, and determine the track data of the specified track as the track data of the target track, or extract the track data of each track in the initial audio data, and display the track data of each track to the user, and determine the track data of the target track according to the user's selection operation for the track data of each track; convert the track data of the target track into track data of the target format, and determine the track data after the conversion format as the audio to be processed; store the audio to be processed in a preset storage space. Among them, the target format can be a pulse code modulation (PCM) format or other formats, and the preset storage space can be a cloud or a local.

[0192] about Figure 5 The specific implementation method of each step can be found in the above description and will not be repeated here.

[0193] After the audio to be processed is determined, Figure 6 As shown, Figure 6 A schematic diagram of a process for segmenting audio provided in an embodiment of the present application. Specifically:

[0194] Each time a user sets a transcription speed, the client can obtain the transcription speed and send it to the server, which stores the transcription speed in a relational database management system, wherein the relational database management system is referred to as MySQL and uses Structured Query Language (SQL) for database management, that is, the server can obtain each transcription speed set by the user and store it in MySQL.

[0195] The server can obtain the maximum transcription speed from MySQL, use the maximum transcription speed to determine each initial segmentation time point of the audio to be processed, and then determine the duration of the initial segment corresponding to each initial segmentation time point.

[0196] If the duration of the initial segment is less than the preset segment duration, each initial segment time point is adjusted according to the preset segment duration, and then at least one silent segment corresponding to each initial segment time point in the audio to be processed is determined; if the duration of the initial segment is not less than the preset segment duration, at least one silent segment corresponding to each initial segment time point in the audio to be processed is determined.

[0197] Then, for any initial segmentation time point, the head of the longest silent segment in at least one silent segment corresponding to the initial segmentation time point is used as the target segmentation time point corresponding to the initial segmentation time point. According to each target segmentation time point, each audio segment of the audio to be processed is obtained, and each audio segment is stored in the database.

[0198] about Figure 6 The specific implementation method of each step can be found in the above description and will not be repeated here.

[0199] After obtaining each audio clip, Figure 7 As shown, Figure 7 A flowchart of a computing power determination process provided in an embodiment of the present application is as follows:

[0200] The server can extract each audio clip from the database, use each audio clip to generate a transcription task, and determine whether the remaining computing power of the server meets the computing power required for the transcription task. If it does, the transcription task enters the execution queue, that is, performs speech recognition and subsequent steps; if it does not, it re-determines whether the remaining computing power of the server meets the computing power required for the transcription task.

[0201] When the remaining computing power of the server meets the computing power required by the transcription task, such as Figure 8 As shown, Figure 8 A flowchart of multi-threaded processing provided for the implementation of this application. Specifically, the server obtains each audio clip, allocates a thread to each audio clip, each thread reports the computing power it occupies, each thread reads the corresponding audio clip, and performs speech recognition on the read audio clip, each thread segments the speech recognition result, and translates it into text corresponding to the target language, stores the text in the database, and then releases the computing power of each thread.

[0202] Among them, for any thread, if the thread releases computing power, it indicates that the thread has completed the task. When all threads have completed the task, the texts obtained by each thread can be spliced ​​according to the timeline to obtain a text file and / or a subtitle file with a timeline.

[0203] Furthermore, after obtaining the text file and / or the subtitle file with the time axis, the text file and / or the subtitle file with the time axis can be sent to the client and displayed by the client. Fig. 9 As shown, Fig. 9 A schematic diagram of a display interface for text display provided in an embodiment of the present application. In the left half of the display interface, the text file is displayed in the form of sentences, with Chinese sentences and English sentences displayed synchronously, and the starting time point and ending time point of each sentence are displayed. For example, Chinese sentence 1 and English sentence 1 are displayed synchronously, and the corresponding starting time 1 and ending time point 1 are displayed synchronously; Chinese sentence 2 and English sentence 2 are displayed synchronously, and the corresponding starting time 2 and ending time point 2 are displayed synchronously; Chinese sentence 3 and English sentence 3 are displayed synchronously, and the corresponding starting time 3 and ending time point 3 are displayed synchronously; in the right half of the display interface, the text is displayed to the user in the form of subtitles in synchronization with the video screen, that is, the screen and Chinese subtitles and English subtitles are displayed synchronously, and a timeline is displayed at the bottom of the display interface to display the subtitle file with a timeline. Among them, Fig. 9 Export controls may also be included to export a text file and / or a subtitle file with a timeline.

[0204] You can click Fig. 9 The export control in Fig.10 The display interface shown, Fig.10 A schematic diagram of a display interface for exporting a file provided in an embodiment of the present application. The display interface includes "File: Video", "Format: Format 1, Format 2", "Export Content: Bilingual, Chinese, English", cancel control, and export control. Among them, the user can select the file format in the format, for example, the file format can be text format subtitles (SubRip Text, SRT), Word (the format of the word processor application), and text document (TXT). The user can also select the content to be exported in the export content, such as the selected export content is bilingual.

[0205] Understandably, Figure 2-Figure 4 ,as well as Figure 9-10 The display interface shown is only for illustration. Other controls may be set on the display interface or certain controls may be deleted according to actual needs. The display interface may be flexibly set according to actual needs.

[0206] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides an audio processing device, such as Fig.11 As shown, the audio processing device 110 may include: an audio acquisition module 1101, a segmentation point determination module 1102, an audio segmentation module 1103 and an audio recognition module 1104, wherein:

[0207] The audio acquisition module 1101 is used to acquire the audio to be processed;

[0208] The segmentation point determination module 1102 is used to determine at least one initial segmentation time point of the audio to be processed, determine at least one silent segment corresponding to each initial segmentation time point in the audio to be processed, and determine the target segmentation time point corresponding to each initial segmentation time point according to the at least one silent segment corresponding to each initial segmentation time point;

[0209] The audio segmentation module 1103 is used to obtain each audio segment of the audio to be processed according to each target segmentation time point;

[0210] The audio recognition module 1104 is used to perform speech recognition on each audio segment and obtain text corresponding to the target language based on the speech recognition result of each audio segment.

[0211] In another possible implementation of the embodiment of the present application, the audio recognition module 1104 is specifically used to:

[0212] Speech recognition is performed on each audio segment in parallel through multi-threading, and a text corresponding to the target language is obtained based on the speech recognition result of each audio segment.

[0213] Another possible implementation method of the embodiment of the present application is that, for any initial segmentation time point, at least one silent segment corresponding to the initial segmentation time point includes a set number of silent segments adjacent to the initial segmentation time point in the audio to be processed, or a silent segment within a preset time length in the audio to be processed with the initial segmentation time point as the base time point.

[0214] In another possible implementation of the embodiment of the present application, the segmentation point determination module 1102 is specifically used to determine the target segmentation time point corresponding to each initial segmentation time point according to at least one silent segment corresponding to each initial segmentation time point:

[0215] For any initial segmentation time point, the target segmentation time point corresponding to the initial segmentation time point is determined according to the longest silent segment in at least one silent segment corresponding to the initial segmentation time point.

[0216] In another possible implementation of the embodiment of the present application, the segmentation point determination module 1102 is specifically used to:

[0217] Determine at least one initial segmentation time point of the audio to be processed according to at least one of the following information:

[0218] Speech recognition speed, number of segments, and preset segment duration.

[0219] In another possible implementation of the embodiment of the present application, each initial segmentation time point is an average time point of the audio to be processed;

[0220] The segmentation point determination module 1102 is further used to determine the duration of the initial segment corresponding to each initial segmentation time point; if the duration of the initial segment is less than the preset duration, each initial segmentation time point is adjusted according to the preset duration.

[0221] In another possible implementation of the embodiment of the present application, the audio processing device 110 further includes a text display module, which is used to display the text to the user through the client.

[0222] Another possible implementation of the embodiment of the present application is when the audio to be processed is the audio in the video;

[0223] The text display module is specifically used to display the text to the user in the form of subtitles synchronously with the video screen through the client.

[0224] In another possible implementation of the embodiment of the present application, the text display module is further used to:

[0225] When a modification operation of the user on the displayed text is obtained, the displayed text is modified accordingly according to the modification operation, and the modified text is displayed to the user.

[0226] In another possible implementation of the embodiment of the present application, the audio acquisition module 1101 is specifically used to:

[0227] Get the initial audio data to be processed;

[0228] Extracting audio track data of a target audio track from the initial audio data;

[0229] The audio track data of the target audio track is determined as audio to be processed.

[0230] In another possible implementation of the embodiment of the present application, when the audio acquisition module 1101 extracts the track data of the target track in the initial audio data, it is specifically used to:

[0231] Extracting the track data of the designated track from the track data of each track included in the initial audio data, and determining the track data of the designated track as the track data of the target track;

[0232] or,

[0233] The track data of each track in the initial audio data is extracted, and the track data of each track is displayed to the user, and the track data of the target track is determined according to the user's selection operation on the track data of each track.

[0234] Another possible implementation of the embodiment of the present application also includes:

[0235] A task generation module is used to generate speech recognition tasks using each audio clip and determine the computing power required for the speech recognition task;

[0236] The audio recognition module 1104 is specifically used for:

[0237] When the remaining computing power of the audio processing device 110 is not less than the computing power required for the speech recognition task, speech recognition is performed on each audio segment, and a text corresponding to the target language is obtained based on the speech recognition result of each audio segment.

[0238] The audio processing device 110 of this embodiment can execute an audio processing method provided by the method embodiment of this application. The implementation principle is similar and will not be repeated here.

[0239] The present application provides an audio processing device. Compared with the prior art, the embodiment of the present application first determines each target segmentation time point according to at least one silent segment corresponding to each initial segmentation time point of the audio, then obtains each audio segment according to each target segmentation time point, and performs speech recognition on each audio segment. Since each silent segment in the audio is usually a sign of the end of a sentence and / or the beginning of the next sentence, each initial segmentation time point is corrected according to at least one silent segment corresponding to each initial segmentation time point to obtain each target segmentation time point. Based on this method, the target segmentation time point can be located at the position of the silent segment, so that each sentence in each audio segment of the audio to be processed obtained according to each target segmentation time point can be guaranteed to be a complete sentence, and then the speech recognition result of each audio segment can be guaranteed to be a complete speech recognition result of each sentence, thereby improving the accuracy of the speech recognition result.

[0240] The above describes the audio processing device of the present application from the perspective of a virtual module. The following describes the electronic device of the present application from the perspective of a physical device.

[0241] The present application embodiment provides an electronic device, such as Fig.12 As shown, Fig.12 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0242] Processor 4001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0243] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI bus or an EISA bus, etc. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0244] The memory 4003 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disk storage (including a compressed optical disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0245] The memory 4003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in any of the above method embodiments.

[0246] The electronic device provided in an embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program.

[0247] The electronic device provided by an embodiment of the present application includes one or more processors, a memory, and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by one or more processors, and the one or more programs are configured to: execute the corresponding content in the aforementioned method embodiment.

[0248] The electronic device of the present application is introduced above from the perspective of a physical device. The following introduces the computer-readable storage medium of the present application from the perspective of a storage medium.

[0249] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the audio processing method provided in the above method embodiment and various possible implementations thereof.

[0250] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0251] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0252] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An audio processing method, characterized in that: include: Get the audio to be processed; Determine at least one initial segmentation time point of the audio to be processed; Determine at least one silent segment corresponding to each of the initial segment time points in the audio to be processed; Determine the target segmentation time point corresponding to each of the initial segmentation time points according to at least one silent segment corresponding to each of the initial segmentation time points; wherein the target segmentation time point is used to mark the end of a sentence and / or the beginning of the next sentence; Determine the duration between any two target segment time points among the target segment time points; If the duration is greater than the duration threshold, at least one silent segment is selected from the silent segments between any two target segment time points; wherein the selected at least one silent segment meets the following conditions: the duration between the silent segment and each of the two target segment time points is not less than the preset segment duration, and / or the duration between any two silent segments is not less than the preset segment duration; According to the selected at least one silent segment, at least one target segment time point is added between any two target segment time points; wherein the duration between any two target segment time points and any two target segment time points of the added at least one target segment time point is not less than the preset segment duration; Obtaining each audio segment of the audio to be processed according to each of the target segmentation time points; Speech recognition is performed on each of the audio segments, and a text corresponding to the target language is obtained based on the speech recognition result of each of the audio segments.

2. The method according to claim 1, characterized in that The determining, according to at least one silent segment corresponding to each of the initial segment time points, a target segment time point corresponding to each of the initial segment time points comprises: For each initial segmentation time point, if the initial segmentation time point corresponds to at least two silent segments, a target silent segment is selected from the at least two silent segments, and the target segmentation time point corresponding to the initial segmentation time point is determined based on the selected target silent segment.

3. The method according to claim 1, characterized in that For any of the initial segmentation time points, at least one silent segment corresponding to the initial segmentation time point includes a set number of silent segments adjacent to the initial segmentation time point in the audio to be processed, or a silent segment within a preset time length in the audio to be processed with the initial segmentation time point as the base time point.

4. The method according to claim 1, characterized in that: The determining, according to at least one silent segment corresponding to each of the initial segment time points, a target segment time point corresponding to each of the initial segment time points comprises: For any of the initial segmentation time points, the target segmentation time point corresponding to the initial segmentation time point is determined according to the longest silence segment in at least one silence segment corresponding to the initial segmentation time point.

5. The method according to claim 1, characterized in that The step of determining at least one initial segmentation time point of the audio to be processed comprises: Determine at least one initial segmentation time point of the audio to be processed according to at least one of the following information: Speech recognition speed, number of segments, and preset segment duration.

6. The method according to claim 5, characterized in that Each of the initial segmentation time points is an average time point of the audio to be processed; Before determining at least one silent segment corresponding to each of the initial segment time points in the audio to be processed, the method further includes: Determine the duration of the initial segment corresponding to each of the initial segment time points; If the duration of the initial segment is less than the preset segment duration, each of the initial segment time points is adjusted according to the preset segment duration so that the duration of the initial segment corresponding to each of the adjusted initial segment time points is greater than or equal to the preset segment duration.

7. The method according to claim 1, characterized in that The step of obtaining text corresponding to the target language based on the speech recognition results of each of the audio segments comprises: If the target language includes one language, segmenting and concatenating the speech recognition results of each audio segment to obtain a text corresponding to the target language, and a start time and an end time corresponding to each sentence in the text of the target language; If the target language includes at least two languages, the speech recognition results of each audio segment are segmented, translated and concatenated to obtain texts corresponding to at least two languages ​​and the start time and end time of each sentence in the text of each language.

8. The method according to claim 1, characterized in that After performing speech recognition on each of the audio segments and obtaining text corresponding to the target language based on the speech recognition results of each of the audio segments, the method further includes: The text is displayed to the user through the client.

9. The method according to claim 8, characterized in that If the audio to be processed is the audio in the video; The presenting the text to the user through the client comprises: Through the client, the text is displayed to the user in the form of subtitles synchronously with the video screen.

10. The method according to claim 8, characterized in that The presenting the text to the user through the client comprises: By means of the client, each sentence in the text and the time point corresponding to each sentence are displayed by means of at least one of the following: Display the starting time point and the ending time point corresponding to each sentence in the text; If the target language includes at least two languages, displaying sentences corresponding to each starting time point and ending time point in each of the at least two languages; If the audio to be processed is audio in a video, the picture of the video with subtitles and the timeline corresponding to the audio to be processed are displayed, the subtitles include sentences in at least one language corresponding to the picture, and the time points corresponding to the picture and its subtitles are displayed on the timeline.

11. The method according to any one of claims 8 to 10, characterized in that: The method further comprises: When a modification operation of the user on the displayed text is obtained, the displayed text is modified accordingly according to the modification operation, and the modified text is displayed to the user; The modification operation includes at least one of the following: Operations to modify sentences in the displayed text; The operation of adjusting the start time point and / or end time point of a sentence in the displayed text.

12. The method according to claim 1, characterized in that The performing speech recognition on each of the audio segments and obtaining text corresponding to the target language based on the speech recognition results of each of the audio segments includes: Speech recognition is performed on each of the audio segments in parallel through multiple threads, and a text corresponding to the target language is obtained based on the speech recognition result of each of the audio segments.

13. The method according to claim 1, characterized in that The step of obtaining the audio to be processed includes: Get the initial audio data to be processed; Extracting audio track data of a target audio track from the initial audio data; The audio track data of the target audio track is determined as the audio to be processed.

14. The method according to claim 13, characterized in that The extracting the audio track data of the target audio track in the initial audio data comprises: Extracting the track data of a designated track from the track data of each track included in the initial audio data, and determining the track data of the designated track as the track data of the target track; or, The audio track data of each audio track in the initial audio data is extracted, and the audio track data of each audio track is displayed to the user, and the audio track data of the target audio track is determined according to the user's selection operation on the audio track data of each audio track.

15. The method according to claim 1, characterized in that The method is performed by an electronic device and further includes: Generating a speech recognition task using each of the audio clips, and determining the computing power required for the speech recognition task; The performing speech recognition on each of the audio segments and obtaining text corresponding to the target language based on the speech recognition results of each of the audio segments includes: If the remaining computing power of the electronic device is not less than the computing power required for the speech recognition task, speech recognition is performed on each of the audio segments, and a text corresponding to the target language is obtained based on the speech recognition results of each of the audio segments.

16. An audio processing device, characterized in that: include: An audio acquisition module, used to acquire audio to be processed; A segmentation point determination module is used to determine at least one initial segmentation time point of the audio to be processed, determine at least one silent segment corresponding to each of the initial segmentation time points in the audio to be processed, and determine the target segmentation time point corresponding to each of the initial segmentation time points according to the at least one silent segment corresponding to each of the initial segmentation time points; determine the duration between any two of the target segmentation time points; If the duration is greater than the duration threshold, at least one silent segment is selected from the silent segments between any two target segment time points; wherein the duration between the silent segment and each of the two target segment time points is not less than the preset segment duration, and / or the duration between any two silent segments is not less than the preset segment duration; according to the selected at least one silent segment, at least one target segment time point is added between any two target segment time points; The target segmentation time point is used to mark the end of a sentence and / or the beginning of the next sentence; the duration between any two target segmentation time points and any two target segmentation time points of the at least one added target segmentation time point is not less than the preset segmentation duration; An audio segmentation module, used for obtaining each audio segment of the audio to be processed according to each target segmentation time point; The audio recognition module is used to perform speech recognition on each of the audio segments, and obtain text corresponding to the target language based on the speech recognition results of each of the audio segments.

17. The device according to claim 16, characterized in that The segmentation point determination module is used for: For each initial segmentation time point, if the initial segmentation time point corresponds to at least two silent segments, a target silent segment is selected from the at least two silent segments, and the target segmentation time point corresponding to the initial segmentation time point is determined based on the selected target silent segment.

18. The device according to claim 16, characterized in that For any of the initial segmentation time points, at least one silent segment corresponding to the initial segmentation time point includes a set number of silent segments adjacent to the initial segmentation time point in the audio to be processed, or a silent segment within a preset time length in the audio to be processed with the initial segmentation time point as the base time point.

19. The device according to claim 16, characterized in that The segmentation point determination module is used for: For any of the initial segmentation time points, the target segmentation time point corresponding to the initial segmentation time point is determined according to the longest silence segment in at least one silence segment corresponding to the initial segmentation time point.

20. The device according to claim 16, characterized in that The segmentation point determination module is used for: Determine at least one initial segmentation time point of the audio to be processed according to at least one of the following information: Speech recognition speed, number of segments, and preset segment duration.

21. The device according to claim 20, characterized in that Each of the initial segmentation time points is an average time point of the audio to be processed; The segmentation point determination module is also used for: Determine the duration of the initial segment corresponding to each of the initial segment time points; If the duration of the initial segment is less than the preset segment duration, each of the initial segment time points is adjusted according to the preset segment duration so that the duration of the initial segment corresponding to each of the adjusted initial segment time points is greater than or equal to the preset segment duration.

22. The device according to claim 16, characterized in that The audio recognition module is used for: If the target language includes one language, segmenting and concatenating the speech recognition results of each audio segment to obtain a text corresponding to the target language, and a start time and an end time corresponding to each sentence in the text of the target language; If the target language includes at least two languages, the speech recognition results of each audio segment are segmented, translated and concatenated to obtain texts corresponding to at least two languages ​​and the start time and end time of each sentence in the text of each language.

23. The device according to claim 16, characterized in that The audio processing device further includes a text display module, which is used to: The text is displayed to the user through the client.

24. The device according to claim 23, characterized in that If the audio to be processed is the audio in the video, the text display module is used to: Through the client, the text is displayed to the user in the form of subtitles synchronously with the video screen.

25. The device according to claim 23, characterized in that The text display module is used for: By means of the client, each sentence in the text and the time point corresponding to each sentence are displayed by means of at least one of the following: Display the starting time point and the ending time point corresponding to each sentence in the text; If the target language includes at least two languages, displaying sentences corresponding to each starting time point and ending time point in each of the at least two languages; If the audio to be processed is audio in a video, the picture of the video with subtitles and the timeline corresponding to the audio to be processed are displayed, the subtitles include sentences in at least one language corresponding to the picture, and the time points corresponding to the picture and its subtitles are displayed on the timeline.

26. The device according to any one of claims 23 to 25, characterized in that: The text display module is also used for: When a modification operation of the user on the displayed text is obtained, the displayed text is modified accordingly according to the modification operation, and the modified text is displayed to the user; The modification operation includes at least one of the following: Operations to modify sentences in the displayed text; The operation of adjusting the start time point and / or end time point of a sentence in the displayed text.

27. The device according to claim 16, characterized in that The audio recognition module is used for: Speech recognition is performed on each of the audio segments in parallel through multiple threads, and a text corresponding to the target language is obtained based on the speech recognition result of each of the audio segments.

28. The device according to claim 16, characterized in that The audio acquisition module is used for: Get the initial audio data to be processed; Extracting audio track data of a target audio track from the initial audio data; The audio track data of the target audio track is determined as the audio to be processed.

29. The device according to claim 28, characterized in that The audio acquisition module is used for: Extracting the track data of a designated track from the track data of each track included in the initial audio data, and determining the track data of the designated track as the track data of the target track; or, The audio track data of each audio track in the initial audio data is extracted, and the audio track data of each audio track is displayed to the user, and the audio track data of the target audio track is determined according to the user's selection operation on the audio track data of each audio track.

30. The device according to claim 16, characterized in that The audio processing device further includes a task generation module, which is used to: Generating a speech recognition task using each of the audio clips, and determining the computing power required for the speech recognition task; The audio recognition module is used for: If the remaining computing power of the electronic device is not less than the computing power required for the speech recognition task, speech recognition is performed on each of the audio segments, and a text corresponding to the target language is obtained based on the speech recognition results of each of the audio segments.

31. An electronic device, characterized in that: It includes: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the audio processing method according to any one of claims 1 to 15.

32. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the audio processing method described in any one of claims 1 to 15 is implemented.

Images