Video processing method, device, equipment, storage medium and program product

Abstract

The present application discloses a video processing method, device, equipment, storage medium and program product, which relates to the field of data processing technology, including: if it is detected that a video playback device outputs a currently played video, then the ambient audio signal of the external environment in which the video playback device is located is collected through a preset microphone; a voice signal is extracted from the ambient audio signal, and the voice signal is input into a preset voice recognition model to obtain a voice recognition text; based on the voice recognition text, the professional field of the currently played video is determined, and the voice recognition text is adjusted based on a preset vocabulary library corresponding to the professional field to obtain a target recognition text; the target recognition text is associated with the currently played video and saved in a preset storage space. The present application achieves improved accuracy of voice transcription.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a video processing method, device, equipment, storage medium and program product. Background Art

[0002] Speech transcription refers to the process of converting speech content into text. With the development of speech recognition technology, automatic speech transcription systems have been widely used in many fields such as conference records, voice assistants, and speech recognition software. Speech systems usually use advanced speech recognition algorithms and models to analyze the spectrum and voice features of speech signals and convert them into corresponding text representations. However, different types of videos have different content structures, and the current speech transcription system lacks adaptive processing of video content, resulting in the inability of the transcribed text to accurately reflect the video content, thus affecting the accuracy of speech transcription.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a video processing method, device, equipment, storage medium and program product, aiming to solve the current technical problem of low accuracy of speech transcription.

[0005] To achieve the above object, the present application proposes a video processing method, which includes:

[0006] If it is detected that the video playback device outputs the currently played video, an ambient audio signal of the external environment where the video playback device is located is collected through a preset microphone;

[0007] Extracting a speech signal from the ambient audio signal, and inputting the speech signal into a preset speech recognition model to obtain a speech recognition text, wherein the speech recognition model is trained using a historical speech signal as input data and a recognition text corresponding to the historical speech signal as a label;

[0008] Determine the professional field of the currently playing video based on the speech recognition text, and adjust the speech recognition text based on a preset vocabulary library corresponding to the professional field to obtain a target recognition text;

[0009] The target recognition text and the currently playing video are associated and saved in a preset storage space.

[0010] In one embodiment, before the step of associating the target recognition text with the currently played video and saving it in a preset storage space, the method further includes:

[0011] Divide the target recognition text into a plurality of recognition subtexts, traverse each of the recognition subtexts, and determine the audio segment corresponding to the recognition subtext from the speech signal;

[0012] Determining, based on the acquisition time of the audio segment, the video duration of the currently playing video, and the playing time of the currently playing video, that the audio segment corresponds to a video segment in the currently playing video;

[0013] Processing the identified subtext based on the video segment to obtain a speech transcription text;

[0014] The step of associating the target recognition text with the currently played video and saving it in a preset storage space comprises:

[0015] The voice transcription text and the currently playing video are associated and saved in a preset storage space.

[0016] In one embodiment, the step of processing the recognition subtext based on the video segment to obtain a speech transcription text includes:

[0017] If the currently playing video is a learning video, determining a target time period of the video segment in the video timeline of the currently playing video, and identifying the recognition subtext by the target time period to obtain a speech transcription text;

[0018] If the currently playing video is a conference video, the speaker is identified from the video segment, and the identity information of the speaker is determined. The identification sub-text is identified by the identity information of the speaker to obtain the speech transcription text.

[0019] In one embodiment, the speech recognition model includes speech recognition models corresponding to different languages, and the step of inputting the speech signal into a preset speech recognition model to obtain a speech recognition text includes:

[0020] Determining a target language corresponding to the speech signal;

[0021] If the target language includes multiple languages, the speech signal is divided into speech sub-signals according to the multiple languages, a target recognition model corresponding to the language of each speech sub-signal is determined from the speech recognition models corresponding to the different languages, each speech sub-signal is input into the corresponding target recognition model to obtain a language text, and each language text is spliced ​​according to the time sequence of each speech sub-signal in the speech signal to obtain a speech recognition text;

[0022] If the target language includes one language, a target recognition model corresponding to the target language is determined from the speech recognition models corresponding to the different languages, and the speech signal is input into the target recognition model corresponding to the target language to obtain the speech recognition text.

[0023] In one embodiment, the step of extracting the speech signal from the ambient audio signal comprises:

[0024] Inputting the environmental audio signal into a noise reduction model to obtain a noise-reduced audio signal, wherein the noise reduction model is trained using historical audio signals as input data and using the noise-reduced historical audio signals as labels;

[0025] A preset frequency band signal in a preset frequency band is extracted from the noise reduction audio signal, and the preset frequency band signal is determined as a speech signal.

[0026] In one embodiment, the noise reduction model includes an encoder, an attention mechanism module, a decoder and a fully connected layer, the attention mechanism module includes a weight calculation block and a weighted processing block, and the step of inputting the ambient audio signal into the noise reduction model to obtain the noise-reduced audio signal includes:

[0027] Inputting the ambient audio signal into the encoder for processing to obtain a first processing result;

[0028] Inputting the first processing result into the decoder for processing to obtain a second processing result;

[0029] Inputting the first processing result and the second processing result into the weight calculation block for processing to obtain an attention weight matrix, and inputting the attention weight matrix and the first processing result into the weighted processing block for processing to obtain a third processing result;

[0030] The second processing result and the third processing result are connected, and the connected result is input into the fully connected layer for processing to obtain a noise-reduced audio signal.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a video processing device, the video processing device comprising:

[0032] A collection module, for collecting an ambient audio signal of an external environment in which the video playback device is located through a preset microphone if it is detected that the video playback device outputs a currently played video;

[0033] A recognition module, used for extracting a speech signal from the environmental audio signal, and inputting the speech signal into a preset speech recognition model to obtain a speech recognition text, wherein the speech recognition model is trained using a historical speech signal as input data and a recognition text corresponding to the historical speech signal as a label;

[0034] A determination module, configured to determine the professional field of the currently playing video based on the speech recognition text, and adjust the speech recognition text based on a preset vocabulary library corresponding to the professional field to obtain a target recognition text;

[0035] The storage module is used to associate and save the target recognition text and the currently played video into a preset storage space.

[0036] In addition, to achieve the above objectives, the present application also proposes a video processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video processing method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video processing method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the video processing method described above are implemented.

[0039] In the present application, if it is detected that the video playback device outputs the currently playing video, the ambient audio signal of the external environment in which the video playback device is located is collected through a preset microphone; a voice signal is extracted from the ambient audio signal, and the voice signal is input into a preset voice recognition model to obtain a voice recognition text, wherein the voice recognition model is trained with historical voice signals as input data and with recognition texts corresponding to the historical voice signals as labels; based on the voice recognition text, the professional field of the currently playing video is determined, and based on a preset vocabulary library corresponding to the professional field, the voice recognition text is adjusted to obtain a target recognition text; the target recognition text and the currently playing video are associated and saved in a preset storage space.

[0040] In this application, by determining the professional field based on the speech recognition text, the speech recognition text is further optimized and adjusted in combination with the preset vocabulary library corresponding to the professional field, so that the target recognition text can adapt to different professional fields, thereby improving the accuracy of speech transcription. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the video processing method of the present application;

[0044] Figure 2 A flowchart of the second embodiment of the video processing method of the present application is provided;

[0045] Figure 3 A brief flowchart of a video processing method provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the module structure of the video processing device according to an embodiment of the present application;

[0047] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the video processing method in the embodiment of the present application.

[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] The main solution of the embodiment of the present application is: if it is detected that the video playback device outputs the currently playing video, the ambient audio signal of the external environment of the video playback device is collected through a preset microphone; the voice signal is extracted from the ambient audio signal, and the voice signal is input into a preset voice recognition model to obtain a voice recognition text, wherein the voice recognition model is trained with historical voice signals as input data and with recognition texts corresponding to the historical voice signals as labels; based on the voice recognition text, the professional field of the currently playing video is determined, and the voice recognition text is adjusted based on a preset vocabulary library corresponding to the professional field to obtain a target recognition text; the target recognition text and the currently playing video are associated and saved in a preset storage space.

[0052] In this embodiment, for ease of description, the following is explained with the video processing device as the execution subject. Voice transcription refers to the process of converting voice content into text form. With the development of voice recognition technology, automatic voice transcription systems have been widely used in many fields such as conference records, voice assistants, and voice recognition software. Voice systems usually use advanced voice recognition algorithms and models to analyze the spectrum and voice features of voice signals and convert them into corresponding text representations. However, different types of videos have different content structures, and the current voice transcription system lacks adaptive processing of video content, resulting in the inability of the transcribed text to accurately reflect the video content, thereby affecting the accuracy of voice transcription.

[0053] The present application provides a solution by determining the professional field based on the speech recognition text, and further optimizing and adjusting the speech recognition text in combination with the preset vocabulary library corresponding to the professional field, so that the target recognition text can adapt to different professional fields, thereby improving the accuracy of speech transcription.

[0054] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a video processing device, etc. The following takes a video processing device as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the embodiment of the present application provides a video processing method, referring to Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the video processing method of the present application.

[0056] In this embodiment, the video processing method includes steps S10 to S40:

[0057] Step S10, if it is detected that the video playback device outputs the currently played video, then the ambient audio signal of the external environment where the video playback device is located is collected through a preset microphone;

[0058] The video playback device is a device that can play video content, such as a TV, a computer monitor, a mobile phone, a tablet computer, etc. The video currently played by the video playback device is called the currently played video, and the environmental audio signal is the sum of all sounds in the environment where the video playback device is located, including the audio of the video playback, background noise, human voice, etc. In this embodiment, by monitoring the startup of the video playback software, the change of the screen content, etc., it is detected whether the video playback device starts to play the video. When the video playback device starts to play the video, the preset microphone on the video processing device is started to collect the environmental audio signal. It can be understood that by timely capturing the environmental audio during video playback, a basis is provided for the subsequent extraction of voice signals.

[0059] Step S20, extracting a speech signal from the environmental audio signal, and inputting the speech signal into a preset speech recognition model to obtain a speech recognition text, wherein the speech recognition model is trained using a historical speech signal as input data and a recognition text corresponding to the historical speech signal as a label;

[0060] A speech signal is extracted from the ambient audio signal using signal processing technology (such as filtering, noise reduction and other signal processing technology), and the extracted speech signal is input into a pre-trained speech recognition model to obtain a speech recognition text. In this embodiment, the speech recognition model is pre-trained using a historical speech signal as input data and the recognition text corresponding to the historical speech signal as a label. The specific training process is not limited here and can be set according to actual needs.

[0061] Step S30, determining the professional field of the currently playing video based on the speech recognition text, and adjusting the speech recognition text based on a preset vocabulary library corresponding to the professional field to obtain a target recognition text;

[0062] In this embodiment, natural language processing technology is used to analyze the speech recognition text, extract keywords and context information, and match the extracted key information with the professional field according to the keywords and context information in combination with the preset professional field classification rules or models to determine the professional field of the currently playing video. The professional field classification rules can be constructed based on vocabulary similarity, topic models, machine learning algorithms, etc., and are not specifically limited here. Among them, the professional field is used to characterize the professional field to which the video content belongs, which can be a field corresponding to an industry, discipline, or conference theme. Each professional field has its own corresponding preset vocabulary library, which is a database containing specific field or theme vocabulary that is pre-collected and organized. The vocabulary library may contain professional terms, common words, and phrases related to the professional field, so as to provide more accurate vocabulary matching and verification when processing speech recognition text. The vocabulary library can be extracted from professional papers, publications, and news reports, and is not limited here.

[0063] The speech recognition text is adjusted according to the vocabulary library corresponding to the professional field, such as filtering, correction or enhancement, to obtain the target recognition text. In this embodiment, the professional vocabulary in the speech recognition text is verified according to the vocabulary library corresponding to the professional field, the professional vocabulary is extracted from the speech recognition text, and the verification vocabulary with the same lexical semantics and pronunciation as the professional vocabulary is retrieved from the vocabulary library corresponding to the professional field. If the verification vocabulary and the professional vocabulary are different, the professional vocabulary is replaced by the verification vocabulary, so as to adjust the speech recognition text based on the preset vocabulary library corresponding to the professional field to obtain the target recognition text.

[0064] Step S40, the target recognition text and the currently playing video are associated and saved in a preset storage space.

[0065] Create an associated record, which contains the target recognition text and the identification information of the currently playing video (such as the file name, play time, etc.), and save the associated record, the target recognition text and the currently playing video to a preset storage space (such as a database, a file system). It is understandable that by storing the associated speech recognition text and the video content, it is convenient to retrieve, analyze and apply the video content later.

[0066] In a feasible implementation manner, the step S10: extracting the speech signal from the ambient audio signal comprises:

[0067] Step S101, inputting the environmental audio signal into a noise reduction model to obtain a noise-reduced audio signal, wherein the noise reduction model is trained using a historical audio signal as input data and the noise-reduced historical audio signal as a label;

[0068] In this embodiment, the noise reduction model is obtained by pre-training the historical audio signal as input data and the noise-reduced historical audio signal as a label. The training method can adopt a conventional neural network training method, which is not described here. The environmental audio signal is input into the noise reduction model to obtain a noise-reduced audio signal. The noise reduction model can effectively reduce the impact of environmental noise on the speech signal and improve the accuracy of speech recognition.

[0069] Step S102: extracting a preset frequency band signal in a preset frequency band from the noise reduction audio signal, and determining the preset frequency band signal as a speech signal.

[0070] The denoised audio signal is subjected to Fourier transform or short-time Fourier transform to obtain its spectrum representation, and the signal components in the frequency band are screened out in the spectrum according to the preset frequency band, that is, the preset frequency band signal, and the screened frequency band signal is subjected to inverse Fourier transform or inverse short-time Fourier transform to obtain the speech signal in the time domain. It can be understood that the preset frequency band refers to one or more frequency ranges set according to the spectral characteristics of the speech signal, for example, it can be 300Hz to 3400Hz, which is the main frequency range of human speech.

[0071] In one feasible implementation, the noise reduction model includes an encoder, an attention mechanism module, a decoder and a fully connected layer, the attention mechanism module includes a weight calculation block and a weighted processing block, and the step S101: inputting the ambient audio signal into the noise reduction model to obtain a noise-reduced audio signal includes:

[0072] Step S1011, inputting the ambient audio signal into the encoder for processing to obtain a first processing result;

[0073] In this embodiment, the noise reduction model may include an encoder, an attention mechanism module, a decoder, and a fully connected layer. Among them, the encoder layer is used to extract data features and downsample the input ambient audio signal; the attention mechanism module is used to focus on the key signal in the ambient audio signal, that is, the voice signal; the decoder is used to upsample the result output by the encoder; the fully connected layer is used to process the result output by the decoder and output the signal after noise elimination. In this embodiment, the ambient audio signal is input into the encoder for processing to obtain the first processing result.

[0074] Step S1012, inputting the first processing result into the decoder for processing to obtain a second processing result;

[0075] The first processing result is input into the decoder for upsampling to obtain a second processing result.

[0076] Step S1013, inputting the first processing result and the second processing result into the weight calculation block for processing to obtain an attention weight matrix, and inputting the attention weight matrix and the first processing result into the weighted processing block for processing to obtain a third processing result;

[0077] Based on the first processing result and the second processing result, an attention weight matrix representing the degree of attention to the information in the first processing result is calculated, and the first processing result is weighted by the attention weight matrix to obtain the third processing result. The attention mechanism module performs weighted processing on the first processing result to achieve weight allocation of features in the first processing result, thereby filtering and enhancing important information in the first processing result, so that the noise reduction model pays attention to key information, that is, speech signals, during the decoding process, thereby improving the accuracy of noise elimination.

[0078] Step S1014: connect the second processing result and the third processing result, and input the connected result into the fully connected layer for processing to obtain a noise-reduced audio signal.

[0079] The second processing result and the third processing result are connected, and the connected result is input into a fully connected layer for processing to obtain a noise-reduced audio signal.

[0080] In this implementation, the first processing result is weighted by introducing an attention mechanism module to achieve weight allocation of features in the first processing result, thereby filtering out important information in the first processing result and enhancing important information, so that the decoding process focuses on key information, that is, the speech signal, thereby improving the accuracy of noise elimination.

[0081] In this embodiment, by determining the professional field based on the speech recognition text, the speech recognition text is further optimized and adjusted in combination with the preset vocabulary library corresponding to the professional field, so that the target recognition text can adapt to different professional fields, thereby improving the accuracy of speech transcription.

[0082] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , before the step S40: associating the target recognition text with the currently played video and saving it in a preset storage space, the method further includes:

[0083] Step S50, dividing the target recognition text into a plurality of recognition sub-texts, traversing each of the recognition sub-texts, and determining the audio segment corresponding to the recognition sub-text from the speech signal;

[0084] According to the length, content structure or specific separators (such as periods, paragraph marks, etc.) of the target recognition text, the text is divided into multiple smaller recognition sub-texts. Audio segment positioning: using the timestamp information or audio signal processing algorithm in speech recognition technology, traverse the original speech signal to find the audio segment corresponding to each recognition sub-text. Specifically, the text content can be compared with the speech features in the audio signal, such as speech rhythm, intonation, pronunciation, etc., which are not limited here. By dividing the target recognition text into multiple recognition sub-texts, it is easier to locate the specific audio segment in the original speech signal, which helps to improve the accuracy of audio and text alignment.

[0085] Step S60, determining that the audio segment corresponds to a video segment in the currently playing video based on the acquisition time of the audio segment, the video duration of the currently playing video, and the playing time of the currently playing video;

[0086] Extract the acquisition time of the audio segment from the audio signal (which may include the start time and the end time), obtain the video duration and playback time of the currently playing video (which may include the start playback time and the current playback time point), and calculate the specific position of the audio segment in the video, i.e. the start and end time points of the video segment, based on the acquisition time of the audio segment and the time information of the video. It can be understood that the precise alignment of the audio segment and the video segment helps to quickly locate the relevant audio content in the video content, thereby improving the efficiency of video retrieval and editing.

[0087] Step S70, processing the identified subtext based on the video segment to obtain a speech transcription text;

[0088] Combined with the content of the video segment (such as pictures, subtitles, background sounds, etc.), the recognized sub-text is further verified and adjusted to generate the final speech transcription text.

[0089] The step S40: the step of associating the target recognition text with the currently played video and saving it in a preset storage space comprises:

[0090] Step S401, the speech transcription text and the currently playing video are associated and saved in a preset storage space.

[0091] The association relationship between the speech transcription text and the currently playing video can usually be established through file naming, database records or specific data structures, and the associated speech transcription text and video files are saved in a preset storage space.

[0092] In a feasible implementation manner, the step S70: the step of processing the recognition subtext based on the video segment to obtain the speech transcription text, comprises:

[0093] Step S701, if the currently playing video is a learning video, determining a target time period of the video segment in the video timeline of the currently playing video, and identifying the recognition subtext by the target time period to obtain a speech transcription text;

[0094] In this implementation, the video type of the currently playing video is determined, and the video types include but are not limited to learning and conference types, and may also include entertainment types.

[0095] If the video is a learning video based on its content, determine the start and end time points of the video segment on the video timeline, that is, the target time period, and associate the identified sub-text with the target time period. This can be achieved by adding a timestamp or time range marker to the speech transcription text, and outputting the speech transcription text with a timestamp. The speech transcription text with a timestamp can help learners quickly locate the key content in the video, improve learning efficiency, and facilitate learners to quickly review specific parts of the video according to the timestamp when reviewing, saving time.

[0096] Step S702: If the currently playing video is a conference video, identify the speaker from the video segment, determine the speaker's identity information, and identify the identification subtext by the identity information to obtain the speech transcription text.

[0097] If it is determined based on the video content that the video is a conference video, facial recognition technology, voice recognition technology or conference recording system is used to identify the speaker in the video segment. By comparing with a known character database or a list of conference participants, the speaker's identity information (such as name, position, etc.) is determined, and the identification sub-text is associated with the speaker's identity information. This can be achieved by adding a speaker tag to the voice transcription text, and outputting a voice transcription text with the speaker information. The voice transcription text with the speaker information can help meeting organizers or participants quickly understand the content and order of everyone's speeches in the meeting, facilitate subsequent analysis and summary of the meeting content, improve meeting efficiency and quality, and help identify key speakers or decision points in the meeting, providing a basis for actions or decisions after the meeting.

[0098] In this implementation, specific processing is performed on learning videos and conference videos respectively, and speech transcription texts with timestamps or speaker information are generated, so that the speech transcription texts have higher practical value and convenience in subsequent applications.

[0099] In one feasible implementation, the speech recognition model includes speech recognition models corresponding to different languages, and the step S20: the step of inputting the speech signal into a preset speech recognition model to obtain a speech recognition text includes:

[0100] Step S201, determining the target language corresponding to the speech signal;

[0101] Preprocess the speech signal, such as denoising, framing, and windowing, for subsequent feature extraction. Extract features from the preprocessed speech signal, such as Mel frequency cepstral coefficients, linear prediction coefficients, etc. Use language recognition algorithms (such as support vector machines, neural networks, etc.) to classify the extracted features and determine the target language of the speech signal. By accurately identifying the target language of the speech signal, it provides a basis for the subsequent selection of a suitable speech recognition model, improves the flexibility and applicability of the multilingual speech recognition system, and can process speech input in multiple languages.

[0102] Step S202: if the target language includes multiple languages, the speech signal is divided into speech sub-signals according to the multiple languages, a target recognition model corresponding to the language of each speech sub-signal is determined from the speech recognition models corresponding to the different languages, each speech sub-signal is input into the corresponding target recognition model to obtain a language text, and each language text is spliced ​​according to the time sequence of each speech sub-signal in the speech signal to obtain a speech recognition text;

[0103] For speech signals containing multiple languages, the speech signals are divided into multiple speech sub-signals according to the languages, and a target recognition model matching the language of each speech sub-signal is selected from the preset speech recognition models corresponding to different languages, and each speech sub-signal is input into its corresponding target recognition model to obtain the respective language texts, and the obtained multiple language texts are spliced ​​together according to the temporal order of the speech sub-signals in the original speech signal to form a complete speech recognition text. This implementation can effectively process complex speech signals containing multiple languages, improve the accuracy and applicability of the speech recognition system, and realize the continuous recognition and output of multi-language speech signals through parallel processing and text splicing.

[0104] Step S203, if the target language includes one language, determine a target recognition model corresponding to the target language from the speech recognition models corresponding to the different languages, input the speech signal into the target recognition model corresponding to the target language, and obtain the speech recognition text.

[0105] For a single language speech signal, select a target recognition model that matches the single language from the predefined speech recognition models corresponding to different languages, input the speech signal into the selected target recognition model, perform speech recognition processing, and obtain and output the speech recognition text. By simplifying the processing flow, the efficiency and accuracy of single language speech recognition are improved, unnecessary computing resource consumption is reduced, and the overall performance of the system is improved.

[0106] It can be understood that this embodiment, by constructing a processing flow for multilingual speech recognition, can accurately recognize and process speech signals containing multiple or single languages, generate corresponding speech recognition text, and improve the flexibility and accuracy of speech transcription.

[0107] For example, to help understand the implementation process of the video processing method obtained by combining the first embodiment with the second embodiment, please refer to Figure 3 , Figure 3 A brief flow chart of a video processing method is provided, specifically:

[0108] Collect ambient audio signals, and collect ambient audio signals of the external environment where the video playback device is located through a preset microphone.

[0109] Noise reduction processing, inputting the ambient audio signal into the noise reduction model to obtain a noise-reduced audio signal, the noise reduction model includes an encoder, an attention mechanism module, a decoder and a fully connected layer, the attention mechanism module includes a weight calculation block and a weighted processing block, and the step of inputting the ambient audio signal into the noise reduction model to obtain the noise-reduced audio signal includes: inputting the ambient audio signal into the encoder for processing to obtain a first processing result; inputting the first processing result into the decoder for processing to obtain a second processing result; inputting the first processing result and the second processing result into the weight calculation block for processing to obtain an attention weight matrix, and inputting the attention weight matrix and the first processing result into the weighted processing block for processing to obtain a third processing result; connecting the second processing result and the third processing result, and inputting the connected result into the fully connected layer for processing to obtain the noise-reduced audio signal.

[0110] A speech signal is extracted, a preset frequency band signal in a preset frequency band is extracted from the noise reduction audio signal, and the preset frequency band signal is determined as a speech signal.

[0111] Speech recognition, determine the target language corresponding to the speech signal; if the target language includes multiple languages, divide the speech signal into speech sub-signals according to the multiple languages, determine the target recognition model corresponding to the language of each speech sub-signal from the speech recognition models corresponding to different languages, input each speech sub-signal into its corresponding target recognition model to obtain a language text, and splice the language texts according to the time sequence of each speech sub-signal in the speech signal to obtain a speech recognition text; if the target language includes one language, determine the target recognition model corresponding to the target language from the speech recognition models corresponding to different languages, input the speech signal into the target recognition model corresponding to the target language, and obtain a speech recognition text.

[0112] Text adjustment: Determine the professional field of the currently playing video based on the speech recognition text, and adjust the speech recognition text based on the preset vocabulary library corresponding to the professional field to obtain the target recognition text.

[0113] Text identification, dividing the target recognition text into multiple recognition sub-texts, traversing each recognition sub-text, and determining the audio segment corresponding to the recognition sub-text from the voice signal; based on the acquisition time of the audio segment, the video length of the currently playing video, and the playing time of the currently playing video, determine the video segment corresponding to the audio segment in the currently playing video; based on the video segment, process the recognition sub-text to obtain a voice transcription text, wherein, if the currently playing video is a learning video, determine the target time period of the video segment in the video timeline of the currently playing video, and identify the recognition sub-text by the target time period to obtain a voice transcription text; if the currently playing video is a conference video, identify the speaker from the video segment, and determine the speaker's identity information, and identify the recognition sub-text by the identity information to obtain a voice transcription text.

[0114] Data storage: associate the voice transcription text with the currently playing video and save it in the preset storage space.

[0115] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the video processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0116] This application also provides a video processing device, please refer to Figure 4 , the video processing device comprises:

[0117] The acquisition module 10 is used to acquire the ambient audio signal of the external environment of the video playback device through a preset microphone if it is detected that the video playback device outputs the currently played video;

[0118] A recognition module 20 is used to extract a speech signal from the environmental audio signal, and input the speech signal into a preset speech recognition model to obtain a speech recognition text, wherein the speech recognition model is trained using a historical speech signal as input data and a recognition text corresponding to the historical speech signal as a label;

[0119] A determination module 30, configured to determine the professional field of the currently playing video based on the speech recognition text, and adjust the speech recognition text based on a preset vocabulary library corresponding to the professional field to obtain a target recognition text;

[0120] The storage module 40 is used to associate the target recognition text with the currently playing video and save it in a preset storage space.

[0121] Optionally, the device further comprises an identification module, configured to:

[0122] Divide the target recognition text into a plurality of recognition sub-texts, traverse each of the recognition sub-texts, and determine the audio segment corresponding to the recognition sub-text from the speech signal;

[0123] Determining, based on the acquisition time of the audio segment, the video duration of the currently playing video, and the playing time of the currently playing video, that the audio segment corresponds to a video segment in the currently playing video;

[0124] Processing the identified subtext based on the video segment to obtain a speech transcription text;

[0125] The storage module 40 is also used for:

[0126] The voice transcription text and the currently playing video are associated and saved in a preset storage space.

[0127] Optionally, the identification module is further used for:

[0128] If the currently playing video is a learning video, determining a target time period of the video segment in the video timeline of the currently playing video, and identifying the recognition subtext by the target time period to obtain a speech transcription text;

[0129] If the currently playing video is a conference video, the speaker is identified from the video segment, and the identity information of the speaker is determined. The identification sub-text is identified by the identity information of the speaker to obtain the speech transcription text.

[0130] Optionally, the speech recognition model includes speech recognition models corresponding to different languages, and the recognition module 20 is further used for:

[0131] Determining a target language corresponding to the speech signal;

[0132] If the target language includes multiple languages, the speech signal is divided into speech sub-signals according to the multiple languages, a target recognition model corresponding to the language of each speech sub-signal is determined from the speech recognition models corresponding to the different languages, each speech sub-signal is input into the corresponding target recognition model to obtain a language text, and each language text is spliced ​​according to the time sequence of each speech sub-signal in the speech signal to obtain a speech recognition text;

[0133] If the target language includes one language, a target recognition model corresponding to the target language is determined from the speech recognition models corresponding to the different languages, and the speech signal is input into the target recognition model corresponding to the target language to obtain the speech recognition text.

[0134] Optionally, the identification module 20 is further used for:

[0135] Inputting the environmental audio signal into a noise reduction model to obtain a noise-reduced audio signal, wherein the noise reduction model is trained using historical audio signals as input data and using the noise-reduced historical audio signals as labels;

[0136] A preset frequency band signal in a preset frequency band is extracted from the noise reduction audio signal, and the preset frequency band signal is determined as a speech signal.

[0137] Optionally, the denoising model includes an encoder, an attention mechanism module, a decoder and a fully connected layer, the attention mechanism module includes a weight calculation block and a weighted processing block, and the recognition module 20 is further used to:

[0138] Inputting the ambient audio signal into the encoder for processing to obtain a first processing result;

[0139] Inputting the first processing result into the decoder for processing to obtain a second processing result;

[0140] Inputting the first processing result and the second processing result into the weight calculation block for processing to obtain an attention weight matrix, and inputting the attention weight matrix and the first processing result into the weighted processing block for processing to obtain a third processing result;

[0141] The second processing result and the third processing result are connected, and the connected result is input into the fully connected layer for processing to obtain a noise-reduced audio signal.

[0142] The video processing device provided by the present application adopts the video processing method in the above embodiment, which can solve the technical problem of low accuracy of current speech transcription. Compared with the prior art, the beneficial effects of the video processing device provided by the present application are the same as the beneficial effects of the video processing method provided by the above embodiment, and other technical features in the video processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0143] The present application provides a video processing device, which may be a video playback device or an electronic device connected to the video playback device. The video processing device includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video processing methods in the above-mentioned embodiments.

[0144] Reference below Figure 5 , which shows a schematic diagram of the structure of a video processing device suitable for implementing an embodiment of the present application. The video processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The video processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0145] like Figure 5As shown, the video processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the video processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the video processing device to communicate with other devices wirelessly or by wire to exchange data. Although the video processing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0146] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0147] The video processing device provided by the present application adopts the video processing method in the above embodiment, which can solve the technical problem of low accuracy of current speech transcription. Compared with the prior art, the beneficial effects of the video processing device provided by the present application are the same as the beneficial effects of the video processing method provided by the above embodiment, and other technical features in the video processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0148] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0149] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the professional field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0150] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the video processing method in the above-mentioned embodiment.

[0151] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0152] The computer-readable storage medium may be included in the video processing device; or may exist independently without being assembled into the video processing device.

[0153] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the video processing device, the video processing device: if it is detected that the video playback device outputs the currently played video, collects the ambient audio signal of the external environment of the video playback device through a preset microphone; extracts the voice signal from the ambient audio signal, and inputs the voice signal into a preset voice recognition model to obtain a voice recognition text, wherein the voice recognition model is trained with historical voice signals as input data and recognition texts corresponding to the historical voice signals as labels; determines the professional field of the currently played video based on the voice recognition text, and adjusts the voice recognition text based on a preset vocabulary library corresponding to the professional field to obtain a target recognition text; and associates the target recognition text with the currently played video and saves it in a preset storage space.

[0154] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0157] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video processing method, and can solve the technical problem of low accuracy of current speech transcription. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the video processing method provided by the above-mentioned embodiment, and will not be described in detail here.

[0158] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video processing method when executed by a processor.

[0159] The computer program product provided by the present application can solve the technical problem of low accuracy of current speech transcription. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the video processing method provided by the above embodiment, which will not be repeated here.

[0160] The above descriptions are only some embodiments of the present application, and do not limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related professional fields are included in the patent protection scope of the present application.

Claims

1. A video processing method, characterized in that: The video processing method comprises: If it is detected that the video playback device outputs the currently played video, an ambient audio signal of the external environment where the video playback device is located is collected through a preset microphone; Extracting a speech signal from the ambient audio signal, and inputting the speech signal into a preset speech recognition model to obtain a speech recognition text, wherein the speech recognition model is trained using a historical speech signal as input data and a recognition text corresponding to the historical speech signal as a label; Determine the professional field of the currently playing video based on the speech recognition text, extract professional vocabulary from the speech recognition text, retrieve verification vocabulary that is consistent with the professional vocabulary in terms of semantics and pronunciation from a preset vocabulary library corresponding to the professional field, and if the verification vocabulary is different from the professional vocabulary, replace the professional vocabulary with the verification vocabulary to obtain a target recognition text, wherein the professional field is the professional field of the video content of the currently playing video; Divide the target recognition text into a plurality of recognition subtexts, traverse each of the recognition subtexts, determine an audio segment corresponding to the recognition subtext from the speech signal, and determine that the audio segment corresponds to a video segment in the currently playing video; If the currently playing video is a learning video, determine the starting time point and the ending time point of the video segment on the video timeline, determine the target time period based on the time point and the ending time point, and add a timestamp to the recognition subtext according to the target time period to obtain a speech transcription text; If the currently playing video is a conference video, identifying a speaker from the video segment, determining the speaker's identity information, and identifying the identification subtext by the identity information to obtain the speech transcription text; The voice transcription text and the currently playing video are associated and saved in a preset storage space; The speech recognition model includes speech recognition models corresponding to different languages, and the step of inputting the speech signal into a preset speech recognition model to obtain a speech recognition text includes: Determining a target language corresponding to the speech signal; If the target language includes multiple languages, the speech signal is divided into speech sub-signals according to the multiple languages, a target recognition model corresponding to the language of each speech sub-signal is determined from the speech recognition models corresponding to the different languages, each speech sub-signal is input into the corresponding target recognition model to obtain a language text, and each language text is spliced ​​according to the time sequence of each speech sub-signal in the speech signal to obtain a speech recognition text; If the target language includes one language, a target recognition model corresponding to the target language is determined from the speech recognition models corresponding to the different languages, and the speech signal is input into the target recognition model corresponding to the target language to obtain the speech recognition text.

2. The video processing method according to claim 1, characterized in that: The step of determining that the audio segment corresponds to the video segment in the currently playing video includes: Based on the acquisition time of the audio segment, the video duration of the currently playing video, and the playing time of the currently playing video, it is determined that the audio segment corresponds to the video segment in the currently playing video.

3. The video processing method according to any one of claims 1 to 2, characterized in that: The step of extracting the speech signal from the ambient audio signal comprises: Inputting the environmental audio signal into a noise reduction model to obtain a noise-reduced audio signal, wherein the noise reduction model is trained using historical audio signals as input data and using the noise-reduced historical audio signals as labels; A preset frequency band signal in a preset frequency band is extracted from the noise reduction audio signal, and the preset frequency band signal is determined as a speech signal.

4. The video processing method according to claim 3, characterized in that: The noise reduction model includes an encoder, an attention mechanism module, a decoder and a fully connected layer, the attention mechanism module includes a weight calculation block and a weighted processing block, and the step of inputting the ambient audio signal into the noise reduction model to obtain a noise-reduced audio signal includes: Inputting the ambient audio signal into the encoder for processing to obtain a first processing result; Inputting the first processing result into the decoder for processing to obtain a second processing result; Inputting the first processing result and the second processing result into the weight calculation block for processing to obtain an attention weight matrix, and inputting the attention weight matrix and the first processing result into the weighted processing block for processing to obtain a third processing result; The second processing result and the third processing result are connected, and the connected result is input into the fully connected layer for processing to obtain a noise-reduced audio signal.

5. A video processing device, characterized in that: The video processing device comprises: A collection module, for collecting an ambient audio signal of an external environment in which the video playback device is located through a preset microphone if it is detected that the video playback device outputs a currently played video; A recognition module, used for extracting a speech signal from the environmental audio signal, and inputting the speech signal into a preset speech recognition model to obtain a speech recognition text, wherein the speech recognition model is trained using a historical speech signal as input data and a recognition text corresponding to the historical speech signal as a label; A determination module, used to determine the professional field of the currently playing video based on the speech recognition text, extract professional vocabulary from the speech recognition text, retrieve verification vocabulary that is consistent with the professional vocabulary in terms of semantics and pronunciation from a preset vocabulary library corresponding to the professional field, and if the verification vocabulary is different from the professional vocabulary, replace the professional vocabulary with the verification vocabulary to obtain a target recognition text, wherein the professional field is the professional field of the video content of the currently playing video; an identification module, configured to divide the target recognition text into a plurality of recognition subtexts, traverse each of the recognition subtexts, determine an audio segment corresponding to the recognition subtext from the speech signal, and determine that the audio segment corresponds to a video segment in the currently playing video; The identification module is further configured to determine a target time period of the video segment in the video timeline of the currently playing video if the currently playing video is a learning video, and identify the recognition subtext by the target time period to obtain a speech transcription text; The identification module is further configured to, if the currently playing video is a conference video, identify a speaker from the video segment, determine the speaker's identity information, and identify the identification subtext by the identity information to obtain the speech transcription text; A storage module, used for associating and saving the speech transcription text and the currently playing video into a preset storage space; Among them, the speech recognition model includes speech recognition models corresponding to different languages, and the recognition module is also used to: determine the target language corresponding to the speech signal; if the target language includes multiple languages, divide the speech signal into speech sub-signals according to the multiple languages, determine the target recognition model corresponding to the language of each speech sub-signal from the speech recognition models corresponding to the different languages, input each of the speech sub-signals into its corresponding target recognition model to obtain a language text, and splice each of the language texts according to the time sequence of each of the speech sub-signals in the speech signal to obtain a speech recognition text; if the target language includes one language, determine the target recognition model corresponding to the target language from the speech recognition models corresponding to the different languages, input the speech signal into the target recognition model corresponding to the target language, and obtain the speech recognition text.

6. A video processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video processing method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 4 are implemented.

8. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 4 are implemented.

Images