Text generation method and device, equipment, storage medium and program product

Abstract

The invention provides a text generation method and device, equipment, a storage medium and a program product. The embodiment of the invention can be applied to text generation scenes such as voice recognition, voice text conversion and the like; the method comprises the steps of performing framing processing on a to-be-processed voice signal to obtain a plurality of voice frames of the to-be-processed voice signal; performing feature extraction on each voice frame to obtain a frame feature corresponding to each voice frame; fusing the plurality of frame features to obtain a voice feature of the to-be-processed voice signal; and based on the voice features, performing text conversion on the to-be-processed voice signal to obtain a matching text corresponding to the to-be-processed voice signal. According to the invention, the generation accuracy of the text content corresponding to the voice signal can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a text generation method, device, equipment, storage medium and program product. Background Art

[0002] Using artificial intelligence technology to convert speech signals into text content is an important application direction of artificial intelligence technology. For example, through artificial intelligence technology, the discussion speech in the meeting can be converted into text to reduce the burden of the meeting recorder, or the oral speech of blind writers can be converted into text to help blind writers write, etc.

[0003] In the related art, speech recognition is usually implemented based on the pronunciation template of proper nouns, so before speech recognition, the proper nouns need to be preprocessed or a whitelist needs to be generated. However, in the related art, the number of preprocessed words or words in the whitelist is small, which affects the accuracy of generating the text content corresponding to the speech signal. Summary of the invention

[0004] The embodiments of the present application provide a text generation method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the generation accuracy of text content corresponding to a speech signal.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a text generation method, including:

[0007] Performing frame processing on the speech signal to be processed to obtain multiple speech frames of the speech signal to be processed;

[0008] Extracting features from each speech frame to obtain frame features corresponding to each speech frame;

[0009] Fusing the plurality of frame features to obtain speech features of the speech signal to be processed;

[0010] Based on the voice features, text conversion is performed on the voice signal to be processed to obtain a matching text corresponding to the voice signal to be processed.

[0011] The present application provides a text generation device, including:

[0012] A frame processing module, used for performing frame processing on the speech signal to be processed to obtain multiple speech frames of the speech signal to be processed;

[0013] A feature extraction module is used to extract features from each of the speech frames to obtain frame features corresponding to each of the speech frames;

[0014] A feature fusion module, used for fusing the plurality of frame features to obtain the speech feature of the speech signal to be processed;

[0015] The text conversion module is used to perform text conversion on the speech signal to be processed based on the speech feature to obtain a matching text corresponding to the speech signal to be processed.

[0016] In some embodiments of the present application, the text conversion module is also used to use a state prediction model to predict the occurrence probability corresponding to each state sequence in the state sequence set from the speech features; wherein the state sequence set is used to provide a sequence of all candidate pronunciation states; from the multiple state sequences in the state sequence set, a target state sequence with the highest occurrence probability is screened; text parsing is performed on the target state sequence to obtain the matching text corresponding to the speech signal to be processed, thereby completing the text conversion of the speech signal to be processed.

[0017] In some embodiments of the present application, the text conversion module is also used to combine the pronunciation states in the target state sequence to obtain a phoneme sequence; perform character integration on the phonemes in the phoneme sequence to obtain multiple character information; connect the multiple character information to obtain the matching text corresponding to the voice signal to be processed, and complete the text conversion of the voice signal to be processed.

[0018] In some embodiments of the present application, the feature extraction module is also used to convert each of the speech frames from the time domain to the frequency domain to obtain the spectrum information of each of the speech frames; divide the spectrum information into at least one frequency band, and calculate the weighted average value of the energy in each of the frequency bands; perform logarithmic operation on the weighted average value of the energy in each of the frequency bands, and integrate the logarithmic operation results of at least one of the frequency bands into the frame features of each of the speech frames.

[0019] In some embodiments of the present application, the text generation device further includes: a text optimization module, configured to perform text optimization on the matching text to obtain an optimized matching text of the speech signal to be processed.

[0020] In some embodiments of the present application, the text optimization module is also used to perform word segmentation processing on the matching text to obtain multiple word segmentation results; perform part-of-speech tagging on each of the word segmentation results to obtain part-of-speech information of each of the word segmentation results; determine the dependency relationship between the multiple word segmentation results based on the part-of-speech information of each of the multiple word segmentation results; according to the dependency relationship, perform text integration on the multiple word segmentation results to obtain the optimized matching text of the speech signal to be processed, and complete the text optimization of the matching text.

[0021] In some embodiments of the present application, the text optimization module is further used to determine the order of appearance between the multiple word segmentation results based on the dependency relationship; arrange the multiple word segmentation results according to the order of appearance to obtain the optimized matching text of the speech signal to be processed.

[0022] In some embodiments of the present application, the text generation device further includes: a signal optimization module, which is used to perform signal optimization processing on the input voice signal to obtain the voice signal to be processed; wherein the input voice signal is collected by an audio collection device.

[0023] In some embodiments of the present application, the signal optimization module is also used to perform signal denoising on the input voice signal to obtain the denoised voice signal; estimate the signal spectrum and the noise spectrum from the denoised voice signal; the noise spectrum refers to the spectrum of the background noise in the environment; based on the signal spectrum and the noise spectrum, suppress the environmental noise of the denoised voice signal to obtain a valid voice signal; filter the valid voice signal to obtain a filtering result, and determine the filtering result as the voice signal to be processed.

[0024] In some embodiments of the present application, the signal optimization module is also used to filter the filter input signal of the i-th round through the filter of the i-th round to obtain the filter output signal of the i-th round; wherein, i≥1, i is an integer, and the filter input signal of the first round is the valid speech signal; calculate the signal error between the filter output signal and the expected output signal; when the signal error is less than the error threshold, determine the filter output signal of the i-th round as the filtering result; when the signal error is greater than the error threshold, adjust the parameters of the filter of the i-th round according to the signal error to obtain the filter of the i+1th round.

[0025] In some embodiments of the present application, the text generation device also includes: a model generation module, which is used to obtain a state prediction model through the following processing: feature extraction is performed on each training speech frame of the training speech to obtain training features, and multiple training features are fused to obtain training fusion features of the training speech; a state sequence set is generated for the pronunciation state of the usage scenario to which the training speech belongs, and a corresponding state transfer matrix is ​​generated; a corresponding observation set is created using the training fusion features, and an observation probability matrix is ​​established; the occurrence probability of each state sequence is calculated using the initial probability distribution, the state transfer matrix and the observation probability matrix; wherein the initial probability of each state in the initial probability distribution is the same, and the transition probability of each state in the state transfer matrix is ​​only related to the previous state; based on the occurrence probability, the expectation of the observation set on each state sequence is calculated, and the initial probability distribution, the state transfer matrix and the observation probability matrix are optimized based on the expectation until the end of the training, and the state prediction model is obtained by integrating the optimized initial probability distribution, the state transfer matrix and the observation probability matrix.

[0026] An embodiment of the present application provides an electronic device, including:

[0027] A memory for storing computer executable instructions;

[0028] The processor is used to implement the text generation method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0029] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute and implement the text generation method provided in the embodiment of the present application.

[0030] The embodiments of the present application have the following beneficial effects: the electronic device divides the voice signal to be processed into voice frames, and then utilizes the frame features of different voice frames to fuse and generate the voice features of the entire voice signal to be processed, and combines the global voice features to automatically convert the voice signal to be processed into the corresponding matching text. In this way, even if the usage scenario lacks pre-processed vocabulary or whitelist vocabulary, the voice in the usage scenario can be recognized, thereby improving the generation accuracy of the text content corresponding to the voice signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the architecture of the text generation system provided in the embodiment of the present application;

[0032] Figure 2 The embodiment of this application provides Figure 1 A schematic diagram of the structure of the server in FIG.

[0033] Figure 3 This is a flowchart of the text generation method provided in the embodiment of the present application. Figure 1 ;

[0034] Figure 4 This is a flowchart of the text generation method provided in the embodiment of the present application. Figure 2 ;

[0035] Figure 5 This is a flowchart of the text generation method provided in the embodiment of the present application. Figure 3 ;

[0036] Figure 6 This is a flowchart of the text generation method provided in the embodiment of the present application. Figure 4 ;

[0037] Figure 7 is a schematic diagram of a process of preprocessing an audio signal provided in an embodiment of the present application;

[0038] Figure 8 It is a schematic diagram of a process of extracting features of a speech signal provided in an embodiment of the present application;

[0039] Fig. 9 It is a schematic diagram of speech recognition using the HMM model provided in the embodiment of the present application;

[0040] Fig.10 It is a schematic diagram of a dependency syntax tree provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0042] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0044] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0045] 1) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0046] 2) Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or abilities, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. Pre-trained models are the latest development in deep learning, integrating the above technologies.

[0047] 3) Hidden Markov Model (HMM) is a statistical model used to model the probability distribution of a sequence. It can be applied to speech recognition, natural language processing, bioinformatics learning, financial analysis and other fields. In the embodiment of the present application, HMM can be used to model the speech signal to recognize and generate text.

[0048] 4) Butterworth band-stop filter, which is used to perform a low-pass filter on the signal of the input filter and then perform a high-pass filter to obtain a band-stop filter. The characteristics of the Butterworth band-stop filter are smooth amplitude-frequency characteristics, no ripples, and phase linearity. In an embodiment of the present application, the Butterworth band-stop filter is used to remove background noise in the environment while retaining the required signals in the environment, such as human voice signals in the environment.

[0049] With the research and advancement of artificial intelligence technology, artificial intelligence technology will be studied and applied in multiple fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless cars, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0050] Using artificial intelligence technology to convert speech signals into text content is an important application direction of artificial intelligence technology. For example, through artificial intelligence technology, the discussion speech in the meeting can be converted into text to reduce the burden of the meeting recorder, or the oral speech of blind writers can be converted into text to help blind writers write, etc.

[0051] The general process of converting voice signals into text content is as follows: first, voice signals are collected through audio collection equipment; the collected voice signals are compressed and encoded according to specific rules for easy transmission and storage; then, at a specific time, the compressed and encoded voice signals are read in through voice recognition to perform voice recognition and obtain the text corresponding to the voice signal; then the obtained text is subjected to subsequent processing such as spelling correction, syntactic analysis, and part-of-speech tagging to improve the accuracy and readability of the text; finally, the processed text is output to the screen, text or other devices for users to view and use.

[0052] In the related art, before performing speech recognition, it is necessary to pre-process the proper nouns in the usage scenario or generate a whitelist, for example, using word_1 to correspond to numbers, word_2 to correspond to names, and word_3 to correspond to variables. In the speech recognition process, recognition is achieved based on the pronunciation templates of these proper nouns. However, in the related art, pre-processing proper nouns or generating a whitelist takes a lot of time to complete, resulting in a small number of pre-processed words or words in the whitelist in the usage scenario, which may cause some speech in the usage scenario to be unable to be accurately recognized, thereby affecting the generation accuracy of the text content corresponding to the speech signal.

[0053] In addition, in the related art, the recognized text is usually processed based on grammar rule analysis, which requires writing a large number of grammar rules. The writing of grammar rules is relatively cumbersome, so a lot of preliminary preparation is required to complete the subsequent processing of the text.

[0054] The embodiments of the present application provide a text generation method, device, electronic device, computer-readable storage medium and computer program product, which can improve the success rate of generating text content corresponding to the voice signal. The exemplary application of the electronic device for text generation provided by the embodiments of the present application is described below. The electronic device provided by the embodiments of the present application can be implemented as various types of terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers. Below, the exemplary application of the electronic device when it is implemented as a server will be described.

[0055] See also Figure 1 , Figure 1 1 is a schematic diagram of the architecture of the text generation system provided in the embodiment of the present application. In order to support a text generation application, in the text generation system 100, the terminal (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 via the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two. In the text generation system 100, a database 500 is also provided for providing data support to the server 200. Among them, the database 500 can be independent of the server 200, and can also be configured in the server 200. Figure 1 The illustrated case is that the database 500 is independent of the server 200 .

[0056] Terminal 400-1 is used to respond to the user's triggering operation of the voice collection mark on the graphical interface 410-1, call the audio collection device, collect the user's voice signal to be processed, and send the voice signal to be processed to the server 200 through the network 300; similarly, terminal 400-2 is used to respond to the user's triggering operation of the voice collection mark on the graphical interface 410-2, call the audio collection device, collect the user's voice signal to be processed, and transmit the voice signal to be processed to the server 200 through the network 300.

[0057] The server 200 is used to perform frame processing on the voice signal to be processed to obtain multiple voice frames of the voice signal to be processed; perform feature extraction on each voice frame to obtain frame features corresponding to each voice frame; fuse multiple frame features to obtain voice features of the voice signal to be processed; perform text conversion on the voice signal to be processed based on the voice features to obtain matching text corresponding to the voice signal to be processed; and send the matching text to the terminal 400-1 and the terminal 400-2.

[0058] Terminal 400 - 1 and terminal 400 - 2 are further used to display the converted text on graphical interface 410 - 1 and graphical interface 410 - 2 , respectively.

[0059] The embodiments of the present application can be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to achieve data computing, storage, processing, and sharing.

[0060] Cloud computing is a general term for network technology, information technology, integration technology, management platform, and application technology based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The system background services of the technical network require a large amount of computing and storage resources, which need to be realized through cloud computing.

[0061] Exemplarily, server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals 400-1 and 400-2 may be smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, etc., but are not limited thereto. The terminals and servers may be directly or indirectly connected via wired or wireless communications, which is not limited in the embodiments of the present application.

[0062] See also Figure 2 , Figure 2 The embodiment of this application provides Figure 1 A schematic diagram of the structure of a server (an implementation of an electronic device) in FIG. Figure 2 The server 200 shown includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0063] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0064] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0065] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0066] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0067] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0068] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0069] A network communication module 252, used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0070] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., display screen, speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripherals and displaying content and information);

[0071] The input processing module 254 is used to detect one or more user inputs or interactions from one of the one or more input devices 232 and translate the detected inputs or interactions.

[0072] In some embodiments, the text generation device provided in the embodiments of the present application can be implemented in software. Figure 2 The text generation device 255 stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a frame processing module 2551, a feature extraction module 2552, a feature fusion module 2553, a text conversion module 2554, a text optimization module 2555, a signal optimization module 2556 and a model generation module 2557. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0073] In other embodiments, the text generation device provided in the embodiments of the present application can be implemented in hardware. As an example, the text generation device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the text generation method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0074] In some embodiments, the terminal or server can implement the text generation method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a recording APP or an instant messaging APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0075] The embodiments of the present application can be applied to text generation scenarios such as speech recognition and speech-to-text conversion. Below, the text generation method provided by the embodiments of the present application is described. As mentioned above, the electronic device implementing the text generation method of the embodiments of the present application can be implemented as a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0076] Below, the text generation method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.

[0077] See also Figure 3 , Figure 3 This is a flowchart of the text generation method provided in the embodiment of the present application. Figure 1 , will combine Figure 3 The steps shown are explained.

[0078] S101 . Perform frame processing on a speech signal to be processed to obtain a plurality of speech frames of the speech signal to be processed.

[0079] The embodiment of the present application is implemented in the scenario of converting a voice signal into text content. Among them, the voice signal to be processed refers to the voice signal that is being processed for text generation at the current moment. After obtaining the voice to be processed, the electronic device will first divide the continuous voice to be processed into short time windows. The signal change rule in each short time window is relatively simple, which is convenient for analysis and processing, so that the signal in each short time window can be called a voice frame, so that the electronic device can obtain multiple voice frames.

[0080] In the embodiment of the present application, the voice signal to be processed can be received from the user's terminal, or can be downloaded from the network, which is not limited in the embodiment of the present application. The voice signal to be processed can include voices of multiple different objects, for example, the voices of multiple participants in a meeting, or the voices of a speaker and a questioner in a speech, etc., which is not limited in the embodiment of the present application.

[0081] It should be noted that the speech signal to be processed in the embodiment of the present application may be an unprocessed original speech signal, or may be a speech signal obtained after a series of preprocessing, such as noise reduction and filtering, on the original speech signal.

[0082] In the embodiment of the present application, the electronic device can divide the voice signal to be processed according to the same time length, for example, uniformly divide the voice signal to be processed according to 25ms, so that the length of each voice frame is 25ms. The electronic device can also divide the voice signal according to different time lengths, for example, divide the voice signal to be processed alternately according to 25ms and 20ms, so that the voice frame can have two lengths of 25ms and 20ms.

[0083] It should be noted that different speech frames may or may not overlap, and this is not limited in the embodiments of the present application.

[0084] S102: Extract features from each speech frame to obtain frame features corresponding to each speech frame.

[0085] After obtaining multiple speech frames, the electronic device can extract features from each speech frame in sequence, or extract features from each speech frame in parallel, and use the obtained audio features as frame features of the speech frame. In other words, the frame feature refers to the audio feature of each speech frame, which is used to describe the characteristics of each speech frame.

[0086] It is understandable that the electronic device can extract features from the speech frame in the time domain to obtain corresponding frame features, and can also extract features from the speech frame in the frequency domain to obtain corresponding frame features.

[0087] See also Figure 4 , Figure 4 This is a flowchart of the text generation method provided in the embodiment of the present application. Figure 2 In some embodiments of the present application, Figure 3 In S102, feature extraction is performed on each speech frame to obtain frame features corresponding to each speech frame, which can be achieved by the following processing:

[0088] S1021. Convert each speech frame from the time domain to the frequency domain to obtain spectrum information of each speech frame.

[0089] The electronic device converts each speech frame from the time domain to the frequency domain to obtain the signal of each speech frame in the frequency domain, that is, to obtain the spectrum information of the speech frame. The spectrum information is used to describe the characteristics or image of the speech frame in the frequency domain.

[0090] It is understandable that the electronic device can convert the speech frame from the time domain signal to the frequency domain signal through fast Fourier transform, Fourier transform or Laplace transform.

[0091] S1022: Divide the spectrum information into at least one frequency band, and calculate a weighted average value of energy in each frequency band.

[0092] After obtaining the spectrum information of the speech frame, the electronic device divides the spectrum information into one or more different frequency bands, and then performs weighted averaging on at least two energy values ​​in each frequency band. The result obtained is the weighted average value of the energy of each frequency band.

[0093] Among them, the electronic device can select the maximum energy value and the minimum energy value in each frequency band for weighted averaging, or can select several energy values ​​for weighted averaging, which is not limited in the embodiments of the present application. The weighted weight of each energy value can be randomly assigned or a set value, which is not limited in the embodiments of the present application.

[0094] S1023: Perform a logarithmic operation on the weighted average value of the energy in each frequency band, and integrate the logarithmic operation result of at least one frequency band into a frame feature of each speech frame.

[0095] The electronic device performs a logarithmic operation on the weighted average value of the energy to obtain a logarithmic operation result of the weighted average value of the energy in each frequency band, so that the electronic device can obtain at least one logarithmic operation result. Afterwards, the electronic device can integrate these logarithmic operation results in a vector manner or in a matrix manner, and use the integrated result as a frame feature of each speech frame.

[0096] It should be noted that when the electronic device performs logarithmic operation on the weighted average, the base may be 10, a natural number e, or any set value, which is not limited in the embodiment of the present application.

[0097] In other embodiments of the present application, Figure 3 In S102, feature extraction is performed on each speech frame to obtain frame features corresponding to each speech frame. This can also be achieved by the following processing: time domain features are calculated for each speech frame, and the obtained time domain features are determined as frame features of each speech frame.

[0098] It should be noted that the time domain features may include at least one of short-time energy, short-time average zero-crossing rate, short-time autocorrelation coefficient and short-time average amplitude difference. Of course, the time domain features may also include other features, such as the duration of a speech frame, etc., which are not limited in the embodiments of the present application.

[0099] S103: Fusing multiple frame features to obtain speech features of the speech signal to be processed.

[0100] The electronic device fuses multiple frame features corresponding to multiple speech frames, and the obtained fusion result can include the characteristics of each speech frame of the speech signal to be processed, so that the speech signal to be processed can be described globally. Therefore, the electronic device will determine the fusion result as the speech feature of the speech signal to be processed.

[0101] It should be noted that the electronic device can fuse multiple frame features by splicing, and in this case, the voice feature is the splicing result of the multiple frame features. The electronic device can also fuse multiple frame features by weighted summation, and in this case, the voice feature is the weighted summation result of the multiple frame features.

[0102] S104: Based on the voice features, perform text conversion on the voice signal to be processed to obtain a matching text corresponding to the voice signal to be processed.

[0103] After obtaining the speech features, the electronic device will model the speech signal to be processed based on the speech features and predict the text sequence involved in the speech signal to be processed, so as to obtain a match for the speech signal to be processed to complete the text generation process.

[0104] It should be noted that, in the embodiment of the present application, the complete speech features of the speech signal to be processed are used for text prediction, which can realize text prediction based on the global features of the speech signal, that is, the features involved in text prediction carry contextual information, thereby improving the generation accuracy of the text corresponding to the speech signal.

[0105] See also Figure 5 , Figure 5 This is a flowchart of the text generation method provided in the embodiment of the present application. Figure 3 In some embodiments of the present application, Figure 3 S104 in the above, i.e., performing text conversion on the speech signal to be processed based on the speech feature to obtain the matching text corresponding to the speech signal to be processed, can be achieved by the following processing:

[0106] S1041. Utilizing the state prediction model, predict the occurrence probability corresponding to each state sequence in the state sequence set from the speech features.

[0107] When the electronic device converts the processed speech signal into text based on the speech features, it first obtains the state prediction model, which is used to predict the probability of different state sequences from the speech features, thereby obtaining the probability of occurrence of each state sequence. Among them, the state sequence set exhaustively lists all state sequences that may appear in the future, and the state sequence is a sequence of pronunciation states, so that the state sequence set actually provides all candidate sequences of pronunciation states, and the occurrence probability indicates how likely the sequence of pronunciation states is to appear.

[0108] It should be noted that in the embodiment of the present application, the state sequence set is associated with the usage scenario, that is, the state sequence set is established based on the usage scenario. For example, when the usage scenario is a conference recording scenario, the state sequence set is the state sequence of the pronunciations that may appear in all participants in the conference; when the usage scenario is a speech scenario, the state sequence set is the state sequence of the pronunciations that may appear in the speaker.

[0109] In some embodiments of the present application, the state prediction model is obtained by the following processing: feature extraction is performed on each training speech frame of the training speech to obtain training features, and multiple training features are fused to obtain training fusion features of the training speech; a state sequence set is generated for the pronunciation state of the usage scenario to which the training speech belongs, and a corresponding state transfer matrix is ​​generated; the corresponding observation set is created using the training fusion features, and an observation probability matrix is ​​established; the occurrence probability of each state sequence is calculated using the initial probability distribution, the state transfer matrix and the observation probability matrix; based on the occurrence probability, the expectation of the observation set on each state sequence is calculated, and the initial probability distribution, the state transfer matrix and the observation probability matrix are optimized based on the expectation, until the end of the training, the state prediction model is obtained by integrating the optimized initial probability distribution, the state transfer matrix and the observation probability matrix.

[0110] It should be noted that in the embodiment of the present application, the initial probability of each state in the initial probability distribution is the same, and the transition probability of each state in the state transfer matrix is ​​only related to the previous state. Therefore, the prerequisites for each state transition can be observed from the state transfer matrix.

[0111] The electronic device can calculate the expectation of observations combined with each state sequence according to the sum of probabilities, and optimize the initial probability distribution, state transfer matrix and observation probability matrix with the goal of maximizing the expectation, and use the optimization results of these three variables to form a state prediction model.

[0112] S1042. Filter out a target state sequence with the highest probability of occurrence from multiple state sequences in the state sequence set.

[0113] After obtaining the occurrence probability of each state sequence, the electronic device may sort the occurrence probabilities of multiple state sequences to determine the state sequence with the highest occurrence probability, and then screen the state sequence with the highest occurrence probability to obtain the target state sequence.

[0114] S1043: Perform text analysis on the target state sequence to obtain the matching text corresponding to the speech signal to be processed, and complete the text conversion of the speech signal to be processed.

[0115] After obtaining the target state sequence, the electronic device converts the target state sequence from the pronunciation state sequence into a text sequence, and the obtained text sequence is the matching text of the speech signal to be processed.

[0116] In some embodiments, Figure 5In S1043, i.e., performing text parsing on the target state sequence to obtain the matching text corresponding to the voice signal to be processed, and completing the text conversion of the voice signal to be processed, it can be achieved by the following processing: combining the pronunciation states in the target state sequence to obtain a phoneme sequence; performing character integration on the phonemes in the phoneme sequence to obtain multiple character information; connecting the multiple character information to obtain the matching text corresponding to the voice signal to be processed, and completing the text conversion of the voice signal to be processed.

[0117] That is, the electronic device will first integrate the pronunciation states in the target state sequence into individual phonemes, then integrate the phonemes into character information, and finally connect the obtained character information into text to obtain the final matching text. Among them, each phoneme can correspond to three pronunciation states in the target state sequence, or it can correspond to a single pronunciation state in the target state sequence. The electronic device can segment the phoneme sequence according to the starting phoneme and ending phoneme of different character information, and determine the corresponding character information.

[0118] In other embodiments of the present application, Figure 5 In S1043, i.e., performing text parsing on the target state sequence, completing the text conversion of the voice signal to be processed, and obtaining the matching text corresponding to the voice signal to be processed, it can also be achieved by the following processing: matching the target state sequence with multiple preset state sequences corresponding to multiple preset texts, and determining the preset text corresponding to the matched (i.e., the most similar) preset state sequence as the matching text of the voice signal to be processed, thereby completing the text conversion of the voice signal to be processed.

[0119] It is understandable that compared with the related art, the number of pre-processed words or white-listed words in the usage scenario is relatively small, resulting in some voices in the usage scenario being unable to be recognized, affecting the generation accuracy of the text content corresponding to the voice signal. In the embodiment of the present application, the electronic device will divide the voice signal to be processed into voice frames, and then use the frame features of different voice frames to fuse and generate the voice features of the entire voice signal to be processed, and automatically convert the voice signal to be processed into the corresponding matching text in combination with the global voice features. In this way, even if the usage scenario lacks pre-processed words or white-listed words, the voices in the usage scenario can be recognized, thereby improving the generation accuracy of the text content corresponding to the voice signal.

[0120] based on Figure 3 , see Figure 6 , Figure 6 This is a flowchart of the text generation method provided in the embodiment of the present application. Figure 4In some embodiments of the present application, after S104, that is, after performing text conversion on the speech signal to be processed based on the speech feature to obtain the matching text corresponding to the speech signal to be processed, the method may further include the processing of S105 as follows:

[0121] S105: Optimize the matching text to obtain the optimized matching text of the speech signal to be processed.

[0122] It should be noted that although the matching text obtained based on speech features can realize the expression of the speech signal to be processed in the form of text content, the matching text may not conform to the grammar, syntax structure or the user's understanding habits of text, resulting in low readability. Therefore, the matching text obtained needs to be further optimized in the future, such as sentence segmentation, proprietary vocabulary recognition, etc., to form a more readable complete text.

[0123] In some embodiments of the present application, Figure 6 S105, i.e., performing text optimization on the matching text to obtain the optimized matching text of the speech signal to be processed, can be achieved by the following processing: performing word segmentation processing on the matching text to obtain multiple word segmentation results; performing part-of-speech tagging on each word segmentation result to obtain the part-of-speech information of each word segmentation result; determining the dependency relationship between the multiple word segmentation results based on the part-of-speech information of each of the multiple word segmentation results; and performing text integration on the multiple word segmentation results according to the dependency relationship to obtain the optimized matching text of the speech signal to be processed, thereby completing the text optimization of the matching text.

[0124] Among them, the word segmentation processing for the matching text can be implemented by the forward maximum matching algorithm, that is, from front to back, the vocabulary existing in the dictionary is searched in the matching text. In this process, the electronic device can set the initial substring length, and perform word segmentation on the matching text according to the initial substring length. If the word segmentation result is not included in the dictionary, the substring length is reduced by 1, and then the word segmentation is performed again to determine whether the word segmentation result exists in the dictionary. This cycle is repeated until the matching text is segmented using the vocabulary existing in the dictionary, and multiple word segmentation results are obtained.

[0125] Afterwards, the electronic device will mark the part of speech of each word segmentation result, thereby obtaining the part of speech information of each word segmentation result. The part of speech information includes but is not limited to nouns, verbs, adjectives, adverbs, etc.

[0126] There is usually a dependency relationship between different parts of speech. For example, a verb depends on a subject and an object (both of which can be nouns), and a noun depends on an adjective or an article. Therefore, based on the part-of-speech information of different word segmentation results, the dependency relationship between different word segmentation results can be clarified, and the dependency relationship can basically determine the order of appearance of different words. Therefore, after clarifying the dependency relationship, the electronic device can reintegrate multiple word segmentations into text based on the dependency relationship to obtain optimized matching text.

[0127] In more detail, in some embodiments of the present application, multiple word segmentation results are textually integrated according to dependency relationships to obtain optimized matching text for the speech signal to be processed, which can be achieved through the following processing: determining the order of appearance between multiple word segmentation results based on the dependency relationship; arranging multiple word segmentation results according to the order of appearance to obtain optimized matching text for the speech signal to be processed.

[0128] Of course, in the case where the same segmentation result depends on different segmentation results, for example, a noun depends on two adjectives at the same time, it is still difficult to determine the order of appearance of these segmentation results based on the dependency relationship alone. In this case, the relative positions of the different segmentation results that the same segmentation result depends on in the original matching text can be combined to help determine the order of appearance of the different segmentation results that the segmentation result depends on. For example, if a segmentation result depends on adjective 1 and adjective 2 at the same time, the order of appearance of the two adjectives can be determined based on whether adjective 1 and adjective 2 are located before or after the segmentation result.

[0129] It can be understood that compared with the related art, which requires a lot of preliminary preparation to complete the subsequent processing of the text, in the embodiment of the present application, the matching text can be directly segmented and tagged with parts of speech, and different segmentation results can be segmented according to the dependency relationship based on the part-of-speech information corresponding to the segmentation results, and then the segmentation results are integrated in combination with the dependency flushing, so as to achieve the subsequent processing of the text without any preliminary preparation, thereby helping to improve the readability of the text content.

[0130] In some embodiments of the present application, before performing frame processing on the voice signal to be processed to obtain multiple voice frames of the voice signal to be processed, that is, before S101, the method may also include the following processing: performing signal optimization processing on the input voice to obtain the voice signal to be processed; wherein the input voice signal is collected by an audio collection device.

[0131] That is to say, in the embodiment of the present application, the voice signal to be processed is not directly collected, but guessed after optimizing the voice signal collected by the audio collection device, that is, the input voice signal. In this way, the quality of the voice signal to be processed can be guaranteed, so as to further improve the accuracy of generating the text content of the voice signal to be processed.

[0132] In some embodiments of the present application, signal optimization processing is performed on an input voice signal to obtain a voice signal to be processed, which can be achieved through the following processing: performing signal denoising processing on the input voice signal to obtain a denoised voice signal; estimating a signal spectrum and a noise spectrum from the denoised voice signal; suppressing environmental noise on the denoised voice signal based on the signal spectrum and the noise spectrum to obtain a valid voice signal; filtering the valid voice signal to obtain a filtering result, and determining the filtering result as the voice signal to be processed.

[0133] De-noising the input voice signal refers to removing the noise introduced by the audio acquisition device itself and the voice signal when it is transmitted to the voice signal. These noises are usually not collected by the audio acquisition device. Suppressing the environmental noise of the denoised voice signal is used to eliminate the noise collected by the audio acquisition device, that is, the environmental noise of the scene itself.

[0134] De-noising the input voice signal can be achieved by wavelet transform denoising, Fourier denoising, etc., which is not limited in the embodiments of the present application. For example, when wavelet transform denoising is used, the electronic device can increase the number of decomposition layers of the input voice signal as much as possible (i.e., make the decomposition more detailed and the number of decomposition layers more), thereby improving the effect of eliminating high-frequency interference signals in the input voice signal.

[0135] It should be noted that within a period of time at the beginning of the speech signal, there may be no valid speech (for example, no human voice), and the speech signal at this stage can be considered as pure environmental noise. Therefore, the noise spectrum can be estimated based on the signal at this stage, and then the electronic device can suppress the environmental noise by subtracting the signal spectrum from the noise spectrum, or superimposing a spectrum that can offset the noise spectrum, to obtain a valid speech signal. Finally, the electronic device can filter the valid speech signal through a filter, and the filtering result is the speech signal to be processed.

[0136] In some embodiments of the present application, filtering is performed on a valid speech signal to obtain a filtering result, which can be achieved through the following processing: filtering the filter input signal of the i-th round through the i-th round filter to obtain the i-th round filter output signal; calculating the signal error between the filtered output signal and the expected output signal; when the signal error is less than the error threshold, determining the i-th round filter output signal as the filtering result; when the signal error is greater than the error threshold, adjusting the parameters of the i-th round filter according to the signal error to obtain the i+1-th round filter.

[0137] It should be noted that, in the embodiment of the present application, i≥1, i is an integer, and the first round of filtering input signal is a valid voice signal. That is, the electronic device will start the first round of filtering from the valid voice signal, and in each subsequent round of filtering, the filtered output signal will be compared with the expected output signal to determine the signal error, and then by judging whether the signal error is less than the error threshold, it is determined whether to determine the filtered output signal as the filtering result, or to continue to adjust each round of filters to obtain a better filter. In this way, it can be ensured that the filtering result takes into account the expected output signal as much as possible, thereby ensuring the quality of the filtering result.

[0138] In the embodiments of the present application, the expected output signal is associated with the actual usage scenario. For example, when the usage scenario is a lecture scenario, the expected output signal should only be the speech waveform of the speaker or the superposition of the speech waveform of the questioner, and should not have other waveforms. When the usage scenario is a meeting scenario, the expected output signal should be the speech waveform of the participants, and should not have other waveforms.

[0139] It can be understood that in the embodiment of the present application, the electronic device can ensure the quality of the voice signal to be processed by performing signal optimization processing on the input voice signal and determining the processing result as the voice signal to be processed, thereby helping to improve the effect of text generation.

[0140] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0141] The embodiment of the present application is implemented in a scenario where a server generates corresponding text for a voice signal input by an external device (referred to as an audio acquisition device).

[0142] First, the server receives the audio signal (referred to as the input voice signal) input by the peripheral device and pre-processes the audio signal, including denoising, noise reduction, filtering and other processing. Among them, denoising is mainly used to remove noise in the audio signal to improve the signal-to-noise ratio. In the embodiment of the present application, wavelet transform denoising, Fourier transform denoising and adaptive filtering denoising can be used, among which the scale function of the wavelet transform is shown in formula (1) (in the embodiment of the present application, a higher decomposition layer can be selected for wavelet transform denoising to improve the filtering effect of high-frequency noise in the input voice); noise reduction refers to eliminating relatively stable noise in the scene, such as background noise, which can be achieved by spectral subtraction in linear noise reduction; filtering is used to remove signals within a specific frequency range, such as background noise, while retaining signals of other frequencies.

[0143]

[0144] Among them, a is the translation parameter, b is the scale parameter, and x(t) is the audio signal.

[0145] For example, Figure 7 It is a schematic diagram of the process of preprocessing an audio signal provided in an embodiment of the present application. First, the server will translate and obtain scale parameters 7-2 for the input audio signal 7-1, and then perform inverse scale transformation 7-3, as well as scaling and translation transformation 7-4, and then integrate the input audio signal over time t to obtain wavelet coefficients 7-5, determine the noise threshold 7-6 according to the statistical characteristics of the current wavelet coefficients, and then increase the number of decomposition layers 7-7 for the wavelet system function, and perform wavelet decomposition to achieve denoising; then, the server will continue to perform linear noise reduction 7-8 on the signal obtained after the denoising process, that is, estimate the spectrum of the signal and the noise 7-9, subtract the estimated noise spectrum from the spectrum of the signal to obtain the spectrum of the signal after denoising, and then obtain the denoised signal 7-10 through inverse transformation to achieve denoising; finally, the server will create a Butterworth bandstop filter 7-11, initialize the filter parameters and covariance, input the signal and the expected output signal into the Butterworth bandstop filter, and update the covariance matrix 7-12 and the filter parameters according to the error between the filter output signal and the expected output signal to minimize the error for filtering. The server repeats the filtering process and determines whether the error converges 7-13, and ends the preprocessing 7-14 if yes, otherwise continues to update the covariance matrix 7-12.

[0146] After completing the preprocessing, the server will first extract features from the speech signal and then model the speech signal using a sequence modeling algorithm.

[0147] For example, Figure 8It is a schematic diagram of the process of feature extraction of speech signals provided in an embodiment of the present application. The server will divide the preprocessed speech signal (referred to as the speech signal to be processed) into several frames through the Hamming window 8-1, each frame is 16ms long, and the overlapping part is half the frame length, and then perform fast Fourier transform 8-2 for each frame (referred to as a speech frame) to convert the time domain signal into a frequency domain signal 8-3; then divide each frequency into frequency bands 8-4 to obtain several frequency bands, perform weighted averaging 8-5 on the energy in each frequency band, and take the logarithm of the energy after weighted averaging to narrow the amplitude range, obtain the features of each frame (referred to as frame features), and remove frames below a certain threshold 8-6 according to the energy size of each frame. Finally, the server performs feature splicing 8-7 to obtain the feature representation of the preprocessed speech signal (referred to as speech features).

[0148] It should be noted that in Figure 8 During the process, the server can adjust parameters for each frame and then perform feature extraction to improve the feature extraction effect. The parameters here may include pre-emphasis coefficient, frame length, frame shift, number of filters, etc.

[0149] Next, the server will use the HMM model to model the probability distribution of the feature representation of the speech signal, and determine the state set according to the different pronunciation states or phonemes of the current scene (called the usage scene). The establishment of the HMM model includes hidden states and observed states. The hidden state is a state that cannot be directly observed, such as the speaker's speaking speed, emotions, etc., while the observed state is a state that can be directly observed, using the speaker's sound signal. The server creates an observation set based on the spliced ​​feature representation to determine the initial probability distribution. In the embodiment of the present application, it is assumed that the initial probability of each state is equal. At the same time, the server establishes an observation probability matrix to describe the probability of observing a certain observation state under a given state. In speech recognition, given a factor, the probability of observing a certain sound signal can be obtained by analyzing the speech data.

[0150] The server establishes a state transition probability matrix to represent the probability of transitioning from one state to another. Generally speaking, the state transition probability conforms to the Markov property, that is, the current state is only related to the previous state. Finally, the server uses the Gaussian mixture model to repeatedly iterate until the HMM model converges, and then determines the probability of the observed variables in the observation set under the current state. The HMM model established in this way can predict the given sequence. In more detail, when predicting, the probability distribution of the sequence can be calculated based on the observation sequence using a forward algorithm or a backward algorithm, so as to obtain the most likely state sequence (called the target state sequence) and the observation sequence to accurately obtain the text. The text obtained in this way has a mapping relationship with the hidden state, and since the hidden state in the state sequence can represent the speaker's speaking speed and emotions, if the text is subsequently restored to speech, the speaker's speaking speed and emotions can still be retained.

[0151] For example, Fig. 9 It is a schematic diagram of speech recognition using the HMM model provided in the embodiment of the present application. The state set 9-1 of the HMM model includes hidden states 9-11 and observed states 9-12. According to the observed states 9-12, an observation matrix 9-13 can be obtained for calculating probabilities 9-14. According to the state set, state transitions 9-15 can be determined. In combination with the Gaussian mixture model, the state transitions 9-15 and the calculated probabilities are iteratively calculated 9-16 until the HMM model converges 9-17. The converged HMM model is used to recognize and generate text 9-18 (called matching text) from speech.

[0152] Finally, the server will post-process the recognized text (matching text), such as converting pinyin to Chinese characters, breaking sentences, etc. In this process, the server will first segment the text. Here, the forward maximum matching algorithm can be used, that is, searching for words in the dictionary from front to back. The dictionary contains all the words for named entity recognition, such as names of people, places, etc. Through the dictionary, named entities can be recognized quickly, thereby reducing the time consumed in vocabulary recognition.

[0153] For example, for the text "Zhang X rejected the manager Su X's arrangement and won over Su X's popular person Hou X's friend", the dictionary will contain the names "Zhang X", "Su X" and "Hou X". The server will first set the initial length of the substring to 5, and output "Zhang X refused" in the first round of word segmentation, but this word is not included in the dictionary, so the length is reduced by 1. In the second round of word segmentation, it outputs "Zhang X refused", which is still not included in the dictionary, and continues to reduce 1. When "Zhang X" is matched, the next substring scan is started until the word segmentation is completed.

[0154] Next, the server will perform part-of-speech tagging on the word segmentation results, such as nouns, verbs, adjectives, adverbs, etc. Finally, when the server generates a complete sentence using the word segmentation results, it will first perform syntactic analysis and grammatical analysis, create a sentence (called optimized matching text), and start indexing until the entire sentence is determined. In an embodiment of the present application, dependency syntactic analysis can be used to analyze the sentence into a dependency syntactic tree to describe the dependency relationship between each word, thereby providing a syntactic collocation relationship between different words, which is not only convenient for the regeneration of the sentence, but also can be combined with the dependency relationship between different words to achieve sentiment analysis of the sentence based on word segmentation.

[0155] For example, Fig.10 It is a schematic diagram of the dependency syntax tree provided in an embodiment of the present application. The central word of the sentence is the verb "prefer", which depends on the subject (nsubj) "I" and the direct object (dobj) "flight". When the subject "I" is detected, the server creates a sentence and associates it with (d obj) "flight" through the constructed root node (root). "Flight" depends on the definite article (det) "the", the noun modifier (nmod) "m orning" and the place name modifier (nmod) "Denxxx", and "Denxxx" depends on (case) "through". Although there is no lexical dependency on "prefer", a special word "root" is constructed to depend on "prefer". At this point, the index is completed and the sentence construction is completed.

[0156] It is understandable that in the embodiments of the present application, user information is involved, such as voice signals, matching text and other related data. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0157] The following is a description of an exemplary structure of the text generation device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the text generation device 255 of the memory 250 may include:

[0158] A frame processing module 2551 is used to perform frame processing on the speech signal to be processed to obtain multiple speech frames of the speech signal to be processed;

[0159] The feature extraction module 2552 is used to extract features from each of the speech frames to obtain frame features corresponding to each of the speech frames;

[0160] A feature fusion module 2553 is used to fuse the multiple frame features to obtain the speech features of the speech signal to be processed;

[0161] The text conversion module 2554 is used to perform text conversion on the voice signal to be processed based on the voice features to obtain a matching text corresponding to the voice signal to be processed.

[0162] In some embodiments of the present application, the text conversion module 2554 is also used to use a state prediction model to predict the occurrence probability corresponding to each state sequence in the state sequence set from the speech features; wherein the state sequence set is used to provide a sequence of all candidate pronunciation states; from the multiple state sequences in the state sequence set, a target state sequence with the highest occurrence probability is screened; the target state sequence is subjected to text parsing to obtain the matching text corresponding to the speech signal to be processed, thereby completing the text conversion of the speech signal to be processed.

[0163] In some embodiments of the present application, the text conversion module 2554 is also used to combine the pronunciation states in the target state sequence to obtain a phoneme sequence; perform character integration on the phonemes in the phoneme sequence to obtain multiple character information; connect the multiple character information to obtain the matching text corresponding to the voice signal to be processed, and complete the text conversion of the voice signal to be processed.

[0164] In some embodiments of the present application, the feature extraction module 2552 is also used to convert each of the speech frames from the time domain to the frequency domain to obtain the spectrum information of each of the speech frames; divide the spectrum information into at least one frequency band, and calculate the weighted average value of the energy in each of the frequency bands; perform logarithmic operation on the weighted average value of the energy in each of the frequency bands, and integrate the logarithmic operation results of at least one of the frequency bands into the frame features of each of the speech frames.

[0165] In some embodiments of the present application, the text generation device 255 further includes: a text optimization module 2555, which is used to perform text optimization on the matching text to obtain the optimized matching text of the speech signal to be processed.

[0166] In some embodiments of the present application, the text optimization module 2555 is also used to perform word segmentation processing on the matching text to obtain multiple word segmentation results; perform part-of-speech tagging on each of the word segmentation results to obtain part-of-speech information of each of the word segmentation results; determine the dependency relationship between the multiple word segmentation results based on the part-of-speech information of each of the multiple word segmentation results; according to the dependency relationship, perform text integration on the multiple word segmentation results to obtain the optimized matching text of the speech signal to be processed, and complete the text optimization of the matching text.

[0167] In some embodiments of the present application, the text optimization module 2555 is also used to determine the order of appearance between the multiple word segmentation results based on the dependency relationship; arrange the multiple word segmentation results according to the order of appearance to obtain the optimized matching text of the speech signal to be processed.

[0168] In some embodiments of the present application, the text generation device 255 also includes: a signal optimization module 2556, which is used to perform signal optimization processing on the input voice signal to obtain the voice signal to be processed; wherein the input voice signal is collected by an audio collection device.

[0169] In some embodiments of the present application, the signal optimization module 2556 is also used to perform signal denoising on the input voice signal to obtain the denoised voice signal; estimate the signal spectrum and the noise spectrum from the denoised voice signal; the noise spectrum refers to the spectrum of the background noise in the environment; based on the signal spectrum and the noise spectrum, suppress the environmental noise of the denoised voice signal to obtain a valid voice signal; filter the valid voice signal to obtain a filtering result, and determine the filtering result as the voice signal to be processed.

[0170] In some embodiments of the present application, the signal optimization module 2556 is also used to filter the filter input signal of the i-th round through the filter of the i-th round to obtain the filter output signal of the i-th round; wherein, i≥1, i is an integer, and the filter input signal of the first round is the valid speech signal; calculate the signal error between the filter output signal and the expected output signal; when the signal error is less than the error threshold, determine the filter output signal of the i-th round as the filtering result; when the signal error is greater than the error threshold, adjust the parameters of the filter of the i-th round according to the signal error to obtain the filter of the i+1th round.

[0171] In some embodiments of the present application, the text generation device 255 also includes: a model generation module 2557, which is used to obtain a state prediction model through the following processing: extracting features from each training speech frame of the training speech to obtain training features, and fusing multiple training features to obtain training fusion features of the training speech; generating a state sequence set for the pronunciation state of the usage scenario to which the training speech belongs, and generating a corresponding state transfer matrix; using the training fusion features to create a corresponding observation set, and establish an observation probability matrix; using the initial probability distribution, the state transfer matrix and the observation probability matrix, the occurrence probability of each state sequence is calculated; wherein the initial probability of each state in the initial probability distribution is the same, and the transition probability of each state in the state transfer matrix is ​​only related to the previous state; based on the occurrence probability, calculating the expectation of the observation set on each state sequence, and optimizing the initial probability distribution, the state transfer matrix and the observation probability matrix based on the expectation, until the end of the training, the state prediction model is obtained by integrating the optimized initial probability distribution, the state transfer matrix and the observation probability matrix.

[0172] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the text generation method described in the embodiment of the present application.

[0173] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the text generation method provided by the embodiment of the present application, for example, Figure 3 The text generation method shown.

[0174] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0175] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0176] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0177] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0178] To summarize, through the embodiments of the present application, the electronic device will divide the voice signal to be processed into voice frames, and then use the frame features of different voice frames to fuse and generate the voice features of the entire voice signal to be processed, and combine the global voice features to automatically convert the voice signal to be processed into the corresponding matching text. In this way, even if the usage scenario lacks pre-processed vocabulary or white-list vocabulary, the voice in the usage scenario can be recognized, thereby improving the generation accuracy of the text content corresponding to the voice signal; the matching text can be directly segmented and tagged with parts of speech, and different segmentation results can be segmented according to the dependency relationship based on the part-of-speech information corresponding to the segmentation results, and then the segmentation results can be integrated in combination with the dependency flushing to achieve subsequent processing of the text without any preliminary preparation work, thereby helping to improve the readability of the text content.

[0179] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A text generation method, characterized in that: The method comprises: Performing frame processing on the speech signal to be processed to obtain multiple speech frames of the speech signal to be processed; Extracting features from each speech frame to obtain frame features corresponding to each speech frame; Fusing the plurality of frame features to obtain speech features of the speech signal to be processed; Based on the voice features, text conversion is performed on the voice signal to be processed to obtain a matching text corresponding to the voice signal to be processed.

2. The method according to claim 1, characterized in that The step of performing text conversion on the voice signal to be processed based on the voice feature to obtain a matching text corresponding to the voice signal to be processed includes: Using the state prediction model, predicting the occurrence probability corresponding to each state sequence in the state sequence set from the speech features; wherein the state sequence set is used to provide a sequence of all candidate pronunciation states; Selecting a target state sequence with the highest probability of occurrence from a plurality of state sequences in the state sequence set; The target state sequence is subjected to text parsing to obtain the matching text corresponding to the speech signal to be processed, thereby completing the text conversion of the speech signal to be processed.

3. The method according to claim 2, characterized in that The text parsing of the target state sequence to obtain the matching text corresponding to the speech signal to be processed and completing the text conversion of the speech signal to be processed includes: Combining the pronunciation states in the target state sequence to obtain a phoneme sequence; Performing character integration on the phonemes in the phoneme sequence to obtain a plurality of character information; Connect the plurality of character information to obtain the matching text corresponding to the voice signal to be processed, and complete the text conversion of the voice signal to be processed.

4. The method according to any one of claims 1 to 3, characterized in that: The extracting features of each speech frame to obtain frame features corresponding to each speech frame includes: Convert each of the speech frames from the time domain to the frequency domain to obtain frequency spectrum information of each of the speech frames; Dividing the spectrum information into at least one frequency band, and calculating a weighted average value of energy in each frequency band; A logarithmic operation is performed on the weighted average value of the energy in each of the frequency bands, and the logarithmic operation result of at least one of the frequency bands is integrated into the frame feature of each of the speech frames.

5. The method according to any one of claims 1 to 3, characterized in that: After performing text conversion on the voice signal to be processed based on the voice feature to obtain a matching text corresponding to the voice signal to be processed, the method further includes: Text optimization is performed on the matching text to obtain an optimized matching text of the speech signal to be processed.

6. The method according to claim 5, characterized in that The performing text optimization on the matching text to obtain the optimized matching text of the speech signal to be processed includes: Performing word segmentation processing on the matching text to obtain multiple word segmentation results; Performing part-of-speech tagging on each of the word segmentation results to obtain part-of-speech information of each of the word segmentation results; Determining a dependency relationship between the plurality of word segmentation results according to the part-of-speech information of each of the plurality of word segmentation results; According to the dependency relationship, the multiple word segmentation results are text-integrated to obtain the optimized matching text of the speech signal to be processed, thereby completing the text optimization of the matching text.

7. The method according to claim 6, characterized in that The step of integrating the multiple word segmentation results according to the dependency relationship to obtain an optimized matching text of the speech signal to be processed includes: Determining the order of appearance of the plurality of word segmentation results according to the dependency relationship; Arrange the plurality of word segmentation results according to the order of appearance to obtain the optimized matching text of the speech signal to be processed.

8. The method according to any one of claims 1 to 3, characterized in that: Before performing frame processing on the speech signal to be processed to obtain a plurality of speech frames of the speech signal to be processed, the method further includes: Signal optimization processing is performed on the input voice signal to obtain the voice signal to be processed; wherein the input voice signal is collected by an audio collection device.

9. The method according to claim 8, characterized in that The step of performing signal optimization processing on the input voice signal to obtain the voice signal to be processed includes: Performing signal denoising processing on the input speech signal to obtain the denoised speech signal; From the denoised speech signal, a signal spectrum and a noise spectrum are estimated; the noise spectrum refers to the spectrum of background noise in the environment; Based on the signal spectrum and the noise spectrum, suppressing the environmental noise of the denoised speech signal to obtain a valid speech signal; The effective speech signal is filtered to obtain a filtering result, and the filtering result is determined as the speech signal to be processed.

10. The method according to claim 9, characterized in that The filtering of the effective speech signal to obtain a filtering result includes: Performing filtering processing on the filter input signal of the i-th round through the filter of the i-th round to obtain the filter output signal of the i-th round; wherein i≥1, i is an integer, and the filter input signal of the first round is the valid speech signal; Calculating a signal error between the filtered output signal and a desired output signal; When the signal error is less than the error threshold, determining the filtered output signal of the i-th round as the filtering result; When the signal error is greater than the error threshold, the parameters of the filter of the i-th round are adjusted according to the signal error to obtain the filter of the (i+1)-th round.

11. The method according to claim 2, characterized in that The state prediction model is obtained by the following processing: Extracting features from each training speech frame of the training speech to obtain training features, and fusing a plurality of the training features to obtain training fusion features of the training speech; Generate a state sequence set and a corresponding state transfer matrix for the pronunciation state of the usage scenario to which the training speech belongs; Using the training fusion features to create a corresponding observation set and establish an observation probability matrix; The occurrence probability of each state sequence is calculated using the initial probability distribution, the state transfer matrix and the observation probability matrix; wherein the initial probability of each state in the initial probability distribution is the same, and the transition probability of each state in the state transfer matrix is ​​only related to the previous state; Based on the occurrence probability, the expectation of the observation set in each state sequence is calculated, and the initial probability distribution, the state transfer matrix and the observation probability matrix are optimized based on the expectation until the end of training, and the state prediction model is obtained by integrating the optimized initial probability distribution, state transfer matrix and observation probability matrix.

12. A text generation device, characterized in that: The device comprises: A frame processing module, used for performing frame processing on the speech signal to be processed to obtain multiple speech frames of the speech signal to be processed; A feature extraction module is used to extract features from each of the speech frames to obtain frame features corresponding to each of the speech frames; A feature fusion module, used for fusing the plurality of frame features to obtain the speech feature of the speech signal to be processed; The text conversion module is used to perform text conversion on the speech signal to be processed based on the speech feature to obtain a matching text corresponding to the speech signal to be processed.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the text generation method according to any one of claims 1 to 11 when executing the computer executable instructions stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the text generation method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the text generation method according to any one of claims 1 to 11 is implemented.

Images