Information processing method and equipment

Abstract

The embodiment of the invention discloses an information processing method and equipment. The information processing method is applied to the information processing equipment comprising an information processing model. The method comprises the steps that first information of a user is received, the first information comprises first audio frequency, and the first audio frequency is audio frequency to be repaired; the information processing model obtains second information according to the first information, the second information comprises a first semantic feature corresponding to the first audio, and the first semantic feature is a semantic feature obtained after semantic restoration is carried out on a first frame number of audio in the first audio; and the information processing model obtains a second audio according to the voiceprint feature of the user and the second information. And real-time sound restoration can be realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an information processing method and device. Background Art

[0002] There are some special groups in society whose speech is difficult to understand due to congenital and acquired factors, which greatly affects their daily social life. Although sound restoration technology can convert difficult-to-understand speech of hearing-impaired people into more intelligible speech and keep the timbre consistent with the original person, the delay is at the sentence level, and real-time sound restoration cannot be achieved. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides an information processing method and device, which can realize real-time sound restoration.

[0004] In a first aspect, an embodiment of the present invention provides an information processing method, which is applied to an information processing device including an information processing model; the method comprises:

[0005] Receive first information from a user, where the first information includes first audio, and the first audio is audio to be repaired;

[0006] The information processing model obtains second information according to the first information, where the second information includes a first semantic feature corresponding to the first audio, where the first semantic feature is a semantic feature after semantic restoration is performed on audio of a first number of frames in the first audio;

[0007] The information processing model obtains the second audio according to the voiceprint feature of the user and the second information.

[0008] In the information processing method provided by the embodiment of the present invention, the information processing model outputs the first semantic feature of the first audio. Semantic features refer to the features of vocabulary, grammar, pronunciation, intonation, etc. in a language, which reflect the structure, function, evolution and other laws of the language, and can reveal the intention, emotion, topic and other information of the speaker or author. Among them, the first semantic feature is generated based on the audio of the first frame number and converted into the second audio by the information processing model, so the delay generated in the process of converting the first audio into the second audio is a delay at the frame level.

[0009] Audio data is streamed. In practical applications, for the convenience of audio algorithm processing and transmission, it is generally agreed that one frame of audio is 2.5ms to 60ms. Usually, the normal speaking speed is about 3 to 4 words per second.

[0010] Therefore, the delay at the frame level is much smaller than the delay at the sentence level, so real-time sound restoration can be achieved.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, before receiving the first information of the user, the method further includes:

[0012] receiving a third audio of the user, where the third audio is used to register a voiceprint;

[0013] The information processing model obtains the voiceprint feature according to the third audio.

[0014] The information processing method provided in the embodiment of the present invention needs to record a sentence of the user to register the voiceprint and extract the voiceprint features of the user in advance before performing voice restoration. Therefore, the user only needs to record one sentence in advance for registration to use the information processing model in the embodiment of the present invention for voice restoration.

[0015] In combination with the first aspect, in some implementations of the first aspect, the information processing model obtains the second information according to the first information, including:

[0016] The information processing model obtains the first semantic feature according to the first audio.

[0017] For example, in a voice call scenario, the input is only the first audio; the information processing model obtains the first semantic feature based on the first audio.

[0018] In combination with the first aspect, in some implementations of the first aspect, the second information further includes a first text corresponding to the first semantic feature;

[0019] The information processing model obtains second information according to the first information, including:

[0020] The information processing model obtains the first semantic feature and the first text according to the first audio.

[0021] For example, in a voice call scenario, the input is only the first audio; the information processing model not only obtains the first semantic feature based on the first audio, but also outputs the first text at the same time, so that subtitles can be generated based on the first text later. In a real-time scenario, the text can be given synchronously to assist the user in understanding the repaired audio content. There is no need for an additional speech recognition model, and the text will not lag behind the repaired audio.

[0022] In combination with the first aspect, in certain implementations of the first aspect, the first information also includes a first video corresponding to the first audio; the second information also includes a second semantic feature corresponding to the first video, and the second semantic feature is a semantic feature after semantic restoration of an image of a second frame number in the first video.

[0023] For example, in a video call scenario, the input includes not only the first audio but also the first video, and the first video contains the user's facial image or lip image. Therefore, the information processing model can also obtain the second semantic feature based on the first video. Among them, the second semantic feature is generated based on the video image of the second frame number. Therefore, the delay generated in the process of converting the first audio and the first video into the second audio is a delay at the frame level, which is much lower than the sentence level delay, thereby achieving real-time sound restoration.

[0024] Exemplarily, the second frame number is equal to or unequal to the first frame number.

[0025] Preferably, the second frame number is equal to the first frame number, so that feature fusion can be performed directly later.

[0026] If the second frame number is not equal to the first frame number, the first semantic feature and the second semantic feature need to be processed before subsequent feature fusion so that the frame numbers of the fused features are equal.

[0027] In combination with the first aspect, in some implementations of the first aspect, the information processing model obtains the second information according to the first information, including:

[0028] The information processing model obtains the first semantic feature according to the first audio;

[0029] The information processing model obtains the second semantic feature according to the first video.

[0030] For example, in a video call scenario, the input includes a first audio and a first video. The information processing model obtains a first semantic feature based on the first audio and obtains a second semantic feature based on the first video.

[0031] In combination with the first aspect, in some implementations of the first aspect, the second information further includes a first text;

[0032] The information processing model obtains second information according to the first information, including:

[0033] The information processing model obtains the first semantic feature and the first text according to the first audio, and the first text corresponds to the first semantic feature;

[0034] The information processing model obtains the second semantic feature according to the first video;

[0035] or

[0036] The information processing model obtains second information according to the first information, including:

[0037] The information processing model obtains the first semantic feature and the second text according to the first audio;

[0038] The information processing model obtains the second semantic feature and the first text according to the first video and the second text, and the first text corresponds to the first semantic feature and the second semantic feature.

[0039] For example, in a video call scenario, the input includes the first audio and the first video. The information processing model obtains the first semantic feature based on the first audio and the second semantic feature based on the first video. At the same time, the information processing model also outputs the first text so that subtitles can be generated based on the first text. In a real-time scenario, the text can be given synchronously to assist the user in understanding the restored audio content. No additional speech recognition model is required, and the text will not lag behind the restored audio.

[0040] The first text may be obtained based on the first audio only, or may be obtained based on the first audio and the first video together.

[0041] In combination with the first aspect, in some implementations of the first aspect, the information processing model obtains the second audio according to the voiceprint feature of the user and the second information, including:

[0042] The information processing model obtains a first fusion feature according to the voiceprint feature and the second information;

[0043] The information processing model obtains the second audio according to the first fusion feature.

[0044] In the embodiment of the present invention, before the information processing model obtains the second audio according to the voiceprint feature and the second information, it first obtains the first fusion feature according to various features in the voiceprint feature and the second information, and then converts the first fusion feature into the second audio.

[0045] In combination with the first aspect, in some implementations of the first aspect, the information processing model obtains a first fusion feature according to the voiceprint feature and the second information, including:

[0046] The information processing model performs feature fusion on the voiceprint feature and the second information to obtain a second fused feature;

[0047] The information processing model performs audio restoration on the second fusion feature to obtain the first fusion feature.

[0048] In an embodiment of the present invention, the information processing model includes an audio repair module. The audio repair module includes: a fusion module and a streaming audio repair module. The information processing model first uses the fusion module to fuse the voiceprint feature and various features in the second information to obtain a second fused feature, and then uses the streaming audio repair module to perform audio repair on the second fused feature to obtain a first fused feature.

[0049] In combination with the first aspect, in some implementations of the first aspect, the information processing model obtains the first fusion feature according to the voiceprint feature and the second information, including: a first reasoning process or an i-th reasoning process, where i is a natural number greater than 1;

[0050] The first reasoning process includes:

[0051] The information processing model performs feature fusion on the voiceprint feature and the second information to obtain a second fused feature;

[0052] The information processing model performs audio restoration on the second fused feature to obtain the first fused feature;

[0053] The i-th reasoning process includes:

[0054] The information processing model performs speech discretization processing on the first fusion feature obtained by the i-1th reasoning to obtain a third fusion feature;

[0055] The information processing model obtains a fourth fusion feature by passing the third fusion feature through the first layer;

[0056] The information processing model performs feature fusion on the voiceprint feature, the second information and the fourth fusion feature to obtain a fifth fusion feature;

[0057] The information processing model performs audio restoration on the fifth fusion feature to obtain the first fusion feature.

[0058] In the embodiment of the present invention, the information processing model includes an audio repair module, and the audio repair module includes: a fusion module, a streaming audio repair module, a discretized speech module and an embedding layer.

[0059] Exemplarily, the streaming audio repair module adopts a generative pre-trained model (GPT) structure. Since the streaming audio repair module adopts the GPT structure, it needs to be matched with the discretized speech module and embedding layer corresponding to the vector quantized variational autoencoder (VQVAE) model for autoregressive reasoning.

[0060] During the autoregressive inference process, the audio restoration module fuses the fourth fusion feature obtained from the previous inference, predicts the fourth fusion feature of the next frame, and gives the corresponding first fusion feature. The first fusion feature can be the probability distribution feature vector of the aforementioned discrete feature, or it can be the feature of the bottleneck layer near the output layer.

[0061] In the autoregressive reasoning process, the fourth fusion feature obtained in the previous reasoning is not available in the first reasoning process, so the voiceprint feature and various features in the second information are directly fused, and then the audio is repaired to obtain the first fusion feature. In the i-th reasoning process, the fourth fusion feature obtained in the previous reasoning exists, so the voiceprint feature, various features in the second information and the fourth feature are fused, and then the audio is repaired to obtain the first fusion feature.

[0062] In combination with the first aspect, in some implementations of the first aspect, the information processing model performs audio restoration on the second fusion feature to obtain the first fusion feature, including:

[0063] The information processing model uses the prosody information to perform audio restoration on the second fused feature to obtain the first fused feature; the prosody information is a prosody feature extracted from the fourth audio;

[0064] The information processing model performs audio restoration on the fifth fusion feature to obtain the first fusion feature, including:

[0065] The information processing model uses the prosody information to perform audio restoration on the fifth fusion feature to obtain the first fusion feature.

[0066] Exemplarily, the prosody information is a prosody feature extracted from the fourth audio. For example, a perceptual resampler model is used to extract the prosody feature from the fourth audio.

[0067] Exemplarily, the fourth audio is a normal person's speech, and normal people's speech is usually highly intelligible. Therefore, the intelligibility of the fourth audio is similar to that of the restored audio, which is higher than the intelligibility of the audio to be restored.

[0068] In the audio restoration process, the embodiments of the present invention use the prosody information to optimize the prosody of the second fusion feature or the fifth fusion feature.

[0069] In combination with the first aspect, in some implementations of the first aspect, after the information processing model obtains the second information according to the first information, the information processing model further includes:

[0070] Subtitles are generated based on the first text.

[0071] The embodiment of the present invention generates subtitles based on the first text. In a real-time scenario, the text can be synchronously provided to assist the user in understanding the restored audio content. No additional speech recognition model is required, and the text will not lag behind the restored audio.

[0072] In a second aspect, an embodiment of the present invention provides an information processing device, including a processor and a memory, wherein the memory is used to store a program, and when the processor runs the program, the information processing device executes the steps of the method described above.

[0073] In a third aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium stores a program, and when the program is executed by an information processing, the information processing is caused to execute the method as described above.

[0074] In a fourth aspect, an embodiment of the present invention provides a program product, which includes a program. When the program product is run on an information processing device or any at least one processor, the information processing device executes the functions / steps in the above method.

[0075] In the technical solution of the information processing method and device provided by the embodiment of the present invention, the information processing method is applied to an information processing device including an information processing model; the method includes: receiving first information of a user, the first information including first audio, the first audio being the audio to be repaired; the information processing model obtains second information according to the first information, the second information including first semantic features corresponding to the first audio, the first semantic features being semantic features after semantic repair of the audio of the first number of frames in the first audio; the information processing model obtains second audio according to the voiceprint features of the user and the second information. Real-time sound repair can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1a A flowchart of an information processing method is provided;

[0077] Figure 1b To adopt Figure 1a Schematic diagram of the sentence-level delay of sound production repaired by the information processing method;

[0078] Figure 2 A schematic diagram of the structure of a device provided by an embodiment of the present invention;

[0079] Figure 3 is a software structure block diagram of the device 100 according to an embodiment of the present invention;

[0080] Figure 4 A schematic diagram of an application scenario of an information processing model in an embodiment of the present invention;

[0081] Figure 5 is a schematic diagram of another application scenario of the information processing model in an embodiment of the present invention;

[0082] Figure 6A schematic diagram of the structure of an information processing model provided by an embodiment of the present invention;

[0083] Figure 7 for Figure 6 A specific structural diagram of the information processing model in FIG.

[0084] Figure 8 for Figure 6 Another specific structural diagram of the information processing model;

[0085] Fig. 9 A schematic diagram of the structure of an audio repair module provided by an embodiment of the present invention;

[0086] Fig.10 A schematic diagram of using an information processing model to repair the time delay of sound in real time in an embodiment of the present invention;

[0087] Fig.11 A flowchart of an information processing method provided by an embodiment of the present invention;

[0088] Fig.12 A flowchart of another information processing method provided by an embodiment of the present invention;

[0089] Fig.13 A schematic diagram of the structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0091] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0093] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0094] In order to better understand the embodiments of the present application, the terms or concepts that may be involved in the embodiments are explained below.

[0095] 1. People with special speech impairments: There are some special groups in society who have defects in their pronunciation due to congenital or acquired factors (such as hearing impairment, amyotrophic lateral sclerosis, damaged vocal cords, etc.), their speech is difficult to be understood by the listeners, and their semantics are difficult to be accurately understood by the listeners.

[0096] 2. Artificial Intelligence (AI): A new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0097] 3. Automatic Speech Recognition (ASR): technology that realizes the conversion from "sound" to "text".

[0098] 4. Text To Speech (TTS): A service that converts text into realistic speech. It provides personalized pronunciation services for businesses and individuals through voice selection, custom volume, and speech speed.

[0099] 5. Emphasized Channel Attention, Propagation and Aggregation in Time Delay Neural Network (ECAPA-TDNN): an industry-leading voiceprint extraction model.

[0100] 6. Recurrent Neural Network Transducer (RNNT): A neural network model for speech recognition.

[0101] 7. Generative Pre-trained Transformers (GPT): A neural network model structure commonly used in the field of artificial intelligence.

[0102] 8. Vector-Quantized Variational AutoEncoders (VQVAE): An industry-leading speech discretization model that converts audio into frame-level discretization units.

[0103] 9. Efficient and High-Fidelity Generative Adversarial Network (HiFi GAN): A vocoder neural network model that converts audio features into audio samples.

[0104] 10. Audio frame: Audio data is streamed, and there is no clear concept of frames. In practical applications, for the convenience of audio algorithm processing and transmission, it is generally agreed that a frame of audio is 2.5ms to 60ms. Usually, the normal speaking speed is about 3 to 4 words per second. Therefore, the delay at the frame level is much smaller than the delay at the sentence level.

[0105] In recent years, the AI ​​industry has developed rapidly. The ability of automatic speech recognition (ASR) is close to the level of normal people, the audio of text-to-speech (TTS) is gradually difficult to distinguish from real audio, and the intelligence of large AI models has emerged, which has greatly improved people's production and living standards.

[0106] However, there are some special groups in society who cannot use these AI products as smoothly as normal people due to congenital and acquired factors. Among them, the speech of people with special speech impairments is difficult to be understood by the listeners, which greatly affects their daily social life.

[0107] For the above-mentioned special speech-impaired people, in order to solve the problem of daily communication, an information processing method that assists language comprehension is needed. This method is required to be able to convert the difficult-to-understand speech of the hearing-impaired into a more intelligible speech and keep the timbre consistent with the original person.

[0108] However, the real-time restoration function for disabled sounds has not yet been implemented in the industry. The current sound restoration technology is unable to complete real-time restoration of disabled sounds and lacks effective algorithms and frameworks.

[0109] Figure 1a FIG. 1 is a flow chart of an information processing method. The information processing method is a cascade solution, that is, a cascade system using an ASR model and a TTS model. Figure 1a As shown, first the user pre-registers by recording voice so that the TTS model can extract the user's timbre information; when in use, after the user inputs the audio, it is converted into text through the ASR model, and then the TTS model converts the text into audio of the user's timbre and plays it out.

[0110] However, the text that the ASR model inputs to the TTS model must be at least a sentence, so that the TTS model can determine the pronunciation and intonation of each word in the sentence based on the text, thereby ensuring the accuracy of playback. Therefore, the time delay between the sound played by the TTS model and the user's spoken sound is at the sentence level, and real-time sound repair cannot be achieved.

[0111] For example, Figure 1b To adopt Figure 1a Schematic diagram of the sentence-level delay of sound generation repaired by the information processing method. Figure 1b As shown in the figure, suppose the user says "I am (si) Xiao Ming". Figure 1a The ASR model in the example first needs to output the complete text of the sentence "I am Xiao Ming" based on the audio of "I am (si) Xiao Ming", and then input the text of "I am Xiao Ming" into the TTS model, which then outputs the sound of "I am Xiao Ming" based on the text of "I am Xiao Ming". Since the ASR model needs to recognize the text of a whole sentence, it needs to wait until the user finishes speaking the whole sentence before outputting the text of the whole sentence, which results in a delay of a whole sentence; coupled with the delay generated during the TTS model processing, the delay between the sound played by the TTS model and the sound spoken by the user is greater than the delay of a sentence.

[0112] Therefore, information processing methods cannot achieve real-time sound restoration.

[0113] In view of the above technical problems, an embodiment of the present invention provides an information processing model that can realize real-time sound restoration. The information processing model provided by the embodiment of the present invention can be deployed in an information processing device.

[0114] Figure 2 A schematic diagram of the structure of a device provided in an embodiment of the present invention.

[0115] like Figure 2 As shown, the device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a user identification module card interface 195, etc.

[0116] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the device 100. In other embodiments of the present application, the device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0117] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor, etc. Different processing units may be independent devices or integrated into one or more processors.

[0118] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0119] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0120] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit interface, an integrated circuit built-in audio interface, a pulse code modulation interface, a universal asynchronous receiver and transmitter interface, a mobile industry processor interface, a general purpose input and output interface, a subscriber identity module interface, and / or a universal serial bus interface, etc.

[0121] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the device 100. In other embodiments of the present application, the device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0122] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0123] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0124] The wireless communication function of the device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0125] The device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0126] The device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0127] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0128] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory, etc. The processor 110 executes various functional applications and data processing of the device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0129] The device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0130] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The device 100 may receive key input and generate key signal input related to user settings and function control of the device 100.

[0131] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0132] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0133] The software system of the device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the device 100.

[0134] Figure 3 It is a software structure block diagram of the device 100 according to an embodiment of the present invention.

[0135] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime (Android runtime) and system library, and the kernel layer.

[0136] The application layer can include a series of application packages.

[0137] like Figure 3 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0138] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0139] like Figure 3 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0140] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0141] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0142] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0143] The phone manager is used to provide communication functions of the device 100, such as management of call status (including connection, disconnection, etc.).

[0144] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.

[0145] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0146] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0147] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0148] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0149] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0150] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0151] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0152] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0153] The 2D graphics engine is a drawing engine for 2D drawings.

[0154] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0155] The hardware structure and software structure of the information processing device provided by the embodiment of the present invention can be found in Figure 2 and Figure 3 For relevant description about device 100.

[0156] In some possible embodiments, the information processing device may be a server.

[0157] The server is a device with centralized computing capabilities. Exemplarily, the server can be implemented by a server, a virtual machine, a cloud, or a robot.

[0158] When the service end includes a server, the type of the server includes but is not limited to a general-purpose computer, a dedicated server computer, a blade server, etc. This application does not impose a strict limit on the number of servers included in the service end, and the number can be one or more (such as a server cluster, etc.).

[0159] A virtual machine is a computing module that runs in a completely isolated environment and has complete hardware system functions simulated by software. Of course, in addition to virtual machines, the server can also be implemented through other computing instances, such as containers.

[0160] The cloud is a software platform that uses application virtualization technology to enable one or more software and applications to be developed and run in an independent virtualized environment. Optionally, when the server is implemented through the cloud, the cloud can be deployed on a public cloud, private cloud, or hybrid cloud.

[0161] For example, the information processing device is the server. Figure 4 is a schematic diagram of an application scenario of the information processing model in an embodiment of the present invention, such as Figure 4 As shown, mobile phone 1 is connected to the server, and mobile phone 1 and mobile phone 2 are connected through a base station; mobile phone 1 and mobile phone 2 are in a voice call scene or a video call scene, and user 1 of mobile phone 1 (for example, a user with damaged vocal cords) clicks the "sound repair" control on the call interface of mobile phone 1. After receiving the voice of user 1, mobile phone 1 sends the voice of user 1 to the server. The information processing model of the server repairs the voice of user 1 to obtain the repaired voice and the subtitles corresponding to the repaired voice, and returns the repaired voice and subtitles to mobile phone 1. Mobile phone 1 sends the repaired voice and subtitles to mobile phone 2 through the base station. User 2 of mobile phone 2 can hear the repaired voice and see the subtitles.

[0162] In some possible embodiments, the information processing device may be a terminal device, such as a handheld terminal, a wearable device, or an entertainment device.

[0163] The information processing system provided by the embodiment of the present invention can be applied to terminal devices such as mobile phones, tablets, and computers. The main users are special speech-impaired people who cannot speak clearly. It can repair the input speech of the disabled in real time and play the repaired audio to the listener or transmit it to the other end device of the call. It has high use value. For example, the use scenario can be a remote communication scenario such as a voice call or a video call.

[0164] For example, the information processing device is a mobile phone 1 . Figure 5 is a schematic diagram of another application scenario of the information processing model in an embodiment of the present invention, such as Figure 5 As shown, mobile phone 1 and mobile phone 2 are connected through a base station; mobile phone 1 and mobile phone 2 are in a voice call scene or a video call scene, and user 1 of mobile phone 1 (for example, a user with damaged vocal cords) clicks the "sound repair" control on the call interface of mobile phone 1. After mobile phone 1 receives the voice of user 1, the information processing model in mobile phone 1 repairs the voice of user 1 to obtain the repaired voice and the subtitles corresponding to the repaired voice, and sends the repaired voice and subtitles to mobile phone 2 through the base station. User 2 of mobile phone 2 can hear the repaired voice and see the subtitles.

[0165] It should be understood that Figure 4 and Figure 5 The application scenario shown is only an example. The information processing model provided by the embodiment of the present invention is not only applicable to the above two scenarios, but can be applied to any scenario that requires real-time sound restoration.

[0166] Figure 6 A schematic diagram of the structure of an information processing model provided by an embodiment of the present invention.

[0167] like Figure 6 As shown, the information processing model includes: a semantic correction module 200 and an audio generation module 300.

[0168] Exemplarily, the semantic correction module 200 is used to correct the content part of the audio to be repaired. The intelligibility of the audio to be repaired is low.

[0169] Exemplarily, the audio generation module 300 is used to fuse various features output by the semantic correction module 200 and generate a repaired audio. The repaired audio has high intelligibility. The audio generation module 300 can convert various features output by the semantic correction module 200 into audio in real time.

[0170] For example, Figure 6 As shown, the audio to be repaired is used as the input of the semantic correction module 200, and the semantic features of the audio to be repaired after semantic repair are output, and then the semantic features after semantic repair are input into the audio generation module 300; the audio generation module 300 outputs the repaired audio.

[0171] Semantic features refer to the characteristics of vocabulary, grammar, pronunciation, intonation, etc. in a language. They reflect the structure, function, evolution and other laws of the language, and can reveal the speaker or author's intentions, emotions, topics and other information.

[0172] Optionally, the output of the semantic correction module 200 also includes a first text. The first text is the text corresponding to the semantic feature after semantic restoration. The semantic correction module inputs low-intelligibility audio and outputs the semantic feature after semantic restoration and its corresponding text in real time. The embodiment of the present invention can also generate subtitles based on the first text, so that the embodiment of the present invention can generate restored audio and subtitles in real time. The role of the subtitles is to allow users to perceive the content of the restored audio, so that users can understand the content of the restored audio.

[0173] Optionally, the first text output by the semantic correction module 200 may also be input to the audio generation module 300 together with the semantic features after semantic repair.

[0174] Optionally, the input of the semantic correction module 200 also includes a first video corresponding to the audio to be repaired, and the output also includes semantic features of the first video after semantic repair; the input of the audio generation module 300 also includes semantic features of the first video after semantic repair. Wherein, the first video includes a face image or a lip image.

[0175] Figure 7 for Figure 6 A specific structural diagram of the information processing model in .

[0176] like Figure 7 As shown, the semantic correction module 200 includes: a voiceprint model 201 and at least one encoder. The at least one encoder in the semantic correction module 200 includes a first stream feature encoder 202 .

[0177] Exemplarily, the first streaming feature encoder 202 is a streaming audio encoder, such as a streaming ASR encoder.

[0178] For example, in a voice call scenario, the input is only audio, so the semantic correction module 200 may have only one streaming ASR encoder.

[0179] Alternatively, if Figure 7 As shown, at least one encoder in the semantic correction module 200 further includes a second streaming feature encoder 203 .

[0180] Exemplarily, the second streaming feature encoder 203 is a streaming image encoder.

[0181] For example, in a video call scenario, when the face video information of the speaker can be obtained, a streaming image encoder can be added. For more scenarios, all inputs containing speech content information can add a streaming feature encoder to extract features.

[0182] Exemplarily, the input of the voiceprint model 201 is audio or audio spectrum features, and the output is a voiceprint feature of a fixed dimension.

[0183] For example, the voiceprint feature A(T1*D1) is a vector with a shape of T1*D1. The length T1 of the vector is a fixed value, which is used to represent the speaker's timbre characteristics; D1 represents the dimension of the voiceprint feature. For example, T1=1, D1=512.

[0184] Exemplarily, the voiceprint model 201 can use the ECAPA-TDNN structure. When in use, in the sound registration process, a third audio is obtained by recording a sentence of the user and input into the voiceprint model 201 to extract the voiceprint features in advance. Therefore, the user only needs to record one sentence in advance for registration to use the information processing model provided by the embodiment of the present invention for voice restoration.

[0185] Exemplarily, the input of the first streaming feature encoder 202 is the first audio (audio to be repaired), and the output is the first semantic feature. The first semantic feature is the semantic feature of the audio of the first number of frames in the first audio after semantic repair.

[0186] Among them, the first semantic feature is generated according to the audio of the first frame number and sent to the audio generation module 300, so the delay generated is a delay at the frame level, which is much lower than the delay at the sentence level, thereby realizing real-time sound restoration.

[0187] For example, the first semantic feature B(T2*D2) is a vector of shape T2*D2, where T2 is the first frame number, that is, the frame number of the first semantic feature, which is related to the length of the input audio, and D2 is the dimension of the semantic feature.

[0188] Exemplarily, the structure of the first stream feature encoder 202 may select a commonly used RNNT structure.

[0189] Exemplarily, the second streaming feature encoder 203 receives as input a first video containing a face or lip image, and outputs a second semantic feature. The second semantic feature is a semantic feature after semantic restoration of the second number of frames in the first video.

[0190] Among them, the second semantic feature is generated according to the second frame number and sent to the audio generation module 300 according to the second frame number. Therefore, the generated delay is a delay at the frame level, which is much lower than the delay at the sentence level, thereby realizing real-time sound restoration.

[0191] For example, the second semantic feature D(T4*D3) is a semantic feature with a shape of T4*D3, T4 is the second frame number, that is, the frame number of the second semantic feature, which is related to the length of the input video, and D3 is the dimension of the second semantic feature.

[0192] Among them, the second semantic feature is generated according to the video image of the second frame number and sent to the audio generation module 300. The resulting delay is a delay at the frame level, which is much lower than the delay at the sentence level, thereby achieving real-time sound restoration.

[0193] Exemplarily, the second frame number is equal to or unequal to the first frame number.

[0194] Preferably, the second frame number is equal to the first frame number, so that feature fusion can be performed directly later.

[0195] If the second frame number is not equal to the first frame number, the first semantic feature and the second semantic feature need to be processed before subsequent feature fusion so that the frame numbers of the fused features are equal.

[0196] Exemplarily, the size of the first frame number and the second frame number can be set during the training phase of the information processing model. The smaller the first frame number and the second frame number are, the lower the latency of the information processing model in repairing the sound will be, but the accuracy will be reduced. The larger the first frame number and the second frame number are, the higher the accuracy of the information processing model in repairing the sound will be, but the latency will be greater. Therefore, the size of the first frame number and the second frame number can be repeatedly adjusted during the model training phase, so that the information processing model can take into account both low latency and high accuracy, so as to achieve real-time sound repair.

[0197] Exemplarily, the second streaming feature encoder 203 may select a (Convolutional Neural Network+Transformer, CNN+Transformer) structure.

[0198] Optionally, the output of the semantic correction module 200 also includes a first text. The first text is the text corresponding to the semantic feature after semantic restoration.

[0199] For example, the first text C(T3*1) is a text with a shape of T3*1, where T3 is the number of frames of the first text, which is related to the length of the input audio, and D3 is the dimension of the first text.

[0200] In some possible embodiments, the first text is output by the first streaming feature encoder 202. Exemplarily, the first streaming feature encoder 202 outputs the first semantic feature and the first text according to the input audio to be repaired. The first text corresponds to the first semantic feature.

[0201] In some possible embodiments, the first text is outputted by the first streaming feature encoder 202 and the second streaming encoder 203. Exemplarily, the first streaming feature encoder 202 outputs the first semantic feature and the second text according to the input audio to be repaired; then the second text and the first video are inputted into the second streaming encoder 203, and the second streaming encoder 203 outputs the first text and the second semantic feature. The first text corresponds to the first semantic feature and the second semantic feature.

[0202] like Figure 7 As shown, the audio generation module 300 includes: an audio repair module 310 and a vocoder 320 .

[0203] Exemplarily, the audio restoration module 310 is used to convert various features output by the semantic correction module 200 into first fusion features.

[0204] In some possible embodiments, Figure 8 for Figure 6 Another specific structural diagram of the information processing model in . Figure 8 As shown, the audio repair module 310 includes: a fusion module 311 and a streaming audio repair module 312. Exemplarily, the streaming audio repair module 312 adopts a (Convolution-augmented Transformer, conformer) structure.

[0205] Exemplarily, the fusion module 311 is used to fuse various features output by the semantic correction module 200 to obtain a second fused feature.

[0206] For example, assuming that T1=1, T2=T3=T4=T, the various features output by the semantic correction module 200 include voiceprint feature A(1*D1), first semantic feature B(T*D2), first text C(T*1), and second semantic feature D(T*D3). The fusion module 311 is responsible for concatenating and fusing the voiceprint feature A(1*D1), first semantic feature B(T*D2), first text C(T*1), and second semantic feature D(T*D3). A direct fusion method is to pass A, B, and D through a fully connected neural network layer respectively, and the dimensions become A(1*D4), B(T*D5), and D(T*D6), and C is transformed into C(T*D7) through an embedding layer. Then A(1*D1) is copied T times to become A(T*D4), and concatenated with B(T*D5), C(T*D7), and D(T*D6) to obtain the initial fused feature E(T*(D4+D5+D6+D7)). Finally, the initial fused feature E(T*(D4+D5+D6+D7)) is passed through a fully connected layer to obtain the second fused feature E(T*D8).

[0207] Even if T2, T3, and T4 are not equal, before fusion, the first semantic feature B (T2*D2), the first text C (T3*1), and the second semantic feature D (T4*D3) need to be processed so that the number of frames of various features used for feature fusion is equal to facilitate fusion.

[0208] Exemplarily, the streaming audio repair module 312 is used to convert the first fused features into the first fused features, that is, new audio features (such as spectrum features).

[0209] In some possible embodiments, Fig. 9 FIG. 1 is a schematic diagram of the structure of an audio repair module provided by an embodiment of the present invention. Fig. 9 As shown, the audio repair module includes: a fusion module 311, a streaming audio repair module 312, a discretized speech module 313 and an embedding layer 314. Exemplarily, the streaming audio repair module 312 adopts a GPT structure. Since the streaming audio repair module 312 adopts a GPT structure, it is necessary to match the discretized speech module 313 and the embedding layer 314 corresponding to the VQVAE model for autoregressive reasoning.

[0210] like Fig. 9 As shown in FIG. 3 , the audio restoration module adds a discretized speech module 313 and an embedding layer 314. During the autoregressive inference process, the audio restoration module fuses the H obtained from the previous inference, predicts the H of the next frame, and gives the corresponding first fused feature F. The first fused feature F can be the probability distribution feature vector of the aforementioned discrete feature, or it can be the feature of the bottleneck layer near the output layer.

[0211] Exemplarily, the audio repair module 310 converts various features output by the semantic correction module 200 into first fusion features, including: the first reasoning process or the i-th reasoning process, where i is a natural number greater than 1.

[0212] Exemplarily, the first reasoning process includes: the fusion module 311 fuses various features output by the semantic correction module 200 to output a second fused feature; the streaming audio repair module 312 converts the second fused feature into a first fused feature.

[0213] For example, Fig. 9As shown, the fusion module 311 fuses the voiceprint feature A (A1, A2...), the first semantic feature B (B1, B2...), the first text C (C1, C2...) and the second semantic feature D (D1, D2...) to output the second fusion feature E (E1, E2...). Then the second fusion feature E (E1, E2...) is input into the streaming audio repair module 312. Since there is a delay in the process of the streaming audio repair module 312 converting the second fusion feature E (E1, E2...) into the first fusion feature F (F1, F2...), F1 lags behind E1, for example, 3 frames.

[0214] Exemplarily, the i-th reasoning process includes: the discretized speech module 313 converts the first fusion feature output by the i-1-th reasoning into the third fusion feature; the embedding layer 314 converts the third fusion feature into the fourth fusion feature; the fusion module 311 fuses the second information and the fourth fusion feature to obtain the fifth fusion feature; the streaming audio repair module 312 converts the fifth fusion feature into the first fusion feature.

[0215] For example, Fig. 9 As shown, the streaming audio repair module 312 outputs the first fusion feature F obtained in the last reasoning to the discretized speech module 313; the discretized speech module 313 outputs the third fusion feature based on the first fusion feature F obtained in the last reasoning; the embedding layer 314 outputs the fourth fusion feature H based on the third fusion feature. Then, the streaming audio repair module 312 will fuse the fourth fusion feature H with the second information to obtain the fifth fusion feature during this reasoning; and then the streaming audio repair module 312 converts the fifth fusion feature into the first fusion feature.

[0216] Exemplarily, the vocoder 320 is used to convert the first fusion feature into a second audio. The second audio is the restored audio. The intelligibility of the second audio is higher than the intelligibility of the audio to be restored.

[0217] Exemplarily, the vocoder 320 may be a HiFiGAN, the input of which is the first fusion feature, and the output of which is the audio sampling point.

[0218] Alternatively, if Figure 8 As shown, the prosody information G(N*D8) can be spliced ​​at the input front end of the streaming audio repair module 312. Where N is a fixed length, and D8 is the dimension of the prosody information. The prosody information G(N*D8) is used to optimize the prosody of the second fusion feature or the fifth fusion feature. The prosody information can also be called a prosody prompt.

[0219] Exemplarily, during the first reasoning process, the streaming audio restoration module 312 converts the prosody information and the second fusion feature into the first fusion feature.

[0220] Exemplarily, in the i-th reasoning process, the streaming audio repair module 312 converts the prosody information and the fifth fused feature into the first fused feature.

[0221] Exemplarily, the prosody information is a prosody feature extracted from the fourth audio. For example, a perceptual resampler model is used to extract the prosody feature from the fourth audio.

[0222] Exemplarily, the fourth audio is a normal person's speech, and normal people's speech is usually highly intelligible. Therefore, the intelligibility of the fourth audio is similar to that of the restored audio, which is higher than the intelligibility of the audio to be restored.

[0223] For example, Fig.10 Schematic diagram of using the information processing model to repair the time delay of the sound in real time in an embodiment of the present invention. The user uses the information processing model to repair the sound in real time. The user says "I am (si) Xiao Ming", and the information processing model outputs the sound of "I am Xiao Ming". Fig.10 As shown in the figure, the audio of the user's sentence received by the information processing model is arranged frame by frame. The first two frames of audio received correspond to "I" said by the user, the third frame received does not contain any words said by the user, and the fourth frame received shows that the user starts to say "si". Assuming that the repair delay of the information processing model is 3 frames, the information processing model will start playing the repaired sound of this sentence when receiving the fourth frame of audio, which is equivalent to outputting the repaired sound as soon as the user finishes saying the first word, so real-time sound repair can be achieved.

[0224] The following describes the training process of the information processing model in the embodiment of the present invention:

[0225] Exemplarily, the training data of the information processing model includes normal person audio data and its corresponding text, and disabled person audio data and its corresponding text.

[0226] Optionally, the training data also includes disabled video frames.

[0227] The voiceprint model 201 and the discretized speech module 313 can be trained in advance. When training the voiceprint model 201, the spectral features are input and the speaker classification result is output. The training of the discretized speech module 313 can refer to the training method of VQVAE.

[0228] The first stream feature encoder 202 and the second stream encoder 203 can each target text and train a speech recognition task; or the first stream feature encoder 202 and the second stream encoder 203 can be combined to train a single audio-visual speech recognition (AVSR) task. Since there are few audio data for the disabled and their corresponding texts, a large amount of normal audio data and their corresponding texts can be used for pre-training, and then the model can be fine-tuned based on the audio data for the disabled.

[0229] After the first streaming feature encoder 202 and the second streaming encoder 203 are trained, the normal person audio data is used to simultaneously train the streaming audio restoration module 312 and the fusion module 311. The input is the previously trained voiceprint model 201, the first streaming feature encoder 202, and the features output by the second streaming encoder 203, and the training target is the discretization unit of normal audio.

[0230] Finally, the normal human audio data is used to train the vocoder 320, the input is the features output by the streaming audio restoration module 312, and the output is the sampling points of the audio.

[0231] Table 1 Evaluation results of two batches of disability test sets with different intelligibility

[0232] Before restoration (recognition rate) After restoration (recognition rate) Test set 1 53.24％ 70.46％ Test Set 2 74.67％ 81.38％

[0233] For the model framework provided by the embodiment of the present invention, the information processing model implemented based on 300h of disabled audio data and its corresponding text and 50wh of normal audio data and its corresponding text has a delay of about 1.8s, which is much lower than the sentence-level delay. Speech restoration was evaluated for two batches of disabled test sets with different intelligibility, and a fixed general ASR model was used for the evaluation. As shown in Table 1, the information processing model provided by the embodiment of the present invention was used to repair test set 1 and test set 2, and the recognition rate after repair was significantly higher than the recognition rate before repair. At the same time, the information processing model provided by the embodiment of the present invention has a much lower delay for sound repair than the sentence level, achieving real-time repair. In addition, the information processing model provided by the embodiment of the present invention provides corresponding text results while performing real-time repair of the sound, helping users understand the content of the repaired audio without the need for an additional recognition model.

[0234] Therefore, the information processing model provided by the embodiment of the present invention can perform real-time repair of disabled audio in remote communication scenarios such as voice and video calls. The repair delay is a delay at the frame level, which is much lower than the delay at the sentence level, thereby achieving real-time repair. In real-time scenarios, text can be given synchronously to assist users in understanding the repaired audio content. There is no need for an additional speech recognition model, and the text will not lag behind the repaired audio.

[0235] For example, the information processing model provided by the embodiment of the present invention is applied to a voice call scenario. The user records the screen and the audio during the voice call, thereby obtaining the screen recording data (including the voice of the other party) and the call recording (including the voice of both parties), and aligning the timestamps according to the voice of the other party (the sound after the call is at the other end, and the call recording saves the repaired audio). According to the screen recording data and the call recording, it can be found that the embodiment of the present invention uses the information processing model to perform real-time repair with a delay far lower than the sentence level.

[0236] For example, the information processing model provided by the embodiment of the present invention is applied to a video call scenario. The user records the screen and audio during the video call, thereby obtaining the screen recording data (including subtitles and the sound of the other party) and the call recording (including the sound of both parties), and aligning the timestamps according to the sound of the other party (the sound after the call is on the other end, and the call recording saves the repaired audio). According to the screen recording data and the call recording, it can be found that the information processing model used in the embodiment of the present invention can perform real-time repair with a delay far lower than the sentence level, and can simultaneously provide the repaired audio and text, with basically the same content, and the words come out before the sound or the sound and words are synchronized, and there is no obvious lag in the words.

[0237] In the embodiment of the present invention, the subtitles are given in advance by the text output by the semantic correction module and will not lag behind the restored audio. If other subtitle solutions are used, such as an external speech recognition model, additional delay will be generated and the text will lag behind the restored audio.

[0238] The model framework of the information processing model provided in the embodiment of the present invention is scalable. In addition to being trained for real-time sound restoration, it can also be trained for real-time conversion of dialects to Mandarin or real-time cross-language translation, etc.

[0239] Based on the information processing model provided by the embodiment of the present invention, the embodiment of the present invention provides an information processing method that can achieve real-time sound restoration.

[0240] Fig.11 Flow chart of an information processing method provided by an embodiment of the present invention. The information processing method provided by an embodiment of the present invention is applied to an information processing device including an information processing model; Fig.11 As shown, the method includes:

[0241] Step 402: Receive first information from a user, where the first information includes first audio, and the first audio is audio to be repaired.

[0242] Step 404: The information processing model obtains second information based on the first information, where the second information includes a first semantic feature corresponding to the first audio; the first semantic feature is a semantic feature after semantic restoration of the audio of a first number of frames in the first audio.

[0243] Step 406: The information processing model obtains the second audio according to the user's voiceprint feature and the second information.

[0244] In the technical solution of the information processing method provided by the embodiment of the present invention, the information processing method is applied to a device including an information processing model; the method includes: receiving first information of a user, the first information including first audio, the first audio being the audio to be repaired; the information processing model obtains second information according to the first information, the second information including first semantic features corresponding to the first audio, the first semantic features being semantic features after semantic repair of the audio of the first number of frames in the first audio; the information processing model obtains second audio according to the voiceprint features of the user and the second information. Real-time sound repair can be achieved.

[0245] Fig.12 A flowchart of another information processing method provided by an embodiment of the present invention. The information processing method provided by an embodiment of the present invention is applied to an information processing device including an information processing model; Fig.12 As shown, the method includes:

[0246] Step 502: Receive a third audio of the user, where the third audio is used to register a voiceprint.

[0247] Step 504: The information processing model obtains voiceprint features according to the third audio.

[0248] Step 506: Receive first information from the user, where the first information includes first audio, and the first audio is the audio to be repaired.

[0249] Step 508: The information processing model obtains second information according to the first information, where the second information includes a first semantic feature corresponding to the first audio, and the first semantic feature is a semantic feature after semantic restoration of the audio of a first number of frames in the first audio.

[0250] In some possible embodiments, step 508 specifically includes: the information processing model obtains the first semantic feature according to the first audio.

[0251] In some possible embodiments, the second information further includes the first semantic feature corresponding to the first text; step 508 specifically includes: the information processing model obtains the first semantic feature and the first text according to the first audio.

[0252] In some possible embodiments, the first information also includes a first video corresponding to the first audio; the second information also includes a second semantic feature of the first video, and the second semantic feature is a semantic feature after semantic restoration of an image of a second frame number in the first video. Step 508 specifically includes: the information processing model obtains the first semantic feature according to the first audio; the information processing model obtains the second semantic feature according to the first video.

[0253] In some possible embodiments, the first information also includes a first video corresponding to the first audio; the second information also includes a first text and a second semantic feature of the first video, and the second semantic feature includes a semantic feature of an image of a second frame number in the first video. Step 508 specifically includes: the information processing model obtains the first semantic feature and the first text according to the first audio, and the first text corresponds to the first semantic feature; the information processing model obtains the second semantic feature according to the first video; or

[0254] Step 508 specifically includes: the information processing model obtains the first semantic feature and the second text according to the first audio; the information processing model obtains the second semantic feature and the first text according to the first video and the second text, and the first text corresponds to the first semantic feature and the second semantic feature.

[0255] In some possible embodiments, the second information includes a first text; and step 508 further includes: generating subtitles according to the first text.

[0256] Step 510: The information processing model obtains the second audio according to the user's voiceprint features and the second information.

[0257] In some possible embodiments, step 510 specifically includes: step 5102-step 5104.

[0258] Step 5102: The information processing model obtains a first fusion feature based on the voiceprint feature and the second information.

[0259] In some possible embodiments, step 5102 specifically includes: the information processing model performs feature fusion on the voiceprint feature and the second information to obtain a second fused feature; and the information processing model performs audio restoration on the second fused feature to obtain a first fused feature.

[0260] In some possible embodiments, step 5102 specifically includes: the first reasoning process or the i-th reasoning process, where i is a natural number greater than 1.

[0261] Exemplarily, the first reasoning process includes: the information processing model performs feature fusion on the voiceprint feature and the second information to obtain a second fused feature; the information processing model performs audio restoration on the second fused feature to obtain a first fused feature.

[0262] Exemplarily, the i-th reasoning process includes: the information processing model performs speech discretization processing on the first fusion feature obtained by the i-1-th reasoning to obtain a third fusion feature; the information processing model passes the third fusion feature through the first layer to obtain a fourth fusion feature; the information processing model performs feature fusion on the voiceprint feature, the second information and the fourth fusion feature to obtain a fifth fusion feature; the information processing model performs audio restoration on the fifth fusion feature to obtain the first fusion feature.

[0263] Exemplarily, the first layer is the Embedding layer.

[0264] In some possible embodiments, the information processing model performs audio restoration on the second fusion feature to obtain the first fusion feature, including: the information processing model uses rhythmic information to perform audio restoration on the second fusion feature to obtain the first fusion feature; the rhythmic information is a rhythmic feature extracted from the fourth audio.

[0265] In some possible embodiments, the information processing model performs audio restoration on the fifth fusion feature to obtain the first fusion feature, including: the information processing model uses prosody information to perform audio restoration on the fifth fusion feature to obtain the first fusion feature.

[0266] Step 5104: The information processing model obtains the second audio according to the first fusion feature.

[0267] Fig.13 This is a schematic diagram of a device structure provided by an embodiment of the present invention. It should be understood that the device 700 can execute each step in the above method embodiment. To avoid repetition, it will not be described in detail here. The device 700 includes: a processor 701 and a memory 702.

[0268] An embodiment of the present application also provides an information processing device, including at least one processor 701 and at least one memory 702, wherein the at least one memory 702 is used to store at least one program, and when the at least one processor 701 runs the at least one program, the information processing device performs the operations involved in the information processing device in the above method embodiment.

[0269] An embodiment of the present application provides a readable storage medium, in which a program is stored. When the program is executed by an information processing device, the information processing device performs the operations involved in the information processing device in the above method embodiment.

[0270] The embodiment of the present application also provides a program product including instructions, which, when executed on a first device or any at least one processor, enables the information processing device to perform operations involved in the information processing device in the above method embodiment.

[0271] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0272] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0273] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An information processing method, It is characterized in that Applied to an information processing device including an information processing model; the method comprises: Receive first information from a user, where the first information includes first audio, and the first audio is audio to be repaired; The information processing model obtains second information according to the first information, where the second information includes a first semantic feature corresponding to the first audio, where the first semantic feature is a semantic feature after semantic restoration is performed on audio of a first number of frames in the first audio; The information processing model obtains the second audio according to the voiceprint feature of the user and the second information.

2. The method according to claim 1, It is characterized in that Before receiving the first information of the user, the method further includes: receiving a third audio of the user, where the third audio is used to register a voiceprint; The information processing model obtains the voiceprint feature according to the third audio.

3. The method according to claim 1 or 2, It is characterized in that The information processing model obtains second information according to the first information, including: The information processing model obtains the first semantic feature according to the first audio.

4. The method according to claim 1 or 2, It is characterized in that The second information further includes a first text corresponding to the first semantic feature; The information processing model obtains second information according to the first information, including: The information processing model obtains the first semantic feature and the first text according to the first audio.

5. The method according to claim 1 or 2, It is characterized in that The first information also includes a first video corresponding to the first audio; the second information also includes a second semantic feature corresponding to the first video, and the second semantic feature is a semantic feature after semantic restoration of an image of a second frame number in the first video.

6. The method according to claim 5, It is characterized in that The information processing model obtains second information according to the first information, including: The information processing model obtains the first semantic feature according to the first audio; The information processing model obtains the second semantic feature according to the first video.

7. The method according to claim 5, It is characterized in that The second information further includes a first text; The information processing model obtains second information according to the first information, including: The information processing model obtains the first semantic feature and the first text according to the first audio, and the first text corresponds to the first semantic feature; The information processing model obtains the second semantic feature according to the first video; or The information processing model obtains second information according to the first information, including: The information processing model obtains the first semantic feature and the second text according to the first audio; The information processing model obtains the second semantic feature and the first text according to the first video and the second text, and the first text corresponds to the first semantic feature and the second semantic feature.

8. The method according to any one of claims 1 to 7, It is characterized in that The information processing model obtains the second audio according to the voiceprint feature of the user and the second information, including: The information processing model obtains a first fusion feature according to the voiceprint feature and the second information; The information processing model obtains the second audio according to the first fusion feature.

9. The method according to claim 8, It is characterized in that The information processing model obtains a first fusion feature according to the voiceprint feature and the second information, including: The information processing model performs feature fusion on the voiceprint feature and the second information to obtain a second fused feature; The information processing model performs audio restoration on the second fusion feature to obtain the first fusion feature.

10. The method according to claim 8, It is characterized in that The information processing model obtains a first fusion feature according to the voiceprint feature and the second information, including: a first reasoning process or an i-th reasoning process, where i is a natural number greater than 1; The first reasoning process includes: The information processing model performs feature fusion on the voiceprint feature and the second information to obtain a second fused feature; The information processing model performs audio restoration on the second fused feature to obtain the first fused feature; The i-th reasoning process includes: The information processing model performs speech discretization processing on the first fusion feature obtained by the i-1th reasoning to obtain a third fusion feature; The information processing model obtains a fourth fusion feature by passing the third fusion feature through the first layer; The information processing model performs feature fusion on the voiceprint feature, the second information and the fourth fusion feature to obtain a fifth fusion feature; The information processing model performs audio restoration on the fifth fusion feature to obtain the first fusion feature.

11. The method according to claim 10, It is characterized in that The information processing model performs audio restoration on the second fusion feature to obtain the first fusion feature, including: The information processing model uses the prosody information to perform audio restoration on the second fused feature to obtain the first fused feature; the prosody information is a prosody feature extracted from the fourth audio; The information processing model performs audio restoration on the fifth fusion feature to obtain the first fusion feature, including: The information processing model uses the prosody information to perform audio restoration on the fifth fusion feature to obtain the first fusion feature.

12. The method according to claim 4 or 7, It is characterized in that After the information processing model obtains the second information according to the first information, the information processing model further includes: Subtitles are generated based on the first text.

13. An information processing device, It is characterized in that The information processing device comprises a processor and a memory, wherein the memory is used to store a program, and when the processor runs the program, the information processing device executes the steps of the method according to any one of claims 1 to 12.

14. A readable storage medium, It is characterized in that The readable storage medium stores a program, and when the program is executed by an information processing device, the information processing device executes the method according to any one of claims 1 to 12.

15. A program product, It is characterized in that The program product stores a program, and when the program is executed by an information processing device, the information processing device executes the method according to any one of claims 1 to 12.

Images